Adaptive mutex spin: 6099802 events in 300.017 seconds (20332 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 1066855 17% 17% 0.00 31827 zio_write_issue taskqueue_thread_loop+0xd0 1037177 17% 34% 0.00 34980 zio_write_issue spa_taskq_dispatch_ent+0x84 612020 10% 45% 0.00 4889 vm active pagequeue vm_page_free_prep+0x184 519276 9% 53% 0.00 1919 dbu_evict taskqueue_thread_loop+0xd0 485989 8% 61% 0.00 1338 dbu_evict dbuf_destroy+0xb0 430985 7% 68% 0.00 14877 vm active pagequeue vm_page_activate+0x74 389128 6% 74% 0.00 21812 zio_write_issue taskqueue_thread_loop+0xc0 244909 4% 78% 0.00 7684 zio_write_intr taskqueue_thread_loop+0xd0 241025 4% 82% 0.00 6404 zio_write_intr spa_taskq_dispatch_ent+0x84 209138 3% 86% 0.00 22568 vm page vm_fault_trap+0x64 195666 3% 89% 0.00 6327 zio_write_intr taskqueue_thread_loop+0xc0 115262 2% 91% 0.00 6289 vm active pagequeue vm_page_deactivate+0x4c 49103 1% 92% 0.00 3252 pmap data_abort+0xc4 44539 1% 92% 0.00 10836 vm active pagequeue vm_page_alloc_domain_after+0x200 30883 1% 93% 0.00 17258 vm page vm_fault+0xd90 27351 0% 93% 0.00 2265 pmap vm_fault+0x1634 26010 0% 94% 0.00 7160 dp_sync_taskq taskq_dispatch+0x6c 24898 0% 94% 0.00 13515 dp_sync_taskq taskqueue_thread_loop+0xd0 23871 0% 95% 0.00 8858 dp_sync_taskq taskqueue_thread_loop+0xc0 20855 0% 95% 0.00 5969 vm active pagequeue vm_page_advise+0x170 17360 0% 95% 0.00 89945 VM reserv domain vm_reserv_alloc_page+0x2b0 14566 0% 96% 0.00 1800 pmap pv chunk list pmap_remove_pages+0x8c 13816 0% 96% 0.00 4397 pmap kmem_back_domain+0x140 13581 0% 96% 0.00 1580 pmap pv chunk list pmap_try_insert_pv_entry+0x24 11044 0% 96% 0.00 2947 CAM device lock g_disk_start+0x30c 10681 0% 96% 0.00 16980 vm page free queue vm_page_alloc_domain_after+0xb0 9076 0% 96% 0.00 4585 AHCI channel lock xpt_action_default+0x72c 8630 0% 97% 0.00 1239 pipe mutex dofilewrite+0x7c 8429 0% 97% 0.00 3757 vnode_free_list zfs_zget+0x38 7826 0% 97% 0.00 21106 VM reserv domain vm_page_free_prep+0x1b8 7578 0% 97% 0.00 2356 CAM device lock xpt_run_allocq+0x180 7483 0% 97% 0.00 1274 zio_free_intr spa_taskq_dispatch_ent+0x84 7478 0% 97% 0.00 1371 vm inactive pagequeue release_page+0x98 7248 0% 97% 0.00 2749 AHCI channel lock ahci_intr+0xd4 6909 0% 97% 0.00 17224 vm page free queue vm_page_free_prep+0x1b8 6867 0% 98% 0.00 3603 pmap vm_fault+0x364 6204 0% 98% 0.00 48846 vm page uiomove_object+0x1c4 5819 0% 98% 0.00 3609 pmap _kmem_unback+0x34 5445 0% 98% 0.00 1310 pipe mutex dofileread+0x7c 5272 0% 98% 0.00 5240 umtxql do_lock_umutex+0x7ec 5214 0% 98% 0.00 5336 pmap vm_fault_prefault+0x17c 5109 0% 98% 0.00 3700 vm active pagequeue vm_pageout+0x1b0 5082 0% 98% 0.00 10191 vm page zfs_freebsd_getpages+0x1c4 4857 0% 98% 0.00 16395 vm active pagequeue vm_page_unwire+0x15c 4617 0% 98% 0.00 2658 CAM device lock xpt_done_process+0x690 4332 0% 98% 0.00 1990 vm inactive pagequeue vm_page_free_prep+0x184 3855 0% 99% 0.00 2601 umtxql do_el0_sync+0x550 3559 0% 99% 0.00 1956 ncvn zfs_lookup+0x66c 3184 0% 99% 0.00 52515 VM reserv domain vm_reserv_alloc_page+0x524 3179 0% 99% 0.00 5504 vnode_free_list _vget_prep+0x68 3134 0% 99% 0.00 945 dbu_evict taskqueue_thread_loop+0xc0 3108 0% 99% 0.00 5246 pmap vm_fault_prefault+0x110 3065 0% 99% 0.00 3610 vnode_free_list zfs_freebsd_create+0x3e0 2589 0% 99% 0.00 1024 vm inactive pagequeue vm_page_alloc_domain_after+0x200 2441 0% 99% 0.00 3027 vnode_free_list vrefl+0xc8 2169 0% 99% 0.00 116631 process lock handle_el0_sync+0xc0 1862 0% 99% 0.00 231 pipe mutex pipe_read+0xa8 1824 0% 99% 0.00 3613 zio_free_intr taskqueue_thread_loop+0xd0 1765 0% 99% 0.00 9063 vm page vm_map_madvise+0x268 1657 0% 99% 0.00 75566 process lock thread_suspend_check+0x434 1591 0% 99% 0.00 4119 pmap pv chunk list pmap_enter+0x6f8 1579 0% 99% 0.00 8566 vm page vm_pageout+0x1b0 1360 0% 99% 0.00 17247 vm page elf64_load_sections+0x1cc 1354 0% 99% 0.00 11249 umtxql __umtx_op_wait_umutex+0x48 1240 0% 99% 0.00 537 CAM queue lock xpt_done_td+0x114 1211 0% 99% 0.00 7989 dp_sync_taskq dmu_objset_sync+0x2a0 1165 0% 99% 0.00 5411 process lock kern_wait6+0x1ec 1093 0% 99% 0.00 2694 UMA Slabs malloc+0x12c 1071 0% 99% 0.00 620 kernel arena domain vmem_alloc+0x58 1067 0% 99% 0.00 4549 pmap vm_map_delete+0x17c 1064 0% 99% 0.00 3183 ncvn tmpfs_lookup1+0x13c 1044 0% 99% 0.00 1125 zio_free_issue_3 taskqueue_thread_loop+0xd0 1037 0% 99% 0.00 987 zio_free_issue_2 taskqueue_thread_loop+0xd0 1033 0% 99% 0.00 1599 vm inactive pagequeue vm_object_madvise+0x290 1026 0% 99% 0.00 1059 zio_free_issue_6 taskqueue_thread_loop+0xd0 1021 0% 99% 0.00 992 zio_free_issue_5 taskqueue_thread_loop+0xd0 1001 0% 99% 0.00 1073 zio_free_issue_4 taskqueue_thread_loop+0xd0 987 0% 99% 0.00 940 zio_free_issue_7 taskqueue_thread_loop+0xd0 961 0% 99% 0.00 2098 zio_free_issue_1 taskqueue_thread_loop+0xd0 954 0% 99% 0.00 662 zio_free_issue_6 taskqueue_thread_loop+0xc0 950 0% 99% 0.00 584 zio_free_issue_3 taskqueue_thread_loop+0xc0 950 0% 99% 0.00 1222 zio_free_issue_0 taskqueue_thread_loop+0xd0 949 0% 99% 0.00 567 zio_free_issue_0 taskqueue_thread_loop+0xc0 915 0% 100% 0.00 609 zio_free_issue_2 taskqueue_thread_loop+0xc0 898 0% 100% 0.00 621 zio_free_issue_5 taskqueue_thread_loop+0xc0 893 0% 100% 0.00 621 zio_free_issue_4 taskqueue_thread_loop+0xc0 892 0% 100% 0.00 582 zio_free_issue_7 taskqueue_thread_loop+0xc0 887 0% 100% 0.00 939 CAM doneq ahci_end_transaction+0x8ec 880 0% 100% 0.00 597 zio_free_issue_1 taskqueue_thread_loop+0xc0 847 0% 100% 0.00 4593 process_exec exec_elf64_imgact+0x9e8 813 0% 100% 0.00 5285 dp_sync_taskq taskqueue_quiesce+0x104 781 0% 100% 0.00 1428 zio_free_issue_3 spa_taskq_dispatch_ent+0x84 768 0% 100% 0.00 2158 zio_free_issue_6 spa_taskq_dispatch_ent+0x84 766 0% 100% 0.00 2539 zio_free_issue_1 spa_taskq_dispatch_ent+0x84 760 0% 100% 0.00 1990 zio_free_issue_5 spa_taskq_dispatch_ent+0x84 760 0% 100% 0.00 1514 zio_free_issue_2 spa_taskq_dispatch_ent+0x84 758 0% 100% 0.00 1270 zio_free_issue_7 spa_taskq_dispatch_ent+0x84 754 0% 100% 0.00 1511 zio_free_issue_0 spa_taskq_dispatch_ent+0x84 730 0% 100% 0.00 1433 zio_free_issue_4 spa_taskq_dispatch_ent+0x84 668 0% 100% 0.00 625 vm inactive pagequeue dmu_read_pages+0x4dc 648 0% 100% 0.00 4097 dp_sync_taskq taskqueue_quiesce+0x1b8 625 0% 100% 0.00 7243 metaslab_group_taskq taskqueue_thread_loop+0xc0 618 0% 100% 0.00 2575 vm active pagequeue vm_pageout_worker+0x123c 604 0% 100% 0.00 9725 metaslab_group_taskq taskqueue_thread_loop+0xd0 600 0% 100% 0.00 2288 process_exit sys_sys_exit+0x10 584 0% 100% 0.00 11725 vm page kern_execve+0x6e4 580 0% 100% 0.00 8750 vnode_free_list tmpfs_alloc_vp+0x3f0 576 0% 100% 0.00 21722 vm page elf64_load_file+0x128 561 0% 100% 0.00 3279 UMA Slabs zone_alloc_item_locked+0x1ec 543 0% 100% 0.00 458 kernel arena domain uma_large_free+0x18 511 0% 100% 0.00 1068 UMA Slabs zone_free_item+0x70 491 0% 100% 0.00 18948 vm page free queue uma_zalloc_arg+0x47c 447 0% 100% 0.00 1077 CAM queue lock xpt_action_default+0x72c 433 0% 100% 0.00 3158 xhci1 fork_exit+0x90 398 0% 100% 0.00 613 vm inactive pagequeue dmu_read_pages+0x1a4 397 0% 100% 0.00 17678 vm page free queue bucket_drain+0x80 364 0% 100% 0.00 7214 CAM queue lock adastart+0x8cc 343 0% 100% 0.00 367 CAM queue lock ahci_end_transaction+0x824 317 0% 100% 0.00 21732 g_disk_start g_io_request+0x27c 309 0% 100% 0.00 3326 CAM doneq fork_exit+0x90 300 0% 100% 0.00 2260 process_ctor uma_zalloc_arg+0x8b0 295 0% 100% 0.00 551 ncnegl cache_zap_locked+0x78 289 0% 100% 0.00 8557 CAM queue lock xpt_release_simq+0x8c 267 0% 100% 0.00 1976 thread_dtor uma_zfree_arg+0x44 216 0% 100% 0.00 3716 pmap pv chunk list pmap_enter+0xbb0 214 0% 100% 0.00 21313 pmap vm_pageout_worker+0x11b0 209 0% 100% 0.00 2649 pipe mutex pipe_close+0x68 205 0% 100% 0.00 25071 sleep mtxpool zio_vdev_io_start+0x278 201 0% 100% 0.00 26557 vm page exec_map_first_page+0x350 174 0% 100% 0.00 278 CAM doneq xpt_done_td+0xac 170 0% 100% 0.00 204 pipe mutex pipe_write+0x65c 167 0% 100% 0.00 2194 process lock sys_sys_exit+0x10 163 0% 100% 0.00 214 ncnegl zfs_lookup+0x66c 162 0% 100% 0.00 1731 vnode interlock VOP_UNSET_TEXT_APV+0x4c 156 0% 100% 0.00 10292 ncneglh cache_zap_locked+0x78 153 0% 100% 0.00 2528 umtxql __umtx_op_wait_uint_private+0x50 152 0% 100% 0.00 271 procid sys_fork+0x30 151 0% 100% 0.00 1412 process lock sys_sigprocmask+0x64 149 0% 100% 0.00 1914 thread_ctor uma_zalloc_arg+0x8b0 143 0% 100% 0.00 5520 struct mount vlist mtx vputx+0x3a0 118 0% 100% 0.00 646 vm inactive pagequeue vm_page_activate+0x44 117 0% 100% 0.00 1627 vnode interlock cache_lookup+0x5c8 113 0% 100% 0.00 12893 ncneglh vfs_cache_lookup+0xb4 113 0% 100% 0.00 5106 CAM queue lock xpt_run_devq+0x360 112 0% 100% 0.00 1215 pipe mutex pipe_read+0x3d8 111 0% 100% 0.00 552 vnode_free_list zfs_freebsd_mkdir+0x31c 100 0% 100% 0.00 314 pmap pv chunk list pmap_pv_insert_l2+0x28 88 0% 100% 0.00 355 procid proc_reap+0x2f0 81 0% 100% 0.00 668 sleep mtxpool g_part_start+0xf4 78 0% 100% 0.00 2174 vm inactive pagequeue vm_page_readahead_finish+0x6c 76 0% 100% 0.00 9114 ncvn zfs_lookup+0x720 72 0% 100% 0.00 5032 dp_sync_taskq dsl_pool_sync+0x1c4 71 0% 100% 0.00 680 struct mount vlist mtx _vget_prep+0x68 68 0% 100% 0.00 500 vnode interlock vn_io_fault+0x130 67 0% 100% 0.00 6873 process lock kern_sigsuspend+0x140 67 0% 100% 0.00 37166 ncvn cache_lookup_nomakeentry+0x310 66 0% 100% 0.00 2426 process lock sys_fork+0x30 65 0% 100% 0.00 3669 struct mount mtx tmpfs_update_mtime+0x4c 64 0% 100% 0.00 6235 xhci1 usbd_do_request_flags+0x33c 63 0% 100% 0.00 245 procid sys_vfork+0x34 59 0% 100% 0.00 1618 process_dtor uma_zfree_arg+0x44 58 0% 100% 0.00 9513 vm page zfs_freebsd_read+0x450 57 0% 100% 0.00 404 vnode interlock VOP_SET_TEXT_APV+0x4c 56 0% 100% 0.00 7161 umtxql do_lock_umutex+0x5a8 56 0% 100% 0.00 557 ncnegl vfs_cache_lookup+0xb4 49 0% 100% 0.00 31661 ncvn zfs_freebsd_rename+0xd10 48 0% 100% 0.00 218 ncnegl tmpfs_lookup1+0x13c 47 0% 100% 0.00 277 process lock sys_execve+0x54 45 0% 100% 0.00 673 vnode_free_list _vdrop+0x424 42 0% 100% 0.00 305 vnode interlock _vget_prep+0x68 42 0% 100% 0.00 2159 vnode interlock lookup+0xb50 42 0% 100% 0.00 10019 ncvn vn_fullpath1+0x144 39 0% 100% 0.00 2521 vnode interlock lookup+0x4ac 38 0% 100% 0.00 314 zio_free_intr taskqueue_thread_loop+0xc0 37 0% 100% 0.00 38109 process lock do_lock_umutex+0x9a8 36 0% 100% 0.00 556 pipe mutex pipe_write+0xb0 35 0% 100% 0.00 400 process lock exit1+0x5ac 33 0% 100% 0.00 2219 process lock exit1+0x4f4 33 0% 100% 0.00 5559 USB device mutex usb_process+0xfc 33 0% 100% 0.00 6405 Giant usbd_do_request_flags+0x354 27 0% 100% 0.00 1613 TID lock uma_zfree_arg+0x44 26 0% 100% 0.00 5285 xhci0 fork_exit+0x90 26 0% 100% 0.00 1496 vm inactive pagequeue vm_page_unwire+0x130 26 0% 100% 0.00 10127 ncvn vfs_cache_lookup+0xb4 25 0% 100% 0.00 3660 umtxql do_wait+0x35c 24 0% 100% 0.00 242 vm map (system) vm_fault+0xa4 22 0% 100% 0.00 312 process lock kern_execve+0x47c 22 0% 100% 0.00 445 metaslab_group_taskq taskq_dispatch+0x6c 22 0% 100% 0.00 2972 dbu_evict dbuf_rele_and_unlock+0x118 20 0% 100% 0.00 168 vnode interlock VOP_LOCK1_APV+0x4c 20 0% 100% 0.00 165 pipe mutex pipe_read+0x410 19 0% 100% 0.00 1849 process lock proc_to_reap+0x454 18 0% 100% 0.00 5460 xhci0 usbd_do_request_flags+0x33c 18 0% 100% 0.00 2348 sleep mtxpool vn_io_fault+0xfc 18 0% 100% 0.00 1043 pipe mutex kern_fstat+0xa8 17 0% 100% 0.00 4411 xhci1 usb_command_wrapper+0x84 17 0% 100% 0.00 1664 vm page vm_page_unhold_pages+0x20 17 0% 100% 0.00 9656 umtxql __umtx_op_wake2_umutex+0x1d8 17 0% 100% 0.00 41520 axe0 if_transmit+0x17c 15 0% 100% 0.00 349 vnode interlock vn_io_fault+0x16c 15 0% 100% 0.00 2314 vnode interlock vputx+0x370 15 0% 100% 0.00 20435 struct mount mtx kern_funlinkat+0x230 15 0% 100% 0.00 2380 USB device mutex usbd_do_request_proc+0x94 14 0% 100% 0.00 1970 struct mount mtx vrecycle+0x6c 14 0% 100% 0.00 7036 process lock filt_procdetach+0x20 13 0% 100% 0.00 1901 reseed mutex read_random_uio+0x30 12 0% 100% 0.00 43992 vm page vm_object_unwire+0x178 12 0% 100% 0.00 6555 ncvn zfsvfs_teardown+0x48 11 0% 100% 0.00 6299 xhci0 softclock+0x74 11 0% 100% 0.00 2880 xhci0 usb_command_wrapper+0x84 11 0% 100% 0.00 8527 struct mount mtx tmpfs_alloc_vp+0x478 11 0% 100% 0.00 11173 struct mount mtx vgonel+0x6c 11 0% 100% 0.00 171 sleep mtxpool vn_io_fault+0x5c 11 0% 100% 0.00 681 dp_zil_clean_taskq taskqueue_thread_loop+0xc0 10 0% 100% 0.00 8670 xhci1 usb_process+0xfc 10 0% 100% 0.00 3473 TID lock free_unr+0x26c 9 0% 100% 0.00 55772 xhci1 softclock+0x74 9 0% 100% 0.00 93071 xhci1 ithread_loop+0x1f4 9 0% 100% 0.00 560 vm inactive pagequeue vm_pageout_worker+0x123c 9 0% 100% 0.00 513 struct mount vlist mtx tmpfs_alloc_vp+0x478 9 0% 100% 0.00 4192 process lock sys_sigaction+0x60 9 0% 100% 0.00 3291 process lock fork_rfppwait+0xf4 9 0% 100% 0.00 3143 jail mutex do_jail_attach+0x23c 8 0% 100% 0.00 222 vnode_free_list null_nodeget+0x134 8 0% 100% 0.00 533 vnode interlock null_nodeget+0x88 8 0% 100% 0.00 1572 resv obj lock vm_page_alloc_domain_after+0xb0 8 0% 100% 0.00 23136 process lock sleepq_wait_sig+0xc 8 0% 100% 0.00 144 4096 free+0x4c 7 0% 100% 0.00 1892 xhci1 ue_start+0x58 7 0% 100% 0.00 403 vnode_free_list zfs_freebsd_symlink+0x2dc 7 0% 100% 0.00 200 pmap pv chunk list pmap_remove_l3_range+0x348 7 0% 100% 0.00 739 dp_zil_clean_taskq taskqueue_thread_loop+0xd0 7 0% 100% 0.00 76330 axe0 usb_process+0xfc 7 0% 100% 0.00 164 4096 malloc+0x7c 6 0% 100% 0.00 7992 xhci0 usb_process+0xfc 6 0% 100% 0.00 152 vm pgcache vm_object_terminate+0x9c 6 0% 100% 0.00 6559 reseed mutex read_random_uio+0x1d8 6 0% 100% 0.00 192 devfs interlock devfs_lookup+0x220 6 0% 100% 0.00 2840 cdev vn_close1+0x174 6 0% 100% 0.00 148 abd_chunk abd_free+0x70 5 0% 100% 0.00 2342 zio_null_issue spa_taskq_dispatch_ent+0x84 5 0% 100% 0.00 1240 vnode interlock VOP_ADD_WRITECOUNT_APV+0x4c 5 0% 100% 0.00 21835 ncvn cache_purge+0xa8 5 0% 100% 0.00 164 jail mutex crcopysafe+0x174 4 0% 100% 0.00 522 vm page kern_execve+0x4d8 4 0% 100% 0.00 622 process lock VOP_CLOSE_APV+0x4c 4 0% 100% 0.00 1955 process lock do_el0_sync+0x89c 4 0% 100% 0.00 3533 process lock sys_kill+0x188 4 0% 100% 0.00 400 cdev devfs_close+0x2ac 4 0% 100% 0.00 1911 USB device mutex usbd_req_get_port_status+0x3c 3 0% 100% 0.00 185 zio_cache zio_done+0xa84 3 0% 100% 0.00 16392 xhci1 usbd_transfer_submit+0xcc 3 0% 100% 0.00 3807 xhci0 usbd_do_request_flags+0x334 3 0% 100% 0.00 1133 vnode_free_list sync_fsync+0x128 3 0% 100% 0.00 1489 vnode interlock VOP_CLOSE_APV+0x4c 3 0% 100% 0.00 2873 vnode interlock lookup+0x940 3 0% 100% 0.00 3866 vm page vm_fault_quick_hold_pages+0x15c 3 0% 100% 0.00 155 umtxql __umtx_op_nwake_private+0x68 3 0% 100% 0.00 8904 ttymtx ttydisc_read+0x12c 3 0% 100% 0.00 1319 ttymtx kern_select+0x670 3 0% 100% 0.00 185 resv obj lock vm_reserv_free_page+0x278 3 0% 100% 0.00 207 process lock bufspace_daemon+0xd0 3 0% 100% 0.00 237 pmap pv chunk list pmap_advise+0x138 3 0% 100% 0.00 185 pmap pv chunk list pmap_remove+0x164 3 0% 100% 0.00 170 dmu_buf_impl_t dbuf_destroy+0x420 3 0% 100% 0.00 41859 axe0 usbd_do_request_proc+0x94 2 0% 100% 0.00 255 zio_null_issue taskqueue_thread_loop+0xc0 2 0% 100% 0.00 4222 xhci1 usbd_req_get_port_status+0x3c 2 0% 100% 0.00 15489 xhci1 usbd_do_request_flags+0x334 2 0% 100% 0.00 214689 xhci0 ithread_loop+0x1f4 2 0% 100% 0.00 5567 xhci0 usbd_req_get_port_status+0x3c 2 0% 100% 0.00 3177 xhci0 xhci_do_command+0x294 2 0% 100% 0.00 433 vnode_free_list _vget_prep+0x140 2 0% 100% 0.00 244 vm pgcache vm_page_alloc_domain_after+0xc8 2 0% 100% 0.00 777 umass1 usb_process+0xfc 2 0% 100% 0.00 1644 ttymtx devfs_ioctl+0xf0 2 0% 100% 0.00 1600 tmpfs allnode lock tmpfs_alloc_vp+0x35c 2 0% 100% 0.00 878 so_rcv soreceive_generic+0x1d0 2 0% 100% 0.00 288 so_rcv sosend_generic+0x488 2 0% 100% 0.00 25633 process lock handle_el0_irq+0xb8 2 0% 100% 0.00 211 process lock buf_daemon+0x1a8 2 0% 100% 0.00 22433 process lock sys_thr_exit+0x60 2 0% 100% 0.00 367 process lock sys_kill+0xb0 2 0% 100% 0.00 5544 process lock kqueue_kevent+0xc8 2 0% 100% 0.00 8322 pmap vmspace_fork+0x26c 2 0% 100% 0.00 1611 pipe mutex pipe_write+0x698 2 0% 100% 0.00 3011 pfs_vncache lookup+0x850 2 0% 100% 0.00 156 ncvn null_vptocnp+0x98 2 0% 100% 0.00 188 kstack_cache thread_reap+0x11c 2 0% 100% 0.00 1622 cdev vget+0x18 2 0% 100% 0.00 222 abd_chunk abd_alloc+0x108 2 0% 100% 0.00 233 TID lock thread_ctor+0x2c 2 0% 100% 0.00 211 RADIX NODE vm_radix_insert+0xf0 2 0% 100% 0.00 10633 Giant usbd_req_get_port_status+0x3c 2 0% 100% 0.00 20733 Giant uhub_explore+0x64 2 0% 100% 0.00 289 8192 malloc+0x7c 2 0% 100% 0.00 122 512 free+0x4c 1 0% 100% 0.00 200 zio_link_cache zio_add_child+0x30 1 0% 100% 0.00 156 zio_null_intr spa_taskq_dispatch_ent+0x84 1 0% 100% 0.00 178 zfs_znode_cache zfs_znode_alloc+0x40 1 0% 100% 0.00 200 xhci0 usbd_transfer_submit+0xcc 1 0% 100% 0.00 2955 xhci1 xpt_run_devq+0x360 1 0% 100% 0.00 200 vnode_free_list vfs_allocate_syncvnode+0x34 1 0% 100% 0.00 5333 xhci0 usbd_callback_wrapper+0x5d4 1 0% 100% 0.00 245 vnode interlock devfs_lookup+0x220 1 0% 100% 0.00 178 vm page free queue vm_object_terminate+0x9c 1 0% 100% 0.00 1666 ttymtx dofileread+0x7c 1 0% 100% 0.00 5422 ttymtx leavepgrp+0xf4 1 0% 100% 0.00 1578 tmpfs allnode lock tmpfs_reclaim+0xd8 1 0% 100% 0.00 155 struct mount vlist mtx vrefl+0xc8 1 0% 100% 0.00 222 so_rcv kern_select+0x670 1 0% 100% 0.00 222 so_rcv soreceive_generic+0x460 1 0% 100% 0.00 2978 ttymtx kern_select+0x894 1 0% 100% 0.00 1488 ue0 ether_output_frame+0xc0 1 0% 100% 0.00 10556 unp soreceive_generic+0x460 1 0% 100% 0.00 245 sleep mtxpool kern_getdirentries+0x2ec 1 0% 100% 0.00 178 session sys_vfork+0x34 1 0% 100% 0.00 112 session kern_proc_out+0x34 1 0% 100% 0.00 3045 sleep mtxpool g_io_deliver+0x244 1 0% 100% 0.00 177 resv obj lock vm_fault+0x1108 1 0% 100% 0.00 5444 resv obj lock vm_reserv_alloc_page+0x50c 1 0% 100% 0.00 2889 sellck pipe_write+0xe40 1 0% 100% 0.00 178 procid proc_reap+0x1fc 1 0% 100% 0.00 289 reseed mutex random_kthread+0xe0 1 0% 100% 0.00 155 procid sys_setpgid+0x1cc 1 0% 100% 0.00 10778 reseed mutex random_fortuna_read+0xa4 1 0% 100% 0.00 444 process lock sys_vfork+0x34 1 0% 100% 0.00 222 ncvn tmpfs_rmdir+0x1d4 1 0% 100% 0.00 356 pipe mutex kern_poll+0x380 1 0% 100% 0.00 2933 process lock filt_procattach+0x88 1 0% 100% 0.00 156 RADIX NODE vm_radix_reclaim_allnodes_int+0x4c 1 0% 100% 0.00 134 RADIX NODE vm_radix_remove+0x104 1 0% 100% 0.00 133 8192 free+0x4c 1 0% 100% 0.00 155 65536 free+0x4c 1 0% 100% 0.00 156 32768 malloc+0x7c 1 0% 100% 0.00 267 2048 free+0x4c 1 0% 100% 0.00 133 exec args kva kern_execve+0x834 1 0% 100% 0.00 200 cdev devfs_allocv+0x128 1 0% 100% 0.00 155 g_bio vdev_geom_io_start+0xd8 1 0% 100% 0.00 111 jail mutex exit1+0x1004 ------------------------------------------------------------------------------- Adaptive mutex block: 12715 events in 300.017 seconds (42 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 2433 19% 19% 0.00 973791 vm page vm_fault_trap+0x64 1210 10% 29% 0.00 317424 vm active pagequeue vm_page_activate+0x74 804 6% 35% 0.00 353864 vm active pagequeue vm_page_free_prep+0x184 723 6% 41% 0.00 8905 xhci1 fork_exit+0x90 687 5% 46% 0.00 2942969 pmap data_abort+0xc4 657 5% 51% 0.00 769191 vm page vm_fault+0xd90 561 4% 56% 0.00 4146 axe0 usb_process+0xfc 486 4% 59% 0.00 578606 zio_write_issue spa_taskq_dispatch_ent+0x84 476 4% 63% 0.00 221838 zio_write_issue taskqueue_thread_loop+0xd0 304 2% 66% 0.00 93446 zio_write_issue taskqueue_thread_loop+0xc0 286 2% 68% 0.00 17982 zio_write_intr taskqueue_thread_loop+0xd0 280 2% 70% 0.00 553473 dbu_evict dbuf_destroy+0xb0 268 2% 72% 0.00 16970 axe0 fork_exit+0x90 255 2% 74% 0.00 15531 xhci0 fork_exit+0x90 215 2% 76% 0.00 211355 vm active pagequeue vm_page_deactivate+0x4c 157 1% 77% 0.00 144928 pmap pv chunk list pmap_try_insert_pv_entry+0x24 155 1% 78% 0.00 454903 vm page free queue vm_page_alloc_domain_after+0xb0 132 1% 79% 0.00 193984 pmap pv chunk list pmap_remove_pages+0x8c 132 1% 80% 0.00 384025 VM reserv domain vm_reserv_alloc_page+0x2b0 124 1% 81% 0.00 16321 zio_write_intr taskqueue_thread_loop+0xc0 111 1% 82% 0.00 14829 AHCI channel lock ahci_intr+0xd4 109 1% 83% 0.00 21701 zio_write_intr spa_taskq_dispatch_ent+0x84 105 1% 84% 0.00 1083533 vm active pagequeue vm_page_alloc_domain_after+0x200 104 1% 85% 0.00 391419 dbu_evict taskqueue_thread_loop+0xd0 102 1% 86% 0.00 481127 vm page free queue vm_page_free_prep+0x1b8 98 1% 86% 0.00 615393 vm page uiomove_object+0x1c4 97 1% 87% 0.00 97205 pipe mutex pipe_read+0x3d8 96 1% 88% 0.00 397118 pmap vm_fault+0x1634 95 1% 89% 0.00 71130 process lock kern_wait6+0x1ec 85 1% 89% 0.00 1014837 umtxql do_lock_umutex+0x7ec 84 1% 90% 0.00 12694 umass2 usb_process+0xfc 84 1% 91% 0.00 128670 pmap _kmem_unback+0x34 82 1% 91% 0.00 12711 umass1 usb_process+0xfc 69 1% 92% 0.00 577687 vnode_free_list zfs_zget+0x38 65 1% 92% 0.00 73352 pmap kmem_back_domain+0x140 63 0% 93% 0.00 269064 umtxql do_wait+0x35c 57 0% 93% 0.00 227370 VM reserv domain vm_page_free_prep+0x1b8 53 0% 94% 0.00 18179 CAM device lock g_disk_start+0x30c 49 0% 94% 0.00 556692 process lock handle_el0_sync+0xc0 44 0% 94% 0.00 1749896 umtxql __umtx_op_wait_umutex+0x48 43 0% 95% 0.00 266988 dbu_evict taskqueue_thread_loop+0xc0 42 0% 95% 0.00 438105 ncvn zfs_lookup+0x66c 35 0% 95% 0.00 148603 process lock fork_rfppwait+0xf4 32 0% 96% 0.00 1458056 vm page zfs_freebsd_getpages+0x1c4 32 0% 96% 0.00 771697 umtxql do_el0_sync+0x550 24 0% 96% 0.00 261074 vnode_free_list zfs_freebsd_create+0x3e0 24 0% 96% 0.00 372600 vm page vm_pageout+0x1b0 24 0% 96% 0.00 1375930 VM reserv domain vm_reserv_alloc_page+0x524 23 0% 97% 0.00 152877 pmap pv chunk list pmap_enter+0x6f8 22 0% 97% 0.00 217976 kernel arena domain vmem_alloc+0x58 21 0% 97% 0.00 753434 vm page elf64_load_sections+0x1cc 21 0% 97% 0.00 33845 CAM device lock xpt_done_process+0x690 20 0% 97% 0.00 996785 vm active pagequeue vm_page_unwire+0x15c 19 0% 97% 0.00 198455 UMA Slabs malloc+0x12c 16 0% 97% 0.00 15776763 pmap vm_fault_prefault+0x17c 15 0% 98% 0.00 379893 vm page free queue uma_zalloc_arg+0x47c 14 0% 98% 0.00 173549 vnode_free_list _vget_prep+0x68 12 0% 98% 0.00 2559277 vm page kern_execve+0x6e4 12 0% 98% 0.00 12127555 pmap vm_fault_prefault+0x110 11 0% 98% 0.00 145347 vm active pagequeue vm_page_advise+0x170 11 0% 98% 0.00 3036953 pmap vm_fault+0x364 11 0% 98% 0.00 13369 dp_sync_taskq taskqueue_thread_loop+0xc0 11 0% 98% 0.00 15438 CAM device lock xpt_run_allocq+0x180 10 0% 98% 0.00 38866 vm active pagequeue vm_pageout+0x1b0 10 0% 98% 0.00 27360 sleep mtxpool zio_vdev_io_start+0x278 10 0% 98% 0.00 430968 ncvn tmpfs_lookup1+0x13c 8 0% 99% 0.00 71813 kernel arena domain uma_large_free+0x18 7 0% 99% 0.00 752911 vm inactive pagequeue vm_page_free_prep+0x184 7 0% 99% 0.00 93882 vm inactive pagequeue release_page+0x98 7 0% 99% 0.00 212711 UMA Slabs zone_free_item+0x70 6 0% 99% 0.00 272174 vm page free queue bucket_drain+0x80 6 0% 99% 0.00 1850562 vm page exec_map_first_page+0x350 6 0% 99% 0.00 275281 process_exit sys_sys_exit+0x10 6 0% 99% 0.00 87315 pmap vm_map_delete+0x17c 6 0% 99% 0.00 3015455 ncnegl cache_zap_locked+0x78 5 0% 99% 0.00 33986 process lock thread_suspend_check+0x434 5 0% 99% 0.00 18089 pipe mutex pipe_read+0xa8 4 0% 99% 0.00 1501522 vm page vm_map_madvise+0x268 4 0% 99% 0.00 471038 vm page elf64_load_file+0x128 4 0% 99% 0.00 98817 vm inactive pagequeue vm_page_alloc_domain_after+0x200 4 0% 99% 0.00 432250 umtxql do_lock_umutex+0x5a8 4 0% 99% 0.00 163728 procid proc_reap+0x2f0 4 0% 99% 0.00 1539122 process_ctor uma_zalloc_arg+0x8b0 4 0% 99% 0.00 14150 g_disk_start g_io_request+0x27c 4 0% 99% 0.00 89138 dp_sync_taskq taskqueue_thread_loop+0xd0 4 0% 99% 0.00 64371 dp_sync_taskq taskq_dispatch+0x6c 3 0% 99% 0.00 227281 vnode_free_list tmpfs_alloc_vp+0x3f0 3 0% 99% 0.00 85600 vnode_free_list vrefl+0xc8 3 0% 99% 0.00 194600 vm inactive pagequeue vm_object_madvise+0x290 3 0% 99% 0.00 800066 thread_ctor uma_zalloc_arg+0x8b0 3 0% 99% 0.00 62266 struct mount vlist mtx _vget_prep+0x68 3 0% 99% 0.00 15585 sleep mtxpool g_part_start+0xf4 3 0% 99% 0.00 16326 sellck seltdwait+0x7c 3 0% 99% 0.00 434385 procid sys_fork+0x30 3 0% 100% 0.00 210215 process lock sys_sys_exit+0x10 2 0% 100% 0.00 36689 vnode interlock cache_lookup+0x5c8 2 0% 100% 0.00 82544 thread_dtor uma_zfree_arg+0x44 2 0% 100% 0.00 1526211 process_exec exec_elf64_imgact+0x9e8 2 0% 100% 0.00 14566 process_dtor uma_zfree_arg+0x44 2 0% 100% 0.00 110289 process lock exit1+0x5ac 2 0% 100% 0.00 246222 pipe mutex pipe_close+0x68 2 0% 100% 0.00 1285544 ncvn zfs_lookup+0x720 2 0% 100% 0.00 20278 dp_sync_taskq taskqueue_quiesce+0x1b8 2 0% 100% 0.00 10522 USB device mutex usbd_req_get_port_status+0x3c 2 0% 100% 0.00 13211 USB device mutex usbd_do_request_proc+0x94 2 0% 100% 0.00 17700 CAM queue lock xpt_done_td+0x114 1 0% 100% 0.00 6089 zio_free_issue_2 taskqueue_thread_loop+0xd0 1 0% 100% 0.00 27600 zio_free_issue_2 spa_taskq_dispatch_ent+0x84 1 0% 100% 0.00 21155 zio_free_issue_3 spa_taskq_dispatch_ent+0x84 1 0% 100% 0.00 28133 xhci1 usb_command_wrapper+0x84 1 0% 100% 0.00 17089 xhci1 usbd_transfer_submit+0xcc 1 0% 100% 0.00 26289 zio_free_issue_0 taskqueue_thread_loop+0xc0 1 0% 100% 0.00 16533 zio_free_issue_0 spa_taskq_dispatch_ent+0x84 1 0% 100% 0.00 1377133 vnode_free_list _vdrop+0x424 1 0% 100% 0.00 839555 vnode interlock _vget_prep+0x68 1 0% 100% 0.00 89067 vnode_free_list zfs_freebsd_mkdir+0x31c 1 0% 100% 0.00 81933 vnode_free_list zfs_freebsd_symlink+0x2dc 1 0% 100% 0.00 49245 vm inactive pagequeue dmu_read_pages+0x4dc 1 0% 100% 0.00 31778 vm active pagequeue vm_pageout_worker+0x123c 1 0% 100% 0.00 1238178 vm page zfs_freebsd_read+0x450 1 0% 100% 0.00 30622 procid sys_vfork+0x34 1 0% 100% 0.00 251356 struct mount vlist mtx vputx+0x3a0 1 0% 100% 0.00 45800 process lock sys_execve+0x54 1 0% 100% 0.00 57178 process lock kern_execve+0x47c 1 0% 100% 0.00 14289 pmap pv chunk list pmap_remove_l3_range+0x348 1 0% 100% 0.00 477311 pmap pv chunk list pmap_pv_insert_l2+0x28 1 0% 100% 0.00 87534 process lock do_lock_umutex+0x9a8 1 0% 100% 0.00 728133 process lock sys_thr_exit+0x60 1 0% 100% 0.00 17933 process lock sys_sigprocmask+0x64 1 0% 100% 0.00 185311 process lock sys_kill+0xb0 1 0% 100% 0.00 21022 USB device mutex usb_process+0xfc 1 0% 100% 0.00 61644 TID lock uma_zfree_arg+0x44 1 0% 100% 0.00 379756 pipe mutex dofilewrite+0x7c 1 0% 100% 0.00 334689 pipe mutex kern_fstat+0xa8 1 0% 100% 0.00 688756 ncnegl zfs_lookup+0x66c 1 0% 100% 0.00 63334 ncneglh vfs_cache_lookup+0xb4 1 0% 100% 0.00 238844 ncvn vfs_cache_lookup+0xb4 1 0% 100% 0.00 476489 ncvn vn_fullpath1+0x144 1 0% 100% 0.00 1759844 pmap vm_pageout_worker+0x11b0 1 0% 100% 0.00 13889 CAM queue lock ahci_end_transaction+0x824 1 0% 100% 0.00 6466 CAM queue lock adastart+0x8cc 1 0% 100% 0.00 1222378 256 free+0x4c 1 0% 100% 0.00 2053467 256 malloc+0x7c 1 0% 100% 0.00 35222 32 malloc+0x7c 1 0% 100% 0.00 23956 32 free+0x4c ------------------------------------------------------------------------------- Spin lock spin: 832291 events in 300.017 seconds (2774 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 494754 59% 59% 0.00 10253 sleepq chain taskqueue_enqueue_locked+0x164 171855 21% 80% 0.00 2215 sleepq chain taskqueue_thread_loop+0xc0 11987 1% 82% 0.00 5848 sleepq chain taskqueue_run_locked+0x80 6828 1% 82% 0.00 585 sleepq chain pipe_write+0xcbc 5495 1% 83% 0.00 68176 turnstile chain __mtx_lock_sleep+0xe4 2824 0% 83% 0.00 1189 sched lock 0 sched_switch+0x6a4 2538 0% 84% 0.00 1351 sched lock 3 sched_switch+0x6a4 2340 0% 84% 0.00 979 sched lock 1 sched_switch+0x6a4 2157 0% 84% 0.00 1654 sched lock 28 mi_switch+0xfc 2148 0% 84% 0.00 815 sleepq chain __umtx_op_wake2_umutex+0x5b0 2130 0% 85% 0.00 1278 sched lock 31 mi_switch+0xfc 2099 0% 85% 0.00 1303 sched lock 30 mi_switch+0xfc 1975 0% 85% 0.00 51107 sleepq chain userquota_updates_task+0x2ec 1970 0% 85% 0.00 1466 sched lock 26 mi_switch+0xfc 1877 0% 86% 0.00 1707 sched lock 3 setrunnable+0x58 1875 0% 86% 0.00 1384 sched lock 25 mi_switch+0xfc 1855 0% 86% 0.00 1407 sched lock 24 mi_switch+0xfc 1853 0% 86% 0.00 955 sched lock 4 sched_switch+0x6a4 1851 0% 87% 0.00 1689 sched lock 29 mi_switch+0xfc 1815 0% 87% 0.00 1254 sched lock 27 mi_switch+0xfc 1801 0% 87% 0.00 1116 sleepq chain xpt_done+0x114 1781 0% 87% 0.00 1409 sched lock 21 mi_switch+0xfc 1751 0% 87% 0.00 1374 sched lock 23 mi_switch+0xfc 1747 0% 88% 0.00 1430 sched lock 18 mi_switch+0xfc 1718 0% 88% 0.00 1017 sched lock 7 sched_switch+0x6a4 1718 0% 88% 0.00 1534 sched lock 19 mi_switch+0xfc 1695 0% 88% 0.00 1342 sched lock 20 mi_switch+0xfc 1681 0% 88% 0.00 1047 sched lock 5 sched_switch+0x6a4 1661 0% 89% 0.00 1074 sched lock 2 sched_switch+0x6a4 1647 0% 89% 0.00 1345 sched lock 17 mi_switch+0xfc 1641 0% 89% 0.00 1347 sched lock 22 mi_switch+0xfc 1598 0% 89% 0.00 1440 sched lock 16 mi_switch+0xfc 1584 0% 89% 0.00 21708 sleepq chain vdev_queue_io+0x1c0 1569 0% 90% 0.00 901 sched lock 8 sched_switch+0x6a4 1552 0% 90% 0.00 1011 sched lock 6 sched_switch+0x6a4 1532 0% 90% 0.00 764 sleepq chain pipe_read+0x4f4 1480 0% 90% 0.00 1346 sched lock 15 mi_switch+0xfc 1430 0% 90% 0.00 1325 sched lock 13 mi_switch+0xfc 1396 0% 90% 0.00 987 sched lock 11 sched_switch+0x6a4 1396 0% 91% 0.00 2004 sched lock 0 setrunnable+0x58 1395 0% 91% 0.00 10571 sleepq chain dsl_dir_diduse_space+0x1a8 1291 0% 91% 0.00 1982 sched lock 10 sched_switch+0x6a4 1277 0% 91% 0.00 1317 sleepq chain kern_umtx_wake+0x108 1269 0% 91% 0.00 1484 sched lock 12 mi_switch+0xfc 1250 0% 91% 0.00 1449 sched lock 11 mi_switch+0xfc 1242 0% 92% 0.00 1415 sched lock 14 mi_switch+0xfc 1235 0% 92% 0.00 944 sched lock 9 sched_switch+0x6a4 1202 0% 92% 0.00 4101 turnstile lock __rw_wunlock_hard+0x94 1175 0% 92% 0.00 1019 sched lock 12 sched_switch+0x6a4 1171 0% 92% 0.00 1527 sched lock 10 mi_switch+0xfc 1148 0% 92% 0.00 1707 sched lock 5 setrunnable+0x58 1148 0% 92% 0.00 1814 sched lock 4 setrunnable+0x58 1144 0% 93% 0.00 1805 sched lock 2 setrunnable+0x58 1142 0% 93% 0.00 2344 sched lock 1 setrunnable+0x58 1137 0% 93% 0.00 1683 sched lock 10 setrunnable+0x58 1091 0% 93% 0.00 973 sched lock 13 sched_switch+0x6a4 1090 0% 93% 0.00 1413 sleepq chain vm_map_find+0x5f8 1051 0% 93% 0.00 1786 sched lock 7 setrunnable+0x58 1049 0% 93% 0.00 1511 sched lock 8 mi_switch+0xfc 1040 0% 93% 0.00 1751 sched lock 9 setrunnable+0x58 1035 0% 94% 0.00 1832 sched lock 6 setrunnable+0x58 1026 0% 94% 0.00 2154 sched lock 13 setrunnable+0x58 1015 0% 94% 0.00 1755 sched lock 11 setrunnable+0x58 1005 0% 94% 0.00 1748 sched lock 12 setrunnable+0x58 998 0% 94% 0.00 1766 sched lock 8 setrunnable+0x58 991 0% 94% 0.00 1571 sched lock 7 mi_switch+0xfc 985 0% 94% 0.00 1002 sched lock 15 sched_switch+0x6a4 957 0% 94% 0.00 1820 sched lock 17 setrunnable+0x58 927 0% 95% 0.00 2177 sched lock 9 mi_switch+0xfc 920 0% 95% 0.00 1766 sched lock 15 setrunnable+0x58 917 0% 95% 0.00 1776 sched lock 14 setrunnable+0x58 912 0% 95% 0.00 1681 sched lock 16 setrunnable+0x58 902 0% 95% 0.00 2078 sched lock 18 setrunnable+0x58 896 0% 95% 0.00 1782 sched lock 19 setrunnable+0x58 890 0% 95% 0.00 1822 sched lock 20 setrunnable+0x58 844 0% 95% 0.00 1710 sched lock 6 mi_switch+0xfc 816 0% 95% 0.00 1114 sched lock 17 sched_switch+0x6a4 811 0% 96% 0.00 2037 sched lock 24 setrunnable+0x58 811 0% 96% 0.00 1716 sched lock 21 setrunnable+0x58 811 0% 96% 0.00 1163 sched lock 16 sched_switch+0x6a4 808 0% 96% 0.00 1751 sched lock 22 setrunnable+0x58 807 0% 96% 0.00 2933 sleepq chain rangelock_calc_block+0x58 805 0% 96% 0.00 1798 sched lock 23 setrunnable+0x58 798 0% 96% 0.00 993 sched lock 14 sched_switch+0x6a4 797 0% 96% 0.00 4261 sleepq chain _vm_map_lock_upgrade+0xe4 782 0% 96% 0.00 1835 sched lock 5 mi_switch+0xfc 781 0% 96% 0.00 1155 sched lock 18 sched_switch+0x6a4 770 0% 96% 0.00 1683 sched lock 25 setrunnable+0x58 751 0% 97% 0.00 1802 sched lock 4 mi_switch+0xfc 741 0% 97% 0.00 3589 turnstile chain __rw_wlock_hard+0x218 731 0% 97% 0.00 2423 sched lock 28 setrunnable+0x58 722 0% 97% 0.00 9237 turnstile lock __mtx_lock_sleep+0xe4 705 0% 97% 0.00 1730 sched lock 30 setrunnable+0x58 700 0% 97% 0.00 1787 sched lock 27 setrunnable+0x58 695 0% 97% 0.00 1070 sched lock 19 sched_switch+0x6a4 666 0% 97% 0.00 1739 sched lock 31 setrunnable+0x58 655 0% 97% 0.00 2357 sched lock 29 setrunnable+0x58 640 0% 97% 0.00 1651 sched lock 26 setrunnable+0x58 606 0% 97% 0.00 1214 sched lock 20 sched_switch+0x6a4 600 0% 97% 0.00 16025 sleepq chain dbuf_write_ready+0x494 562 0% 98% 0.00 3837 turnstile lock __mtx_unlock_sleep+0x7c 515 0% 98% 0.00 2985 sched lock 3 mi_switch+0xfc 510 0% 98% 0.00 1109 sched lock 21 sched_switch+0x6a4 494 0% 98% 0.00 1385 sched lock 3 sched_idletd+0x2a0 456 0% 98% 0.00 3350 turnstile lock __rw_wlock_hard+0x218 436 0% 98% 0.00 30749 sleepq chain dsl_pool_undirty_space+0xec 432 0% 98% 0.00 1122 sched lock 22 sched_switch+0x6a4 431 0% 98% 0.00 2697 callout _callout_stop_safe+0xc0 426 0% 98% 0.00 19563 sleepq chain vdev_queue_io_done+0x184 425 0% 98% 0.00 2335 sched lock 2 mi_switch+0xfc 395 0% 98% 0.00 1175 sched lock 23 sched_switch+0x6a4 381 0% 98% 0.00 5794 sleepq chain pipe_read+0x3d8 348 0% 98% 0.00 7319 sleepq chain pipe_write+0xe04 333 0% 98% 0.00 6322 sleepq chain exit1+0x101c 328 0% 98% 0.00 3082 sched lock 1 mi_switch+0xfc 321 0% 98% 0.00 6388 sleepq chain metaslab_alloc_dva+0xab0 315 0% 98% 0.00 17961 turnstile chain taskqueue_run_locked+0x114 314 0% 98% 0.00 1144 sched lock 25 sched_switch+0x6a4 310 0% 98% 0.00 6495 sleepq chain pipeclose+0x240 307 0% 98% 0.00 1061 sched lock 24 sched_switch+0x6a4 296 0% 98% 0.00 7638 sleepq chain dsl_dir_tempreserve_impl+0x2f8 273 0% 98% 0.00 3384 sched lock 0 mi_switch+0xfc 271 0% 99% 0.00 1638 callout callout_reset_sbt_on+0x84 256 0% 99% 0.00 21555 sleepq chain vdev_queue_io_done+0x154 250 0% 99% 0.00 12834 turnstile chain unlock_mtx+0x4c 248 0% 99% 0.00 21651 sleepq chain vdev_queue_io_to_issue+0x9e0 247 0% 99% 0.00 2587 sleepq chain dsl_dir_tempreserve_impl+0x308 233 0% 99% 0.00 17272 sleepq chain kern_wait6+0x5a8 193 0% 99% 0.00 6347 sleepq chain umtxq_sleep+0x13c 190 0% 99% 0.00 3074 sleepq chain xpt_done_td+0xac 190 0% 99% 0.00 1193 sched lock 27 sched_switch+0x6a4 185 0% 99% 0.00 1022 sched lock 26 sched_switch+0x6a4 184 0% 99% 0.00 51689 sleepq chain userquota_updates_task+0x3f8 181 0% 99% 0.00 14098 sleepq chain dsl_dir_tempreserve_clear+0x100 177 0% 99% 0.00 2002 sched lock 0 sched_idletd+0x2a0 174 0% 99% 0.00 7241 sleepq chain vdev_queue_io+0x1d0 172 0% 99% 0.00 16015 sleepq chain metaslab_alloc_dva+0xa88 170 0% 99% 0.00 23590 turnstile chain taskqueue_enqueue_locked+0x28c 168 0% 99% 0.00 4044 asid set vmspace_exit+0xe8 158 0% 99% 0.00 2357 asid set pmap_switch+0x2c 155 0% 99% 0.00 1735 turnstile chain pmap_try_insert_pv_entry+0x11c 152 0% 99% 0.00 15308 sleepq chain metaslab_alloc_dva+0xc64 134 0% 99% 0.00 1001 sched lock 28 sched_switch+0x6a4 119 0% 99% 0.00 2262 sleepq chain lockmgr_unlock_fast_path+0x60 116 0% 99% 0.00 1583 turnstile chain pmap_remove_pages+0x514 116 0% 99% 0.00 2185 sleepq chain vm_map_lookup_done+0x4c 106 0% 99% 0.00 7982 sleepq chain dsl_dir_tempreserve_clear+0xec 106 0% 99% 0.00 940 sched lock 29 sched_switch+0x6a4 100 0% 99% 0.00 420 sleepq chain txg_sync_thread+0x14c 99 0% 99% 0.00 6060 sleepq chain zio_allocate_dispatch+0x18c 97 0% 99% 0.00 2603 sched lock 2 sched_idletd+0x2a0 94 0% 99% 0.00 1253 sleepq chain vm_map_protect+0x200 89 0% 99% 0.00 7960 sleepq chain vdev_queue_io_done+0x13c 87 0% 99% 0.00 2414 sleepq chain usb_proc_msignal+0x100 85 0% 99% 0.00 12225 sleepq chain spa_config_enter+0x19c 84 0% 99% 0.00 2975 sched lock 29 fork_exit+0x90 83 0% 99% 0.00 1443 sched lock 3 fork_exit+0x90 81 0% 99% 0.00 2838 sched lock 4 sched_idletd+0x2a0 80 0% 99% 0.00 6145 sleepq chain vm_map_lookup+0xac 80 0% 99% 0.00 1053 sleepq chain __umtx_op_rw_unlock+0x358 80 0% 99% 0.00 8140 sleepq chain usb_process+0xe0 80 0% 99% 0.00 2611 sched lock 18 fork_exit+0x90 78 0% 99% 0.00 3365 sched lock 31 fork_exit+0x90 78 0% 99% 0.00 3197 sched lock 30 fork_exit+0x90 77 0% 99% 0.00 7256 sched lock 3 fork1+0x1b14 76 0% 99% 0.00 2846 sched lock 1 sched_idletd+0x2a0 74 0% 99% 0.00 2806 sleepq chain metaslab_alloc_dva+0xc8c 74 0% 99% 0.00 40071 sleepq chain dsl_pool_need_dirty_delay+0x114 73 0% 99% 0.00 3201 sleepq chain pipe_read+0xa8 72 0% 99% 0.00 3062 sched lock 5 sched_idletd+0x2a0 71 0% 99% 0.00 2577 sched lock 20 fork_exit+0x90 71 0% 99% 0.00 2973 sched lock 17 fork_exit+0x90 69 0% 99% 0.00 3171 sched lock 28 fork_exit+0x90 68 0% 99% 0.00 2776 sched lock 27 fork_exit+0x90 66 0% 99% 0.00 595 sleepq chain dbuf_read_done+0xcc 64 0% 99% 0.00 2561 sched lock 25 fork_exit+0x90 61 0% 99% 0.00 85246 turnstile chain vm_page_pqbatch_submit+0x218 61 0% 99% 0.00 9788 sleepq chain spa_config_exit+0x130 61 0% 99% 0.00 18493 sleepq chain dsl_pool_need_dirty_delay+0x124 61 0% 99% 0.00 2762 sched lock 9 sched_idletd+0x2a0 61 0% 99% 0.00 2569 sched lock 19 fork_exit+0x90 59 0% 99% 0.00 2400 sched lock 7 sched_idletd+0x2a0 59 0% 99% 0.00 2593 sched lock 6 sched_idletd+0x2a0 59 0% 99% 0.00 2632 sched lock 23 fork_exit+0x90 55 0% 99% 0.00 6124 sleepq chain dsl_dir_willuse_space+0x124 55 0% 99% 0.00 2823 sched lock 21 fork_exit+0x90 55 0% 99% 0.00 3444 sched lock 11 sched_idletd+0x2a0 54 0% 99% 0.00 3043 sched lock 26 fork_exit+0x90 52 0% 99% 0.00 2473 sched lock 22 fork_exit+0x90 50 0% 99% 0.00 1720 sched lock 7 fork_exit+0x90 50 0% 99% 0.00 1518 sched lock 0 fork1+0x1b14 49 0% 99% 0.00 2368 sched lock 12 fork_exit+0x90 48 0% 99% 0.00 3257 sched lock 24 fork_exit+0x90 48 0% 99% 0.00 2993 sched lock 15 fork_exit+0x90 48 0% 99% 0.00 3411 sched lock 13 sched_idletd+0x2a0 47 0% 99% 0.00 23589 sleepq chain trim_map_free+0xfc 47 0% 99% 0.00 5121 sleepq chain vdev_queue_io_to_issue+0x9f4 46 0% 99% 0.00 6529 sleepq chain metaslab_alloc_dva+0xa98 46 0% 99% 0.00 2532 sched lock 2 intr_event_schedule_thread+0x94 46 0% 99% 0.00 2528 sched lock 16 fork_exit+0x90 45 0% 99% 0.00 2937 sched lock 15 sched_idletd+0x2a0 45 0% 99% 0.00 2273 sched lock 10 fork_exit+0x90 44 0% 99% 0.00 2599 sched lock 8 sched_idletd+0x2a0 44 0% 99% 0.00 3090 sched lock 12 sched_idletd+0x2a0 42 0% 100% 0.00 2958 sched lock 9 fork_exit+0x90 42 0% 100% 0.00 2432 sched lock 0 intr_event_schedule_thread+0x94 41 0% 100% 0.00 2556 sched lock 10 sched_idletd+0x2a0 41 0% 100% 0.00 3781 callout softclock+0x74 40 0% 100% 0.00 2703 sched lock 11 fork_exit+0x90 39 0% 100% 0.00 7076 sleepq chain metaslab_class_throttle_unreserve+0xc8 38 0% 100% 0.00 1575 sched lock 9 intr_event_schedule_thread+0x94 38 0% 100% 0.00 2312 sched lock 5 intr_event_schedule_thread+0x94 37 0% 100% 0.00 2600 sleepq chain dsl_dir_willuse_space+0x13c 37 0% 100% 0.00 256 asid set pmap_activate+0x24 36 0% 100% 0.00 2124 sched lock 14 fork_exit+0x90 34 0% 100% 0.00 1729 sched lock 8 fork_exit+0x90 34 0% 100% 0.00 2091 sched lock 21 intr_event_schedule_thread+0x94 34 0% 100% 0.00 2820 sched lock 0 turnstile_unpend+0x1b4 33 0% 100% 0.00 620 turnstile chain vm_map_insert+0x388 33 0% 100% 0.00 21673 sleepq chain zio_dva_throttle+0x18c 32 0% 100% 0.00 189603 turnstile chain vm_page_deactivate+0x4c 32 0% 100% 0.00 1995 sched lock 8 intr_event_schedule_thread+0x94 32 0% 100% 0.00 2550 sched lock 13 fork_exit+0x90 31 0% 100% 0.00 2116 sleepq chain exit1+0x100 31 0% 100% 0.00 873 sched lock 4 fork1+0x1b14 31 0% 100% 0.00 3579 sched lock 17 sched_idletd+0x2a0 31 0% 100% 0.00 2467 sched lock 16 sched_idletd+0x2a0 30 0% 100% 0.00 3459 sleepq chain zio_allocate_dispatch+0x198 30 0% 100% 0.00 2237 sleepq chain release_page+0x48 30 0% 100% 0.00 1466 sched lock 6 fork1+0x1b14 30 0% 100% 0.00 1768 sched lock 20 intr_event_schedule_thread+0x94 30 0% 100% 0.00 3199 sched lock 1 intr_event_schedule_thread+0x94 29 0% 100% 0.00 673 td_contested __mtx_lock_sleep+0x140 29 0% 100% 0.00 2791 sched lock 6 fork_exit+0x90 29 0% 100% 0.00 1511 sched lock 19 intr_event_schedule_thread+0x94 28 0% 100% 0.00 3456 sleepq chain metaslab_alloc_dva+0x1b4 28 0% 100% 0.00 1073 sched lock 6 intr_event_schedule_thread+0x94 28 0% 100% 0.00 2015 sched lock 4 intr_event_schedule_thread+0x94 28 0% 100% 0.00 3252 sched lock 10 intr_event_schedule_thread+0x94 27 0% 100% 0.00 1809 sched lock 3 turnstile_unpend+0x1b4 27 0% 100% 0.00 2176 sched lock 1 turnstile_unpend+0x1b4 26 0% 100% 0.00 1074 sched lock 9 fork1+0x1b14 26 0% 100% 0.00 1351 sched lock 31 intr_event_schedule_thread+0x94 26 0% 100% 0.00 1308 sched lock 18 intr_event_schedule_thread+0x94 26 0% 100% 0.00 1891 sched lock 13 intr_event_schedule_thread+0x94 25 0% 100% 0.00 42844 turnstile chain vm_fault+0x171c 25 0% 100% 0.00 4108 sleepq chain dbuf_read+0xa38 25 0% 100% 0.00 1239 sched lock 30 intr_event_schedule_thread+0x94 25 0% 100% 0.00 999 sched lock 23 intr_event_schedule_thread+0x94 25 0% 100% 0.00 3065 sched lock 14 sched_idletd+0x2a0 24 0% 100% 0.00 3030 sleepq chain spa_config_enter+0x48 24 0% 100% 0.00 14451 sleepq chain dnode_hold_impl+0x210 24 0% 100% 0.00 1701 sleepq chain vm_fault+0xd98 24 0% 100% 0.00 2398 sched lock 9 turnstile_unpend+0x1b4 24 0% 100% 0.00 1203 sched lock 7 fork1+0x1b14 24 0% 100% 0.00 1917 sched lock 25 intr_event_schedule_thread+0x94 24 0% 100% 0.00 1189 sched lock 22 intr_event_schedule_thread+0x94 24 0% 100% 0.00 646 sched lock 19 fork1+0x1b14 24 0% 100% 0.00 3885 sched lock 18 sched_idletd+0x2a0 24 0% 100% 0.00 2845 sched lock 17 intr_event_schedule_thread+0x94 23 0% 100% 0.00 298364 sleepq chain aggsum_add+0x24c 23 0% 100% 0.00 805 sched lock 8 fork1+0x1b14 23 0% 100% 0.00 1840 sched lock 28 intr_event_schedule_thread+0x94 23 0% 100% 0.00 1888 sched lock 27 intr_event_schedule_thread+0x94 23 0% 100% 0.00 1450 sched lock 24 intr_event_schedule_thread+0x94 23 0% 100% 0.00 2438 sched lock 20 sched_idletd+0x2a0 23 0% 100% 0.00 1580 sched lock 18 turnstile_unpend+0x1b4 23 0% 100% 0.00 1694 sched lock 10 turnstile_unpend+0x1b4 22 0% 100% 0.00 642 turnstile lock __rw_runlock_hard+0xe0 22 0% 100% 0.00 5579 sleepq chain spa_config_exit+0x70 22 0% 100% 0.00 2365 sched lock 8 turnstile_unpend+0x1b4 22 0% 100% 0.00 2974 sched lock 5 turnstile_unpend+0x1b4 22 0% 100% 0.00 1120 sched lock 5 fork1+0x1b14 22 0% 100% 0.00 588 sched lock 30 sched_switch+0x6a4 22 0% 100% 0.00 689 sched lock 29 turnstile_unpend+0x1b4 22 0% 100% 0.00 1211 sched lock 20 fork1+0x1b14 22 0% 100% 0.00 269 callout ithread_loop+0x1f4 21 0% 100% 0.00 2756 sleepq chain dsl_dir_diduse_space+0x1b8 21 0% 100% 0.00 12773 sleepq chain taskqueue_quiesce+0x1b8 21 0% 100% 0.00 1195 sched lock 7 intr_event_schedule_thread+0x94 21 0% 100% 0.00 2382 sched lock 29 intr_event_schedule_thread+0x94 21 0% 100% 0.00 1547 sched lock 21 turnstile_unpend+0x1b4 21 0% 100% 0.00 1616 sched lock 20 turnstile_unpend+0x1b4 21 0% 100% 0.00 1069 sched lock 2 fork1+0x1b14 21 0% 100% 0.00 1038 sched lock 17 fork1+0x1b14 21 0% 100% 0.00 1498 sched lock 16 fork1+0x1b14 21 0% 100% 0.00 1733 sched lock 11 intr_event_schedule_thread+0x94 20 0% 100% 0.00 2480 sched lock 7 turnstile_unpend+0x1b4 20 0% 100% 0.00 2135 sched lock 5 fork_exit+0x90 20 0% 100% 0.00 1735 sched lock 4 fork_exit+0x90 20 0% 100% 0.00 2414 sched lock 26 intr_event_schedule_thread+0x94 20 0% 100% 0.00 1315 sched lock 23 fork1+0x1b14 20 0% 100% 0.00 1136 sched lock 22 fork1+0x1b14 20 0% 100% 0.00 2419 sched lock 19 sched_idletd+0x2a0 20 0% 100% 0.00 1958 sched lock 17 turnstile_unpend+0x1b4 20 0% 100% 0.00 1516 sched lock 14 turnstile_unpend+0x1b4 20 0% 100% 0.00 1632 sched lock 13 fork1+0x1b14 20 0% 100% 0.00 1172 sched lock 12 intr_event_schedule_thread+0x94 20 0% 100% 0.00 1096 sched lock 10 fork1+0x1b14 19 0% 100% 0.00 11595 sched lock 30 turnstile_unpend+0x1b4 19 0% 100% 0.00 3333 sched lock 21 sched_idletd+0x2a0 19 0% 100% 0.00 2039 sched lock 16 intr_event_schedule_thread+0x94 19 0% 100% 0.00 2388 sched lock 15 intr_event_schedule_thread+0x94 19 0% 100% 0.00 578 sched lock 1 fork1+0x1b14 18 0% 100% 0.00 6277 turnstile chain pmap_enter+0x5ac 18 0% 100% 0.00 5202 sleepq chain kern_execve+0xd34 18 0% 100% 0.00 1307 sched lock 14 intr_event_schedule_thread+0x94 18 0% 100% 0.00 3246 sched lock 13 turnstile_unpend+0x1b4 18 0% 100% 0.00 1999 sched lock 11 turnstile_unpend+0x1b4 18 0% 100% 0.00 582 sched lock 11 fork1+0x1b14 17 0% 100% 0.00 2145 sleepq chain zio_done+0x2b4 17 0% 100% 0.00 1275 sleepq chain pipe_write+0xb0 17 0% 100% 0.00 929 sched lock 3 intr_event_schedule_thread+0x94 17 0% 100% 0.00 3210 sched lock 2 turnstile_unpend+0x1b4 17 0% 100% 0.00 674 sched lock 18 fork1+0x1b14 17 0% 100% 0.00 976 sched lock 14 fork1+0x1b14 16 0% 100% 0.00 4158 sleepq chain trim_map_write_done+0xec 16 0% 100% 0.00 1779 sleepq chain lockmgr_sunlock_hard+0x24 16 0% 100% 0.00 1352 sched lock 4 turnstile_unpend+0x1b4 16 0% 100% 0.00 934 sched lock 27 fork1+0x1b14 16 0% 100% 0.00 2458 sched lock 22 sched_idletd+0x2a0 16 0% 100% 0.00 1440 sched lock 2 fork_exit+0x90 15 0% 100% 0.00 2444 sleepq chain dsl_dataset_block_born+0x2e0 15 0% 100% 0.00 2697 sleepq chain arc_buf_access+0x36c 15 0% 100% 0.00 6564 sleepq chain fork_rfppwait+0xf4 15 0% 100% 0.00 1699 sched lock 31 turnstile_unpend+0x1b4 15 0% 100% 0.00 1897 sched lock 31 fork1+0x1b14 15 0% 100% 0.00 992 sched lock 30 fork1+0x1b14 15 0% 100% 0.00 1758 sched lock 29 fork1+0x1b14 15 0% 100% 0.00 1770 sched lock 25 turnstile_unpend+0x1b4 15 0% 100% 0.00 1231 sched lock 25 fork1+0x1b14 15 0% 100% 0.00 1115 sched lock 21 fork1+0x1b14 15 0% 100% 0.00 1099 sched lock 15 fork1+0x1b14 14 0% 100% 0.00 6282 turnstile chain adastart+0xbcc 14 0% 100% 0.00 5715 sleepq chain trim_map_write_start+0x2c4 14 0% 100% 0.00 6999 sleepq chain metaslab_class_throttle_reserve+0x144 14 0% 100% 0.00 936 sleepq chain dsl_dir_diduse_space+0x54 14 0% 100% 0.00 1644 sleepq chain vm_map_lookup_entry+0x200 14 0% 100% 0.00 1358 sched lock 26 turnstile_unpend+0x1b4 14 0% 100% 0.00 5174 sched lock 24 turnstile_unpend+0x1b4 14 0% 100% 0.00 2630 sched lock 23 sched_idletd+0x2a0 13 0% 100% 0.00 186 td_contested __rw_wunlock_hard+0xa0 13 0% 100% 0.00 2505 sleepq chain dbuf_dirty+0x290 13 0% 100% 0.00 8307 sleepq chain dmu_buf_will_dirty+0x158 13 0% 100% 0.00 2841 sched lock 6 turnstile_unpend+0x1b4 13 0% 100% 0.00 1480 sched lock 27 turnstile_unpend+0x1b4 13 0% 100% 0.00 984 sched lock 19 turnstile_unpend+0x1b4 13 0% 100% 0.00 2159 sched lock 15 turnstile_unpend+0x1b4 13 0% 100% 0.00 307 callout handleevents+0x194 12 0% 100% 0.00 5677 turnstile chain vm_page_alloc_domain_after+0x200 12 0% 100% 0.00 2831 td_contested __rw_wunlock_hard+0xa8 12 0% 100% 0.00 520 td_contested __mtx_unlock_sleep+0x88 12 0% 100% 0.00 4929 sleepq chain trim_map_write_start+0x290 12 0% 100% 0.00 1860 sleepq chain metaslab_class_throttle_reserve+0x154 12 0% 100% 0.00 1003 sched lock 3 thread_create+0x41c 12 0% 100% 0.00 1023 sched lock 26 fork1+0x1b14 12 0% 100% 0.00 2677 sched lock 25 sched_idletd+0x2a0 12 0% 100% 0.00 3359 sched lock 16 turnstile_unpend+0x1b4 12 0% 100% 0.00 1294 sched lock 12 fork1+0x1b14 12 0% 100% 0.00 257 entropy harvest mutex intr_event_schedule_thread+0x58 11 0% 100% 0.00 1503 turnstile chain pmap_enter+0xd38 11 0% 100% 0.00 10828 sleepq chain dsl_pool_dirty_space+0xec 11 0% 100% 0.00 2779 sched lock 24 sched_idletd+0x2a0 11 0% 100% 0.00 2226 sched lock 23 turnstile_unpend+0x1b4 11 0% 100% 0.00 1149 sched lock 24 fork1+0x1b14 11 0% 100% 0.00 1329 sched lock 12 turnstile_unpend+0x1b4 11 0% 100% 0.00 284 entropy harvest mutex swi_sched+0x40 9 0% 100% 0.00 856 zombie lock sched_idletd+0x36c 9 0% 100% 0.00 3449 turnstile chain pmap_enter+0x598 9 0% 100% 0.00 9345 sleepq chain zio_notify_parent+0x290 9 0% 100% 0.00 2476 sched lock 27 sched_idletd+0x2a0 9 0% 100% 0.00 2106 sched lock 22 turnstile_unpend+0x1b4 8 0% 100% 0.00 2072 turnstile chain pmap_fault+0x4fc 8 0% 100% 0.00 4733 turnstile chain _kmem_unback+0x158 8 0% 100% 0.00 203 td_contested __rw_wlock_hard+0x454 8 0% 100% 0.00 10941 sleepq chain zio_vdev_io_start+0x5c 8 0% 100% 0.00 2366 sleepq chain zio_dva_throttle+0x174 8 0% 100% 0.00 1555 sleepq chain lf_advlockasync+0x934 8 0% 100% 0.00 8133 sleepq chain fork1+0x1280 8 0% 100% 0.00 669 sched lock 28 fork1+0x1b14 7 0% 100% 0.00 2409 turnstile chain kmem_back_domain+0x224 7 0% 100% 0.00 1714 turnstile chain release_page+0x124 7 0% 100% 0.00 6834 sleepq chain zio_remove_child+0x140 7 0% 100% 0.00 676 sleepq chain zio_remove_child+0x174 7 0% 100% 0.00 1698 sleepq chain zio_vdev_io_assess+0x368 7 0% 100% 0.00 2844 sleepq chain zio_wait+0x90 7 0% 100% 0.00 11177 sleepq chain zio_add_child+0x188 7 0% 100% 0.00 5136 sleepq chain dsl_dataset_block_kill+0x414 7 0% 100% 0.00 6032 sleepq chain vm_map_lookup+0xd30 7 0% 100% 0.00 1743 sleepq chain pipe_write+0x65c 7 0% 100% 0.00 7269 sleepq chain exit1+0xacc 7 0% 100% 0.00 641 sleepq chain usbd_callback_wrapper+0x6c4 7 0% 100% 0.00 485 sleepq chain usbd_do_request_flags+0x354 7 0% 100% 0.00 539 sched lock 4 sched_clock+0x44c 7 0% 100% 0.00 1415 sched lock 18 thread_create+0x41c 6 0% 100% 0.00 203 zombie lock proc_reap+0x488 6 0% 100% 0.00 6822 turnstile chain g_disk_start+0x30c 6 0% 100% 0.00 3737 sleepq chain arc_buf_access+0x318 6 0% 100% 0.00 2040 sleepq chain vm_map_find+0x608 6 0% 100% 0.00 1029 sched lock 7 thread_create+0x41c 6 0% 100% 0.00 1411 sched lock 30 thread_create+0x41c 6 0% 100% 0.00 1785 sched lock 28 turnstile_unpend+0x1b4 6 0% 100% 0.00 2152 sched lock 26 sched_idletd+0x2a0 6 0% 100% 0.00 1018 sched lock 22 thread_create+0x41c 6 0% 100% 0.00 766 sched lock 19 thread_create+0x41c 6 0% 100% 0.00 1389 sched lock 17 thread_create+0x41c 6 0% 100% 0.00 429 sched lock 11 thread_create+0x41c 6 0% 100% 0.00 1018 sched lock 1 thread_create+0x41c 5 0% 100% 0.00 444 turnstile chain pmap_copy+0x5c0 5 0% 100% 0.00 283577 turnstile chain vm_reserv_alloc_page+0x2b0 5 0% 100% 0.00 3120 turnstile chain kern_umtx_wake+0x220 5 0% 100% 0.00 4133 td_contested __mtx_unlock_sleep+0x90 5 0% 100% 0.00 2764 sleepq chain dsl_dataset_check_quota+0x248 5 0% 100% 0.00 148146 sleepq chain userquota_updates_task+0x2fc 5 0% 100% 0.00 1968 sleepq chain buf_daemon+0x174 5 0% 100% 0.00 1266 sched lock 9 thread_create+0x41c 5 0% 100% 0.00 1457 sched lock 16 thread_create+0x41c 5 0% 100% 0.00 1719 sched lock 1 fork_exit+0x90 5 0% 100% 0.00 38151 process slock thread_suspend_check+0x434 4 0% 100% 0.00 1700 turnstile chain pmap_enter_quick+0x100 4 0% 100% 0.00 8122 sleepq chain txg_list_add+0xdc 4 0% 100% 0.00 939 sleepq chain dsl_dataset_block_born+0x2f0 4 0% 100% 0.00 1166 sleepq chain dsl_dataset_block_born+0xe4 4 0% 100% 0.00 1655 sleepq chain fault_deallocate+0x14 4 0% 100% 0.00 3072 sleepq chain do_lock_umutex+0x5a8 4 0% 100% 0.00 2272 sched lock 6 thread_create+0x41c 4 0% 100% 0.00 605 sched lock 27 thread_create+0x41c 4 0% 100% 0.00 1666 sched lock 25 statclock+0x128 4 0% 100% 0.00 10966 sched lock 15 propagate_priority+0xb8 4 0% 100% 0.00 2044 sched lock 12 thread_create+0x41c 3 0% 100% 0.00 2881 turnstile chain free_pv_chunk+0xf4 3 0% 100% 0.00 2451 turnstile chain get_pv_entry+0x21c 3 0% 100% 0.00 4756 turnstile chain __rw_rlock_hard+0x1d8 3 0% 100% 0.00 318 sleepq chain txg_rele_to_sync+0x78 3 0% 100% 0.00 778 sleepq chain dbuf_find+0x210 3 0% 100% 0.00 2740 sleepq chain vm_map_remove+0x158 3 0% 100% 0.00 2437 sleepq chain pipe_write+0x698 3 0% 100% 0.00 1200 sleepq chain pipe_read+0x548 3 0% 100% 0.00 815 sched lock 8 sched_clock+0x44c 3 0% 100% 0.00 10540 sched lock 5 thread_create+0x41c 3 0% 100% 0.00 1326 sched lock 4 thread_create+0x41c 3 0% 100% 0.00 2422 sched lock 31 thread_create+0x41c 3 0% 100% 0.00 385 sched lock 29 thread_create+0x41c 3 0% 100% 0.00 5555 sched lock 28 sched_idletd+0x2a0 3 0% 100% 0.00 896 sched lock 28 statclock+0x128 3 0% 100% 0.00 2541 sched lock 26 thread_create+0x41c 3 0% 100% 0.00 1029 sched lock 25 thread_create+0x41c 3 0% 100% 0.00 630 sched lock 23 statclock+0x128 3 0% 100% 0.00 1103 sched lock 20 thread_create+0x41c 3 0% 100% 0.00 1214 sched lock 2 propagate_priority+0xb8 3 0% 100% 0.00 718 sched lock 15 thread_create+0x41c 3 0% 100% 0.00 1066 sched lock 14 thread_create+0x41c 3 0% 100% 0.00 763 sched lock 10 propagate_priority+0xb8 3 0% 100% 0.00 548 sched lock 10 thread_create+0x41c 3 0% 100% 0.00 503 sched lock 0 thread_create+0x41c 3 0% 100% 0.00 333 callout callout_reset_sbt_on+0x17c 2 0% 100% 0.00 3033 turnstile chain pmap_fault+0x360 2 0% 100% 0.00 922 turnstile chain pmap_remove_pages+0x658 2 0% 100% 0.00 1244 turnstile chain vm_fault+0x19a8 2 0% 100% 0.00 3266 turnstile chain getnewvnode_reserve+0x250 2 0% 100% 0.00 1044 turnstile chain __rw_try_upgrade_int+0x68 2 0% 100% 0.00 4166 turnstile chain kern_execve+0x113c 2 0% 100% 0.00 4078 turnstile chain xpt_run_devq+0x4d8 2 0% 100% 0.00 711 sleepq chain zio_vdev_io_done+0x124 2 0% 100% 0.00 4133 sleepq chain zio_done+0x99c 2 0% 100% 0.00 255 sleepq chain txg_list_add+0xec 2 0% 100% 0.00 5600 sleepq chain txg_wait_synced_impl+0x90 2 0% 100% 0.00 2877 sleepq chain dsl_dataset_block_kill+0x424 2 0% 100% 0.00 1544 sleepq chain dsl_dir_space_available+0x1a4 2 0% 100% 0.00 5411 sleepq chain dbuf_new_size+0x1a8 2 0% 100% 0.00 5488 sleepq chain arc_write_ready+0x124 2 0% 100% 0.00 264366 sleepq chain aggsum_add+0x2a0 2 0% 100% 0.00 3145 sleepq chain VOP_LOCK1_APV+0x4c 2 0% 100% 0.00 4122 sleepq chain vm_map_protect+0x6bc 2 0% 100% 0.00 2855 sleepq chain _vm_map_lock_upgrade+0xcc 2 0% 100% 0.00 2778 sleepq chain do_lock_umutex+0x7d8 2 0% 100% 0.00 299 sleepq chain __umtx_op_wake2_umutex+0x1d8 2 0% 100% 0.00 1367 sched lock 9 propagate_priority+0xb8 2 0% 100% 0.00 244 sched lock 9 sched_clock+0x44c 2 0% 100% 0.00 1255 sched lock 7 propagate_priority+0xb8 2 0% 100% 0.00 722 sched lock 6 sched_clock+0x44c 2 0% 100% 0.00 366 sched lock 3 propagate_priority+0xb8 2 0% 100% 0.00 289 sched lock 3 sched_clock+0x44c 2 0% 100% 0.00 1700 sched lock 3 statclock+0x128 2 0% 100% 0.00 666 sched lock 29 sched_idletd+0x2a0 2 0% 100% 0.00 1088 sched lock 29 statclock+0x128 2 0% 100% 0.00 1400 sched lock 28 thread_create+0x41c 2 0% 100% 0.00 1199 sched lock 25 propagate_priority+0xb8 2 0% 100% 0.00 3334 sched lock 26 statclock+0x128 2 0% 100% 0.00 956 sched lock 23 thread_create+0x41c 2 0% 100% 0.00 766 sched lock 23 propagate_priority+0xb8 2 0% 100% 0.00 633 sched lock 22 propagate_priority+0xb8 2 0% 100% 0.00 1622 sched lock 22 statclock+0x128 2 0% 100% 0.00 12077 sched lock 20 propagate_priority+0xb8 2 0% 100% 0.00 277 sched lock 20 statclock+0x128 2 0% 100% 0.00 455 sched lock 2 thread_create+0x41c 2 0% 100% 0.00 177 sched lock 18 propagate_priority+0xb8 2 0% 100% 0.00 1600 sched lock 17 propagate_priority+0xb8 2 0% 100% 0.00 311 sched lock 12 propagate_priority+0xb8 2 0% 100% 0.00 10422 sched lock 1 propagate_priority+0xb8 2 0% 100% 0.00 577 sched lock 1 sched_clock+0x44c 2 0% 100% 0.00 3267 callout callout_process+0x160 1 0% 100% 0.00 400 zombie lock ast+0x228 1 0% 100% 0.00 222 zombie lock sched_idletd+0x74 1 0% 100% 0.00 311 zombie lock thread_alloc+0x14 1 0% 100% 0.00 289 zombie lock critical_exit_preempt+0x58 1 0% 100% 0.00 845 turnstile chain pmap_is_prefaultable+0x158 1 0% 100% 0.00 200 turnstile chain pmap_enter_object+0x248 1 0% 100% 0.00 245 turnstile chain pmap_remove+0x248 1 0% 100% 0.00 2889 turnstile chain vm_reserv_free_page+0x464 1 0% 100% 0.00 311 turnstile chain vm_page_zone_import+0x138 1 0% 100% 0.00 400 turnstile chain vm_map_insert+0x498 1 0% 100% 0.00 600 turnstile chain cache_purgevfs+0x588 1 0% 100% 0.00 556 turnstile chain pipeclose+0x418 1 0% 100% 0.00 289 turnstile chain pipe_write+0xee0 1 0% 100% 0.00 2800 turnstile chain doselwakeup+0x170 1 0% 100% 0.00 1533 turnstile chain getnewvnode+0x2e8 1 0% 100% 0.00 889 turnstile chain fault_deallocate+0xa4 1 0% 100% 0.00 1533 turnstile chain adadone+0xc84 1 0% 100% 0.00 177 td_contested __rw_runlock_hard+0xf4 1 0% 100% 0.00 400 turnstile chain g_disk_start+0x3e4 1 0% 100% 0.00 5600 turnstile chain kern_thr_exit+0x120 1 0% 100% 0.00 155 sleepq chain vdev_cache_write+0x1a8 1 0% 100% 0.00 5533 sleepq chain txg_quiesce_thread+0xb8 1 0% 100% 0.00 1556 sleepq chain sa_bulk_lookup+0xd4 1 0% 100% 0.00 666 sleepq chain zio_ready+0x2ac 1 0% 100% 0.00 600 sleepq chain zio_create+0x290 1 0% 100% 0.00 5445 sleepq chain hdr_full_cons+0xbc 1 0% 100% 0.00 5445 sleepq chain arc_get_data_impl+0x1f4 1 0% 100% 0.00 200 sleepq chain arc_read+0xdbc 1 0% 100% 0.00 5445 sleepq chain aggsum_add+0x228 1 0% 100% 0.00 244 sleepq chain vm_pageout_worker+0x27c 1 0% 100% 0.00 3111 sleepq chain pipe_write+0x338 1 0% 100% 0.00 1622 sleepq chain pipe_read+0x410 1 0% 100% 0.00 223 sleepq chain seltdwait+0x7c 1 0% 100% 0.00 289 sleepq chain doselwakeup+0xd8 1 0% 100% 0.00 3178 sleepq chain pipe_read+0x34c 1 0% 100% 0.00 511 sleepq chain vnlru_proc+0xac 1 0% 100% 0.00 155 sleepq chain dnode_new_blkid+0x354 1 0% 100% 0.00 689 sleepq chain dsl_dir_space_available+0x194 1 0% 100% 0.00 178 sleepq chain dmu_buf_hold_array_by_dnode+0x4d8 1 0% 100% 0.00 3089 sleepq chain userquota_updates_task+0x408 1 0% 100% 0.00 1533 sleepq chain dbuf_destroy+0x4e8 1 0% 100% 0.00 1000 sleepq chain dbuf_read+0x5f8 1 0% 100% 0.00 30933 sleepq chain dbuf_read+0x98 1 0% 100% 0.00 5711 sleepq chain dbuf_read+0x3e8 1 0% 100% 0.00 2844 sleepq chain dbuf_dirty+0x75c 1 0% 100% 0.00 20667 sleepq chain dsl_dataset_block_kill+0x400 1 0% 100% 0.00 955 sleepq chain dsl_pool_mos_diduse_space+0xcc 1 0% 100% 0.00 289 sched lock 4 statclock+0x128 1 0% 100% 0.00 933 sched lock 30 propagate_priority+0xb8 1 0% 100% 0.00 5422 sched lock 30 sched_idletd+0x2a0 1 0% 100% 0.00 245 sched lock 30 statclock+0x128 1 0% 100% 0.00 467 sched lock 27 statclock+0x128 1 0% 100% 0.00 10644 sched lock 26 propagate_priority+0xb8 1 0% 100% 0.00 1578 sched lock 27 sched_clock+0x44c 1 0% 100% 0.00 667 sched lock 27 propagate_priority+0xb8 1 0% 100% 0.00 10555 sched lock 28 sched_clock+0x44c 1 0% 100% 0.00 178 sched lock 4 propagate_priority+0xb8 1 0% 100% 0.00 267 sched lock 5 statclock+0x128 1 0% 100% 0.00 222 sched lock 6 statclock+0x128 1 0% 100% 0.00 2888 sched lock 5 propagate_priority+0xb8 1 0% 100% 0.00 533 sched lock 5 sched_clock+0x44c 1 0% 100% 0.00 511 sleepq chain exit1+0x70c 1 0% 100% 0.00 222 sched lock 9 statclock+0x128 1 0% 100% 0.00 312 sleepq chain killpg1+0x32c 1 0% 100% 0.00 556 sleepq chain kern_wait6+0x5dc 1 0% 100% 0.00 222 sleepq chain do_lock_umutex+0x884 1 0% 100% 0.00 978 sched lock 24 thread_create+0x41c 1 0% 100% 0.00 889 sched lock 22 sched_clock+0x44c 1 0% 100% 0.00 400 sched lock 19 propagate_priority+0xb8 1 0% 100% 0.00 1088 sched lock 18 sched_clock+0x44c 1 0% 100% 0.00 2822 sched lock 19 statclock+0x128 1 0% 100% 0.00 1089 sched lock 2 sched_clock+0x44c 1 0% 100% 0.00 378 sched lock 20 sched_clock+0x44c 1 0% 100% 0.00 1733 sched lock 14 statclock+0x128 1 0% 100% 0.00 266 sched lock 13 sched_clock+0x44c 1 0% 100% 0.00 845 sched lock 13 propagate_priority+0xb8 1 0% 100% 0.00 400 sched lock 12 sched_clock+0x44c 1 0% 100% 0.00 1711 sched lock 13 statclock+0x128 1 0% 100% 0.00 3111 sched lock 10 sched_clock+0x44c 1 0% 100% 0.00 311 sched lock 11 propagate_priority+0xb8 1 0% 100% 0.00 244 sched lock 11 sched_clock+0x44c 1 0% 100% 0.00 2978 sched lock 12 statclock+0x128 1 0% 100% 0.00 289 sched lock 16 statclock+0x128 1 0% 100% 0.00 3156 sched lock 14 sched_clock+0x44c 1 0% 100% 0.00 10423 sched lock 14 propagate_priority+0xb8 1 0% 100% 0.00 333 sched lock 16 sched_clock+0x44c 1 0% 100% 0.00 2978 sched lock 17 sched_clock+0x44c 1 0% 100% 0.00 688 sched lock 18 statclock+0x128 1 0% 100% 0.00 1111 sched lock 0 sched_clock+0x44c 1 0% 100% 0.00 955 sched lock 0 propagate_priority+0xb8 1 0% 100% 0.00 1711 sched lock 10 statclock+0x128 ------------------------------------------------------------------------------- Thread lock spin: 87658 events in 300.017 seconds (292 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 25075 29% 29% 0.00 1767 sleepq chain wakeup_any+0x24 5964 7% 35% 0.00 1984 sleepq chain wakeup+0x24 2241 3% 38% 0.00 649523 sched lock 3 fork_exit+0x90 863 1% 39% 0.00 170917 sched lock 0 fork_exit+0x90 732 1% 40% 0.00 2698 turnstile lock __rw_wunlock_hard+0xa8 721 1% 41% 0.00 65023 sched lock 2 fork_exit+0x90 487 1% 41% 0.00 1611 sched lock 0 _sleep+0x11c 450 1% 42% 0.00 1852679 sched lock 10 fork_exit+0x90 438 0% 42% 0.00 4199 sched lock 3 intr_irq_handler+0xe8 428 0% 43% 0.00 1771 sched lock 2 _sleep+0x11c 425 0% 43% 0.00 1563 sched lock 1 _sleep+0x11c 423 0% 44% 0.00 2664 sched lock 3 turnstile_wait+0x2c8 410 0% 44% 0.00 10886426 sched lock 17 fork_exit+0x90 406 0% 45% 0.00 1461 sched lock 7 _sleep+0x11c 405 0% 45% 0.00 2721 sched lock 4 _sleep+0x11c 391 0% 45% 0.00 1613 sched lock 5 _sleep+0x11c 379 0% 46% 0.00 2887 sched lock 10 _sleep+0x11c 373 0% 46% 0.00 1439 sched lock 9 _sleep+0x11c 372 0% 47% 0.00 11715 sleepq chain cv_broadcastpri+0x44 370 0% 47% 0.00 1489 sched lock 11 _sleep+0x11c 369 0% 48% 0.00 1430 sched lock 6 _sleep+0x11c 368 0% 48% 0.00 38666 sched lock 6 fork_exit+0x90 355 0% 48% 0.00 1703 sched lock 3 _sleep+0x11c 355 0% 49% 0.00 1355 sched lock 13 _sleep+0x11c 351 0% 49% 0.00 2974 sched lock 12 _sleep+0x11c 344 0% 50% 0.00 2850 sched lock 18 _sleep+0x11c 339 0% 50% 0.00 2765 sched lock 0 arm_gic_v3_intr+0x100 334 0% 50% 0.00 1606 sched lock 20 _sleep+0x11c 326 0% 51% 0.00 1685948 sched lock 19 fork_exit+0x90 325 0% 51% 0.00 12682 sched lock 0 userret+0x114 323 0% 51% 0.00 3606406 sched lock 20 fork_exit+0x90 318 0% 52% 0.00 4437 sched lock 15 _sleep+0x11c 315 0% 52% 0.00 355799 sched lock 5 fork_exit+0x90 313 0% 53% 0.00 3539 sched lock 3 intr_event_handle+0x110 311 0% 53% 0.00 1653 sched lock 8 _sleep+0x11c 311 0% 53% 0.00 1657 sched lock 14 _sleep+0x11c 310 0% 54% 0.00 2849 sched lock 4 arm_gic_v3_intr+0x100 305 0% 54% 0.00 1529 sched lock 16 _sleep+0x11c 301 0% 54% 0.00 1583 sched lock 17 _sleep+0x11c 298 0% 55% 0.00 5324 sleepq chain cv_signal+0x68 297 0% 55% 0.00 2757 sched lock 6 arm_gic_v3_intr+0x100 292 0% 55% 0.00 3005 sched lock 23 _sleep+0x11c 287 0% 56% 0.00 2793 sched lock 5 arm_gic_v3_intr+0x100 285 0% 56% 0.00 1455 sched lock 28 _sleep+0x11c 284 0% 56% 0.00 2967 sched lock 29 _sleep+0x11c 283 0% 57% 0.00 4911 sched lock 9 arm_gic_v3_intr+0x100 282 0% 57% 0.00 1462 sched lock 19 _sleep+0x11c 275 0% 57% 0.00 1422 sched lock 21 _sleep+0x11c 275 0% 58% 0.00 3057 sched lock 13 arm_gic_v3_intr+0x100 272 0% 58% 0.00 2845 sched lock 1 arm_gic_v3_intr+0x100 271 0% 58% 0.00 5085144 sched lock 4 fork_exit+0x90 270 0% 59% 0.00 2890 sched lock 11 arm_gic_v3_intr+0x100 269 0% 59% 0.00 2847 sched lock 2 arm_gic_v3_intr+0x100 269 0% 59% 0.00 2716 sched lock 14 arm_gic_v3_intr+0x100 267 0% 59% 0.00 1310 sched lock 24 _sleep+0x11c 265 0% 60% 0.00 1487 sched lock 30 _sleep+0x11c 264 0% 60% 0.00 2834 sched lock 8 arm_gic_v3_intr+0x100 263 0% 60% 0.00 1540 sched lock 22 _sleep+0x11c 262 0% 61% 0.00 2835 sched lock 10 arm_gic_v3_intr+0x100 259 0% 61% 0.00 2610 sched lock 12 arm_gic_v3_intr+0x100 258 0% 61% 0.00 1404 sched lock 31 _sleep+0x11c 257 0% 62% 0.00 2676 sched lock 19 arm_gic_v3_intr+0x100 254 0% 62% 0.00 193104 sched lock 9 fork_exit+0x90 252 0% 62% 0.00 3108 sched lock 17 arm_gic_v3_intr+0x100 251 0% 62% 0.00 2718 sched lock 16 arm_gic_v3_intr+0x100 249 0% 63% 0.00 3472 sched lock 26 _sleep+0x11c 248 0% 63% 0.00 1464 sched lock 25 _sleep+0x11c 248 0% 63% 0.00 3026 sched lock 18 arm_gic_v3_intr+0x100 247 0% 64% 0.00 12365520 sched lock 7 fork_exit+0x90 244 0% 64% 0.00 2806 sleepq chain _sx_xunlock_hard+0xb4 242 0% 64% 0.00 2875 sched lock 23 arm_gic_v3_intr+0x100 241 0% 64% 0.00 2655 sched lock 3 arm_gic_v3_intr+0x100 232 0% 65% 0.00 2718 sched lock 20 arm_gic_v3_intr+0x100 230 0% 65% 0.00 1418 sched lock 27 _sleep+0x11c 230 0% 65% 0.00 2747 sched lock 22 arm_gic_v3_intr+0x100 228 0% 65% 0.00 3112 sched lock 21 arm_gic_v3_intr+0x100 225 0% 66% 0.00 17769195 sched lock 23 fork_exit+0x90 221 0% 66% 0.00 2993 sched lock 15 arm_gic_v3_intr+0x100 221 0% 66% 0.00 3944 sched lock 0 intr_irq_handler+0xe8 220 0% 66% 0.00 2535 sched lock 24 arm_gic_v3_intr+0x100 217 0% 67% 0.00 3125 sched lock 7 arm_gic_v3_intr+0x100 211 0% 67% 0.00 1819012 sched lock 11 fork_exit+0x90 210 0% 67% 0.00 13756 sched lock 2 userret+0x114 209 0% 67% 0.00 2736 sched lock 29 arm_gic_v3_intr+0x100 208 0% 68% 0.00 24142 sched lock 4 userret+0x114 208 0% 68% 0.00 816887 sched lock 30 fork_exit+0x90 207 0% 68% 0.00 2744 sched lock 26 arm_gic_v3_intr+0x100 206 0% 68% 0.00 9872286 sched lock 28 fork_exit+0x90 206 0% 69% 0.00 3078 sched lock 25 arm_gic_v3_intr+0x100 206 0% 69% 0.00 426317 sched lock 1 fork_exit+0x90 203 0% 69% 0.00 3025 sched lock 30 arm_gic_v3_intr+0x100 200 0% 69% 0.00 3000 turnstile lock __mtx_unlock_sleep+0x90 198 0% 69% 0.00 8015 sched lock 7 userret+0x114 198 0% 70% 0.00 16976636 sched lock 21 fork_exit+0x90 195 0% 70% 0.00 11618 sched lock 18 userret+0x114 193 0% 70% 0.00 744340 sched lock 27 fork_exit+0x90 192 0% 70% 0.00 15029 sched lock 3 userret+0x114 192 0% 71% 0.00 5183 sched lock 28 arm_gic_v3_intr+0x100 191 0% 71% 0.00 19586 sched lock 1 userret+0x114 187 0% 71% 0.00 11879 sched lock 6 userret+0x114 186 0% 71% 0.00 2888 sched lock 27 arm_gic_v3_intr+0x100 186 0% 71% 0.00 6687534 sched lock 22 fork_exit+0x90 185 0% 72% 0.00 2685 sched lock 31 arm_gic_v3_intr+0x100 179 0% 72% 0.00 12798 sched lock 20 userret+0x114 177 0% 72% 0.00 8074680 sched lock 12 fork_exit+0x90 176 0% 72% 0.00 16360461 sched lock 15 fork_exit+0x90 173 0% 72% 0.00 4363680 sched lock 31 fork_exit+0x90 171 0% 73% 0.00 12485 sched lock 10 userret+0x114 163 0% 73% 0.00 11778842 sched lock 16 fork_exit+0x90 162 0% 73% 0.00 12135 sched lock 9 userret+0x114 160 0% 73% 0.00 4036 sched lock 0 exit1+0x1034 157 0% 73% 0.00 1654 sched lock 0 proc_reap+0x488 155 0% 74% 0.00 10587 sched lock 17 userret+0x114 151 0% 74% 0.00 4492415 sched lock 8 fork_exit+0x90 151 0% 74% 0.00 1627957 sched lock 18 fork_exit+0x90 150 0% 74% 0.00 1258950 sched lock 13 fork_exit+0x90 149 0% 74% 0.00 30981 sched lock 19 userret+0x114 147 0% 74% 0.00 14632 sched lock 22 userret+0x114 146 0% 75% 0.00 15482 sched lock 12 userret+0x114 145 0% 75% 0.00 11601 sched lock 11 userret+0x114 144 0% 75% 0.00 18148 sched lock 15 userret+0x114 141 0% 75% 0.00 19214 sched lock 8 userret+0x114 140 0% 75% 0.00 9580 sched lock 29 userret+0x114 138 0% 75% 0.00 22409348 sched lock 14 fork_exit+0x90 136 0% 76% 0.00 4044 sched lock 3 proc_reap+0x488 135 0% 76% 0.00 12762 sched lock 25 userret+0x114 135 0% 76% 0.00 15747 sched lock 21 userret+0x114 134 0% 76% 0.00 15732 sched lock 5 userret+0x114 132 0% 76% 0.00 3193 sched lock 6 proc_reap+0x488 131 0% 76% 0.00 9693 sched lock 31 userret+0x114 130 0% 76% 0.00 3167 sched lock 5 proc_reap+0x488 130 0% 77% 0.00 16954 sched lock 13 userret+0x114 128 0% 77% 0.00 9668 sched lock 16 userret+0x114 127 0% 77% 0.00 20431 sched lock 24 userret+0x114 126 0% 77% 0.00 2086 sched lock 17 proc_reap+0x488 125 0% 77% 0.00 3497 sched lock 4 proc_reap+0x488 125 0% 77% 0.00 6386 sched lock 30 proc_reap+0x488 124 0% 77% 0.00 3089 sched lock 3 exit1+0x1034 124 0% 78% 0.00 2477 sched lock 2 proc_reap+0x488 123 0% 78% 0.00 23142 sched lock 27 userret+0x114 121 0% 78% 0.00 15506 sched lock 30 userret+0x114 120 0% 78% 0.00 4366 sched lock 1 proc_reap+0x488 118 0% 78% 0.00 4245 sched lock 6 intr_irq_handler+0xe8 118 0% 78% 0.00 4214 sched lock 4 intr_irq_handler+0xe8 117 0% 78% 0.00 17051 sched lock 23 userret+0x114 117 0% 79% 0.00 3817 sched lock 1 intr_irq_handler+0xe8 116 0% 79% 0.00 3148 sched lock 12 proc_reap+0x488 114 0% 79% 0.00 17309 sched lock 28 userret+0x114 114 0% 79% 0.00 14508 sched lock 14 userret+0x114 113 0% 79% 0.00 1669315 sched lock 24 fork_exit+0x90 112 0% 79% 0.00 2518 sched lock 31 proc_reap+0x488 111 0% 79% 0.00 3952 sched lock 2 intr_irq_handler+0xe8 110 0% 79% 0.00 2492 sched lock 9 proc_reap+0x488 110 0% 80% 0.00 2548 sched lock 15 proc_reap+0x488 107 0% 80% 0.00 4468 sched lock 7 proc_reap+0x488 107 0% 80% 0.00 2955 sched lock 4 exit1+0x1034 106 0% 80% 0.00 3291 sched lock 20 proc_reap+0x488 105 0% 80% 0.00 3991 sched lock 8 intr_irq_handler+0xe8 105 0% 80% 0.00 4018 sched lock 5 intr_irq_handler+0xe8 105 0% 80% 0.00 2362 sched lock 29 proc_reap+0x488 105 0% 80% 0.00 3610 sched lock 11 proc_reap+0x488 104 0% 81% 0.00 2079 sleepq chain _sx_sunlock_hard+0xe8 104 0% 81% 0.00 2241514 sched lock 26 fork_exit+0x90 103 0% 81% 0.00 10361 sched lock 26 userret+0x114 103 0% 81% 0.00 2997 sched lock 21 proc_reap+0x488 103 0% 81% 0.00 5380 sched lock 16 proc_reap+0x488 102 0% 81% 0.00 2024 sched lock 18 proc_reap+0x488 101 0% 81% 0.00 1917580 sched lock 29 fork_exit+0x90 99 0% 81% 0.00 3492 sched lock 24 proc_reap+0x488 98 0% 81% 0.00 2535 sched lock 27 proc_reap+0x488 97 0% 82% 0.00 3405 sched lock 6 exit1+0x1034 95 0% 82% 0.00 2628 sched lock 25 proc_reap+0x488 94 0% 82% 0.00 11774141 sched lock 25 fork_exit+0x90 93 0% 82% 0.00 2843 sched lock 8 proc_reap+0x488 92 0% 82% 0.00 4248 sched lock 7 intr_irq_handler+0xe8 92 0% 82% 0.00 4345 sched lock 19 proc_reap+0x488 91 0% 82% 0.00 4042 sched lock 23 proc_reap+0x488 90 0% 82% 0.00 4001 sched lock 19 intr_irq_handler+0xe8 90 0% 82% 0.00 3443 sched lock 10 proc_reap+0x488 89 0% 83% 0.00 4735 sched lock 14 proc_reap+0x488 88 0% 83% 0.00 2317 sched lock 26 proc_reap+0x488 87 0% 83% 0.00 2794 sched lock 2 exit1+0x1034 86 0% 83% 0.00 4237 sched lock 21 intr_irq_handler+0xe8 86 0% 83% 0.00 4007 sched lock 18 intr_irq_handler+0xe8 86 0% 83% 0.00 4542 sched lock 13 intr_irq_handler+0xe8 86 0% 83% 0.00 3975 sched lock 10 intr_irq_handler+0xe8 85 0% 83% 0.00 4217 sched lock 23 intr_irq_handler+0xe8 85 0% 83% 0.00 2306 sched lock 1 exit1+0x1034 83 0% 83% 0.00 4305 sched lock 12 intr_irq_handler+0xe8 82 0% 83% 0.00 4605 sched lock 16 intr_irq_handler+0xe8 82 0% 84% 0.00 1639 sched lock 0 turnstile_wait+0x2c8 81 0% 84% 0.00 1392 sched lock 6 turnstile_wait+0x2c8 81 0% 84% 0.00 5632 sched lock 28 proc_reap+0x488 81 0% 84% 0.00 4000 sched lock 22 intr_irq_handler+0xe8 81 0% 84% 0.00 3629 sched lock 19 exit1+0x1034 81 0% 84% 0.00 6073 sched lock 13 proc_reap+0x488 80 0% 84% 0.00 4286 sched lock 15 intr_irq_handler+0xe8 80 0% 84% 0.00 5610 sched lock 13 exit1+0x1034 79 0% 84% 0.00 4275 sched lock 0 intr_event_handle+0x110 78 0% 84% 0.00 3164 sched lock 6 intr_event_handle+0x110 78 0% 85% 0.00 5005 sched lock 11 intr_irq_handler+0xe8 78 0% 85% 0.00 3415 sched lock 10 exit1+0x1034 76 0% 85% 0.00 4037 sched lock 9 intr_irq_handler+0xe8 75 0% 85% 0.00 3923 sched lock 20 intr_irq_handler+0xe8 75 0% 85% 0.00 2455 sched lock 2 turnstile_wait+0x2c8 74 0% 85% 0.00 3784 sched lock 25 intr_irq_handler+0xe8 72 0% 85% 0.00 3884 sched lock 29 intr_irq_handler+0xe8 72 0% 85% 0.00 3873 sched lock 14 intr_irq_handler+0xe8 69 0% 85% 0.00 1213 sched lock 9 turnstile_wait+0x2c8 69 0% 85% 0.00 2484 sched lock 22 proc_reap+0x488 68 0% 85% 0.00 4009 sched lock 5 intr_event_handle+0x110 68 0% 85% 0.00 3837 sched lock 30 intr_irq_handler+0xe8 68 0% 85% 0.00 4419 sched lock 24 intr_irq_handler+0xe8 68 0% 86% 0.00 3715 sched lock 10 intr_event_handle+0x110 67 0% 86% 0.00 3848 sched lock 9 exit1+0x1034 67 0% 86% 0.00 3803 sched lock 15 intr_event_handle+0x110 67 0% 86% 0.00 3201 sched lock 11 intr_event_handle+0x110 67 0% 86% 0.00 1850 sched lock 1 turnstile_wait+0x2c8 65 0% 86% 0.00 4715 sched lock 7 exit1+0x1034 65 0% 86% 0.00 3469 sched lock 4 _sleep+0x1b0 64 0% 86% 0.00 4747 sched lock 9 intr_event_handle+0x110 64 0% 86% 0.00 1599 sched lock 5 turnstile_wait+0x2c8 63 0% 86% 0.00 3432 sched lock 17 intr_irq_handler+0xe8 63 0% 86% 0.00 23879 sched lock 10 handle_el0_sync+0xc0 63 0% 86% 0.00 3687 sched lock 0 _sleep+0x1b0 62 0% 86% 0.00 3415 sched lock 21 intr_event_handle+0x110 62 0% 87% 0.00 11573 sched lock 0 handle_el0_sync+0xc0 61 0% 87% 0.00 2379 sched lock 21 exit1+0x1034 60 0% 87% 0.00 3443 sched lock 8 exit1+0x1034 60 0% 87% 0.00 6554 sched lock 5 exit1+0x1034 60 0% 87% 0.00 7039 sched lock 30 exit1+0x1034 60 0% 87% 0.00 4651 sched lock 2 intr_event_handle+0x110 60 0% 87% 0.00 74448 sched lock 18 handle_el0_sync+0xc0 60 0% 87% 0.00 2880 sched lock 18 intr_event_handle+0x110 60 0% 87% 0.00 1250 sched lock 16 turnstile_wait+0x2c8 60 0% 87% 0.00 2062 sched lock 11 exit1+0x1034 60 0% 87% 0.00 3634 sched lock 1 intr_event_handle+0x110 59 0% 87% 0.00 12078 turnstile lock turnstile_wait+0x2c8 59 0% 87% 0.00 2193 sched lock 4 turnstile_wait+0x2c8 59 0% 87% 0.00 3476 sched lock 24 intr_event_handle+0x110 59 0% 87% 0.00 3925 sched lock 20 exit1+0x1034 59 0% 88% 0.00 3867 sched lock 17 _sleep+0x1b0 58 0% 88% 0.00 4361 sched lock 16 _sleep+0x1b0 58 0% 88% 0.00 3709 sched lock 10 _sleep+0x1b0 57 0% 88% 0.00 2883 sched lock 4 intr_event_handle+0x110 57 0% 88% 0.00 2023 sched lock 22 turnstile_wait+0x2c8 57 0% 88% 0.00 4104 sched lock 17 intr_event_handle+0x110 56 0% 88% 0.00 1360 sched lock 17 turnstile_wait+0x2c8 55 0% 88% 0.00 4537 sched lock 9 _sleep+0x1b0 55 0% 88% 0.00 17041 sched lock 3 handle_el0_sync+0xc0 55 0% 88% 0.00 12372 sched lock 2 handle_el0_sync+0xc0 55 0% 88% 0.00 2665 sched lock 13 turnstile_wait+0x2c8 54 0% 88% 0.00 1439 sched lock 7 turnstile_wait+0x2c8 54 0% 88% 0.00 3991 sched lock 27 intr_irq_handler+0xe8 54 0% 88% 0.00 3949 sched lock 12 exit1+0x1034 54 0% 88% 0.00 16245 sched lock 10 turnstile_wait+0x2c8 53 0% 88% 0.00 1490 sched lock 8 turnstile_wait+0x2c8 53 0% 89% 0.00 4242 sched lock 28 intr_irq_handler+0xe8 53 0% 89% 0.00 3883 sched lock 27 intr_event_handle+0x110 53 0% 89% 0.00 1431 sched lock 23 turnstile_wait+0x2c8 53 0% 89% 0.00 1479 sched lock 19 turnstile_wait+0x2c8 53 0% 89% 0.00 963 sched lock 18 turnstile_wait+0x2c8 53 0% 89% 0.00 2841 sched lock 16 intr_event_handle+0x110 53 0% 89% 0.00 4686 sched lock 13 intr_event_handle+0x110 52 0% 89% 0.00 3338 sched lock 7 _sleep+0x1b0 52 0% 89% 0.00 2410 sched lock 29 exit1+0x1034 52 0% 89% 0.00 4437 sched lock 25 intr_event_handle+0x110 51 0% 89% 0.00 3408 sched lock 8 intr_event_handle+0x110 51 0% 89% 0.00 3339 sched lock 31 intr_irq_handler+0xe8 51 0% 89% 0.00 3219 sched lock 23 intr_event_handle+0x110 51 0% 89% 0.00 1792 sched lock 20 turnstile_wait+0x2c8 51 0% 89% 0.00 5367 sched lock 16 exit1+0x1034 51 0% 89% 0.00 1408 sched lock 15 turnstile_wait+0x2c8 51 0% 90% 0.00 2932 sched lock 12 intr_event_handle+0x110 50 0% 90% 0.00 3301 sched lock 3 _sleep+0x1b0 50 0% 90% 0.00 3243 sched lock 28 intr_event_handle+0x110 50 0% 90% 0.00 2984 sched lock 26 intr_irq_handler+0xe8 50 0% 90% 0.00 3652 sched lock 14 intr_event_handle+0x110 50 0% 90% 0.00 3116 sched lock 14 exit1+0x1034 50 0% 90% 0.00 1409 sched lock 11 turnstile_wait+0x2c8 49 0% 90% 0.00 3103 sched lock 8 _sleep+0x1b0 49 0% 90% 0.00 4751 sched lock 31 handle_el0_sync+0xc0 49 0% 90% 0.00 2585 sched lock 29 turnstile_wait+0x2c8 49 0% 90% 0.00 1373 sched lock 28 turnstile_wait+0x2c8 49 0% 90% 0.00 1981 sched lock 26 turnstile_wait+0x2c8 49 0% 90% 0.00 4130 sched lock 22 exit1+0x1034 49 0% 90% 0.00 2045 sched lock 12 turnstile_wait+0x2c8 49 0% 90% 0.00 3959 sched lock 11 _sleep+0x1b0 48 0% 90% 0.00 1239 sched lock 27 turnstile_wait+0x2c8 48 0% 90% 0.00 3578 sched lock 26 intr_event_handle+0x110 48 0% 90% 0.00 1312 sched lock 14 turnstile_wait+0x2c8 47 0% 91% 0.00 3333 sched lock 6 _sleep+0x1b0 47 0% 91% 0.00 3573 sched lock 23 _sleep+0x1b0 47 0% 91% 0.00 1246 sched lock 21 turnstile_wait+0x2c8 46 0% 91% 0.00 9781 sched lock 7 handle_el0_sync+0xc0 46 0% 91% 0.00 1830 sched lock 30 turnstile_wait+0x2c8 46 0% 91% 0.00 3715 sched lock 30 intr_event_handle+0x110 46 0% 91% 0.00 3791 sched lock 2 _sleep+0x1b0 46 0% 91% 0.00 3843 sched lock 17 exit1+0x1034 45 0% 91% 0.00 9958 sched lock 7 intr_event_handle+0x110 45 0% 91% 0.00 4460 sched lock 22 intr_event_handle+0x110 45 0% 91% 0.00 4620 sched lock 20 intr_event_handle+0x110 45 0% 91% 0.00 3872 sched lock 18 _sleep+0x1b0 45 0% 91% 0.00 5123 sched lock 15 exit1+0x1034 45 0% 91% 0.00 19379 sched lock 11 handle_el0_sync+0xc0 44 0% 91% 0.00 1427 sched lock 6 thread_exit+0xcc 44 0% 91% 0.00 3241 sched lock 3 __rw_wunlock_hard+0xa8 44 0% 91% 0.00 3650 sched lock 29 handle_el0_sync+0xc0 44 0% 91% 0.00 1625 sched lock 24 turnstile_wait+0x2c8 44 0% 91% 0.00 3036 sched lock 24 exit1+0x1034 44 0% 91% 0.00 8732 sched lock 19 handle_el0_sync+0xc0 44 0% 92% 0.00 4233 sched lock 13 _sleep+0x1b0 43 0% 92% 0.00 71683 sched lock 6 handle_el0_sync+0xc0 43 0% 92% 0.00 8470 sched lock 4 handle_el0_sync+0xc0 43 0% 92% 0.00 1014 sched lock 25 turnstile_wait+0x2c8 43 0% 92% 0.00 8794 sched lock 20 handle_el0_sync+0xc0 43 0% 92% 0.00 4146 sched lock 19 intr_event_handle+0x110 42 0% 92% 0.00 2595 sched lock 27 exit1+0x1034 42 0% 92% 0.00 5809 sched lock 18 exit1+0x1034 41 0% 92% 0.00 10592 sched lock 30 handle_el0_sync+0xc0 41 0% 92% 0.00 3785 sched lock 30 _sleep+0x1b0 41 0% 92% 0.00 1686 sched lock 3 exit1+0x1160 41 0% 92% 0.00 4201 sched lock 25 exit1+0x1034 41 0% 92% 0.00 3746 sched lock 14 _sleep+0x1b0 40 0% 92% 0.00 3027 sched lock 23 exit1+0x1034 39 0% 92% 0.00 12049 sched lock 9 handle_el0_sync+0xc0 39 0% 92% 0.00 14571 sched lock 6 __rw_wunlock_hard+0xa8 39 0% 92% 0.00 1810 sched lock 31 turnstile_wait+0x2c8 39 0% 92% 0.00 14153 sched lock 27 handle_el0_sync+0xc0 39 0% 92% 0.00 3718 sched lock 20 _sleep+0x1b0 39 0% 92% 0.00 4510 sched lock 19 _sleep+0x1b0 39 0% 92% 0.00 3221 sched lock 15 _sleep+0x1b0 39 0% 93% 0.00 3889 sched lock 12 _sleep+0x1b0 39 0% 93% 0.00 1695 sched lock 0 exit1+0x1160 38 0% 93% 0.00 77255 turnstile lock exit1+0x1034 38 0% 93% 0.00 12946 sched lock 24 handle_el0_sync+0xc0 38 0% 93% 0.00 3605 sched lock 24 _sleep+0x1b0 38 0% 93% 0.00 18356 sched lock 23 handle_el0_sync+0xc0 38 0% 93% 0.00 3719 sched lock 22 _sleep+0x1b0 38 0% 93% 0.00 14495 sched lock 14 handle_el0_sync+0xc0 37 0% 93% 0.00 28048 sched lock 5 handle_el0_sync+0xc0 37 0% 93% 0.00 4553 sched lock 5 _sleep+0x1b0 37 0% 93% 0.00 26873 sched lock 28 handle_el0_sync+0xc0 37 0% 93% 0.00 3571 sched lock 25 _sleep+0x1b0 37 0% 93% 0.00 3552 sched lock 1 _sleep+0x1b0 36 0% 93% 0.00 4097 sched lock 31 _sleep+0x1b0 36 0% 93% 0.00 9442 sched lock 21 handle_el0_sync+0xc0 36 0% 93% 0.00 3350 sched lock 21 _sleep+0x1b0 36 0% 93% 0.00 7551 sched lock 17 handle_el0_sync+0xc0 35 0% 93% 0.00 3631 sched lock 29 intr_event_handle+0x110 35 0% 93% 0.00 9439 sched lock 28 exit1+0x1034 35 0% 93% 0.00 41998 sched lock 15 handle_el0_sync+0xc0 34 0% 93% 0.00 8626 sched lock 8 handle_el0_sync+0xc0 34 0% 93% 0.00 1398 sched lock 3 thread_exit+0xcc 34 0% 93% 0.00 3723 sched lock 26 _sleep+0x1b0 34 0% 93% 0.00 17294 sched lock 1 handle_el0_sync+0xc0 33 0% 94% 0.00 4026 sched lock 8 __rw_wunlock_hard+0xa8 33 0% 94% 0.00 13574 sched lock 16 handle_el0_sync+0xc0 32 0% 94% 0.00 2800 sched lock 31 intr_event_handle+0x110 32 0% 94% 0.00 4561 sched lock 31 exit1+0x1034 32 0% 94% 0.00 3412 sched lock 2 __rw_wunlock_hard+0xa8 31 0% 94% 0.00 3891 sched lock 28 _sleep+0x1b0 31 0% 94% 0.00 3208 sched lock 22 handle_el0_sync+0xc0 31 0% 94% 0.00 956 sched lock 18 thread_exit+0xcc 31 0% 94% 0.00 4085 sched lock 1 __rw_wunlock_hard+0xa8 30 0% 94% 0.00 2155 sched lock 9 thread_exit+0xcc 30 0% 94% 0.00 1082 sched lock 17 thread_exit+0xcc 30 0% 94% 0.00 1270 sched lock 14 thread_exit+0xcc 30 0% 94% 0.00 11464 sched lock 13 handle_el0_sync+0xc0 29 0% 94% 0.00 1225 sched lock 2 exit1+0x1160 29 0% 94% 0.00 1860 sched lock 16 exit1+0x1160 29 0% 94% 0.00 1739 sched lock 10 thread_exit+0xcc 29 0% 94% 0.00 3927 sched lock 0 __rw_wunlock_hard+0xa8 28 0% 94% 0.00 3297 sched lock 5 __rw_wunlock_hard+0xa8 28 0% 94% 0.00 1312 sched lock 5 exit1+0x1160 28 0% 94% 0.00 3801 sched lock 27 _sleep+0x1b0 28 0% 94% 0.00 1661 sched lock 21 exit1+0x1160 27 0% 94% 0.00 1868 turnstile lock __rw_runlock_hard+0xf4 27 0% 94% 0.00 7098 sched lock 26 exit1+0x1034 27 0% 94% 0.00 17937 sched lock 25 handle_el0_sync+0xc0 27 0% 94% 0.00 1852 sched lock 17 exit1+0x1160 27 0% 94% 0.00 3962 sched lock 13 __rw_wunlock_hard+0xa8 27 0% 94% 0.00 10107 sched lock 12 handle_el0_sync+0xc0 26 0% 94% 0.00 2035 sleepq chain wakeup_one+0x24 26 0% 94% 0.00 4386 sched lock 4 __rw_wunlock_hard+0xa8 26 0% 94% 0.00 1395 sched lock 31 thread_exit+0xcc 26 0% 94% 0.00 1435 sched lock 25 exit1+0x1160 26 0% 95% 0.00 1715 sched lock 13 exit1+0x1160 26 0% 95% 0.00 1632 sched lock 0 thread_exit+0xcc 25 0% 95% 0.00 2038 sched lock 8 exit1+0x1160 25 0% 95% 0.00 911 sched lock 7 exit1+0x1160 25 0% 95% 0.00 1660 sched lock 29 thread_exit+0xcc 25 0% 95% 0.00 2063 sched lock 27 exit1+0x1160 25 0% 95% 0.00 3975 sched lock 26 __rw_wunlock_hard+0xa8 25 0% 95% 0.00 1147 sched lock 20 exit1+0x1160 25 0% 95% 0.00 1450 sched lock 19 exit1+0x1160 25 0% 95% 0.00 2952 sched lock 15 sleepq_wait_sig+0xc 25 0% 95% 0.00 1836 sched lock 1 thread_exit+0xcc 24 0% 95% 0.00 4500 sched lock 9 __rw_wunlock_hard+0xa8 24 0% 95% 0.00 4315 sched lock 7 __rw_wunlock_hard+0xa8 24 0% 95% 0.00 880 sched lock 6 exit1+0x1160 24 0% 95% 0.00 2070 sched lock 30 exit1+0x1160 24 0% 95% 0.00 4410 sched lock 29 _sleep+0x1b0 24 0% 95% 0.00 1604 sched lock 23 exit1+0x1160 24 0% 95% 0.00 1381 sched lock 16 thread_exit+0xcc 24 0% 95% 0.00 26222 sched lock 10 umtx_thread_cleanup+0xdc 23 0% 95% 0.00 1316 sched lock 4 exit1+0x1160 23 0% 95% 0.00 9509 sched lock 26 handle_el0_sync+0xc0 23 0% 95% 0.00 2046 sched lock 22 exit1+0x1160 23 0% 95% 0.00 3183 sched lock 15 thread_exit+0xcc 23 0% 95% 0.00 1267 sched lock 12 exit1+0x1160 22 0% 95% 0.00 108694 sched lock 5 umtx_thread_cleanup+0xdc 22 0% 95% 0.00 3643 sched lock 3 __mtx_unlock_sleep+0x90 22 0% 95% 0.00 42137 sched lock 2 umtx_thread_cleanup+0xdc 22 0% 95% 0.00 4814 sched lock 18 __rw_wunlock_hard+0xa8 22 0% 95% 0.00 4097 sched lock 16 __rw_wunlock_hard+0xa8 22 0% 95% 0.00 1263 sched lock 0 handleevents+0x108 21 0% 95% 0.00 1368 sched lock 9 exit1+0x1160 21 0% 95% 0.00 1342 sched lock 4 thread_exit+0xcc 21 0% 95% 0.00 1300 sched lock 31 exit1+0x1160 21 0% 95% 0.00 3537 sched lock 23 __rw_wunlock_hard+0xa8 21 0% 95% 0.00 4457 sched lock 21 __rw_wunlock_hard+0xa8 21 0% 95% 0.00 51379 sched lock 20 umtx_thread_cleanup+0xdc 21 0% 95% 0.00 4182 sched lock 19 __rw_wunlock_hard+0xa8 21 0% 96% 0.00 24891 sched lock 17 umtx_thread_cleanup+0xdc 21 0% 96% 0.00 4077 sched lock 14 __rw_wunlock_hard+0xa8 21 0% 96% 0.00 4111 sched lock 12 __rw_wunlock_hard+0xa8 21 0% 96% 0.00 1292 sched lock 10 exit1+0x1160 21 0% 96% 0.00 1330 sched lock 1 handleevents+0x108 21 0% 96% 0.00 1248 sched lock 1 exit1+0x1160 20 0% 96% 0.00 44043 sched lock 9 umtx_thread_cleanup+0xdc 20 0% 96% 0.00 44463 sched lock 8 umtx_thread_cleanup+0xdc 20 0% 96% 0.00 30496 sched lock 4 umtx_thread_cleanup+0xdc 20 0% 96% 0.00 3372 sched lock 30 __rw_wunlock_hard+0xa8 20 0% 96% 0.00 3973 sched lock 28 __rw_wunlock_hard+0xa8 20 0% 96% 0.00 2299 sched lock 20 sleepq_wait_sig+0xc 20 0% 96% 0.00 3866 sched lock 20 __rw_wunlock_hard+0xa8 20 0% 96% 0.00 60497 sched lock 13 umtx_thread_cleanup+0xdc 20 0% 96% 0.00 3902 sched lock 10 __rw_wunlock_hard+0xa8 19 0% 96% 0.00 2910 sleepq chain exit1+0xb0 19 0% 96% 0.00 78367 sched lock 3 umtx_thread_cleanup+0xdc 19 0% 96% 0.00 3664 sched lock 29 __rw_wunlock_hard+0xa8 19 0% 96% 0.00 1816 sched lock 24 thread_exit+0xcc 19 0% 96% 0.00 784 sched lock 23 thread_exit+0xcc 19 0% 96% 0.00 4395 sched lock 22 __rw_wunlock_hard+0xa8 19 0% 96% 0.00 2512 sched lock 18 sleepq_wait_sig+0xc 19 0% 96% 0.00 1557 sched lock 15 exit1+0x1160 19 0% 96% 0.00 2276 sched lock 11 exit1+0x1160 19 0% 96% 0.00 1145 sched lock 10 handleevents+0x108 18 0% 96% 0.00 2875 sched lock 7 sleepq_wait_sig+0xc 18 0% 96% 0.00 1481 sched lock 30 thread_exit+0xcc 18 0% 96% 0.00 1376 sched lock 28 handleevents+0x108 18 0% 96% 0.00 3134 sched lock 24 __rw_wunlock_hard+0xa8 18 0% 96% 0.00 1965 sched lock 24 exit1+0x1160 18 0% 96% 0.00 1716 sched lock 19 thread_exit+0xcc 18 0% 96% 0.00 4298 sched lock 17 __rw_wunlock_hard+0xa8 18 0% 96% 0.00 4133 sched lock 15 __rw_wunlock_hard+0xa8 18 0% 96% 0.00 1408 sched lock 12 handleevents+0x108 18 0% 96% 0.00 981 sched lock 11 thread_exit+0xcc 18 0% 96% 0.00 3815 sched lock 0 __mtx_unlock_sleep+0x90 17 0% 96% 0.00 1118 sched lock 5 thread_exit+0xcc 17 0% 96% 0.00 2712 sched lock 31 sleepq_wait_sig+0xc 17 0% 96% 0.00 2091 sched lock 27 thread_exit+0xcc 17 0% 96% 0.00 4134 sched lock 25 __rw_wunlock_hard+0xa8 17 0% 96% 0.00 877 sched lock 21 thread_exit+0xcc 17 0% 96% 0.00 1890 sched lock 2 thread_exit+0xcc 17 0% 96% 0.00 3207 sched lock 17 sleepq_wait_sig+0xc 16 0% 96% 0.00 3754 sched lock 31 __rw_wunlock_hard+0xa8 16 0% 96% 0.00 2827 sched lock 29 exit1+0x1160 16 0% 96% 0.00 30291 sched lock 28 sys_fork+0x30 16 0% 96% 0.00 2816 sched lock 28 exit1+0x1160 16 0% 97% 0.00 17818 sched lock 27 umtx_thread_cleanup+0xdc 16 0% 97% 0.00 2961 sched lock 26 thread_exit+0xcc 16 0% 97% 0.00 2373 sched lock 26 exit1+0x1160 16 0% 97% 0.00 1269 sched lock 20 thread_exit+0xcc 16 0% 97% 0.00 42645 sched lock 19 sys_fork+0x30 16 0% 97% 0.00 2733 sched lock 14 sleepq_wait_sig+0xc 16 0% 97% 0.00 926 sched lock 13 handleevents+0x108 16 0% 97% 0.00 19997 sched lock 1 umtx_thread_cleanup+0xdc 15 0% 97% 0.00 1899 sleepq chain lockmgr_xunlock_hard+0x12c 15 0% 97% 0.00 1800 sched lock 8 thread_exit+0xcc 15 0% 97% 0.00 995 sched lock 7 handleevents+0x108 15 0% 97% 0.00 2338 sched lock 7 thread_exit+0xcc 15 0% 97% 0.00 1238 sched lock 4 handleevents+0x108 15 0% 97% 0.00 1933 sched lock 3 sleepq_wait_sig+0xc 15 0% 97% 0.00 1284 sched lock 28 thread_exit+0xcc 15 0% 97% 0.00 4398 sched lock 20 __mtx_unlock_sleep+0x90 15 0% 97% 0.00 11905 sched lock 19 umtx_thread_cleanup+0xdc 15 0% 97% 0.00 15991 sched lock 18 umtx_thread_cleanup+0xdc 15 0% 97% 0.00 1260 sched lock 18 exit1+0x1160 15 0% 97% 0.00 41256 sched lock 14 sys_fork+0x30 15 0% 97% 0.00 1850 sched lock 14 exit1+0x1160 15 0% 97% 0.00 26946 sched lock 11 umtx_thread_cleanup+0xdc 15 0% 97% 0.00 4343 sched lock 11 __rw_wunlock_hard+0xa8 14 0% 97% 0.00 3515 sched lock 6 sleepq_wait_sig+0xc 14 0% 97% 0.00 1008 sched lock 3 handleevents+0x108 14 0% 97% 0.00 1573 sched lock 29 handleevents+0x108 14 0% 97% 0.00 4041 sched lock 27 __rw_wunlock_hard+0xa8 14 0% 97% 0.00 13160 sched lock 25 umtx_thread_cleanup+0xdc 14 0% 97% 0.00 1252 sched lock 16 handleevents+0x108 14 0% 97% 0.00 1508 sched lock 14 handleevents+0x108 14 0% 97% 0.00 27219 sched lock 12 umtx_thread_cleanup+0xdc 14 0% 97% 0.00 3772 sched lock 10 __mtx_unlock_sleep+0x90 13 0% 97% 0.00 2981 sched lock 9 sleepq_wait_sig+0xc 13 0% 97% 0.00 33317 sched lock 7 sys_fork+0x30 13 0% 97% 0.00 30087 sched lock 6 umtx_thread_cleanup+0xdc 13 0% 97% 0.00 1463 sched lock 5 handleevents+0x108 13 0% 97% 0.00 1223 sched lock 25 thread_exit+0xcc 13 0% 97% 0.00 20620 sched lock 21 sys_fork+0x30 13 0% 97% 0.00 45776 sched lock 20 sys_fork+0x30 13 0% 97% 0.00 1049 sched lock 18 handleevents+0x108 13 0% 97% 0.00 3251 sched lock 16 sleepq_wait_sig+0xc 13 0% 97% 0.00 89386 sched lock 14 umtx_thread_cleanup+0xdc 13 0% 97% 0.00 3447 sched lock 13 sleepq_wait_sig+0xc 13 0% 97% 0.00 1651 sched lock 13 thread_exit+0xcc 13 0% 97% 0.00 23589 sched lock 13 sys_fork+0x30 13 0% 97% 0.00 4194 sched lock 1 sleepq_wait_sig+0xc 13 0% 97% 0.00 1844 sched lock 1 _cv_wait+0x9c 12 0% 97% 0.00 829 sched lock 9 handleevents+0x108 12 0% 97% 0.00 1040 sched lock 8 handleevents+0x108 12 0% 97% 0.00 11970 sched lock 7 umtx_thread_cleanup+0xdc 12 0% 97% 0.00 1544 sched lock 7 __rw_wlock_hard+0x454 12 0% 97% 0.00 4225 sched lock 6 _cv_timedwait_sbt+0xac 12 0% 97% 0.00 5270 sched lock 5 __mtx_unlock_sleep+0x90 12 0% 97% 0.00 32507 sched lock 5 sys_fork+0x30 12 0% 97% 0.00 2589 sched lock 4 sleepq_wait_sig+0xc 12 0% 97% 0.00 25485 sched lock 4 sys_fork+0x30 12 0% 97% 0.00 23488 sched lock 30 sys_fork+0x30 12 0% 97% 0.00 40294 sched lock 29 sys_fork+0x30 12 0% 97% 0.00 23207 sched lock 28 umtx_thread_cleanup+0xdc 12 0% 97% 0.00 1081 sched lock 27 handleevents+0x108 12 0% 97% 0.00 2227 sched lock 27 sleepq_wait_sig+0xc 12 0% 97% 0.00 4768 sched lock 23 sleepq_wait_sig+0xc 12 0% 97% 0.00 2909 sched lock 21 sleepq_wait_sig+0xc 12 0% 97% 0.00 3166 sched lock 2 handleevents+0x108 12 0% 97% 0.00 21427 sched lock 2 sys_fork+0x30 12 0% 98% 0.00 955 sched lock 19 handleevents+0x108 12 0% 98% 0.00 1102 sched lock 15 handleevents+0x108 12 0% 98% 0.00 3127 sched lock 12 sleepq_wait_sig+0xc 12 0% 98% 0.00 1357 sched lock 12 thread_exit+0xcc 12 0% 98% 0.00 13862 sched lock 10 sys_fork+0x30 12 0% 98% 0.00 4301 sched lock 1 __mtx_unlock_sleep+0x90 11 0% 98% 0.00 46640 sched lock 7 _cv_wait+0x9c 11 0% 98% 0.00 3080 sched lock 6 kern_thr_exit+0x1e0 11 0% 98% 0.00 14000 sched lock 31 umtx_thread_cleanup+0xdc 11 0% 98% 0.00 1003 sched lock 30 handleevents+0x108 11 0% 98% 0.00 14785 sched lock 30 umtx_thread_cleanup+0xdc 11 0% 98% 0.00 4529 sched lock 29 sleepq_wait_sig+0xc 11 0% 98% 0.00 38143 sched lock 26 umtx_thread_cleanup+0xdc 11 0% 98% 0.00 4711 sched lock 25 handleevents+0x108 11 0% 98% 0.00 2074 sched lock 24 sleepq_wait_sig+0xc 11 0% 98% 0.00 21888 sched lock 24 umtx_thread_cleanup+0xdc 11 0% 98% 0.00 4515 sched lock 23 __mtx_unlock_sleep+0x90 11 0% 98% 0.00 1092 sched lock 20 handleevents+0x108 11 0% 98% 0.00 4741 sched lock 19 _cv_timedwait_sbt+0xac 11 0% 98% 0.00 24294 sched lock 18 sys_fork+0x30 11 0% 98% 0.00 1201 sched lock 17 handleevents+0x108 11 0% 98% 0.00 33399 sched lock 17 sys_fork+0x30 11 0% 98% 0.00 14949 sched lock 16 umtx_thread_cleanup+0xdc 11 0% 98% 0.00 490 sched lock 12 __rw_wlock_hard+0x454 11 0% 98% 0.00 42834 sched lock 0 umtx_thread_cleanup+0xdc 10 0% 98% 0.00 25340 sched lock 9 sys_fork+0x30 10 0% 98% 0.00 38611 sched lock 8 sys_fork+0x30 10 0% 98% 0.00 1095 sched lock 6 handleevents+0x108 10 0% 98% 0.00 913 sched lock 31 handleevents+0x108 10 0% 98% 0.00 3704 sched lock 30 sleepq_wait_sig+0xc 10 0% 98% 0.00 7295 sched lock 29 umtx_thread_cleanup+0xdc 10 0% 98% 0.00 1497 sched lock 26 sleepq_wait_sig+0xc 10 0% 98% 0.00 22313 sched lock 24 sys_fork+0x30 10 0% 98% 0.00 724 sched lock 23 handleevents+0x108 10 0% 98% 0.00 34753 sched lock 23 sys_fork+0x30 10 0% 98% 0.00 635 sched lock 21 handleevents+0x108 10 0% 98% 0.00 3293 sched lock 2 kern_thr_exit+0x1e0 10 0% 98% 0.00 604 sched lock 2 __rw_wlock_hard+0x454 10 0% 98% 0.00 18493 sched lock 15 umtx_thread_cleanup+0xdc 10 0% 98% 0.00 40204 sched lock 11 sys_fork+0x30 10 0% 98% 0.00 2842 sched lock 0 sleepq_wait_sig+0xc 10 0% 98% 0.00 1757 sched lock 0 kern_thr_exit+0x1e0 10 0% 98% 0.00 43646 sched lock 0 sys_fork+0x30 9 0% 98% 0.00 3303 sleepq chain refcount_release_last+0x60 9 0% 98% 0.00 2589 sched lock 9 __mtx_unlock_sleep+0x90 9 0% 98% 0.00 2410 sched lock 9 _cv_timedwait_sbt+0xac 9 0% 98% 0.00 3681 sched lock 8 sleepq_wait_sig+0xc 9 0% 98% 0.00 2933 sched lock 7 __mtx_lock_sleep+0x140 9 0% 98% 0.00 1397 sched lock 6 __rw_wlock_hard+0x454 9 0% 98% 0.00 3508 sched lock 5 sleepq_wait_sig+0xc 9 0% 98% 0.00 4064 sched lock 4 __mtx_unlock_sleep+0x90 9 0% 98% 0.00 120891 sched lock 27 uiomove_faultflag+0xd4 9 0% 98% 0.00 2446 sched lock 27 kern_thr_exit+0x1e0 9 0% 98% 0.00 77328 sched lock 27 sys_fork+0x30 9 0% 98% 0.00 1377 sched lock 25 kern_thr_exit+0x1e0 9 0% 98% 0.00 3980 sched lock 25 __mtx_unlock_sleep+0x90 9 0% 98% 0.00 2365 sched lock 22 sleepq_wait_sig+0xc 9 0% 98% 0.00 715629 sched lock 22 umtx_thread_cleanup+0xdc 9 0% 98% 0.00 3970 sched lock 2 sleepq_wait_sig+0xc 9 0% 98% 0.00 3706 sched lock 19 sleepq_wait_sig+0xc 9 0% 98% 0.00 528 sched lock 19 __rw_wlock_hard+0x454 9 0% 98% 0.00 41355 sched lock 19 sys_vfork+0x34 9 0% 98% 0.00 23010 sched lock 12 sys_fork+0x30 9 0% 98% 0.00 2153 sched lock 11 handleevents+0x108 9 0% 98% 0.00 4266 sched lock 11 __mtx_unlock_sleep+0x90 9 0% 98% 0.00 3444 sched lock 10 sleepq_wait_sig+0xc 9 0% 98% 0.00 3400 sched lock 10 kern_thr_exit+0x1e0 9 0% 98% 0.00 59219 sched lock 1 _cv_timedwait_sbt+0xac 8 0% 98% 0.00 5744 sched lock 8 __mtx_unlock_sleep+0x90 8 0% 98% 0.00 4397 sched lock 7 __mtx_unlock_sleep+0x90 8 0% 98% 0.00 4986 sched lock 6 __mtx_unlock_sleep+0x90 8 0% 98% 0.00 20472 sched lock 31 sys_fork+0x30 8 0% 98% 0.00 2424 sched lock 30 kern_thr_exit+0x1e0 8 0% 98% 0.00 2764 sched lock 3 __mtx_lock_sleep+0x140 8 0% 98% 0.00 2063 sched lock 28 sleepq_wait_sig+0xc 8 0% 98% 0.00 7808 sched lock 27 __mtx_unlock_sleep+0x90 8 0% 98% 0.00 1125 sched lock 26 handleevents+0x108 8 0% 98% 0.00 82663 sched lock 26 sys_fork+0x30 8 0% 98% 0.00 22441 sched lock 25 sys_fork+0x30 8 0% 98% 0.00 15852 sched lock 23 umtx_thread_cleanup+0xdc 8 0% 98% 0.00 2822 sched lock 23 kern_thr_exit+0x1e0 8 0% 98% 0.00 719 sched lock 23 __rw_wlock_hard+0x454 8 0% 98% 0.00 5563 sched lock 21 __mtx_unlock_sleep+0x90 8 0% 98% 0.00 2044 sched lock 18 kern_thr_exit+0x1e0 8 0% 98% 0.00 377 sched lock 16 __rw_wlock_hard+0x454 8 0% 98% 0.00 399183 sched lock 15 sys_fork+0x30 8 0% 98% 0.00 1647 sched lock 13 __rw_wlock_hard+0x454 8 0% 98% 0.00 3108 sched lock 12 _cv_timedwait_sbt+0xac 8 0% 98% 0.00 922 sched lock 11 __rw_wlock_hard+0x454 7 0% 98% 0.00 3022 sched lock 8 _cv_timedwait_sbt+0xac 7 0% 98% 0.00 2190 sched lock 7 kern_thr_exit+0x1e0 7 0% 98% 0.00 1758 sched lock 6 __mtx_lock_sleep+0x140 7 0% 99% 0.00 12644 sched lock 6 sys_fork+0x30 7 0% 99% 0.00 3197 sched lock 31 _cv_timedwait_sbt+0xac 7 0% 99% 0.00 3336 sched lock 30 __mtx_unlock_sleep+0x90 7 0% 99% 0.00 3368 sched lock 24 __mtx_unlock_sleep+0x90 7 0% 99% 0.00 819 sched lock 22 handleevents+0x108 7 0% 99% 0.00 5355 sched lock 22 __mtx_unlock_sleep+0x90 7 0% 99% 0.00 68698 sched lock 22 sys_fork+0x30 7 0% 99% 0.00 866 sched lock 20 __rw_wlock_hard+0x454 7 0% 99% 0.00 4451 sched lock 2 __mtx_unlock_sleep+0x90 7 0% 99% 0.00 3685 sched lock 19 __mtx_unlock_sleep+0x90 7 0% 99% 0.00 4634 sched lock 17 __mtx_unlock_sleep+0x90 7 0% 99% 0.00 1428 sched lock 16 kern_thr_exit+0x1e0 7 0% 99% 0.00 3698 sched lock 16 __mtx_unlock_sleep+0x90 7 0% 99% 0.00 34739 sched lock 16 sys_fork+0x30 7 0% 99% 0.00 96666 sched lock 15 uiomove_faultflag+0xd4 7 0% 99% 0.00 3187 sched lock 15 _cv_timedwait_sbt+0xac 7 0% 99% 0.00 1644 sched lock 14 kern_thr_exit+0x1e0 7 0% 99% 0.00 463 sched lock 14 __rw_wlock_hard+0x454 7 0% 99% 0.00 4060 sched lock 14 __mtx_unlock_sleep+0x90 7 0% 99% 0.00 19523 sched lock 13 uiomove_faultflag+0xd4 7 0% 99% 0.00 4571 sched lock 13 __mtx_unlock_sleep+0x90 7 0% 99% 0.00 70968 sched lock 10 uiomove_faultflag+0xd4 7 0% 99% 0.00 571 sched lock 10 __rw_wlock_hard+0x454 7 0% 99% 0.00 44977 sched lock 1 sys_fork+0x30 7 0% 99% 0.00 876 sched lock 0 __rw_wlock_hard+0x454 7 0% 99% 0.00 4152 sched lock 0 _cv_timedwait_sbt+0xac 6 0% 99% 0.00 13822 sleepq chain tdsendsignal+0x640 6 0% 99% 0.00 22126 sched lock 9 uiomove_faultflag+0xd4 6 0% 99% 0.00 2244 sched lock 9 __rw_wlock_hard+0x454 6 0% 99% 0.00 518 sched lock 8 __rw_wlock_hard+0x454 6 0% 99% 0.00 3314 sched lock 7 _cv_timedwait_sbt+0xac 6 0% 99% 0.00 10481 sched lock 6 uiomove_faultflag+0xd4 6 0% 99% 0.00 3611 sched lock 5 _cv_timedwait_sbt+0xac 6 0% 99% 0.00 1488 sched lock 4 _cv_wait+0x9c 6 0% 99% 0.00 551 sched lock 3 kern_thr_exit+0x1e0 6 0% 99% 0.00 5337 sched lock 3 _cv_timedwait_sbt+0xac 6 0% 99% 0.00 6144 sched lock 29 __mtx_unlock_sleep+0x90 6 0% 99% 0.00 1540 sched lock 29 _cv_wait+0x9c 6 0% 99% 0.00 1351 sched lock 28 __rw_wlock_hard+0x454 6 0% 99% 0.00 4981 sched lock 26 __mtx_unlock_sleep+0x90 6 0% 99% 0.00 1466 sched lock 24 handleevents+0x108 6 0% 99% 0.00 481 sched lock 24 __rw_wlock_hard+0x454 6 0% 99% 0.00 2452 sched lock 24 _cv_timedwait_sbt+0xac 6 0% 99% 0.00 55696 sched lock 22 uiomove_faultflag+0xd4 6 0% 99% 0.00 19877 sched lock 21 umtx_thread_cleanup+0xdc 6 0% 99% 0.00 2922 sched lock 20 _cv_timedwait_sbt+0xac 6 0% 99% 0.00 5329 sched lock 18 __mtx_unlock_sleep+0x90 6 0% 99% 0.00 4214 sched lock 17 _cv_timedwait_sbt+0xac 6 0% 99% 0.00 848 sched lock 15 __rw_wlock_hard+0x454 6 0% 99% 0.00 1081 sched lock 15 _cv_wait+0x9c 6 0% 99% 0.00 1826 sched lock 14 _cv_timedwait_sbt+0xac 6 0% 99% 0.00 2059 sched lock 13 __mtx_lock_sleep+0x140 6 0% 99% 0.00 1874 sched lock 12 kern_thr_exit+0x1e0 6 0% 99% 0.00 1978 sched lock 11 sleepq_wait_sig+0xc 6 0% 99% 0.00 38089 sched lock 11 sys_vfork+0x34 6 0% 99% 0.00 3522 sched lock 11 _cv_timedwait_sbt+0xac 6 0% 99% 0.00 2044 sched lock 11 _cv_wait+0x9c 6 0% 99% 0.00 13981 sched lock 10 sys_vfork+0x34 6 0% 99% 0.00 47148 sched lock 1 uiomove_faultflag+0xd4 6 0% 99% 0.00 1310 sched lock 0 _cv_wait+0x9c 5 0% 99% 0.00 1670 sched lock 9 _sx_xlock_hard+0x404 5 0% 99% 0.00 1728 sched lock 9 _cv_timedwait_sbt+0x114 5 0% 99% 0.00 2271 sched lock 8 __mtx_lock_sleep+0x140 5 0% 99% 0.00 100862 sched lock 8 _cv_wait+0x9c 5 0% 99% 0.00 13049 sched lock 7 uiomove_faultflag+0xd4 5 0% 99% 0.00 902 sched lock 6 _cv_timedwait_sbt+0x114 5 0% 99% 0.00 2933 sched lock 5 kern_thr_exit+0x1e0 5 0% 99% 0.00 444 sched lock 5 __rw_wlock_hard+0x454 5 0% 99% 0.00 848 sched lock 4 kern_thr_exit+0x1e0 5 0% 99% 0.00 10391 sched lock 31 uiomove_faultflag+0xd4 5 0% 99% 0.00 831 sched lock 31 __rw_wlock_hard+0x454 5 0% 99% 0.00 70813 sched lock 3 uiomove_faultflag+0xd4 5 0% 99% 0.00 63351 sched lock 3 sys_fork+0x30 5 0% 99% 0.00 2031 sched lock 3 _cv_timedwait_sbt+0x114 5 0% 99% 0.00 511 sched lock 28 __mtx_lock_sleep+0x140 5 0% 99% 0.00 27489 sched lock 26 uiomove_faultflag+0xd4 5 0% 99% 0.00 3386 sched lock 25 sleepq_wait_sig+0xc 5 0% 99% 0.00 978 sched lock 23 _cv_timedwait_sbt+0xac 5 0% 99% 0.00 644 sched lock 22 thread_exit+0xcc 5 0% 99% 0.00 2444 sched lock 22 sys_vfork+0x34 5 0% 99% 0.00 2520 sched lock 22 _cv_timedwait_sbt+0xac 5 0% 99% 0.00 3448 sched lock 20 kern_thr_exit+0x1e0 5 0% 99% 0.00 1275 sched lock 20 __mtx_lock_sleep+0x140 5 0% 99% 0.00 4640 sched lock 2 _cv_timedwait_sbt+0xac 5 0% 99% 0.00 2786 sched lock 19 __mtx_lock_sleep+0x140 5 0% 99% 0.00 488 sched lock 18 __rw_wlock_hard+0x454 5 0% 99% 0.00 3688 sched lock 17 _cv_wait+0x9c 5 0% 99% 0.00 5546 sched lock 15 __mtx_unlock_sleep+0x90 5 0% 99% 0.00 60684 sched lock 14 uiomove_faultflag+0xd4 5 0% 99% 0.00 622 sched lock 13 kern_thr_exit+0x1e0 5 0% 99% 0.00 3613 sched lock 12 __mtx_unlock_sleep+0x90 5 0% 99% 0.00 2551 sched lock 12 _cv_wait+0x9c 5 0% 99% 0.00 5324 sched lock 10 _cv_timedwait_sbt+0xac 5 0% 99% 0.00 6306 sched lock 1 handle_el0_irq+0xb8 5 0% 99% 0.00 41680 sched lock 0 uiomove_faultflag+0xd4 5 0% 99% 0.00 506 sched lock 0 swi_sched+0x80 4 0% 99% 0.00 1933 sleepq chain wakeupshlk+0x12c 4 0% 99% 0.00 2972 sched lock 9 kern_thr_exit+0x1e0 4 0% 99% 0.00 3094 sched lock 9 __mtx_lock_sleep+0x140 4 0% 99% 0.00 2783 sched lock 9 _cv_wait+0x9c 4 0% 99% 0.00 644 sched lock 8 kern_thr_exit+0x1e0 4 0% 99% 0.00 716 sched lock 7 _sx_xlock_hard+0x404 4 0% 99% 0.00 8922 sched lock 7 sys_vfork+0x34 4 0% 99% 0.00 667 sched lock 6 _sx_xlock_hard+0x404 4 0% 99% 0.00 18000 sched lock 5 handle_el0_irq+0xb8 4 0% 99% 0.00 850 sched lock 5 _sx_xlock_hard+0x404 4 0% 99% 0.00 17177 sched lock 4 uiomove_faultflag+0xd4 4 0% 99% 0.00 388 sched lock 4 __rw_wlock_hard+0x454 4 0% 99% 0.00 3966 sched lock 31 swi_sched+0x80 4 0% 99% 0.00 2683 sched lock 30 _cv_timedwait_sbt+0xac 4 0% 99% 0.00 3427 sched lock 3 handle_el0_irq+0xb8 4 0% 99% 0.00 505 sched lock 3 __rw_wlock_hard+0x454 4 0% 99% 0.00 2944 sched lock 3 _cv_wait+0x9c 4 0% 99% 0.00 26017 sched lock 29 handle_el0_irq+0xb8 4 0% 99% 0.00 3349 sched lock 29 kern_thr_exit+0x1e0 4 0% 99% 0.00 4555 sched lock 28 kern_thr_exit+0x1e0 4 0% 99% 0.00 128316 sched lock 28 __mtx_unlock_sleep+0x90 4 0% 99% 0.00 2716 sched lock 28 _cv_timedwait_sbt+0xac 4 0% 99% 0.00 30916 sched lock 27 sys_vfork+0x34 4 0% 99% 0.00 317 sched lock 26 __rw_wlock_hard+0x454 4 0% 99% 0.00 3766 sched lock 26 _cv_timedwait_sbt+0xac 4 0% 99% 0.00 922 sched lock 25 __mtx_lock_sleep+0x140 4 0% 99% 0.00 5111 sched lock 25 _cv_timedwait_sbt+0xac 4 0% 99% 0.00 2217 sched lock 24 kern_thr_exit+0x1e0 4 0% 99% 0.00 29466 sched lock 23 uiomove_faultflag+0xd4 4 0% 99% 0.00 3172 sched lock 22 handle_el0_irq+0xb8 4 0% 99% 0.00 416 sched lock 22 __mtx_lock_sleep+0x140 4 0% 99% 0.00 4261 sched lock 22 _cv_timedwait_sbt+0x114 4 0% 99% 0.00 655 sched lock 21 __rw_wlock_hard+0x454 4 0% 99% 0.00 2944 sched lock 21 _cv_wait+0x9c 4 0% 99% 0.00 209705 sched lock 20 uiomove_faultflag+0xd4 4 0% 99% 0.00 2105 sched lock 20 _cv_wait+0x9c 4 0% 99% 0.00 87455 sched lock 2 uiomove_faultflag+0xd4 4 0% 99% 0.00 83150 sched lock 2 sys_thr_new+0x8c 4 0% 99% 0.00 3594 sched lock 19 kern_thr_exit+0x1e0 4 0% 99% 0.00 1761 sched lock 18 _cv_timedwait_sbt+0x114 4 0% 99% 0.00 3939 sched lock 18 _cv_timedwait_sbt+0xac 4 0% 99% 0.00 622 sched lock 18 _cv_wait+0x9c 4 0% 99% 0.00 375105 sched lock 17 uiomove_faultflag+0xd4 4 0% 99% 0.00 1705 sched lock 17 __rw_wlock_hard+0x454 4 0% 99% 0.00 3272 sched lock 16 __mtx_lock_sleep+0x140 4 0% 99% 0.00 9777 sched lock 15 sys_vfork+0x34 4 0% 99% 0.00 1027 sched lock 14 __mtx_lock_sleep+0x140 4 0% 99% 0.00 37049 sched lock 14 sys_vfork+0x34 4 0% 99% 0.00 450 sched lock 14 _cv_wait+0xf0 4 0% 99% 0.00 1833 sched lock 12 _cv_wait+0xf0 4 0% 99% 0.00 13200 sched lock 10 sys_thr_new+0x8c 4 0% 99% 0.00 1244 sched lock 10 _cv_wait+0x9c 4 0% 99% 0.00 972 sched lock 1 kern_thr_exit+0x1e0 4 0% 99% 0.00 1205 sched lock 0 _sx_xlock_hard+0x404 3 0% 99% 0.00 1970 sleepq chain softclock_call_cc+0x130 3 0% 99% 0.00 1548 sleepq chain sx_downgrade_int+0x94 3 0% 99% 0.00 911 sleepq chain tdsendsignal+0x460 3 0% 99% 0.00 28992 sched lock 9 handle_el0_irq+0xb8 3 0% 99% 0.00 1570 sched lock 9 _sx_slock_hard+0x264 3 0% 99% 0.00 139629 sched lock 8 uiomove_faultflag+0xd4 3 0% 99% 0.00 43504 sched lock 8 sys_vfork+0x34 3 0% 99% 0.00 3970 sched lock 8 _cv_wait+0xf0 3 0% 99% 0.00 23629 sched lock 7 sys_thr_new+0x8c 3 0% 99% 0.00 1466 sched lock 7 _cv_wait+0xf0 3 0% 99% 0.00 1999 sched lock 5 __mtx_lock_sleep+0x140 3 0% 99% 0.00 2444 sched lock 4 _sx_slock_hard+0x264 3 0% 99% 0.00 4348 sched lock 4 _sx_xlock_hard+0x404 3 0% 99% 0.00 14688 sched lock 4 sys_vfork+0x34 3 0% 99% 0.00 2029 sched lock 31 __mtx_lock_sleep+0x140 3 0% 99% 0.00 18903 sched lock 30 uiomove_faultflag+0xd4 3 0% 99% 0.00 6407 sched lock 30 _sleep+0x194 3 0% 99% 0.00 17303 sched lock 30 sys_thr_new+0x8c 3 0% 99% 0.00 1296 sched lock 30 _sx_xlock_hard+0x404 3 0% 99% 0.00 2904 sched lock 30 _cv_timedwait_sbt+0x114 3 0% 99% 0.00 34851 sched lock 3 sys_vfork+0x34 3 0% 99% 0.00 11170 sched lock 28 uiomove_faultflag+0xd4 3 0% 99% 0.00 4229 sched lock 28 _cv_wait+0xf0 3 0% 99% 0.00 16274 sched lock 27 sys_thr_new+0x8c 3 0% 99% 0.00 385 sched lock 27 __rw_wlock_hard+0x454 3 0% 99% 0.00 2963 sched lock 27 _cv_timedwait_sbt+0xac 3 0% 99% 0.00 27303 sched lock 26 sys_thr_new+0x8c 3 0% 99% 0.00 3851 sched lock 26 kern_thr_exit+0x1e0 3 0% 99% 0.00 36762 sched lock 26 sys_vfork+0x34 3 0% 99% 0.00 18348 sched lock 25 uiomove_faultflag+0xd4 3 0% 99% 0.00 681 sched lock 25 __rw_wlock_hard+0x454 3 0% 99% 0.00 26310 sched lock 25 sys_vfork+0x34 3 0% 100% 0.00 34356 sched lock 24 sys_thr_new+0x8c 3 0% 100% 0.00 1622 sched lock 24 _cv_wait+0x9c 3 0% 100% 0.00 578 sched lock 23 _cv_timedwait_sbt+0x114 3 0% 100% 0.00 2252 sched lock 22 kern_thr_exit+0x1e0 3 0% 100% 0.00 392 sched lock 22 __rw_wlock_hard+0x454 3 0% 100% 0.00 171052 sched lock 22 _cv_wait+0x9c 3 0% 100% 0.00 1081 sched lock 21 _sx_xlock_hard+0x404 3 0% 100% 0.00 9340 sched lock 21 sys_vfork+0x34 3 0% 100% 0.00 2378 sched lock 21 _cv_timedwait_sbt+0xac 3 0% 100% 0.00 7407 sched lock 21 _cv_wait+0xf0 3 0% 100% 0.00 437 sched lock 2 _sx_xlock_hard+0x404 3 0% 100% 0.00 61311 sched lock 2 sys_vfork+0x34 3 0% 100% 0.00 1777 sched lock 2 _cv_wait+0x9c 3 0% 100% 0.00 10163 sched lock 19 uiomove_faultflag+0xd4 3 0% 100% 0.00 3755 sched lock 19 _cv_timedwait_sbt+0x114 3 0% 100% 0.00 3792 sched lock 19 _cv_wait+0x9c 3 0% 100% 0.00 904 sched lock 18 handle_el0_irq+0xb8 3 0% 100% 0.00 881 sched lock 18 _sx_xlock_hard+0x404 3 0% 100% 0.00 733 sched lock 18 __mtx_lock_sleep+0x140 3 0% 100% 0.00 859 sched lock 17 kern_thr_exit+0x1e0 3 0% 100% 0.00 563 sched lock 17 _sx_xlock_hard+0x404 3 0% 100% 0.00 2296 sched lock 17 __mtx_lock_sleep+0x140 3 0% 100% 0.00 1022 sched lock 16 _sx_xlock_hard+0x404 3 0% 100% 0.00 25052 sched lock 16 sys_vfork+0x34 3 0% 100% 0.00 1970 sched lock 15 __mtx_lock_sleep+0x140 3 0% 100% 0.00 1022 sched lock 14 _sx_xlock_hard+0x404 3 0% 100% 0.00 1644 sched lock 14 _cv_wait+0x9c 3 0% 100% 0.00 6251 sched lock 13 handle_el0_irq+0xb8 3 0% 100% 0.00 39400 sched lock 13 sys_vfork+0x34 3 0% 100% 0.00 7636 sched lock 13 _cv_timedwait_sbt+0xac 3 0% 100% 0.00 1222 sched lock 13 _cv_wait+0x9c 3 0% 100% 0.00 21022 sched lock 12 uiomove_faultflag+0xd4 3 0% 100% 0.00 346526 sched lock 12 sys_vfork+0x34 3 0% 100% 0.00 800 sched lock 11 _sx_xlock_hard+0x404 3 0% 100% 0.00 1437 sched lock 10 _sx_xlock_hard+0x404 3 0% 100% 0.00 563 sched lock 1 __rw_wlock_hard+0x454 3 0% 100% 0.00 24074 sched lock 1 sys_vfork+0x34 3 0% 100% 0.00 10103 sched lock 0 sys_vfork+0x34 2 0% 100% 0.00 644 sleepq chain rufetch+0x44 2 0% 100% 0.00 102022 sched lock 9 sys_thr_new+0x8c 2 0% 100% 0.00 94955 sched lock 6 handle_el0_irq+0xb8 2 0% 100% 0.00 93733 sched lock 6 sys_vfork+0x34 2 0% 100% 0.00 81866 sched lock 5 uiomove_faultflag+0xd4 2 0% 100% 0.00 2444 sched lock 5 handle_el0_sync+0x90 2 0% 100% 0.00 24545 sched lock 5 sys_vfork+0x34 2 0% 100% 0.00 5611 sched lock 5 _cv_timedwait_sbt+0x114 2 0% 100% 0.00 1133 sched lock 5 _cv_wait+0x9c 2 0% 100% 0.00 6311 sched lock 4 handle_el0_irq+0xb8 2 0% 100% 0.00 2933 sched lock 4 __mtx_lock_sleep+0x140 2 0% 100% 0.00 3044 sched lock 31 kern_thr_exit+0x1e0 2 0% 100% 0.00 1433 sched lock 31 _sx_xlock_hard+0x404 2 0% 100% 0.00 5511 sched lock 31 __mtx_unlock_sleep+0x90 2 0% 100% 0.00 20311 sched lock 31 sys_vfork+0x34 2 0% 100% 0.00 5700 sched lock 31 _cv_timedwait_sbt+0x114 2 0% 100% 0.00 711 sched lock 30 handle_el0_irq+0xb8 2 0% 100% 0.00 700 sched lock 30 __rw_wlock_hard+0x454 2 0% 100% 0.00 3522 sched lock 30 _sx_slock_hard+0x264 2 0% 100% 0.00 2389 sched lock 30 _cv_wait+0x9c 2 0% 100% 0.00 12211 sched lock 3 sys_thr_new+0x8c 2 0% 100% 0.00 8533 sched lock 29 sys_thr_new+0x8c 2 0% 100% 0.00 711 sched lock 29 _sx_xlock_hard+0x404 2 0% 100% 0.00 54822 sched lock 29 sys_setpriority+0x4b0 2 0% 100% 0.00 266 sched lock 29 __rw_wlock_hard+0x454 2 0% 100% 0.00 2933 sched lock 29 __mtx_lock_sleep+0x140 2 0% 100% 0.00 4211 sched lock 29 _cv_timedwait_sbt+0xac 2 0% 100% 0.00 466 sched lock 28 handle_el0_irq+0xb8 2 0% 100% 0.00 700 sched lock 28 _sx_xlock_hard+0x404 2 0% 100% 0.00 146222 sched lock 28 sys_rtprio_thread+0xe8 2 0% 100% 0.00 8844 sched lock 28 sys_vfork+0x34 2 0% 100% 0.00 30311 sched lock 27 handle_el0_irq+0xb8 2 0% 100% 0.00 5622 sched lock 27 __mtx_lock_sleep+0x140 2 0% 100% 0.00 1711 sched lock 27 _cv_wait+0x9c 2 0% 100% 0.00 2133 sched lock 26 handle_el0_irq+0xb8 2 0% 100% 0.00 366 sched lock 26 _sx_xlock_hard+0x404 2 0% 100% 0.00 4200 sched lock 26 __mtx_lock_sleep+0x140 2 0% 100% 0.00 333 sched lock 26 swi_sched+0x80 2 0% 100% 0.00 1588 sched lock 26 _cv_wait+0xf0 2 0% 100% 0.00 333 sched lock 25 _sx_xlock_hard+0x404 2 0% 100% 0.00 289 sched lock 25 _cv_timedwait_sbt+0x114 2 0% 100% 0.00 2000 sched lock 25 _cv_wait+0xf0 2 0% 100% 0.00 1011 sched lock 25 _cv_wait+0x9c 2 0% 100% 0.00 4288 sched lock 24 handle_el0_irq+0xb8 2 0% 100% 0.00 1966 sched lock 24 __mtx_lock_sleep+0x140 2 0% 100% 0.00 4311 sched lock 24 _cv_timedwait_sbt+0x114 2 0% 100% 0.00 14566 sched lock 23 handle_el0_irq+0xb8 2 0% 100% 0.00 35922 sched lock 23 sys_vfork+0x34 2 0% 100% 0.00 1278 sched lock 23 _cv_wait+0x9c 2 0% 100% 0.00 466 sched lock 22 swi_sched+0x80 2 0% 100% 0.00 3677 sched lock 21 uiomove_faultflag+0xd4 2 0% 100% 0.00 4255 sched lock 21 kern_thr_exit+0x1e0 2 0% 100% 0.00 5611 sched lock 21 sys_thr_new+0x8c 2 0% 100% 0.00 5511 sched lock 20 handle_el0_irq+0xb8 2 0% 100% 0.00 110955 sched lock 20 sys_thr_new+0x8c 2 0% 100% 0.00 866 sched lock 20 _sx_xlock_hard+0x404 2 0% 100% 0.00 3099 sched lock 20 _sx_slock_hard+0x264 2 0% 100% 0.00 3900 sched lock 20 sys_vfork+0x34 2 0% 100% 0.00 1900 sched lock 2 _sx_slock_hard+0x264 2 0% 100% 0.00 711 sched lock 2 __mtx_lock_sleep+0x140 2 0% 100% 0.00 18200 sched lock 18 uiomove_faultflag+0xd4 2 0% 100% 0.00 1078 sched lock 18 _sx_xlock_hard+0x418 2 0% 100% 0.00 38467 sched lock 17 sys_thr_new+0x8c 2 0% 100% 0.00 24767 sched lock 16 handle_el0_irq+0xb8 2 0% 100% 0.00 21845 sched lock 16 uiomove_faultflag+0xd4 2 0% 100% 0.00 4333 sched lock 16 _cv_timedwait_sbt+0xac 2 0% 100% 0.00 1200 sched lock 15 handle_el0_irq+0xb8 2 0% 100% 0.00 8143 sched lock 15 kern_thr_exit+0x1e0 2 0% 100% 0.00 3400 sched lock 14 _sleep+0x194 2 0% 100% 0.00 1911 sched lock 13 _sx_slock_hard+0x264 2 0% 100% 0.00 3011 sched lock 13 _sx_xlock_hard+0x404 2 0% 100% 0.00 1633 sched lock 13 _cv_timedwait_sbt+0x114 2 0% 100% 0.00 3044 sched lock 12 _sx_xlock_hard+0x404 2 0% 100% 0.00 6078 sched lock 12 __mtx_lock_sleep+0x140 2 0% 100% 0.00 5489 sched lock 11 handle_el0_irq+0xb8 2 0% 100% 0.00 1478 sched lock 10 handle_el0_irq+0xb8 2 0% 100% 0.00 2267 sched lock 10 __mtx_lock_sleep+0x140 2 0% 100% 0.00 3355 sched lock 10 _cv_timedwait_sbt+0x114 2 0% 100% 0.00 5822 sched lock 1 vm_page_free_prep+0x184 2 0% 100% 0.00 544 sched lock 1 _sx_xlock_hard+0x404 2 0% 100% 0.00 489 sched lock 1 _cv_timedwait_sbt+0x114 2 0% 100% 0.00 13755 sched lock 0 handle_el0_irq+0xb8 2 0% 100% 0.00 800 sched lock 0 _sx_slock_hard+0x264 2 0% 100% 0.00 1688 sched lock 0 __mtx_lock_sleep+0x140 2 0% 100% 0.00 955 sched lock 0 _cv_wait+0xf0 1 0% 100% 0.00 10689 sched lock 9 handle_el0_sync+0x90 1 0% 100% 0.00 267 sched lock 9 vm_page_activate+0x74 1 0% 100% 0.00 400 sched lock 8 release_page+0x98 1 0% 100% 0.00 822 sched lock 8 handle_el0_irq+0xb8 1 0% 100% 0.00 2911 sched lock 8 _sx_slock_hard+0x278 1 0% 100% 0.00 1689 sched lock 8 _sleep+0x194 1 0% 100% 0.00 489 sched lock 8 rufetch+0x44 1 0% 100% 0.00 21466 sched lock 8 sys_setpriority+0x4b0 1 0% 100% 0.00 2889 sched lock 8 _sx_xlock_hard+0x404 1 0% 100% 0.00 1022 sched lock 7 handle_el0_sync+0x90 1 0% 100% 0.00 733 sched lock 7 vm_page_activate+0x74 1 0% 100% 0.00 1134 sched lock 7 handle_el0_irq+0xb8 1 0% 100% 0.00 1089 sched lock 8 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 1200 sched lock 6 uma_zalloc_arg+0x980 1 0% 100% 0.00 3155 sched lock 7 swi_sched+0x80 1 0% 100% 0.00 5955 sched lock 7 _cv_timedwait_sbt+0xc0 1 0% 100% 0.00 1045 sched lock 7 _sx_slock_hard+0x264 1 0% 100% 0.00 688 sched lock 7 __rw_rlock_hard+0x260 1 0% 100% 0.00 822 sched lock 7 _sleep+0x194 1 0% 100% 0.00 1733 sched lock 4 uma_zalloc_arg+0x980 1 0% 100% 0.00 1844 sched lock 4 rufetch+0x44 1 0% 100% 0.00 267 sched lock 4 _sleep+0x194 1 0% 100% 0.00 121022 sched lock 31 handle_el0_irq+0xb8 1 0% 100% 0.00 267 sched lock 31 thread_single+0x474 1 0% 100% 0.00 289 sched lock 4 _cv_timedwait_sbt+0xac 1 0% 100% 0.00 3000 sched lock 4 _cv_wait+0xf0 1 0% 100% 0.00 778 sched lock 4 swi_sched+0x80 1 0% 100% 0.00 712733 sched lock 5 sys_thr_new+0x8c 1 0% 100% 0.00 2933 sched lock 5 _sx_slock_hard+0x264 1 0% 100% 0.00 600 sched lock 5 _cv_timedwait_sbt+0xc0 1 0% 100% 0.00 645 sched lock 6 _cv_timedwait_sbt+0xc0 1 0% 100% 0.00 10688 sched lock 6 _cv_wait+0xf0 1 0% 100% 0.00 222 sched lock 6 _cv_wait+0x9c 1 0% 100% 0.00 34867 sched lock 29 uiomove_faultflag+0xd4 1 0% 100% 0.00 1134 sched lock 29 _sleep+0x194 1 0% 100% 0.00 267 sched lock 29 exit1+0xb0 1 0% 100% 0.00 486556 sched lock 29 cpuset_setproc_update_set+0x3c 1 0% 100% 0.00 556 sched lock 29 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 1800 sched lock 29 _cv_timedwait_sbt+0xc0 1 0% 100% 0.00 18712 sched lock 29 sys_vfork+0x34 1 0% 100% 0.00 445 sched lock 29 swi_sched+0x80 1 0% 100% 0.00 38911 sched lock 29 kern_proc_out+0x34 1 0% 100% 0.00 1622 sched lock 28 _cv_wait+0x9c 1 0% 100% 0.00 1111 sched lock 28 _sx_slock_hard+0x278 1 0% 100% 0.00 1866 sched lock 28 _sx_xlock_hard+0x418 1 0% 100% 0.00 16933 sched lock 28 sys_thr_new+0x8c 1 0% 100% 0.00 1689 sched lock 28 _sleep+0x194 1 0% 100% 0.00 1200 sched lock 30 _cv_wait+0xf0 1 0% 100% 0.00 200 sched lock 30 exit1+0xb0 1 0% 100% 0.00 778 sched lock 3 vm_page_activate+0x74 1 0% 100% 0.00 6422 sched lock 3 free+0x4c 1 0% 100% 0.00 289 sched lock 3 swi_sched+0x80 1 0% 100% 0.00 1022 sched lock 3 _cv_timedwait_sbt+0xc0 1 0% 100% 0.00 11201 sched lock 3 _sx_slock_hard+0x278 1 0% 100% 0.00 2977 sched lock 31 _cv_wait+0x9c 1 0% 100% 0.00 311 sched lock 30 vm_page_activate+0x74 1 0% 100% 0.00 5733 sched lock 31 _cv_wait+0xf0 1 0% 100% 0.00 822 sched lock 31 sys_rtprio_thread+0xe8 1 0% 100% 0.00 5489 sched lock 31 _sx_xlock_hard+0x418 1 0% 100% 0.00 778 sched lock 21 handle_el0_irq+0xb8 1 0% 100% 0.00 5578 sched lock 21 _sx_xlock_hard+0x418 1 0% 100% 0.00 755 sched lock 20 uiomove_fromphys+0x138 1 0% 100% 0.00 444 sched lock 21 _cv_wait_sig+0x9c 1 0% 100% 0.00 355 sched lock 21 swi_sched+0x80 1 0% 100% 0.00 1667 sched lock 2 handle_el0_irq+0xb8 1 0% 100% 0.00 33978 sched lock 2 uiomove_fromphys+0x138 1 0% 100% 0.00 1000 sched lock 2 vm_page_free_prep+0x184 1 0% 100% 0.00 6000 sched lock 20 _cv_timedwait_sbt+0xc0 1 0% 100% 0.00 3022 sched lock 20 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 689 sched lock 18 vfs_msync+0xdc 1 0% 100% 0.00 219267 sched lock 18 sys_thr_new+0x8c 1 0% 100% 0.00 822 sched lock 17 vm_page_free_prep+0x184 1 0% 100% 0.00 666 sched lock 18 exit1+0xb0 1 0% 100% 0.00 6311 sched lock 18 sys_vfork+0x34 1 0% 100% 0.00 5689 sched lock 19 _sleep+0x194 1 0% 100% 0.00 19089 sched lock 19 sys_thr_new+0x8c 1 0% 100% 0.00 378 sched lock 19 _sx_xlock_hard+0x404 1 0% 100% 0.00 5466 sched lock 19 _sx_xlock_hard+0x418 1 0% 100% 0.00 844 sched lock 19 _sx_slock_hard+0x264 1 0% 100% 0.00 3156 sched lock 19 rufetch+0x44 1 0% 100% 0.00 32444 sched lock 19 handle_el0_irq+0xb8 1 0% 100% 0.00 1289 sched lock 2 free+0x4c 1 0% 100% 0.00 10667 sched lock 2 kern_proc_out+0x34 1 0% 100% 0.00 14022 sched lock 24 uiomove_fromphys+0x138 1 0% 100% 0.00 467 sched lock 24 _sx_slock_hard+0x264 1 0% 100% 0.00 3089 sched lock 24 _sx_xlock_hard+0x404 1 0% 100% 0.00 10600 sched lock 24 _sleep+0x194 1 0% 100% 0.00 334 sched lock 23 release_page+0x98 1 0% 100% 0.00 5488 sched lock 22 _sx_xlock_hard+0x404 1 0% 100% 0.00 5467 sched lock 22 vm_page_free_prep+0x184 1 0% 100% 0.00 5467 sched lock 23 __mtx_lock_sleep+0x140 1 0% 100% 0.00 622 sched lock 23 _sx_xlock_hard+0x404 1 0% 100% 0.00 645 sched lock 26 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 800 sched lock 25 handle_el0_irq+0xb8 1 0% 100% 0.00 21311 sched lock 25 sys_thr_new+0x8c 1 0% 100% 0.00 10667 sched lock 25 vmspace_exec+0xb4 1 0% 100% 0.00 355 sched lock 25 exit1+0xb0 1 0% 100% 0.00 756 sched lock 25 __rw_runlock_hard+0xf4 1 0% 100% 0.00 5578 sched lock 25 _sx_slock_hard+0x264 1 0% 100% 0.00 1578 sched lock 26 _sx_slock_hard+0x264 1 0% 100% 0.00 5555 sched lock 26 _sleep+0x134 1 0% 100% 0.00 10934 sched lock 26 _sleep+0x194 1 0% 100% 0.00 9311 sched lock 26 sys_rtprio_thread+0xe8 1 0% 100% 0.00 1134 sched lock 26 _sx_xlock_hard+0x418 1 0% 100% 0.00 1088 sched lock 27 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 3178 sched lock 27 _sx_slock_hard+0x264 1 0% 100% 0.00 2977 sched lock 17 sleepq_timedwait_sig+0x14 1 0% 100% 0.00 1600 sched lock 17 _sx_slock_hard+0x264 1 0% 100% 0.00 24645 sched lock 17 sys_vfork+0x34 1 0% 100% 0.00 21266 sched lock 17 sys_rtprio_thread+0xe8 1 0% 100% 0.00 289 sched lock 16 release_page+0x98 1 0% 100% 0.00 10645 sched lock 17 _cv_timedwait_sbt+0xc0 1 0% 100% 0.00 378 sched lock 17 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 1600 sched lock 15 _sx_slock_hard+0x264 1 0% 100% 0.00 1022 sched lock 15 _sx_xlock_hard+0x418 1 0% 100% 0.00 1134 sched lock 15 _sleep+0x134 1 0% 100% 0.00 11222 sched lock 15 _sleep+0x194 1 0% 100% 0.00 2866 sched lock 15 _sx_xlock_hard+0x404 1 0% 100% 0.00 778 sched lock 16 _cv_wait+0x9c 1 0% 100% 0.00 1044 sched lock 15 vm_page_free_prep+0x184 1 0% 100% 0.00 1067 sched lock 15 handle_el0_sync+0x90 1 0% 100% 0.00 5689 sched lock 16 free+0x4c 1 0% 100% 0.00 1667 sched lock 13 _sleep+0x134 1 0% 100% 0.00 5600 sched lock 13 _sx_xlock_hard+0x418 1 0% 100% 0.00 10600 sched lock 12 _sleep+0x194 1 0% 100% 0.00 600 sched lock 12 sys_thr_new+0x8c 1 0% 100% 0.00 11022 sched lock 12 spinlock_exit+0x4c 1 0% 100% 0.00 5600 sched lock 13 exit1+0xb0 1 0% 100% 0.00 1867 sched lock 13 uiomove_fromphys+0x138 1 0% 100% 0.00 5534 sched lock 14 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 29867 sched lock 14 cpuset_setproc_update_set+0x3c 1 0% 100% 0.00 5555 sched lock 14 handle_el0_irq+0xb8 1 0% 100% 0.00 10578 sched lock 15 _cv_wait_sig+0x9c 1 0% 100% 0.00 5755 sched lock 15 _cv_wait+0xf0 1 0% 100% 0.00 2955 sched lock 15 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 5533 sched lock 1 _sx_slock_hard+0x278 1 0% 100% 0.00 1733 sched lock 1 sys_thr_new+0x8c 1 0% 100% 0.00 667 sched lock 1 __mtx_lock_sleep+0x140 1 0% 100% 0.00 13045 sched lock 1 sys_setpriority+0x4b0 1 0% 100% 0.00 1000 sched lock 1 _sx_slock_hard+0x264 1 0% 100% 0.00 1089 sched lock 0 vm_page_activate+0x74 1 0% 100% 0.00 5644 sched lock 0 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 333 sched lock 0 exit1+0xb0 1 0% 100% 0.00 10511 sched lock 0 malloc+0x114 1 0% 100% 0.00 1044 sched lock 11 _cv_timedwait_sbt+0x114 1 0% 100% 0.00 867 sched lock 10 uma_zalloc_arg+0x980 1 0% 100% 0.00 1178 sched lock 10 sys_ppoll+0x80 1 0% 100% 0.00 356 sched lock 10 __umtx_op_nwake_private+0x78 1 0% 100% 0.00 733 sched lock 10 swi_sched+0x80 1 0% 100% 0.00 512 sched lock 11 uiomove_faultflag+0xd4 1 0% 100% 0.00 5422 sched lock 11 _sx_slock_hard+0x264 1 0% 100% 0.00 333 sched lock 11 kern_thr_exit+0x1e0 1 0% 100% 0.00 5511 sched lock 11 __mtx_lock_sleep+0x140 1 0% 100% 0.00 1645 sched lock 11 swi_sched+0x80 1 0% 100% 0.00 1133 sched lock 12 exit1+0xb0 1 0% 100% 0.00 90866 sched lock 12 sys_rtprio_thread+0xe8 ------------------------------------------------------------------------------- R/W writer blocked by writer: 31775 events in 300.017 seconds (106 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 10695 34% 34% 0.00 440751 pmap pv list vmspace_exit+0xc0 8588 27% 61% 0.00 406513 pmap pv list pmap_copy+0x340 4368 14% 74% 0.00 437833 pmap pv list pmap_enter_quick_locked+0x1c0 3896 12% 87% 0.00 444829 pmap pv list exec_new_vmspace+0x1d0 2808 9% 96% 0.00 547194 pmap pv list vm_fault+0x1634 541 2% 97% 0.00 393370 kernel vm object kmem_malloc_domainset+0xc0 370 1% 98% 0.00 196401 kernel vm object kmem_free+0x28 80 0% 99% 0.00 294758 pmap pv list pmap_remove+0x198 79 0% 99% 0.00 1548095 vmobject vm_map_delete+0xa8 64 0% 99% 0.00 614059 vmobject vm_fault_trap+0x64 48 0% 99% 0.00 547668 vmobject vm_mmap_vnode+0x260 46 0% 99% 0.00 419693 vmobject vm_map_lookup+0x9ac 33 0% 99% 0.00 1096872 vmobject vm_map_fixed+0x134 27 0% 100% 0.00 410120 pmap pv list vm_fault+0x364 25 0% 100% 0.00 404518 pmap pv list vm_pageout_worker+0x11b0 18 0% 100% 0.00 887969 vmobject vm_map_find+0x514 18 0% 100% 0.00 1241539 vmobject fork1+0x33c 18 0% 100% 0.00 551200 tidhash kern_thr_exit+0x120 14 0% 100% 0.00 569280 vmobject vm_map_process_deferred+0x88 10 0% 100% 0.00 363664 vmobject vmspace_fork+0x910 6 0% 100% 0.00 207888 tidhash fork1+0x984 5 0% 100% 0.00 613808 ncbuc cache_lookup+0x94 4 0% 100% 0.00 187455 vmobject vm_object_deallocate+0x478 4 0% 100% 0.00 915527 vmobject vm_map_entry_delete+0xe0 3 0% 100% 0.00 756333 pmap pv list pmap_enter_l2+0x260 2 0% 100% 0.00 225722 vmobject zfs_freebsd_getpages+0x1c4 1 0% 100% 0.00 103311 vmobject elf64_load_file+0xf0 1 0% 100% 0.00 15444 pmap pv list vm_page_try_remove_all+0x3c 1 0% 100% 0.00 67911 tidhash exit1+0x5e0 1 0% 100% 0.00 52022 ncbuc zfsvfs_teardown+0x48 1 0% 100% 0.00 730511 ncbuc zfs_lookup+0x720 ------------------------------------------------------------------------------- R/W writer blocked by readers: 3071 events in 300.017 seconds (10 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 1494 49% 49% 0.00 1217432 vmobject vm_map_delete+0xa8 738 24% 73% 0.00 1140591 vmobject vm_fault_trap+0x64 516 17% 89% 0.00 1237720 vmobject vm_mmap_vnode+0x260 127 4% 94% 0.00 1172746 vmobject vm_map_lookup+0x9ac 70 2% 96% 0.00 1227746 vmobject fork1+0x33c 41 1% 97% 0.00 1591195 vmobject vmspace_fork+0x910 30 1% 98% 0.00 883197 vmobject vm_map_fixed+0x134 29 1% 99% 0.00 1871306 vmobject vm_map_process_deferred+0x88 8 0% 99% 0.00 190880 vmobject fault_deallocate+0xa4 6 0% 100% 0.00 852318 vmobject elf64_load_file+0xf0 5 0% 100% 0.00 2041435 vmobject vm_map_entry_delete+0xe0 4 0% 100% 0.00 430438 vmobject vm_map_find+0x514 1 0% 100% 0.00 26489 vmobject zfs_freebsd_getpages+0x1c4 1 0% 100% 0.00 208955 ncbuc cache_lookup+0x94 1 0% 100% 0.00 170378 ncbuc tmpfs_lookup1+0x13c ------------------------------------------------------------------------------- R/W reader blocked by writer: 159 events in 300.017 seconds (1 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 48 30% 30% 0.00 691991 ncbuc vfs_cache_lookup+0xb4 45 28% 58% 0.00 356140 vmobject vm_map_lookup+0x9ac 44 28% 86% 0.00 401202 vmobject vm_fault_trap+0x64 6 4% 90% 0.00 185003 vmobject fault_deallocate+0xa4 4 3% 92% 0.00 526777 vmobject vm_map_process_deferred+0x88 4 3% 95% 0.00 554944 vmobject exec_new_vmspace+0x248 3 2% 97% 0.00 844362 vmobject vmspace_fork+0x308 3 2% 99% 0.00 851926 vmobject elf64_map_insert+0x244 2 1% 100% 0.00 136055 vmobject vm_map_process_deferred+0x78 ------------------------------------------------------------------------------- R/W reader blocked by write wanted: 27 events in 300.017 seconds (0 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 13 48% 48% 0.00 1344970 vmobject vm_map_lookup+0x9ac 11 41% 89% 0.00 863634 vmobject vm_fault_trap+0x64 3 11% 100% 0.00 1904562 vmobject vm_map_process_deferred+0x88 ------------------------------------------------------------------------------- R/W writer spin on writer: 2569758 events in 300.017 seconds (8565 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 833047 32% 32% 0.00 12696 pmap pv list vmspace_exit+0xc0 708826 28% 60% 0.00 12286 pmap pv list pmap_copy+0x340 338298 13% 73% 0.00 11716 pmap pv list exec_new_vmspace+0x1d0 336527 13% 86% 0.00 12584 pmap pv list pmap_enter_quick_locked+0x1c0 253892 10% 96% 0.00 8559 pmap pv list vm_fault+0x1634 36972 1% 98% 0.00 30958 kernel vm object kmem_free+0x28 32591 1% 99% 0.00 42731 kernel vm object kmem_malloc_domainset+0xc0 9462 0% 99% 0.00 4354 vmobject vm_fault_trap+0x64 4979 0% 99% 0.00 11520 pmap pv list pmap_remove+0x198 4222 0% 100% 0.00 4551 vmobject vm_map_lookup+0x9ac 1994 0% 100% 0.00 5234 vmobject vm_map_fixed+0x134 1976 0% 100% 0.00 5078 vmobject vm_mmap_vnode+0x260 1657 0% 100% 0.00 9179 pmap pv list vm_fault+0x364 1369 0% 100% 0.00 11122 pmap pv list vm_pageout_worker+0x11b0 1061 0% 100% 0.00 5332 vmobject vm_map_delete+0xa8 441 0% 100% 0.00 1915 vmobject vm_map_process_deferred+0x88 361 0% 100% 0.00 7112 vmobject elf64_load_sections+0x190 330 0% 100% 0.00 1200 vmobject fork1+0x33c 291 0% 100% 0.00 6716 vmobject vm_map_find+0x514 263 0% 100% 0.00 318 tidhash exit1+0x5e0 172 0% 100% 0.00 735 vmobject vmspace_fork+0x910 160 0% 100% 0.00 296 tidhash fork1+0x984 145 0% 100% 0.00 10358 pmap pv list pmap_enter_l2+0x260 115 0% 100% 0.00 1495 vmobject vm_object_deallocate+0x478 100 0% 100% 0.00 1948 tidhash kern_thr_exit+0x120 88 0% 100% 0.00 7215 vmobject fault_deallocate+0xa4 64 0% 100% 0.00 69414 ncbuc cache_lookup+0x94 55 0% 100% 0.00 16911 pmap pv list vm_page_try_remove_all+0x3c 50 0% 100% 0.00 14356 vmobject zfs_freebsd_getpages+0x1c4 45 0% 100% 0.00 14154 vmobject VOP_GETPAGES_APV+0x4c 45 0% 100% 0.00 6369 vmobject vm_map_entry_delete+0xe0 42 0% 100% 0.00 18679 vmobject vm_pager_get_pages+0x28 36 0% 100% 0.00 2601 vmobject elf64_load_file+0xf0 27 0% 100% 0.00 466 tidhash thread_create+0x3cc 12 0% 100% 0.00 1277 pmap pv list pmap_advise+0x138 12 0% 100% 0.00 47994 ncbuc zfs_lookup+0x720 8 0% 100% 0.00 94624 ncbuc zfs_lookup+0x66c 4 0% 100% 0.00 54633 ncbuc cache_purge_negative+0xfc 4 0% 100% 0.00 54683 ncbuc cache_purge+0xa8 3 0% 100% 0.00 881 vmobject kern_execve+0x238 2 0% 100% 0.00 3522 vmobject vm_fault+0xbdc 2 0% 100% 0.00 1644 pmap pv list pmap_copy+0x3f8 2 0% 100% 0.00 3011 ncbuc tmpfs_lookup1+0x13c 1 0% 100% 0.00 5533 vmobject vmspace_fork+0x92c 1 0% 100% 0.00 21555 pmap pv list pmap_enter_l2+0x168 1 0% 100% 0.00 467 pmap pv list pmap_remove+0x164 1 0% 100% 0.00 1667 tcpinp soreceive_generic+0x460 1 0% 100% 0.00 79578 ncbuc cache_zap_locked_vnode_kl2+0xcc 1 0% 100% 0.00 2444 ncbuc zfsvfs_teardown+0x48 ------------------------------------------------------------------------------- R/W writer spin on readers: 20854 events in 300.017 seconds (70 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 7532 36% 36% 0.00 7530 vmobject vm_fault_trap+0x64 3603 17% 53% 0.00 18495 vmobject vm_map_delete+0xa8 3556 17% 70% 0.00 5672 vmobject fault_deallocate+0xa4 1650 8% 78% 0.00 4440 vmobject vm_map_fixed+0x134 1198 6% 84% 0.00 20991 vmobject vm_mmap_vnode+0x260 1111 5% 89% 0.00 6335 vmobject vm_map_lookup+0x9ac 646 3% 93% 0.00 6266 vmobject fork1+0x33c 363 2% 94% 0.00 4076 vmobject elf64_load_file+0xf0 362 2% 96% 0.00 2291 vmobject elf64_load_sections+0x190 142 1% 97% 0.00 14137 vmobject vmspace_fork+0x910 109 1% 97% 0.00 12462 vmobject vm_map_process_deferred+0x88 94 0% 98% 0.00 10828 vmobject vm_pager_get_pages+0x28 84 0% 98% 0.00 6984 vmobject vm_map_find+0x514 77 0% 98% 0.00 834 ncbuc cache_lookup+0x94 75 0% 99% 0.00 5569 vmobject VOP_GETPAGES_APV+0x4c 52 0% 99% 0.00 1981 vmobject vm_object_deallocate+0x478 40 0% 99% 0.00 4428 vmobject zfs_freebsd_getpages+0x1c4 37 0% 99% 0.00 429 ncbuc zfs_lookup+0x720 29 0% 100% 0.00 416 ncbuc zfs_lookup+0x66c 24 0% 100% 0.00 10929 vmobject vm_map_entry_delete+0xe0 20 0% 100% 0.00 2094 ncbuc tmpfs_lookup1+0x13c 11 0% 100% 0.00 1589 nullhs null_lookup+0x15c 10 0% 100% 0.00 382 ncbuc zfsvfs_teardown+0x48 8 0% 100% 0.00 2024 vmobject kern_execve+0x238 4 0% 100% 0.00 3122 vmobject vm_fault+0xbdc 4 0% 100% 0.00 439 ncbuc cache_purge_negative+0xfc 3 0% 100% 0.00 3252 nullhs null_reclaim+0x28 2 0% 100% 0.00 355 ncbuc cache_zap_locked_vnode_kl2+0xcc 2 0% 100% 0.00 322 ncbuc cache_purge+0xa8 2 0% 100% 0.00 422 ncbuc cache_purge+0x100 1 0% 100% 0.00 200 vmobject vm_object_deallocate+0x1ac 1 0% 100% 0.00 7000 vmobject vmspace_fork+0x92c 1 0% 100% 0.00 3177 nullhs nullfs_mount+0x1b8 1 0% 100% 0.00 445 nullhs null_bypass+0x18c ------------------------------------------------------------------------------- R/W reader spin on writer: 24419 events in 300.017 seconds (81 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 7223 30% 30% 0.00 4685 vmobject fault_deallocate+0xa4 7115 29% 59% 0.00 2007 vmobject vm_fault_trap+0x64 3654 15% 74% 0.00 878 vmobject vm_map_lookup+0x9ac 2346 10% 83% 0.00 70376 ncbuc vfs_cache_lookup+0xb4 1571 6% 90% 0.00 6186 vmobject elf64_map_insert+0x244 1280 5% 95% 0.00 2823 vmobject vm_map_process_deferred+0x88 499 2% 97% 0.00 3628 vmobject vm_map_insert+0x388 327 1% 98% 0.00 3591 vmobject vm_map_process_deferred+0x78 127 1% 99% 0.00 346 vmobject vmspace_fork+0x308 98 0% 99% 0.00 4793 vmobject vm_fault+0x19a0 97 0% 100% 0.00 727 vmobject exec_new_vmspace+0x248 42 0% 100% 0.00 2477 vmobject vmspace_fork+0x27c 25 0% 100% 0.00 576 vmobject kern_execve+0x230 8 0% 100% 0.00 4838 nullhs null_lookup+0x15c 5 0% 100% 0.00 604 vmobject kern_execve+0x770 2 0% 100% 0.00 5511 nullhs nullfs_unlink_lowervp+0x10 ------------------------------------------------------------------------------- R/W reader spin on write wanted: 601 events in 300.017 seconds (2 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 325 54% 54% 0.00 9335 vmobject vm_fault_trap+0x64 253 42% 96% 0.00 7531 vmobject fault_deallocate+0xa4 8 1% 98% 0.00 36836 vmobject vm_map_lookup+0x9ac 6 1% 99% 0.00 13125 vmobject vm_map_process_deferred+0x88 6 1% 100% 0.00 10196 vmobject elf64_map_insert+0x244 1 0% 100% 0.00 10911 vmobject vmspace_fork+0x308 1 0% 100% 0.00 6600 vmobject vm_map_process_deferred+0x78 1 0% 100% 0.00 5800 vmobject exec_new_vmspace+0x248 ------------------------------------------------------------------------------- SX exclusive block: 19638 events in 300.017 seconds (65 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 6866 35% 35% 0.00 127817 dd->dd_lock dsl_dir_diduse_space+0x158 1886 10% 45% 0.00 1488600 os->os_userused_lock taskq_run+0x14 1494 8% 52% 0.00 329351 dd->dd_lock dsl_dir_tempreserve_impl+0x24c 1348 7% 59% 0.00 100087 vq->vq_lock zio_vdev_io_start+0x1ec 862 4% 63% 0.00 54345 dp->dp_lock zio_vdev_io_assess+0x368 824 4% 68% 0.00 246137 dd->dd_lock dmu_tx_commit+0x110 647 3% 71% 0.00 476045 vm map (user) vm_map_find_min+0x64 643 3% 74% 0.00 18244 ms->ms_lock metaslab_alloc+0xb0 567 3% 77% 0.00 206090 vq->vq_lock zio_vdev_io_done+0x12c 542 3% 80% 0.00 289537 dn->dn_struct_rwlock arc_write_ready+0x124 479 2% 82% 0.00 68055 dd->dd_lock dsl_dataset_block_born+0x254 439 2% 85% 0.00 904369 vm map (user) vm_map_lookup+0x8f0 233 1% 86% 0.00 554088 buf_hash_table.ht_locks[i].ht_lock __dbuf_hold_impl+0x64 225 1% 87% 0.00 282304 dp->dp_lock dmu_tx_assign+0x22c 225 1% 88% 0.00 191781 dd->dd_lock dmu_objset_willuse_space+0x30 223 1% 89% 0.00 37717 mg->mg_lock metaslab_alloc+0xb0 191 1% 90% 0.00 40311 spa->spa_alloc_locks[i] zio_done+0x2c0 149 1% 91% 0.00 572534 tx->tx_cpu[c].tc_open_lock dmu_tx_assign+0xfc 147 1% 92% 0.00 148253 ds->ds_lock dbuf_write_done+0x54 132 1% 92% 0.00 110245 vq->vq_lock vdev_queue_io_done+0xb4 104 1% 93% 0.00 985021 as->as_buckets[i].asc_lock arc_get_data_impl+0x1f4 77 0% 93% 0.00 26918 mc->mc_lock zio_done+0x2b4 70 0% 94% 0.00 1526570 spa->spa_alloc_locks[i] zio_execute+0x60 70 0% 94% 0.00 21918 db->db_mtx dsl_dir_diduse_space+0x54 56 0% 94% 0.00 156342 proctree sys_wait4+0x80 50 0% 94% 0.00 282688 proctree sys_sys_exit+0x10 50 0% 95% 0.00 220500 h->hash_mutexes[i] __dbuf_hold_impl+0x38 49 0% 95% 0.00 18289 scl->scl_lock metaslab_alloc+0x114 46 0% 95% 0.00 743657 as->as_buckets[i].asc_lock arc_get_data_impl+0x1e4 45 0% 95% 0.00 257922 buf_hash_table.ht_locks[i].ht_lock arc_write_done+0xbc 45 0% 96% 0.00 85091 allproc sys_fork+0x30 44 0% 96% 0.00 148224 allproc sys_sys_exit+0x10 42 0% 96% 0.00 55292 scl->scl_lock zio_vdev_io_assess+0x108 39 0% 96% 0.00 32540 tm->tm_lock zio_vdev_io_done+0x124 39 0% 96% 0.00 9444 scl->scl_lock zio_vdev_io_start+0x244 38 0% 97% 0.00 66189 tm->tm_lock zio_vdev_io_start+0x220 38 0% 97% 0.00 18199 scl->scl_lock metaslab_alloc+0x4c 38 0% 97% 0.00 210830 dp->dp_lock dbuf_dirty+0x290 36 0% 97% 0.00 38859 vq->vq_lock vdev_queue_io+0x140 35 0% 97% 0.00 225446 vm map (user) kern_mprotect+0xb8 35 0% 98% 0.00 294085 proctree sys_fork+0x30 29 0% 98% 0.00 220378 vm map (user) pipeclose+0x2bc 29 0% 98% 0.00 19911 tm->tm_lock zio_vdev_io_start+0x5c 28 0% 98% 0.00 706775 vm map (user) pipespace_new+0x6c 26 0% 98% 0.00 14373 zio->io_lock zio_done+0x988 22 0% 98% 0.00 772751 tl->tl_lock dsl_dir_willuse_space+0xf4 20 0% 98% 0.00 57079 ds->ds_lock dbuf_write_done+0x44 19 0% 98% 0.00 630155 mls->mls_lock dbuf_rele_and_unlock+0x430 16 0% 99% 0.00 64847 dd->dd_lock dsl_dataset_block_kill+0x1e0 15 0% 99% 0.00 10855 zio->io_lock zio_ready+0x2ac 15 0% 99% 0.00 10182 db->db_mtx dsl_dataset_block_born+0xe4 15 0% 99% 0.00 88863 allproc sys_vfork+0x34 14 0% 99% 0.00 338376 mc->mc_lock zio_dva_throttle+0xf8 14 0% 99% 0.00 495549 buf_hash_table.ht_locks[i].ht_lock arc_freed+0x1c 13 0% 99% 0.00 125692 zio->io_lock zio_create+0x290 13 0% 99% 0.00 2754721 db->db_mtx dbuf_sync_list+0x60 10 0% 99% 0.00 805750 as->as_buckets[i].asc_lock arc_hdr_free_pabd+0x48 8 0% 99% 0.00 104902 dn->dn_mtx dbuf_write_ready+0x70 7 0% 99% 0.00 72952 ms->ms_lock metaslab_free+0x11c 7 0% 99% 0.00 1851977 as->as_buckets[i].asc_lock arc_get_data_impl+0x1d4 6 0% 99% 0.00 49862 proctree sys_vfork+0x34 6 0% 99% 0.00 18270 mc->mc_lock zio_allocate_dispatch+0x88 6 0% 99% 0.00 4921425 h->hash_mutexes[i] arc_write_done+0x334 6 0% 99% 0.00 2337596 dd->dd_lock dsl_dir_space_available+0x40 6 0% 99% 0.00 307237 as->as_buckets[i].asc_lock buf_dest+0x20 5 0% 99% 0.00 149391 db->db_mtx arc_write_done+0x334 4 0% 99% 0.00 13111 scl->scl_lock metaslab_free+0xa8 4 0% 99% 0.00 111689 dp->dp_lock dbuf_new_size+0x1a8 4 0% 99% 0.00 33538 db->db_mtx __dbuf_hold_impl+0x38 4 0% 99% 0.00 242711 db->db_mtx arc_write_ready+0x124 4 0% 100% 0.00 1116483 as->as_buckets[i].asc_lock arc_free_data_impl+0x130 4 0% 100% 0.00 4388711 as->as_buckets[i].asc_lock arc_space_return+0x8c 3 0% 100% 0.00 1167081 zp->z_acl_lock zfs_zaccess+0xd8 3 0% 100% 0.00 243030 tl->tl_lock dsl_dataset_dirty+0x38 3 0% 100% 0.00 173415 mls->mls_lock __dbuf_hold_impl+0x98 3 0% 100% 0.00 3805437 h->hash_mutexes[i] dnode_sync+0x66c 3 0% 100% 0.00 536155 h->hash_mutexes[i] dnode_hold_impl+0xab4 3 0% 100% 0.00 2265985 h->hash_mutexes[i] dmu_read_uio_dnode+0xe8 3 0% 100% 0.00 41044 dp->dp_lock dsl_dataset_block_kill+0x3c0 3 0% 100% 0.00 3260185 dn->dn_struct_rwlock sync_dnodes_task+0x6c 3 0% 100% 0.00 77822 db->db_mtx dsl_dataset_block_kill+0xd8 3 0% 100% 0.00 1265340 buf_hash_table.ht_locks[i].ht_lock dbuf_write+0x94 3 0% 100% 0.00 217807 buf_hash_table.ht_locks[i].ht_lock dbuf_destroy+0x30 3 0% 100% 0.00 5541807 as->as_buckets[i].asc_lock hdr_full_cons+0xac 3 0% 100% 0.00 1499756 as->as_buckets[i].asc_lock arc_free_data_impl+0xf4 3 0% 100% 0.00 407133 as->as_buckets[i].asc_lock arc_space_return+0xcc 2 0% 100% 0.00 10644 zio->io_lock zio_execute+0x60 2 0% 100% 0.00 6789 vd->vdev_stat_lock zio_done+0x554 2 0% 100% 0.00 9166 vc->vc_lock zio_vdev_io_done+0x140 2 0% 100% 0.00 13255 scl->scl_lock zio_dva_free+0x1c 2 0% 100% 0.00 32477 hdl->sa_lock zfs_freebsd_getattr+0x1b0 2 0% 100% 0.00 499900 h->hash_mutexes[i] __dbuf_hold_impl+0x5fc 2 0% 100% 0.00 2979511 db->db_mtx dnode_sync+0xe8 2 0% 100% 0.00 28455 db->db_mtx dmu_read_uio_dnode+0x40 2 0% 100% 0.00 124600 as->as_buckets[i].asc_lock dbuf_create+0x534 2 0% 100% 0.00 61400 as->as_buckets[i].asc_lock dbuf_destroy+0x70 2 0% 100% 0.00 36822 as->as_buckets[i].asc_lock buf_dest+0x5c 2 0% 100% 0.00 89022 as->as_buckets[i].asc_lock buf_cons+0x58 2 0% 100% 0.00 865311 as->as_buckets[i].asc_lock hdr_full_cons+0x9c 1 0% 100% 0.00 8955 zio->io_lock zio_done+0x99c 1 0% 100% 0.00 23655134 zfsvfs->z_hold_mtx[i] zfs_dirent_lookup+0x160 1 0% 100% 0.00 109800 tx->tx_cpu[c].tc_lock dmu_tx_commit+0x160 1 0% 100% 0.00 229112 tl->tl_lock dsl_pool_sync+0xcc 1 0% 100% 0.00 19311 rl->rl_lock zfs_freebsd_read+0x5e4 1 0% 100% 0.00 230778 mls->mls_lock dbuf_evict_one+0x30 1 0% 100% 0.00 869178 mls->mls_lock arc_change_state+0x3fc 1 0% 100% 0.00 383377 ds->ds_lock dsl_dir_tempreserve_impl+0xe4 1 0% 100% 0.00 14067 h->hash_mutexes[i] dbuf_evict_one+0x134 1 0% 100% 0.00 166111 dd->dd_lock dsl_dir_tempreserve_space+0xac 1 0% 100% 0.00 731534 db->db_mtx dnode_sync+0x66c 1 0% 100% 0.00 241555 dn->dn_mtx taskq_run+0x14 1 0% 100% 0.00 66244 dp->dp_lock dbuf_write_done+0x54 1 0% 100% 0.00 46377 as->as_buckets[i].asc_lock dbuf_sync_leaf+0x224 1 0% 100% 0.00 265645 as->as_lock arc_buf_destroy_impl+0xc0 1 0% 100% 0.00 3001889 as->as_lock arc_hdr_free_pabd+0x48 1 0% 100% 0.00 23133 as->as_lock arc_space_return+0x8c 1 0% 100% 0.00 542711 as->as_lock arc_get_data_impl+0x1f4 1 0% 100% 0.00 61223 as->as_lock buf_cons+0x38 1 0% 100% 0.00 6644 bpl->bpl_lock dsl_dataset_block_kill+0x3a0 1 0% 100% 0.00 5845 as->as_buckets[i].asc_lock arc_buf_destroy_impl+0xc0 1 0% 100% 0.00 237178 as->as_buckets[i].asc_lock arc_kstat_update+0xc4 1 0% 100% 0.00 133089 as->as_buckets[i].asc_lock buf_cons+0x38 ------------------------------------------------------------------------------- SX shared block: 1477 events in 300.017 seconds (5 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 1472 100% 100% 0.00 1491916 vm map (user) vm_fault+0xa4 2 0% 100% 0.00 62100 zf->zf_rwlock dbuf_read+0x1fc 2 0% 100% 0.00 651999 vm map (user) vm_fault+0x13c 1 0% 100% 0.00 34356 dn->dn_struct_rwlock dbuf_sync_leaf+0x124 ------------------------------------------------------------------------------- SX exclusive spin: 2975271 events in 300.017 seconds (9917 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 356170 12% 12% 0.00 14254 ms->ms_lock metaslab_alloc+0xb0 276264 9% 21% 0.00 26290 vq->vq_lock zio_vdev_io_start+0x1ec 224316 8% 29% 0.00 34132 tm->tm_lock zio_vdev_io_start+0x5c 214059 7% 36% 0.00 5822 zio->io_lock zio_ready+0x2ac 199437 7% 43% 0.00 6149 zio->io_lock zio_done+0x988 171369 6% 48% 0.00 4334 zio->io_lock zio_create+0x290 142210 5% 53% 0.00 3195 dd->dd_lock dmu_tx_commit+0x110 136277 5% 58% 0.00 8538 spa->spa_alloc_locks[i] zio_done+0x2c0 126684 4% 62% 0.00 7884 mg->mg_lock metaslab_alloc+0xb0 101268 3% 65% 0.00 1584 buf_hash_table.ht_locks[i].ht_lock __dbuf_hold_impl+0x64 91017 3% 69% 0.00 3504 dd->dd_lock dsl_dir_tempreserve_impl+0x24c 81696 3% 71% 0.00 9823 zio->io_lock zio_done+0x99c 81551 3% 74% 0.00 3606 spa->spa_alloc_locks[i] zio_execute+0x60 71518 2% 76% 0.00 31271 vq->vq_lock zio_vdev_io_done+0x12c 60505 2% 78% 0.00 3935 mc->mc_lock zio_done+0x2b4 49379 2% 80% 0.00 5394 scl->scl_lock metaslab_alloc+0x4c 44988 2% 82% 0.00 98969 os->os_userused_lock taskq_run+0x14 44758 2% 83% 0.00 12446 scl->scl_lock zio_vdev_io_assess+0x108 43553 1% 85% 0.00 27952 scl->scl_lock zio_vdev_io_start+0x244 42891 1% 86% 0.00 4925 scl->scl_lock metaslab_alloc+0x114 40967 1% 87% 0.00 10362 dd->dd_lock dsl_dir_diduse_space+0x158 35230 1% 89% 0.00 6001 dp->dp_lock zio_vdev_io_assess+0x368 30524 1% 90% 0.00 4222 mc->mc_lock zio_allocate_dispatch+0x88 28672 1% 91% 0.00 43731 vq->vq_lock vdev_queue_io_done+0xb4 27172 1% 92% 0.00 4273 vm map (user) vm_map_find_min+0x64 24814 1% 92% 0.00 3816 dp->dp_lock dmu_tx_assign+0x22c 24811 1% 93% 0.00 1559 mc->mc_lock zio_dva_throttle+0xf8 23260 1% 94% 0.00 3035 dd->dd_lock dmu_objset_willuse_space+0x30 17244 1% 95% 0.00 18251 scl->scl_lock zio_dva_free+0x1c 17128 1% 95% 0.00 31274 scl->scl_lock metaslab_free+0xa8 13670 0% 96% 0.00 47368 vq->vq_lock vdev_queue_io+0x140 12302 0% 96% 0.00 1576 h->hash_mutexes[i] __dbuf_hold_impl+0x38 10740 0% 96% 0.00 5966 ds->ds_lock dbuf_write_done+0x44 10381 0% 97% 0.00 6391 vm map (user) vm_map_lookup+0x8f0 10250 0% 97% 0.00 3429 db->db_mtx dsl_dir_diduse_space+0x54 8150 0% 97% 0.00 2308 db->db_mtx dsl_dataset_block_born+0xe4 7611 0% 98% 0.00 5076 ds->ds_lock dbuf_write_done+0x54 5896 0% 98% 0.00 1711 ms->ms_lock metaslab_free+0x11c 5756 0% 98% 0.00 5242 dd->dd_lock dsl_dataset_block_born+0x254 5090 0% 98% 0.00 4386 tl->tl_lock dsl_dir_willuse_space+0xf4 5016 0% 98% 0.00 6066 vm map (user) kern_mprotect+0xb8 4262 0% 98% 0.00 5981 tm->tm_lock zio_vdev_io_start+0x220 4226 0% 99% 0.00 8112 tm->tm_lock zio_vdev_io_done+0x124 4012 0% 99% 0.00 1308 proctree sys_wait4+0x80 3582 0% 99% 0.00 19630 dn->dn_struct_rwlock arc_write_ready+0x124 3141 0% 99% 0.00 3493 h->hash_mutexes[i] __dbuf_hold_impl+0x5fc 2796 0% 99% 0.00 4753 dp->dp_lock dbuf_dirty+0x290 2382 0% 99% 0.00 3908 db->db_mtx dsl_dataset_block_kill+0xd8 2050 0% 99% 0.00 1233 proctree sys_sys_exit+0x10 1605 0% 99% 0.00 1464 proctree sys_fork+0x30 1506 0% 99% 0.00 1550 h->hash_mutexes[i] dnode_hold_impl+0xab4 1213 0% 99% 0.00 1478 bpl->bpl_lock dsl_dataset_block_kill+0x104 1163 0% 99% 0.00 319 tl->tl_lock dsl_dataset_dirty+0x38 1150 0% 99% 0.00 905 zp->z_acl_lock zfs_zaccess+0xd8 934 0% 99% 0.00 2762 mls->mls_lock dbuf_rele_and_unlock+0x430 912 0% 99% 0.00 3038 vm map (user) pipespace_new+0x6c 824 0% 99% 0.00 2958 zio->io_lock zio_execute+0x60 773 0% 100% 0.00 6620 dd->dd_lock dsl_dataset_block_kill+0x1e0 771 0% 100% 0.00 3167 vm map (user) pipeclose+0x2bc 770 0% 100% 0.00 4025 dd->dd_lock dsl_dir_space_available+0x40 768 0% 100% 0.00 1097 ds->ds_lock dsl_dir_tempreserve_impl+0xe4 664 0% 100% 0.00 506 dd->dd_lock dsl_dir_tempreserve_space+0xac 642 0% 100% 0.00 2917 bpl->bpl_lock dsl_dataset_block_kill+0x3a0 487 0% 100% 0.00 4470 dp->dp_lock dsl_dataset_block_kill+0x3c0 473 0% 100% 0.00 1372 proctree sys_vfork+0x34 449 0% 100% 0.00 1364 dn->dn_mtx dbuf_write_ready+0x70 425 0% 100% 0.00 2336 rrl->rr_lock zfs_freebsd_access+0x4c 418 0% 100% 0.00 1584 dp->dp_lock dbuf_new_size+0x1a8 410 0% 100% 0.00 446 vd->vdev_stat_lock zio_done+0x554 390 0% 100% 0.00 3398 rrl->rr_lock zfs_freebsd_access+0x9c 370 0% 100% 0.00 2758 dn->dn_mtx arc_write_ready+0x124 349 0% 100% 0.00 4548 dp->dp_lock dbuf_write_done+0x54 343 0% 100% 0.00 4287 db->db_mtx dmu_buf_hold_array_by_dnode+0x250 339 0% 100% 0.00 2268 db->db_mtx __dbuf_hold_impl+0x38 310 0% 100% 0.00 11898 buf_hash_table.ht_locks[i].ht_lock dbuf_destroy+0x30 304 0% 100% 0.00 539 mls->mls_lock __dbuf_hold_impl+0x98 282 0% 100% 0.00 5741 buf_hash_table.ht_locks[i].ht_lock arc_write_done+0xbc 215 0% 100% 0.00 3402 db->db_mtx dmu_read_uio_dnode+0x40 214 0% 100% 0.00 2603 h->hash_mutexes[i] dmu_tx_check_ioerr+0xb4 180 0% 100% 0.00 8272 buf_hash_table.ht_locks[i].ht_lock arc_freed+0x1c 169 0% 100% 0.00 8092 as->as_lock arc_space_return+0x8c 161 0% 100% 0.00 11370 allproc sys_sys_exit+0x10 155 0% 100% 0.00 2544 db->db_mtx dmu_read_uio_dnode+0xe8 144 0% 100% 0.00 2666 rrl->rr_lock zfs_freebsd_getattr+0x6b8 144 0% 100% 0.00 1981 rrl->rr_lock zfs_freebsd_getattr+0x8c 142 0% 100% 0.00 3467 db->db_mtx dmu_buf_will_dirty+0x1d8 129 0% 100% 0.00 13006 allproc sys_fork+0x30 116 0% 100% 0.00 294 h->hash_mutexes[i] dnode_evict_dbufs+0xa8 114 0% 100% 0.00 2979 tl->tl_lock vdev_dirty+0x30 106 0% 100% 0.00 300 mls->mls_lock dbuf_destroy+0xec 104 0% 100% 0.00 4598 ds->ds_lock free_blocks+0x7c 103 0% 100% 0.00 5033 h->hash_mutexes[i] dmu_read_uio_dnode+0xe8 87 0% 100% 0.00 67267 as->as_lock arc_get_data_impl+0x1f4 87 0% 100% 0.00 895 as->as_lock arc_space_consume+0x88 85 0% 100% 0.00 631 hdl->sa_lock zfs_freebsd_getattr+0x1b0 82 0% 100% 0.00 1986 dp->dp_lock dbuf_free_range+0x1cc 76 0% 100% 0.00 1027 db->db_mtx arc_read_done+0x318 66 0% 100% 0.00 1072 as->as_lock arc_free_data_impl+0x130 65 0% 100% 0.00 1552 dd->dd_lock dsl_dir_name+0x28 62 0% 100% 0.00 7057 buf_hash_table.ht_locks[i].ht_lock arc_read+0x7c 57 0% 100% 0.00 3423 dn->dn_mtx dnode_setdirty+0xb8 55 0% 100% 0.00 4943 db->db_mtx dnode_hold_impl+0x1a0 54 0% 100% 0.00 2478 rrl->rr_lock zfs_freebsd_open+0x38 54 0% 100% 0.00 857 db->db_mtx userquota_updates_task+0x1c4 54 0% 100% 0.00 1444 as->as_lock hdr_full_cons+0x9c 53 0% 100% 0.00 2421 db->db_mtx dnode_hold_impl+0xea4 52 0% 100% 0.00 4667 as->as_lock arc_buf_destroy_impl+0xc0 51 0% 100% 0.00 2477 dp->dp_lock zfs_rmnode+0x60 50 0% 100% 0.00 1436 as->as_lock buf_cons+0x38 46 0% 100% 0.00 3405 proctree sys_procctl+0x10c 46 0% 100% 0.00 485 as->as_lock buf_dest+0x20 46 0% 100% 0.00 120571 as->as_lock arc_hdr_free_pabd+0x48 44 0% 100% 0.00 558 zilog->zl_itxg[i].itxg_lock zfs_log_write+0xf8 44 0% 100% 0.00 1765 as->as_lock arc_get_data_impl+0x1e4 43 0% 100% 0.00 1205 as->as_lock arc_get_data_impl+0x1d4 42 0% 100% 0.00 10592 tx->tx_sync_lock txg_wait_synced+0xc 42 0% 100% 0.00 4978 tx->tx_cpu[c].tc_open_lock dmu_tx_assign+0xfc 42 0% 100% 0.00 1500 h->hash_mutexes[i] dbuf_prefetch+0x90 41 0% 100% 0.00 12942 allproc sys_vfork+0x34 40 0% 100% 0.00 3313 as->as_lock dbuf_create+0x534 39 0% 100% 0.00 595 rl->rl_lock zfs_freebsd_read+0xd8 34 0% 100% 0.00 239 dn->dn_mtx dnode_sync+0x85c 33 0% 100% 0.00 949 rrl->rr_lock zfs_freebsd_close+0x30 33 0% 100% 0.00 1336 as->as_lock hdr_full_dest+0x34 32 0% 100% 0.00 882 rl->rl_lock zfs_freebsd_read+0x5e4 31 0% 100% 0.00 1392 dn->dn_dbufs_mtx userquota_updates_task+0x1c4 31 0% 100% 0.00 859 as->as_lock arc_space_return+0xcc 29 0% 100% 0.00 2937 tx->tx_sync_lock fork_exit+0x90 29 0% 100% 0.00 1348 rrl->rr_lock zfs_freebsd_read+0x40 29 0% 100% 0.00 2842 buf_hash_table.ht_locks[i].ht_lock dbuf_dirty+0x300 28 0% 100% 0.00 14546 db->db_mtx dbuf_sync_list+0x60 28 0% 100% 0.00 5952 db->db_mtx dbuf_dirty+0x768 27 0% 100% 0.00 1689 as->as_lock arc_free_data_impl+0xf4 25 0% 100% 0.00 2893 dn->dn_dbufs_mtx __dbuf_hold_impl+0x5fc 25 0% 100% 0.00 13263 buf_hash_table.ht_locks[i].ht_lock dbuf_write+0x94 25 0% 100% 0.00 3364 allprison do_el0_sync+0x550 22 0% 100% 0.00 9374 dn->dn_mtx dbuf_dirty+0x768 21 0% 100% 0.00 1799 tx->tx_cpu[c].tc_open_lock txg_quiesce_thread+0xfc 21 0% 100% 0.00 50663 dp->dp_lock zfs_freesp+0x184 20 0% 100% 0.00 9210 zio->io_lock spa_sync+0x984 20 0% 100% 0.00 755 dd->dd_lock dsl_pool_sync+0x288 20 0% 100% 0.00 552 db->db_mtx __dbuf_hold_impl+0x5fc 20 0% 100% 0.00 895 as->as_lock dnode_destroy+0x180 19 0% 100% 0.00 2645 zio->io_lock spa_sync+0xc18 19 0% 100% 0.00 1656397 os->os_lock dnode_buf_evict_async+0xa4 18 0% 100% 0.00 271 h->hash_mutexes[i] zap_lookup_norm+0x124 18 0% 100% 0.00 2242 as->as_lock arc_kstat_update+0x16c 18 0% 100% 0.00 3187 as->as_lock arc_space_consume+0x98 17 0% 100% 0.00 320 mls->mls_lock dbuf_evict_one+0x30 17 0% 100% 0.00 230 mls->mls_lock arc_buf_destroy+0x19c 17 0% 100% 0.00 3393 as->as_buckets[i].asc_lock arc_buf_destroy_impl+0xc0 16 0% 100% 0.00 56597 vc->vc_lock zio_vdev_io_done+0x140 16 0% 100% 0.00 8008 as->as_lock buf_dest+0x5c 15 0% 100% 0.00 2281 dn->dn_mtx dmu_buf_hold_noread+0xe4 15 0% 100% 0.00 4699 as->as_lock dbuf_destroy+0x42c 15 0% 100% 0.00 72306 as->as_lock hdr_full_cons+0xbc 14 0% 100% 0.00 361 dd->dd_lock dsl_prop_get_all_ds+0xe4 14 0% 100% 0.00 4231 as->as_lock uma_zfree_arg+0x44 13 0% 100% 0.00 2603 rrl->rr_lock zfs_freebsd_read+0x620 13 0% 100% 0.00 1647 db->db_mtx dbuf_hold+0x58 13 0% 100% 0.00 2558 db->db_mtx arc_write_done+0x334 12 0% 100% 0.00 1427 tm->tm_lock trim_map_commit_done+0x48 12 0% 100% 0.00 1266 rrl->rr_lock zfs_freebsd_close+0x84 12 0% 100% 0.00 457 h->hash_mutexes[i] dbuf_evict_one+0x134 12 0% 100% 0.00 835 dn->dn_dbufs_mtx dbuf_rele_and_unlock+0x368 12 0% 100% 0.00 4307 db->db_mtx arc_write_ready+0x124 11 0% 100% 0.00 5074 tl->tl_lock dsl_pool_sync+0x27c 11 0% 100% 0.00 1283 rrl->rr_lock zfs_freebsd_open+0xa4 11 0% 100% 0.00 3304 proctree exit1+0x5ac 10 0% 100% 0.00 1937 as->as_lock buf_cons+0x48 9 0% 100% 0.00 879 tm->tm_lock trim_map_commit+0x8c 9 0% 100% 0.00 508 rrl->rr_lock zfs_freebsd_write+0x54 9 0% 100% 0.00 1869 buf_hash_table.ht_locks[i].ht_lock dbuf_write_done+0x104 9 0% 100% 0.00 530 bpl->bpl_lock dsl_dataset_block_kill+0x180 8 0% 100% 0.00 277 dn->dn_mtx dmu_buf_hold_noread+0x2c 8 0% 100% 0.00 5677 as->as_lock buf_cons+0x58 8 0% 100% 0.00 3903 as->as_lock hdr_full_cons+0xac 7 0% 100% 0.00 854 proctree do_el0_sync+0x550 7 0% 100% 0.00 3555 db->db_mtx zap_lookup_norm+0x124 7 0% 100% 0.00 2492 db->db_mtx dbuf_sync_list+0xd0 7 0% 100% 0.00 6053 buf_hash_table.ht_locks[i].ht_lock traverse_visitbp+0x790 7 0% 100% 0.00 2165 as->as_lock dbuf_destroy+0x70 7 0% 100% 0.00 7063 as->as_buckets[i].asc_lock arc_free_data_impl+0x130 7 0% 100% 0.00 4971 as->as_buckets[i].asc_lock uma_zfree_arg+0x44 6 0% 100% 0.00 3748 zio->io_lock dsl_scan_sync+0x390 6 0% 100% 0.00 9503 vm map (user) pipespace_new+0x188 6 0% 100% 0.00 2136 rrl->rr_lock zfs_lookup+0x80 6 0% 100% 0.00 192 mls->mls_lock arc_change_state+0x3fc 6 0% 100% 0.00 2018 dn->dn_mtx dmu_tx_destroy+0x60 6 0% 100% 0.00 474 dn->dn_mtx arc_write_done+0x334 6 0% 100% 0.00 7477 as->as_buckets[i].asc_lock dbuf_destroy+0x42c 6 0% 100% 0.00 3789 as->as_buckets[i].asc_lock arc_free_data_impl+0xf4 6 0% 100% 0.00 7629 as->as_buckets[i].asc_lock arc_space_return+0xcc 5 0% 100% 0.00 1906 tx->tx_cpu[c].tc_lock dmu_tx_commit+0x160 5 0% 100% 0.00 382 rrl->rr_lock zfs_freebsd_write+0xadc 5 0% 100% 0.00 5000 dn->dn_mtx dnode_sync+0xe8 5 0% 100% 0.00 284 db->db_mtx zap_cursor_fini+0x8c 5 0% 100% 0.00 3386 db->db_mtx zap_add_impl+0x23c 5 0% 100% 0.00 1822 as->as_lock hdr_full_dest+0x70 5 0% 100% 0.00 515 as->as_buckets[i].asc_lock hdr_full_cons+0x9c 5 0% 100% 0.00 4378 as->as_buckets[i].asc_lock arc_get_data_impl+0x1f4 4 0% 100% 0.00 722 zio->io_lock dsl_pool_sync+0x30c 4 0% 100% 0.00 3178 vm map (user) kern_munmap+0x84 4 0% 100% 0.00 9961 h->hash_mutexes[i] arc_write_done+0x334 4 0% 100% 0.00 24455 dn->dn_struct_rwlock sync_dnodes_task+0x6c 4 0% 100% 0.00 233 dn->dn_mtx dmu_tx_hold_sa+0xa8 4 0% 100% 0.00 1527 dn->dn_mtx dmu_tx_hold_zap+0x80 4 0% 100% 0.00 4261 dn->dn_mtx sync_dnodes_task+0x6c 4 0% 100% 0.00 449 dn->dn_dbufs_mtx arc_write_done+0x334 4 0% 100% 0.00 3028 dd->dd_lock dsl_dataset_space+0x48 4 0% 100% 0.00 222 db->db_mtx dnode_sync+0x85c 4 0% 100% 0.00 6027 as->as_buckets[i].asc_lock dbuf_create+0x534 4 0% 100% 0.00 1644 as->as_buckets[i].asc_lock buf_dest+0x5c 4 0% 100% 0.00 2822 as->as_buckets[i].asc_lock hdr_full_cons+0xbc 4 0% 100% 0.00 2827 as->as_buckets[i].asc_lock arc_space_return+0x8c 4 0% 100% 0.00 239 as->as_buckets[i].asc_lock arc_space_consume+0x98 3 0% 100% 0.00 237 tx->tx_cpu[c].tc_lock dmu_tx_assign+0xfc 3 0% 100% 0.00 237 tl->tl_lock dsl_pool_sync+0xcc 3 0% 100% 0.00 2022 tl->tl_lock dsl_pool_sync+0x9c 3 0% 100% 0.00 926 rrl->rr_lock zfs_freebsd_readdir+0x54 3 0% 100% 0.00 2496 rrl->rr_lock zfs_statfs+0x108 3 0% 100% 0.00 9600 os->os_lock dsl_dataset_clone_swap_sync_impl+0x108 3 0% 100% 0.00 370 os->os_lock dmu_objset_evict+0x38 3 0% 100% 0.00 259 ds->ds_lock dmu_objset_write_done+0x34 3 0% 100% 0.00 1081 devfsmount devfs_setattr+0xb4 3 0% 100% 0.00 1385 devfsmount devfs_getattr+0x1c 3 0% 100% 0.00 1088 db->db_mtx dsl_dir_hold+0x190 3 0% 100% 0.00 229 db->db_mtx dmu_bonus_hold+0x1cc 3 0% 100% 0.00 3837 db->db_mtx dmu_buf_hold+0x44 3 0% 100% 0.00 3459 as->as_lock dnode_create+0x190 3 0% 100% 0.00 5666 as->as_buckets[i].asc_lock hdr_full_dest+0x70 3 0% 100% 0.00 6289 as->as_buckets[i].asc_lock arc_get_data_impl+0x1d4 2 0% 100% 0.00 5610 zilog->zl_itxg[i].itxg_lock zfs_freebsd_remove+0x290 2 0% 100% 0.00 611 zilog->zl_itxg[i].itxg_lock zfs_freebsd_setattr+0x1240 2 0% 100% 0.00 30900 vq->vq_lock zio_change_priority+0x3c 2 0% 100% 0.00 278 rrl->rr_lock zfs_freebsd_getpages+0x48 2 0% 100% 0.00 1045 rrl->rr_lock zfs_statfs+0x3c 2 0% 100% 0.00 355 h->hash_mutexes[i] zap_cursor_fini+0x8c 2 0% 100% 0.00 266 dp->dp_lock spa_sync+0x938 2 0% 100% 0.00 2899 dn->dn_mtx dmu_tx_hold_sa+0x60 2 0% 100% 0.00 378 dn->dn_mtx dmu_tx_hold_zap+0x38 2 0% 100% 0.00 267 dn->dn_mtx sync_dnodes_task+0x54 2 0% 100% 0.00 2844 db->db_mtx zap_put_leaf_maybe_grow_ptrtbl+0x128 2 0% 100% 0.00 2022 db->db_mtx sa_attr_op+0x37c 2 0% 100% 0.00 14100 db->db_mtx dmu_tx_check_ioerr+0xa0 2 0% 100% 0.00 1611 db->db_mtx dmu_read_pages+0x8c 2 0% 100% 0.00 17633 db->db_mtx dmu_write_uio_dnode+0xb4 2 0% 100% 0.00 5389 db->db_mtx __dbuf_hold_impl+0x4cc 2 0% 100% 0.00 11578 buf_hash_table.ht_locks[i].ht_lock dmu_objset_sync+0xbc 2 0% 100% 0.00 2111 as->as_lock dbuf_create_bonus+0x94 2 0% 100% 0.00 2822 as->as_buckets[i].asc_lock buf_cons+0x58 2 0% 100% 0.00 333 as->as_buckets[i].asc_lock hdr_full_cons+0xac 2 0% 100% 0.00 333 as->as_buckets[i].asc_lock arc_kstat_update+0x134 1 0% 100% 0.00 556 zio->io_lock vdev_uberblock_sync_list+0x80 1 0% 100% 0.00 2889 zio->io_lock dsl_pool_sync+0x228 1 0% 100% 0.00 289 zilog->zl_itxg[i].itxg_lock zfs_freebsd_rmdir+0x20c 1 0% 100% 0.00 378 zilog->zl_itxg[i].itxg_lock zfs_freebsd_create+0x510 1 0% 100% 0.00 1622 zfsvfs->z_hold_mtx[i] zfs_dirent_lookup+0x160 1 0% 100% 0.00 444 tx->tx_cpu[c].tc_lock txg_quiesce_thread+0xfc 1 0% 100% 0.00 2800 tl->tl_lock metaslab_alloc_dva+0xa24 1 0% 100% 0.00 5534 tl->tl_lock dsl_pool_sync+0x1e8 1 0% 100% 0.00 267 tl->tl_lock dsl_pool_sync+0x218 1 0% 100% 0.00 222 rrl->rr_lock zfs_freebsd_getpages+0x20c 1 0% 100% 0.00 400 rrl->rr_lock zfs_freebsd_readdir+0x5b0 1 0% 100% 0.00 645 rrl->rr_lock zfs_lookup+0x3b4 1 0% 100% 0.00 3134 rrl->rr_lock zfs_freebsd_setattr+0x430 1 0% 100% 0.00 200 h->hash_mutexes[i] dnode_sync+0x66c 1 0% 100% 0.00 245 h->hash_mutexes[i] userquota_updates_task+0x1c4 1 0% 100% 0.00 311 ds->ds_lock dmu_objset_write_done+0x44 1 0% 100% 0.00 266 h->hash_mutexes[i] dbuf_free_range+0xdc 1 0% 100% 0.00 222 h->hash_mutexes[i] dbuf_rele_and_unlock+0x368 1 0% 100% 0.00 133 hdl->sa_lock zfs_freebsd_readdir+0xc8 1 0% 100% 0.00 11089 dn->dn_struct_rwlock dbuf_dirty+0x588 1 0% 100% 0.00 2822 dn->dn_mtx dmu_tx_hold_write+0x84 1 0% 100% 0.00 2866 dn->dn_mtx zfs_freebsd_write+0x3e8 1 0% 100% 0.00 311 dn->dn_mtx dbuf_read+0x494 1 0% 100% 0.00 178 dn->dn_mtx dmu_objset_sync+0x144 1 0% 100% 0.00 5578 dn->dn_mtx dmu_bonus_hold+0x1bc 1 0% 100% 0.00 222 dn->dn_mtx userquota_updates_task+0x1c4 1 0% 100% 0.00 355 db->db_mtx dnode_sync+0x3d8 1 0% 100% 0.00 2867 db->db_mtx dnode_set_blksz+0x30c 1 0% 100% 0.00 4777 db->db_mtx dnode_set_blksz+0x288 1 0% 100% 0.00 1111 db->db_mtx dnode_setdirty+0xb8 1 0% 100% 0.00 5622 db->db_mtx dmu_write_uio_dnode+0x138 1 0% 100% 0.00 21000 db->db_mtx dmu_tx_check_ioerr+0xb4 1 0% 100% 0.00 2822 db->db_mtx dbuf_sync_leaf+0x124 1 0% 100% 0.00 5978 db->db_mtx zap_lookup_norm_by_dnode+0x12c 1 0% 100% 0.00 1556 db->db_mtx zap_get_leaf_byblk+0x1e4 1 0% 100% 0.00 5912 db->db_mtx fzap_lookup+0x2f0 1 0% 100% 0.00 10689 db->db_mtx zap_remove_norm+0xfc 1 0% 100% 0.00 2934 as->as_buckets[i].asc_lock buf_cons+0x48 1 0% 100% 0.00 10467 as->as_buckets[i].asc_lock dbuf_destroy+0x70 1 0% 100% 0.00 5533 as->as_buckets[i].asc_lock buf_dest+0x20 1 0% 100% 0.00 1756 as->as_buckets[i].asc_lock dnode_create+0x190 1 0% 100% 0.00 289 as->as_buckets[i].asc_lock arc_kstat_update+0xc4 1 0% 100% 0.00 267 as->as_buckets[i].asc_lock arc_hdr_free_pabd+0x48 1 0% 100% 0.00 200 as->as_buckets[i].asc_lock arc_space_consume+0x88 1 0% 100% 0.00 311 as->as_buckets[i].asc_lock arc_kstat_update+0x154 1 0% 100% 0.00 245 as->as_buckets[i].asc_lock arc_kstat_update+0x148 1 0% 100% 0.00 244 as->as_buckets[i].asc_lock arc_kstat_update+0xdc 1 0% 100% 0.00 289 as->as_buckets[i].asc_lock arc_kstat_update+0x120 1 0% 100% 0.00 245 as->as_buckets[i].asc_lock arc_get_data_impl+0x1e4 1 0% 100% 0.00 10712 as->as_lock dbuf_sync_leaf+0x224 1 0% 100% 0.00 223 as->as_lock dbuf_read+0x3e8 1 0% 100% 0.00 5555 buf->b_evict_lock __dbuf_hold_impl+0x64 1 0% 100% 0.00 289 buf->b_evict_lock dbuf_write+0x94 1 0% 100% 0.00 1244 db->db_mtx arc_read+0xc78 1 0% 100% 0.00 333 db->db_mtx dbuf_hold_level+0x5c ------------------------------------------------------------------------------- SX shared spin: 43651 events in 300.017 seconds (145 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 41203 94% 94% 0.00 5710 vm map (user) vm_fault+0xa4 1240 3% 97% 0.00 10866 dn->dn_struct_rwlock dnode_hold_impl+0x1a0 235 1% 98% 0.00 5351 dn->dn_struct_rwlock dmu_tx_hold_zap+0x38 211 0% 98% 0.00 3239 dn->dn_struct_rwlock dmu_buf_hold_noread+0x2c 164 0% 99% 0.00 2520 dn->dn_struct_rwlock dmu_tx_hold_sa+0x60 161 0% 99% 0.00 7239 dn->dn_struct_rwlock dmu_tx_hold_write+0x38 153 0% 99% 0.00 3499 dn->dn_struct_rwlock dmu_bonus_hold+0x20 101 0% 100% 0.00 416 zf->zf_rwlock dbuf_read+0x1fc 38 0% 100% 0.00 1546 dn->dn_struct_rwlock dmu_object_set_blocksize+0x2c 37 0% 100% 0.00 3551 proctree sys_procctl+0x10c 22 0% 100% 0.00 1838 dn->dn_struct_rwlock dmu_tx_hold_sa+0x2f4 14 0% 100% 0.00 2991 dn->dn_struct_rwlock dbuf_sync_leaf+0x124 12 0% 100% 0.00 1225 dn->dn_struct_rwlock dmu_object_alloc_impl+0x228 11 0% 100% 0.00 12820 vm map (user) vm_fault+0x13c 6 0% 100% 0.00 2155 proctree ttydev_ioctl+0x248 5 0% 100% 0.00 1449 zap->zap_rwlock zap_lookup_norm+0x74 5 0% 100% 0.00 3871 dn->dn_struct_rwlock dmu_buf_hold+0x44 5 0% 100% 0.00 27271 dn->dn_struct_rwlock dmu_buf_hold+0x1c 4 0% 100% 0.00 9805 dn->dn_struct_rwlock dmu_tx_check_ioerr+0xa0 4 0% 100% 0.00 63172 dn->dn_struct_rwlock dmu_write_uio_dnode+0xb4 4 0% 100% 0.00 22661 dn->dn_struct_rwlock dmu_buf_hold_by_dnode+0x1c 3 0% 100% 0.00 181666 dn->dn_struct_rwlock dmu_write_uio_dnode+0x48 2 0% 100% 0.00 165866 dn->dn_struct_rwlock dmu_tx_count_write+0xdc 2 0% 100% 0.00 153533 dn->dn_struct_rwlock dmu_buf_hold_by_dnode+0x44 1 0% 100% 0.00 200 zf->zf_rwlock dmu_buf_hold_array_by_dnode+0x3c4 1 0% 100% 0.00 21645 vm map (user) umtx_key_get+0x58 1 0% 100% 0.00 1222 dn->dn_struct_rwlock zap_get_leaf_byblk+0x1e4 1 0% 100% 0.00 5844 dn->dn_struct_rwlock zfs_freebsd_readdir+0x408 1 0% 100% 0.00 5666 dn->dn_struct_rwlock dnode_setdirty+0xb8 1 0% 100% 0.00 288 dn->dn_struct_rwlock dmu_tx_hold_zap+0x108 1 0% 100% 0.00 1689 dn->dn_struct_rwlock dmu_object_free+0x28 1 0% 100% 0.00 81266 dn->dn_struct_rwlock dbuf_dirty+0x768 1 0% 100% 0.00 2844 dn->dn_struct_rwlock dmu_buf_will_dirty+0x1d8 ------------------------------------------------------------------------------- lockmgr writer blocked by writer: 16 events in 300.017 seconds (0 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 6 38% 38% 0.00 11237 devfs _vn_lock+0x48 4 25% 62% 0.00 341983 zfs _vn_lock+0x48 3 19% 81% 0.00 15918 devfs vn_truncate+0x110 2 12% 94% 0.00 24967 devfs VOP_LOCK1_APV+0x4c 1 6% 100% 0.00 36778 zfs VOP_LOCK1_APV+0x4c ------------------------------------------------------------------------------- lockmgr writer blocked by readers: 25 events in 300.017 seconds (0 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 10 40% 40% 0.00 53566 zfs _vn_lock+0x48 4 16% 56% 0.00 14955 zfs VOP_LOCK1_APV+0x4c 4 16% 72% 0.00 17955 devfs VOP_LOCK1_APV+0x4c 4 16% 88% 0.00 18700 devfs _vn_lock+0x48 2 8% 96% 0.00 20711 tmpfs VOP_LOCK1_APV+0x4c 1 4% 100% 0.00 10756 devfs vn_truncate+0x110 ------------------------------------------------------------------------------- lockmgr reader blocked by write wanted: 3 events in 300.017 seconds (0 events/sec) Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------------- 3 100% 100% 0.00 19074 zfs _vn_lock+0x48 -------------------------------------------------------------------------------