This document describes some ideas to implement instruction caching in bhyve.

The motivation behind this is to improve the performance of VM exits that
require instruction emulation.

Instruction emulation in bhyve can be broken down into the following steps:

1) Fetch the instruction.
   a. Convert the guest %rip into a guest physical address.
      This involves walking the guest's page table rooted at %cr3 and at
      each level translating the page table guest address into a host
      physical address via vm_gpa_hold().

   b. Copy the instruction bytes into a buffer
      This involves converting the guest physical address obtained from (1a)
      into a host physical address via vm_gpa_hold().

2) Decode the instruction via vmm_decode_instruction() into a 'struct vie'.

3) Emulate the instruction.

Steps (1) and (2) can be potentially eliminated if we can maintain a lookup
table that maps the guest %rip into the decoded instruction i.e. struct vie.

We can do this if we can guarantee that the instruction pointed to by the
guest's %rip is not changing without bhyve knowing about it. There are two
ways in which this could happen:

1) The guest could modify the contents of the page pointed to by %rip.

2) The guest could change its page tables such that %rip now points to a
   different guest physical address.

We can deal with (1) by write protecting the page pointed to by %rip.

We can deal with (2) by write protecting all the page table pages traversed
when translation %rip to the guest physical address.

The efficacy of the instruction caching will depend on how often the
mappings need to be invalidated when either of the write-protected pages
is written to.

Assuming a FreeBSD guest the instructions to touch the Local APIC are
part of the kernel and are guaranteed to be in the top 2GB of the 64-bit
address space (i.e. 0xffffffff80000000 - 0xffffffffffffffff). This area
of the address space is pretty much left untouched after bootup except
for loading kernel modules. So, we expect that instruction caching will
work pretty well for a FreeBSD guest.

XXX behavior of other *BSD, Linux, Windows guests?

Implmentation details:
key: struct vm *vm, uint64_t rip, uint64_t cr3
value: struct vie
lock: rmlock

There is another table that maps all the pages that are write-protected to
the 'key' above. When a write access is detected to a guest physical page
in this table all the keys associated with it are invalidated.

There may also be a need to garbage collect references to keys that are
invalidated.