Hot code reloading

## What is hot code reloading? Ability to upgrade a RabbitMQ node<br/> <strong style="color: #6a994e;"><em>without interrupting</em></strong> its own service. --- ## Why would we want that? --- ### No service interruption * Clients stay connected and notice no disruption * Queues are still running * No leader re-election or re-balancing of things --- ### Faster upgrades * Nothing to stop * Nothing to restart --- <span style="color: #538d22;">No downtime</span><br/> ⬇<br/> <span style="color: #73a942;">No coordination with other teams</span><br/> ⬇<br/> <span style="color: #aad576;"><strong>More frequent upgrades</strong></span>

Huge added value

+

Differentiator

## What about Feature flags? They solve a different problem: <dl> <dt style="color: #6a994e;">Hot code reloading</dt> <dd style="margin-bottom: 20px;">Upgrading of a single node while running</dd> <dt style="color: #0077b6;">Feature flags</dt> <dd>Running different versions inside a cluster</dd> </dl> ➡ Both are required --- <div class="mermaid"> <pre> %%{init: {'theme': 'dark', 'themeVariables': { 'darkMode': true }}}%% flowchart LR subgraph node_a [Node A] direction TB av1(3.12.9) -- "Hot code reloading" --> av2(3.13.0) end subgraph node_b [Node B] bv1(3.12.9) end node_a <-- "Feature flags" --> node_b </pre> </div>

Demo

## Erlang release handling 101 Two building blocks: 1. Erlang releases 2. `relup` and appup files --- ### Erlang releases * An Erlang release has a name and a version. * It is made of: * A set of applications; their version is pinned * A configuration (`sys.config`), possibly empty * A boot script * Optionaly, the runtime (ERTS executables) --- Erlang/OTP is distributed as an Erlang release: ```text erlang-25.3.2.7 ├── bin │ ├── erl │ └── escript ├── erts-13.2.2.4 ├── lib │ ├── kernel-8.5.4.2 │ └── stdlib-4.3.1.3 └── releases ├── 25 │ ├── start_clean.boot │ ├── start_sasl.script │ └── ... └── ... ``` --- Example of a RabbitMQ Erlang release: ```text rabbitmq-3.12.9 ├── bin │ ├── rabbitmq │ └── rabbitmqctl ├── lib │ ├── rabbit-3.12.9 │ └── rabbit_common-3.12.9 └── releases ├── 3.12.9 │ ├── start.boot │ ├── start.script │ └── ... └── ... ``` --- ### Appup files They describe how to go from one version to another: 1. load or unload this module 2. coordinate code change with the process that uses it 3. call this arbitrary function 4. restart the entire application --- Example of `rabbit.appup` in RabbitMQ 3.13.0: ```erlang {"3.13.0", % Upgrade from 3.12.9 to 3.13.0: [{"3.12.9", [{add_module, mc}, {load_module, rabbit_db}, {update, rabbit_amqqueue_process, {advanced, none}}, {apply, {rabbit_khepri, setup, []}} ]}], % Downgrade from 3.13.0 to 3.12.9: [{"3.12.9", [{delete_module, mc}, {delete_module, rabbit_khepri}]}]}. ``` --- ### `relup` file * Automatically generated from: * two Erlang releases: source and destination * the appup files from the destination * Describe the entire process to go from the source state to the destination state * Can handle an upgrade of Erlang/OTP itself --- Example of RabbitMQ 3.12.9 → 3.13.0 relup: ```erlang {"3.13.0", [{"3.12.9",[], [{load_object_code,{horus,"0.2.4",[horus,horus_cover,horus_utils]}}, {load_object_code,{khepri,"0.8.0",[khepri,khepri_adv,...]}}, point_of_no_return, {suspend,[rabbit_amqqueue_process]}, {load,{rabbit_amqqueue_process,brutal_purge,brutal_purge}}, {code_change,up,[{rabbit_amqqueue_process,none}]}, {resume,[rabbit_amqqueue_process]}, ... ``` --- <div class="mermaid"> <pre> %%{init: {'theme': 'dark', 'themeVariables': { 'darkMode': true }}}%% flowchart LR subgraph rel312 [RabbitMQ 3.12.9] direction TB rc312(rabbit_common-3.12.9) r312(rabbit-3.12.9) end subgraph rel313 [RabbitMQ 3.13.0] khepri312(khepri-0.9.0) rc313(rabbit_common-3.13.0) r313(rabbit-3.13.0) end rel312 ==>|relup| rel313 rc312 <-.->|rabbit_common.appup| rc313 r312 <-.->|rabbit.appup| r313 </pre> </div> --- <p class="r-fit-text">A release upgrade is <strong style="color: #ffb703;">NOT</strong> an <strong style="color: #ffb703;">atomic</strong> operation!</p> --- The service continues to run <strong style="color: #ffb703;">while</strong> `relup` instructions are processed <strong style="color: #ffb703;">one by one</strong>. --- ## What is required from us? * How to code * How to package --- ### How to code #### We need to <span style="color: #ffb703;">track every single changes</span> * added/removed exported functions * modified records and states * inter-module dependencies * passing/storing function references * supervision tree changes * use of `application:get_env/2` ➡ Part of this can be <span style="color: #6a994e;">automated</span> --- #### RabbitMQ-specific details * new/updated/removed bootsteps * core vs. plugins * non-OTP-compliant processes such as `rabbit_reader` --- #### Third-party dependencies * missing appup files * tracking their changes ourselves --- #### Ra machine upgrade * machine module reloaded at any time * constraints on the machine to work with old state? * coordination from Ra server's `code_change/3`? --- ### How to package Dependent on the package + how it is deployed. * need to <span style="color: #ffb703;">upgrade a running instance</span> obviously * need to <span style="color: #ffb703;">add new files on top of existing files</span> * should we include the Erlang runtime? * build of the release deferred to install time? [[some leads for Kubernetes](https://ferd.ca/my-favorite-erlang-container.html)] --- ## Where do we go from here? <strong style="color: #6a994e;">Is this something we want?</strong> <img src="cat.jpg"/> (yes) --- ### Suggestions 1. <span style="color: #6a994e;">Practice with workflow changes</span> internally first before we commit to something: * Record our changes in appup files * Evaluate feasability w.r.t. 3rd-party dependencies * Play with live upgrades locally 2. <span style="color: #6a994e;">Target patch releases initially</span> and decide later about minor releases if this goes well --- <h2 class="r-fit-text">Questions & discussion</h2>