Octez & Protocol overview
*************************

Octez, bundled with the Tezos protocol, is a complete implementation
of the Tezos blockchain.

Tezos is a distributed consensus platform with meta-consensus
capability. This means that Tezos not only comes to consensus about
the state of its ledger, like Bitcoin or Ethereum. It also attempts to
come to consensus about how the protocol and the nodes should adapt
and upgrade.

Tezos is backed up by a vibrant Tezos community, which may also be
contacted for technical support when needed.

Octez includes a node, a client, a baker, an accuser, and other tools,
distributed with the Tezos economic protocols of Mainnet for
convenience. The source code is placed under the MIT Open Source
License, and is available at https://gitlab.com/tezos/tezos.

This website contains technical documentation about both Octez and the
Tezos protocol.

The current latest release of Octez is Version 22.1.

For installing instructions, see Installing Octez.


Mainnet & Test Networks
=======================

The Tezos network, called "mainnet", is the current incarnation of the
Tezos blockchain. It runs with real tez that have been allocated to
the donors of July 2017 fundraiser (see Getting keys for fundraiser
accounts). It has been live and open since June 30th 2018.

All the instructions in this documentation are valid for Mainnet
however we **strongly** encourage users to first try all the
introduction tutorials on some test network to familiarize themselves
without risks.

There are several test networks for the Tezos blockchain with faucets
to obtain free tez. These networks are intended for developers wanting
to test their software before going to beta and for users who want to
familiarize themselves with Tezos before using their real tez.


Documentation source
====================

The source of this technical documentation website is an integral part
of the Octez repository, following Docs as Code best practices. The
documentation is automatically built from the current version of the
master branch.

Installing Octez
****************

In this how-to we explain how to get up-to-date binaries to run Tezos
(more precisely, the "Octez" implementation of Tezos software) on any
network (either on the mainnet or on one of the test networks). Octez
consists of several binaries (i.e., executable files), including: a
client, a node, and a baker.

There are several options for getting the binaries, depending on how
you plan to use Octez:

* installing packages. This is the easiest way to install native
  binaries for the latest stable release, together with their
  dependencies, using a package manager.

* using docker images. This is the easiest way to run the latest
  stable release of the binaries in Docker containers, on any OS
  supported by Docker.

* getting static binaries. This is the easiest way to get native
  binaries for the latest stable release, requiring no dependencies,
  under Linux.

* using homebrew. For macOS (and also Linux) you can use our Homebrew
  formula to compile and install octez.

* building the binaries via the OPAM source package manager. Take this
  way to install the latest stable release in your native OS
  environment, automatically built from sources.

* Building Octez by compiling the sources like developers do. This is
  the way to take if you plan to contribute to the source code. It
  allows to install any version you want (typically, the current
  development version on the master branch) by compiling it yourself
  from the sources.

These different options are described in the following sections,
except the last one, described in its own page.

Some Octez binaries also require certain parameter files to run. Only
some of the packaged distributions include such parameter files.
Therefore, depending on the type of installation and your user
profile, you may have to install some extra parameter files
separately. Their installation is described in page Building Octez
from source, but those instructions may be used for other installation
types:

* Install Zcash Parameters

* Install DAL trusted setup

Note that some of the packaged distributions are not only available
for the latest stable release. For instance, static binaries are also
available for release candidates, and Docker images are also available
for the current development version (see Release System for more
information).

When choosing between the installation options, you may take into
account the convenience of the installation step (and of upgrading
steps), but also efficiency and security considerations. For instance,
static binaries have a different memory footprint compared to
dynamically-linked binaries. Also, compiling the sources in the
official Octez repository is more secure than installing OPAM packages
from a repository that is not under Tezos control. In particular,
compiling from sources enforces a fixed set of dependencies; when
compiling via OPAM, this set of dependencies may change, which may or
may not be compatible with your security practices.

All our installation scenarios are tested daily, including by
automated means, to ensure that they are correct and up to date. These
tests are performed by applying scenarios in several standard
environments, from scratch. However, if you encounter problems when
performing one of the installation scenarios in your own environment,
you may want to take a look at Installation troubleshooting.


Installing binary packages
==========================

When it comes to installing software, especially for critical
applications like Tezos/Octez, it’s crucial to ensure a secure and
stable environment. While compiling from source can provide
customization options, it often introduces complexities and risks.
Instead, opting for binary packages sucha as Deb or RPM packages from
a trusted source simplifies installation and enhances security.

Deb or RPM packages compiled for a specific platform should be always
preferred over statically compiled binaries. Deb or RPM packages can
also be used to simplify the creation of OCI images or simply deployed
on bare metal using provisioning tools such as Ansible.

Using the official Deb or RPM packages offers several advantages:

* **Security**: Packages are pre-compiled and thoroughly tested,
  reducing the risk of vulnerabilities introduced during compilation.
  All our packages are signed and our supply chain is strictly
  monitored to make sure the packages that we deliver only use
  components that were vetted by our engineering team.

* **Stability**: Packages from a trusted repository undergo rigorous
  testing, ensuring stability and compatibility with the target
  system. We make sure to compile our binaries in a clean environment
  and using an up-to-date software distribution. We use LTS (long-term
  service) distributions to enhance stability and reduce the attack
  surface.

* **Ease of Installation**: Packages can be installed using standard
  package management tools, streamlining the process. For instance,
  "apt" is ubiquitous in the Debian world. These tools allow us to
  sign our packages that can be automatically verified by the end user
  during installation. We provide packages that allow the end user to
  easily tailor their installation for different use cases.

* **Reduced Downtime**: With reliable binaries and straightforward
  installation, system downtime due to installation errors or
  compatibility issues is minimized. We carefully test the upgrade
  process of our packages to make sure that end users can enjoy a
  click and go upgrade process with near to zero downtime.


Ubuntu and Debian Octez packages
--------------------------------

If you're using Ubuntu or Debian, you can install the Debian packages
using "apt" directly from our APT repository.

We support the following distribution/releases:

* "debian/bookworm"

* "ubuntu/noble"

* "ubuntu/jammy"

both on "amd64" and "arm64" architectures.

In order to add the Tezos package repository to your machine, do:

   export distribution=debian
   export release=bookworm

and run:


     curl -s "https://packages.nomadic-labs.com/$distribution/octez.asc" |
       sudo gpg --dearmor -o /etc/apt/keyrings/octez.gpg
     echo "deb [signed-by=/etc/apt/keyrings/octez.gpg] https://packages.nomadic-labs.com/$distribution $release main" |
       sudo tee /etc/apt/sources.list.d/octez.list
     sudo apt-get update

We also maintain a separate repository for release candidates. To
install the last release candidate simply prepend "RC/" to the
distribution name as in "export distribution=RC/debian"

Then, to install the binaries, run the following command to install
the octez-baker and all its dependencies:

   sudo apt install octez-baker

Once the Octez binary packages are installed, they can be set up as
services as explained in Setting up Octez Services.

To remove the Octez packages you can simply run the following command.

   sudo apt-get autopurge -y octez-baker

If migrating from Serokell packages you can check out migration
documentation Migrating from Serokell's to Nomadic Labs' packages.

To upgrade packages, use "apt-get update" and "apt-get upgrade". If
runnning Octez as services, see also how to restart them.


RPM Octez packages
------------------

If you're using Fedora or Rocky Linux, you can install RPM packages
with Octez binaries from the Octez from our DNF repository. Currently
we support the latest LTS release for Fedora and for RockyLinux :

* "rockylinux/9.3"

* "fedora/39"

* "fedora/42"

both on "amd64" and "arm64" architectures.

In order to add the Tezos package repository to your machine, do:

   export distribution=rockylinux
   export release=9.3

and run:

     # Update and install the config-mananger plugin
     dnf -y update
     dnf -y install dnf-plugins-core

     # Add the repository
     dnf -y config-manager --add-repo "https://packages.nomadic-labs.com/$distribution/dists/$release/"
     # Install public key
     rpm --import "https://packages.nomadic-labs.com/$distribution/octez.asc"

For "rockylinux" user you also need to add the "devel" repository

   dnf -y config-manager --set-enabled devel

To update the local dnf registry run:

   dnf update

We also maintain a separate repository for release candidates. To
install the last release candidate simply prepend "RC/" to the
distribution name as in "export distribution=RC/rockylinux"

Then, to install the binaries, run the following commands:

   dnf -y install octez-node
   dnf -y install octez-client
   dnf -y install octez-baker
   dnf -y install octez-dal-node
   dnf -y install octez-smart-rollup-node

To remove the Octez packages you can simply run the following command.

   dnf -y remove octez-node octez-client octez-baker \
                 octez-dal-node octez-smart-rollup-node

To upgrade packages, use "dnf update". If runnning Octez as services,
see also how to restart them.


Getting static binaries
=======================

You can get static Linux binaries of the latest release from the Octez
package registry.

This repository provides static binaries for x86_64 and arm64
architectures. Since these binaries are static, they can be used on
any Linux distribution without any additional prerequisites. However,
note that, by embedding all dependencies, static binary executables
are typically much larger than dynamically-linked executables.

For upgrading to a newer release, you just have to download and run
the new versions of the binaries.

Octez static binaries are signed via GPG. To verify that a binary was
not tampered with, you can download and import our public GPG key and
use "gpg" to verify the signature associated with the binary.

   curl -O https://packages.nomadic-labs.com/octez.asc
   gpg --import octez.asc
   ... # download the static binary you want to verify and the associated sig file
   gpg --verify <bin>.sig <bin>


Using Homebrew
==============

On macOS and Linux, you can compile and install Octez using Homebrew.
If Homebrew is not yet installed on your system, please refer to the
official [Homebrew installation guide](https://brew.sh/) for detailed
instructions.

Once Homebrew is set up, follow these steps to install Octez.

Download the Octez formula by executing the following command in your
terminal:

   curl -q "https://packages.nomadic-labs.com/homebrew/Formula/octez.rb" -O

For "RC" versions, do rather:

   curl -q "https://packages.nomadic-labs.com/homebrew/RC/Formula/octez.rb" -O

Install Octez using the downloaded formula with the following command:

   brew install -v ./octez.rb

Depending on the speed of your system, the build can take more than 10
minutes. We regularly test the build in our CI using macOS 14 (Sonoma)
with Xcode 15 on an ARM-based Mac. More recent configurations should
also work.


Using Docker images
===================

For every change committed in the GitLab repository, Docker images are
automatically generated and published on DockerHub. This provides a
convenient way to run an always up-to-date "octez-node".

From version 22.0 all Octez Docker images are signed using Cosign. You
can verify if the images are correctly signed, using Cosign.

You can use the Docker images either directly or using Docker compose
files, as explained next. In both cases, you need to have Docker
installed and started (Docker Desktop would suffice for the
instructions below).


Plain Docker images
-------------------

The Docker images can be directly used to run the different Octez
binaries. To make sure you use the most recent version of Octez, run:

   docker pull tezos/tezos-bare:latest

For instance, to run a node on the Ghostnet test network, starting
from a snapshot, in Rolling history mode, start with a fresh directory
and configure the node:

   mkdir $HOME/rolling-data-directory
   docker run -it --rm \
     --volume "$HOME/rolling-data-directory:/home/tezos/.tezos-node" \
     tezos/tezos-bare:latest \
     octez-node config init --network ghostnet --rpc-addr 127.0.0.1 \
       --history-mode rolling

(You may use another location than "$HOME", but note that option "--
volume" requires absolute paths.)

Then, download and import a snapshot, and finally run the node:

   wget -O $HOME/rolling <snapshot-url>
   docker run -it --rm \
     --volume "$HOME/rolling-data-directory:/home/tezos/.tezos-node" \
     --volume "$HOME/rolling:/rolling:ro" \
     tezos/tezos-bare:latest \
     octez-node snapshot import /rolling
   docker run --name octez-local-node -d \
     --volume "$HOME/rolling-data-directory:/home/tezos/.tezos-node" \
     tezos/tezos-bare:latest \
     octez-node run

You may check when your node is bootstrapped by executing in another
terminal:

   docker exec -it octez-local-node octez-client bootstrapped

You may stop and restart the node as needed, for instance if you need
to upgrade the version of the storage:

   docker stop octez-local-node
   docker run --rm --volumes-from octez-local-node tezos/tezos-bare:latest \
     octez-node upgrade storage
   docker start octez-local-node


Docker compose files
--------------------

Another way to run those Docker images is with docker-compose.
"docker-compose" files are available for all active protocols in
directory scripts/docker. Each compose file is able to launch services
for a node, a baker, and an accuser for the given protocol.

First, you have to make some choices:

* choose one of the above protocols and download its compose file

* choose a network to connect to (a testnet name, "sandbox",  or
  "mainnet"); that network must currently run your protocol

* choosing the desired history mode ("rolling", "full", or "archive")

* specify a vote for the liquidity baking feature ("on", "pass", or
  "off")

For instance, to configure and run the node on the active protocol on
Ghostnet, in Rolling history mode:

   export PROTO=parisC
   wget https://gitlab.com/tezos/tezos/-/raw/master/scripts/docker/$PROTO.yml
   export LIQUIDITY_BAKING_VOTE=pass
   docker compose -f $PROTO.yml run --rm -it \
     octez-node octez-node --network ghostnet --history-mode rolling

(Note in the command above that "octez-node" is the name of both the
container and executable.)

Note:

  If the node complains that it is configured for another network,
  you'll have to remove the node configuration file before running it:

     docker compose -f $PROTO.yml run --rm \
       --entrypoint='sh -c "rm /var/run/tezos/node/data/config.json"' \
       octez-node

The node is now configured and started correctly, but may take a very
long time to bootstrap. In most practical cases, you have to restart
it from a snapshot file. For that, you have to stop the node by
hitting ^C (or executing in another terminal "docker compose -f
$PROTO.yml stop octez-node"), then clean up its data directory and
import a snapshot that you previously downloaded:

   docker compose -f $PROTO.yml run --rm \
     --entrypoint='sh -c "cd /var/run/tezos/node/data/; \
       rm -fr lock store context daily_logs"' \
     octez-node
   wget -O $HOME/rolling <snapshot-url>
   docker compose -f $PROTO.yml run --rm -it -v "$HOME/rolling:/snapshot:ro" \
     octez-node octez-snapshot-import

Now you can start all the services:

   docker compose -f $PROTO.yml up

You may check when your node is bootstrapped by running "octez-client"
inside the node's container:

   docker compose -f $PROTO.yml exec octez-node octez-client bootstrapped

You may stop and restart the node as needed. For instance if the Octez
version you are using requires to upgrade the version of the storage,
you can restart the node after upgrading the storage:

   docker compose -f $PROTO.yml stop octez-node
   docker compose -f $PROTO.yml run octez-node octez-upgrade-storage
   docker compose -f $PROTO.yml up octez-node


Building Docker Images Locally
------------------------------

The docker image used throughout the docker-compose files is fetched
from upstream, but you can also build one locally and reference it.
Run the following command to build the image:

   ./scripts/create_docker_image.sh

And then update the docker-compose file (e.g., "alpha.yml") with the
docker tag:

   node:
     image: tezos:latest
     ...


Docker Image Configuration
--------------------------

Lastly, the entrypoint script (scripts/docker/entrypoint.sh) provides
the following configurable environment variables:

* "DATA_DIR": The directory to store the node's data (defaults to
  "/var/run/tezos").

* "NODE_HOST": The name of the node container (defaults to "node").

* "NODE_RPC_PORT": The RPC port **inside the container** the node
  listens to (defaults to "8732").

* "NODE_RPC_ADDR": The RPC address **inside the container** the node
  binds to (defaults to "[::]").

* "PROTOCOL": The protocol used.

These variables can be set in the docker-compose file, as demonstrated
in scripts/docker/alpha.yml:

   octez-node:
     ...
     environment:
       PROTOCOL: alpha
     ...

If the above options are not enough, you can always replace the
default "entrypoint" and "command" fields.

   version: "3"
   services:
     octez-node:
       container_name: octez-node-alpha
       entrypoint: /bin/sh
       command: /etc/my-init-script.sh
       volumes:
         - ./my-init-script.sh:/etc/my-init-script.sh
         - ...
       environment:
         PROTOCOL: alpha
    ...


Installing binaries
===================

Depending on your operating system, you may install Octez
(dynamically-linked) binaries and their dependencies by first
downloading the packages for your distribution from the Octez release
page, browsing to your distribution and then installing them with your
package tool manager. Most of the configuration options are accessible
by the user in "/etc/default/<package>".

If you are upgrading from a different package distributor such as
Serokell's tezos-packaging, please pay attention to the possible
differences between the two packages, in particular regarding the home
directory for the "tezos" user.

There are several packages:

* "octez-client": the client for manipulating wallets and signing
  items

* "octez-node": the Octez node

* "octez-baker": the Octez baking and VDF daemons

* "octez-smartrollup": the Octez Smart Rollup daemons

* "octez-signer": the remote signer, to hold keys on (and sign from) a
  different machine from the baker or client


Building from sources via OPAM
==============================

The easiest way to build the binaries from the source code is to use
the OPAM source package manager for OCaml.

This is easier than setting up a complete development environment as
described in Building Octez from source, like developers do. However,
this method is recommended for expert users as it requires basic
knowledge of the OPAM package manager and the OCaml packages workflow.
In particular, upgrading Octez from release to release might require
tinkering with different options of the OPAM package manager to adjust
the local environment for the new dependencies.


Environment
-----------

Currently Octez is being developed for Linux x86_64, mostly for Ubuntu
and Fedora Linux. The following OSes are also reported to work: macOS
(x86_64), Arch Linux ARM (aarch64), Debian Linux (x86_64). A Windows
port is feasible and might be developed in the future.

Note:

  If you build the binaries by using the following instructions inside
  a Docker container, you have to give extended privileges to this
  container, by passing option "--privileged" to the "docker run"
  command.

Warning:

  Mixing LLVM and GNU binutils toolchains can cause issues when
  building Octez. If you encounter an error like this, it may be that
  you have tools from both LLVM and GNU in scope.

     Error: ExternalToolError { reason: "Failed to create archive index with `ranlib`", tool: "ranlib", args: ["liboctez_rust_deps.a"], stdout: "", stderr: "LLVM ERROR: Invalid encoding\n" }

  In this case, refer to Mixing LLVM and GNU binutils.


Install OPAM
------------

First, you need to install the OPAM package manager, at least version
2.1, that you can get by following the install instructions.

After the first install of OPAM, use "opam init --bare" to set it up
while avoiding to compile an OCaml compiler now, as this will be done
in the next step.


Install Octez OPAM packages
---------------------------

The latest Octez release is available (as soon as possible after the
release) directly as OPAM packages.

Note:

  Every file related to OPAM is (by default) in "$HOME/.opam" which
  means that, first, OPAM installs are user-specific and, second, you
  can get rid of everything by removing this directory (+ updating
  your rc files ("$HOME/.bashrc", "$HOME/.profile", "$HOME/.zshrc",
  "$HOME/.emacs", ...) if you asked/allowed OPAM to add some lines in
  them).

The binaries need a specific version of the OCaml compiler (see the
value of variable "$ocaml_version" in file "scripts/version.sh"). To
get an environment with it do:

   wget -O latest-release:version.sh https://gitlab.com/tezos/tezos/raw/latest-release/scripts/version.sh
   source latest-release:version.sh
   opam switch create for_tezos $ocaml_version
   eval $(opam env)

Note:

  The "opam switch create" command may fail if the switch already
  exists; you are probably re-installing or upgrading an existing
  installation. If the required compiler version has not changed since
  the last time, you may simply ignore this error. Otherwise, you are
  upgrading to a new compiler, so look at the relevant section
  below.The command "eval $(opam env)" sets up required environment
  variables. OPAM will suggest to add it in your rc file. If, at any
  point, you get an error like "octez-something: command not found",
  first thing to try is to (re)run "eval $(opam env --switch
  $ocaml_version)" (replace "$ocaml_version" with its value in
  "scripts/version.sh") to see if it fixes the problem.

Note:

  If an OPAM commands times out, you may allocate it more time for its
  computation by setting the OPAMSOLVERTIMEOUT environment variable
  (to a number of seconds), e.g. by adding "OPAMSOLVERTIMEOUT=1200"
  before the command. If no timeout occurs, you may omit this part.

Now, install all the binaries by:

   opam install octez

You can be more specific and only "opam install octez-node", "opam
install octez-baker-alpha", ...

Warning:

  Note that "opam install octez-client" and "opam install octez-
  signer" are "minimal" and do not install the support for Ledger Nano
  devices. To enable it, run "opam install ledgerwallet-tezos" in
  addition to installing the binaries. (The macro meta-package "tezos"
  installs "ledgerwallet-tezos".)


Updating via OPAM
-----------------

Installation via OPAM is especially convenient for updating to newer
versions. Once some libraries/binaries are installed and new versions
released, you can update by:

   opam update
   opam upgrade

It is recommended to also run the command "opam remove -a" in order to
remove the dependencies installed automatically and not needed
anymore. Beware not uninstall too much though.

Identified situations where it will be more tricky are:

* When the OCaml compiler version requirement changes. In this case,
  you have several possibilities:

  * Be explicit about the "upgrade" and do "opam upgrade --unlock-base
    ocaml.$new_version tezos". Note that starting from OPAM version
    2.1, this option is replaced by "--update-invariant" (see the
    opam-switch manual).

  * Remove the existing switch (e.g., "opam switch remove for_tezos",
    but be aware that this will delete the previous installation), and
    replay the installation instructions.

  * Replay the installation instructions while creating a different
    switch (e.g. "ocaml_${ocaml_version}_for_tezos"), but be aware
    that each switch consumes a significant amount of disk space.

* When there are Rust dependencies involved. The way to go is still
  unclear. The solution will be defined when delivering the first
  release with Rust dependencies.


Appendix
========

* Building Octez from source

  * Install Rust

  * Install Zcash Parameters

  * Install DAL trusted setup

  * Get the sources

  * Install Octez dependencies

  * Compile

  * Update

* Verifying Octez Docker Images with Cosign

  * Prerequisites

  * Obtaining the Public Key

  * Verifying the Docker Image

* Installation troubleshooting

  * Generic issues

  * Mixing LLVM and GNU binutils

  * Compiling the sources

Building Octez from source
**************************

If you plan to contribute to the Octez codebase or if you want to
build a version of Octez that is based on the most recent code, the
way to go is to set up a complete development environment by cloning
the repository and compiling the sources using the provided makefile.
If you rather want to install prebuilt binaries, refer to Installing
Octez.

You can either build all the executables, as illustrated below, or
only a subset of the executables, as detailed in section Compile.

**TL;DR**: From a fresh Debian Bookworm x86_64, you typically want to
select a source branch in the Octez repository, e.g.:

     REPO="tezos/tezos"
     BRANCH="latest-release"

and then do:

   # [Temporary fix: removes tezos folder from PATH if added with Octez <= v13 instructions]
   PATH=${PATH##"$HOME"/tezos/:}
   sudo apt-get install -y rsync git m4 build-essential patch unzip wget opam jq bc
   # [install rust]
   wget https://sh.rustup.rs/rustup-init.sh
   chmod +x rustup-init.sh
   ./rustup-init.sh --profile minimal --default-toolchain 1.86.0 -y
   # [source cargo]
   . $HOME/.cargo/env
   # [get sources]
   git clone https://gitlab.com/"$REPO".git tezos
   cd tezos
   git checkout $BRANCH
   # [install Octez dependencies]
   opam init --bare
   make build-deps
   # [compile sources]
   eval $(opam env)
   make
   # [optional setup]
   # puts Octez binaries in PATH:
   # export PATH=$PWD/_build/install/default/bin/:$PATH
   # if using bash, enables autocompletion:
   # source ./src/bin_client/bash-completion.sh
   # removes Mainnet/testnet disclaimers:
   # export TEZOS_CLIENT_UNSAFE_DISABLE_DISCLAIMER=Y

Warning:

  If you rerun the procedure above in an already cloned repository,
  typically by restarting from "git checkout $BRANCH" after upgrading
  the sources to a new version of Octez, you should do a "git fetch"
  before, to ensure that the tags for the latest release are known to
  Git. Otherwise, the sources compile perfectly fine but Octez
  executables may report their version incorrectly (e.g. "21.2+dev"
  instead of "22.0").

The following sections describe the individual steps above in more
detail.

Note:

  Besides compiling the sources, it is recommended to also install
  Python and some related tools, which are needed, among others, to
  build the documentation and to use the Git pre-commit hook.


Install Rust
============

Compiling Octez requires the Rust compiler (see recommended version in
variable "$recommended_rust_version" in file "scripts/version.sh") and
the Cargo package manager to be installed. If you have rustup
installed, it should work without any additional steps on your side.
You can use rustup to install both. If you do not have "rustup",
please avoid installing it from Snapcraft; you can rather follow the
simple installation process shown below:

   wget https://sh.rustup.rs/rustup-init.sh
   chmod +x rustup-init.sh
   ./rustup-init.sh --profile minimal --default-toolchain 1.86.0 -y

Once Rust is installed, note that your "PATH" environment variable (in
".profile") may be updated and you will need to restart your session
so that changes can be taken into account. Alternatively, you can do
it manually without restarting your session:

   . $HOME/.cargo/env

Note that the command line above assumes that rustup installed Cargo
in "$HOME/.cargo", but this may change depending on how you installed
rustup. See the documentation of your rustup distribution if file
".cargo" does not exist in your home directory.


Install Zcash Parameters
========================

Octez binaries require the Zcash parameter files to run. Docker images
come with those files, and the source distribution also includes those
files. But if you compile from source and move Octez to another
location (such as "/usr/local/bin"), the Octez binaries may prompt you
to install the Zcash parameter files. The easiest way is to download
and run this script:

   wget https://raw.githubusercontent.com/zcash/zcash/713fc761dd9cf4c9087c37b078bdeab98697bad2/zcutil/fetch-params.sh
   chmod +x fetch-params.sh
   ./fetch-params.sh

The node will try to find Zcash parameters in the following
directories, in this order:

1. "$XDG_DATA_HOME/.local/share/zcash-params"

2. "$XDG_DATA_DIRS/zcash-params" (if "$XDG_DATA_DIRS" contains several
   paths separated by colons ":", each path is considered)

3. "$OPAM_SWITCH_PREFIX/share/zcash-params"

4. "./_opam/share/zcash-params"

5. "~/.zcash-params"

6. "~/.local/share/zcash-params"

7. "/usr/local/share/zcash-params"

8. "/usr/share/zcash-params"

If the node complains that it cannot find Zcash parameters, check that
at least one of those directories contains both files "sapling-
spend.params" and "sapling-output.params". Here is where you should
expect to find those files:

* if you are compiling from source, parameters should be in
  "_opam/share/zcash-params" (you may need to run "eval $(opam env)"
  before running the node);

* if you used "fetch-params.sh", parameters should be in "~/.zcash-
  params".

Note:

  Some operating systems may not be covered by the list of directories
  above. If Zcash is located elsewhere on your system (typically, on
  MacOS X), you may try creating a symbolic link such as: "ln -s
  ~/Library/Application\ Support/ZcashParams ~/.zcash-params".

Note that the script "fetch-params.sh" downloads a third file
containing parameters for Sprout (currently called "sprout-
groth16.params"), which is not loaded by Sapling and can be deleted to
save a significant amount of space (this file is *much* bigger than
the two other files).


Install DAL trusted setup
=========================

Users running DAL in **operator or observer** profiles need to have a
set of cryptographic parameters (known as an SRS) installed in order
to run their DAL node. In particular, these are needed when executing
the Octez test suite, which involves DAL nodes running in various
profiles. However, for simplicity, on some test networks the
initialization parameters are mocked-up and built-in.

The cryptographic parameters can be retrieved via the following
script:

   scripts/install_dal_trusted_setup.sh


Get the sources
===============

Octez "git" repository is hosted at GitLab. All development happens
here. Do **not** use our GitHub mirror which we don't use anymore and
only mirrors what happens on GitLab.

Checkout the "latest-release" branch to use the latest release.
Alternatively, you can checkout a specific version based on its tag.


Install Octez dependencies
==========================

Install the OCaml compiler and the libraries that Octez depends on:

   make build-deps

Alternatively, if you want to install extra development packages such
as "merlin", you may use the following command instead:

   make build-dev-deps

Note:

  * These commands create a local OPAM switch ("_opam" folder at the
    root of the repository) where the required version of OCaml and
    OCaml Octez dependencies are compiled and installed (this takes a
    while but it's only done once).

  * Be sure to "eval $(scripts/env.sh)" when you "cd" into the
    repository in order to be sure to load this local environment.

  * As the opam hook would overwrite the effects of "eval
    $(scripts/env.sh)" the script will disable the opam hook
    temporarily.

  * OPAM is meant to handle correctly the OCaml libraries but it is
    not always able to handle all external C libraries we depend on.
    On most systems, it is able to suggest a call to the system
    package manager but it currently does not handle version checking.

  * As a last resort, removing the "_opam" folder (as part of a "git
    clean -dxf" for example) allows to restart in a fresh environment.


Compile
=======

Once the dependencies are installed we can update OPAM's environment
to refer to the new switch and compile the project to build all the
executables:

   eval $(opam env)
   make

Note:

  Instead of the simple "make" command above, you may use more
  restrictive targets in the makefile  to build only some subset of
  the executables. For instance, you may exclude experimental
  executables using "make release"; furthermore exclude executables
  such as the EVM node using "make octez"; or even restrict to Layer 1
  executables using "make octez-layer1".

Lastly, you can also add the Octez binaries to your "PATH" variable,
and after reading the Disclaimer a few hundred times you are allowed
to disable it with "TEZOS_CLIENT_UNSAFE_DISABLE_DISCLAIMER=Y".

You may also activate Bash autocompletion by executing:

   source ./src/bin_client/bash-completion.sh

Warning:

  Note that if your shell is "zsh", you may need extra configuration
  to customize shell completion (refer to the "zsh" documentation).


Update
======

For updating to a new version, you typically have to update the
sources by doing "git pull" in the "tezos/" directory and replay the
compilation scenario starting from "make build-deps". You may also use
"make clean" (and "rm -Rf _opam/" if needed) before that, for
restarting compilation in a fresh state.

Verifying Octez Docker Images with Cosign
*****************************************

Cosign is a tool developed by Sigstore to sign and verify container
images and other artifacts. This document provides instructions on how
to verify Docker images of Octez signed using Cosign.


Prerequisites
=============

Before you can verify Docker images signed with Cosign, ensure you
have Docker and Cosign installed on your system.

* Docker Installation Guide

* Cosign Installation Guide


Obtaining the Public Key
========================

To verify a signed Docker image, you need the public key that
corresponds to the private key used for signing. The pem certificate
is available at https://keyserver.nomadic-labs.com/cosign/nl-prod-
docker-sign-key.pem

Saving the public key:

   # Save the public key to a file (e.g., octez.pub)
   curl -o octez.pub https://keyserver.nomadic-labs.com/cosign/nl-prod-docker-sign-key.pem


Verifying the Docker Image
==========================

To verify the Octez Docker image, follow these steps:

1. **Pull the Docker Image** (if not already pulled):

      docker pull tezos/tezos-bare:master

2. **Use Cosign to Verify the Image**:

   Replace the image name with the name of your Docker image and tag
   with the specific tag ( for example "tezos/tezos:22.0" )

      cosign verify --key octez.pub tezos/tezos-bare:master

   Or more directly:

      cosign verify --key https://keyserver.nomadic-labs.com/cosign/nl-prod-docker-sign-key.pem tezos/tezos-bare:master

3. **Check the Output**:

   You can use tools like "jq" to parse the json output of Cosign:

      cosign verify --key https://keyserver.nomadic-labs.com/cosign/nl-prod-docker-sign-key.pem tezos/tezos-bare:master | jq

   If the verification is successful, Cosign will output the
   signatures and their claims in JSON format:

      The following checks were performed on each of these signatures:
      - The cosign claims were validated
      - Existence of the claims in the transparency log was verified offline
      - The signatures were verified against the specified public key

      [
        {
          "critical": {
            "identity": {
                "docker-reference": "<image>"
            },
            "image": {
              "docker-manifest-digest": "sha256:<digest>"
            },
            "type": "cosign container image signature"
          },
          "optional": {
            "Bundle": {
              "SignedEntryTimestamp": "<timestamp's signature>",
              "Payload": {
                "body": "<payload>",
                "integratedTime": "<time>",
                "logIndex": "<logIndex>",
                "logID": "<logId>"
              }
            }
          }
        }
      ]

   If the verification fails, an error message will be displayed
   indicating the failure reason.

Installation troubleshooting
****************************

This page groups information about known problems when installing
Tezos (more precisely, the "Octez" implementation of Tezos software).
The different issues and their solutions are grouped per installation
method, except for generic issues concerning all installation
scenarios.

This page lists only the most frequent problems. If you don't find
your problem in this page, chances are that the problem is too
specific. Consult the Tezos community (e.g. the Tezos Stack Exchange)
to see if others have encountered a similar problem, and whether a
solution is known.


Generic issues
==============


Case-sensitive file paths
-------------------------

On case-insensitive file systems, which is the default on Windows and
MacOS, beware of some related problems.

In the Octez repository, there are no conflicting file paths differing
only in case. However, depending on how Git extracts your clone, the
case of some file paths in your clone may differ from the expected
one.

For example, file script-inputs/slim-mode-dune contains a list of
source directories (belonging to old protocols) to ignore when
building the documentation locally. If the case of your extracted
files does not match the case in that file, you may encounter
unexpected behaviors, such as errors when building the documentation
on those parts. (Note that these documentation errors cannot be fixed
in old protocols whose code is frozen.)

If you run into such errors, simply rename your file paths to match
the case in the source repository.


Mixing LLVM and GNU binutils
============================

Mixing LLVM and GNU binutils toolchains can cause issues when building
Octez. If you encounter errors like the following, it may be that you
have tools from both LLVM and GNU in scope.

   Error: ExternalToolError { reason: "Failed to create archive index with `ranlib`", tool: "ranlib", args: ["liboctez_rust_deps.a"], stdout: "", stderr: "LLVM ERROR: Invalid encoding\n" }

   LLVM ERROR: Invalid encoding

   bfd plugin: LLVM gold plugin has failed to create LTO module: Opaque pointers are only supported in -opaque-pointers mode (Producer: 'LLVM17.0.4-rust-1.74.0-stable' Reader: 'LLVM 14.0.0')

You can check "objcopy", for example, like this.

   objcopy --version

If the output of this command indicates an LLVM version of "objcopy"
and you have encountered the above error message, then you are mixing
toolchains. In this case, you ought to remove, for example, the LLVM
toolchain (e.g. through your system's package manager) or ensure that
GNU's tools and libraries, like "objcopy", have higher precedence in
your "$PATH" environment variable.


Compiling the sources
=====================

These issues concern installing by compiling the sources.

Currently, the "CONFIG_SITE" environment variable must not be set
during the installation process, or the "stdcompat" package may be
installed incorrectly. See thierry-martinez/stdcompat#13.

Getting started with Octez
**************************

This short tutorial illustrates the use of the various Octez binaries
as well as some concepts about the network.


The Binaries
============

After a successful compilation, you should have the following
binaries:

* "octez-node": the Octez daemon itself (see Node);

* "octez-client": a command-line client and basic wallet (see Client);

* "octez-admin-client": administration tool for the node (see Admin
  Client);

* "octez-{baker,accuser}-*": daemons to bake and accuse on the Tezos
  network (see Running Octez);

* "octez-signer": a client to remotely sign operations or blocks (see
  Signer);

* "octez-smart-rollup-node": executable for using and running a smart
  rollup node as Layer 2 (see Smart rollup node);

* "octez-smart-rollup-wasm-debugger": debugger for smart rollup
  kernels (see Smart rollup node)

* "octez-codec": a utility for documenting the data encodings and for
  performing data encoding/decoding (see Codec)

* "octez-protocol-compiler": a domain-specific compiler for Tezos
  protocols (see Protocol compiler)

* "octez-snoop": a tool for modeling the performance of any piece of
  OCaml code, based on benchmarking (see Benchmarking with Snoop)

The daemons other than the node are suffixed with the name of the
protocol they are bound to. More precisely, the suffix consists of the
first 8 characters of the protocol hash; except for protocol Alpha,
for which the suffix is simply "-alpha". For instance, "octez-baker-
PsQuebec" is the baker for the Quebec protocol, and "octez-baker-
alpha" is the baker of the development protocol. The "octez-node"
daemon is not suffixed by any protocol name, because it is independent
of the economic protocol. See also the Node's Protocol section below.


Read the Manual
===============

All the Octez binaries provide the "--help" option to display
information about their usage, including the available options and the
possible parameters.

Additionally, most of the above binaries (i.e., all but the node, the
validator, and the compiler) provide a textual manual that can be
obtained with the command "man", whose verbosity can be increased with
"-v", for example:

   octez-client man -v 3

Note that the result of this command is also available online at
Client manual, and this is true for the other binaries.

It is also possible to get information on a specific command in the
manual with "man <command>":

   octez-client man set

To see the usage of one specific command, you may also type the
command without arguments, which displays its possible completions and
options:

   octez-client transfer

Warning:

  Beware that the commands available on the client depend on the
  specific protocol run by the node. For instance, "transfer" is not
  available when the node runs the genesis protocol, which may happen
  for a few minutes when launching a node for the first time, **or
  when the client is not connected to a node**. In the last case, the
  above command generates a warning followed by an error:

     Warning:
       Failed to acquire the protocol version from the node
       [...]
     Error:
       Unrecognized command.
       Try using the man command to get more information.
     Usage:
       [...]

To make the client command behave as for a protocol other than that
used by the node (or even when not connected to a node), use the
option "--protocol" (or "-p"), e.g.:

   octez-client --protocol ProtoALphaAL man transfer

Note that you can get the list of protocols known to the client with:

   octez-client list understood protocols

The full command line documentation of the Octez binaries supporting
the "man" command is also available online: Command Line Interface.


Node
====

The node is the main actor of the Tezos blockchain and it has two main
functions: running the gossip network and updating the context. The
gossip network is where all Tezos nodes exchange blocks and operations
with each other (see Admin Client to monitor p2p connections). Using
this peer-to-peer network, an operation originated by a user can hop
several times through other nodes until it finds its way into a block
baked by a baker. Using the blocks it receives on the gossip network
the node also keeps up to date the current *context*, that is the full
state of the blockchain shared by all peers. Approximately every 8
seconds a new block is created and, when the node receives it, it
applies each operation in the block to its current context and
computes a new context. The last block received on a chain is also
called the *head* of that chain. Each new head is then advertised by
the node to its peers, disseminating this information to build a
consensus across the network.

Other than passively observing the network, your node can also inject
its own new operations when instructed by the "octez-client" and even
send new blocks when guided by the "octez-baker-*". The node has also
a view of the multiple chains that may exist concurrently and selects
the best one based on its fitness (see The consensus algorithm).

Note:

  The "octez-node" uses (unless the option "--singleprocess" is given)
  an auxiliary daemon in order to validate, apply and compute the
  resulting context of blocks, in parallel to its main process. Thus,
  an "octez-validator" process can appear while monitoring the active
  processes of the machine.

Warning:

  To ensure the best conditions to run a node, we recommend users to
  use NTP to avoid clock drift. Clock drift may result in not being
  able to get recent blocks in case of negative lag time, and in not
  being able to inject new blocks in case of positive lag time.


Node Identity
-------------

First, we need to generate a new identity for the node to connect to
the network:

   octez-node identity generate

Note:

  If the node prompts you to install the Zcash parameter file, follow
  the corresponding instructions.

The identity comprises a pair of cryptographic keys that nodes use to
encrypt messages sent to each other, and an antispam proof-of-work
stamp proving that enough computing power has been dedicated to
creating this identity. Note that this is merely a network identity
and it is not related in any way to a Tezos address on the blockchain.

If you wish to run your node on a test network, now is also a good
time to configure your node for it (see Connecting to a Network).


Node Synchronization
--------------------

Whenever a node starts, it tries to retrieve the most current head of
the chain from its peers. This can be a long process if there are many
blocks to retrieve (e.g. when a node is launched for the first time or
has been out of sync for a while), or on a slow network connection.
The mechanism of Snapshots can help in reducing the synchronization
time.

Once the synchronization is complete, the node is said to be
*bootstrapped*. Some operations require the node to be bootstrapped.


Node's Protocol
---------------

A Tezos node can switch from one protocol to another during its
execution.  This typically happens during the synchronization phase
when a node launches for the first time. The node starts with the
genesis protocol and then goes through all previous protocols until it
finally switches to the current protocol.

Throughout the documentation, "Alpha" refers to the protocol in the
"src/proto_alpha" directory of the "master" branch, that is, a
protocol under development, which serves as a basis to propose
replacements for the currently active protocol. The Alpha protocol is
used by default in sandbox mode and in the various test suites.


Storage
-------

All blockchain data is stored by the node under a data directory,
which by default is "$HOME/.tezos-node/".

If for some reason your node is misbehaving or there has been an
upgrade of the network, it is safe to remove this directory, it just
means that your node will take some time to resync the chain.

If removing this directory, please note that if it took you a long
time to compute your node identity, keep the "identity.json" file and
instead only remove its child "store", "context" and "protocol" (if
any) sub-directories.

If you are also running a baker, make sure that it is configured to
access the data directory of the node (see how to run a baker).


RPC Interface
-------------

The only programming interface to the node is through JSON RPC calls
and it is disabled by default.  More detailed documentation can be
found in the RPC index. The RPC interface must be enabled for the
clients to communicate with the node but it should not be publicly
accessible on the internet. With the following command, it is
available uniquely on the "localhost" address of your machine, on the
default port "8732".

   octez-node run --rpc-addr 127.0.0.1


Node configuration
------------------

Many options of the node can be configured when running the node:

* RPC parameters (e.g. the port number for listening to RPC requests
  using option "--rpc-addr")

* The directory where the node stores local data (using option "--
  data-dir")

* Network parameters (e.g. the network to connect to, using option "--
  network", the number of connections to peers, using option "--
  connections")

* Validator and mempool parameters

* Logging options.

The list of configurable options can be obtained using the following
command:

   octez-node run --help

You can read more about the node configuration and its private mode.

Besides listening to requests from the client, the node listens to
connections from peers, by default on port "9732" (this can be changed
using option "--net-addr"), so it's advisable to open incoming
connections to that port.


Quickstart Octez node
---------------------

Putting together all the above instructions, you may want to quickly
start a node on a testnet as follows:

   # Download a snapshot for your target network, e.g. ghostnet:
   wget https://snapshots.tzinit.org/ghostnet/rolling -O rolling
   # Configure the node for running on your testnet, e.g. ghostnet:
   octez-node config init --network ghostnet --rpc-addr 127.0.0.1
   # Import the snapshot file into the node:
   octez-node snapshot import rolling
   # Run the node:
   octez-node run


Client
======

Octez client can be used to interact with the node, it can query its
status or ask the node to perform some actions. A complete manual page
of octez-client is available here (for the currently active protocol).

Note:

  The rest of this page assumes that you have launched a local node,
  as explained in the previous section. But it is useful to know that
  the client can be configured to interact with a public node instead,
  either using the configuration file or by supplying option "-E
  <node-url>" with a public RPC node.

After starting your local node you can check if it has finished
synchronizing (see Synchronisation heuristic) using:

   octez-client bootstrapped

This call will hang and return only when the node is synchronized
(recall that this is much faster when starting a node from a
snapshot). Once the above command returns, we can check what is the
current timestamp of the head of the chain (time is in UTC so it may
differ from your local time):

   octez-client get timestamp

You can also use the above command before the node is bootstrapped,
from another terminal. However, recall that the commands available on
the client depend on the specific protocol run by the node. For
instance, "get timestamp" isn't available when the node runs the
genesis protocol, which may happen for a few minutes when launching a
node for the first time.

The behaviour of the client can be customized using various
mechanisms, including command-line options, a configuration file, and
environment variables. For details, refer to the client manual and
client set-up instructions.


A Simple Wallet
---------------

The client is also a basic wallet. We can, for example, generate a new
pair of keys, which can be used locally with the alias *alice*:

   $ octez-client gen keys alice

To check the account (also called a contract) for Alice has been
created:

   $ octez-client list known contracts

You will notice that the client data directory (by default, "~/.tezos-
client") has been populated with 3 files "public_key_hashs",
"public_keys" and "secret_keys". The content of each file is in JSON
and keeps the mapping between aliases (e.g., "alice") and the kind of
keys indicated by the name of each file. Secret keys should be stored
on disk encrypted with a password except when using a hardware wallet
(see Ledger support). An additional file "contracts" contains the
addresses of smart contracts, which have the form *KT1…*.

Secret keys are stored by default unencrypted on testnets (as can be
seen here). On mainnet, the default is to store keys encrypted,
because this is the recommended practice for most use cases, except
when using a hardware wallet (see Ledger support).

If you want to force using encryption on a testnet, you must supply
the option "--encrypted" when generating a new account:

   $ octez-client gen keys bob --encrypted

Tezos supports four different ECC (Elliptic-Curve Cryptography)
schemes: *Ed25519*, *secp256k1* (the one used in Bitcoin), *P-256*
(also called *secp256r1*), and *BLS* (variant *MinPk*, for aggregated
signatures). The secp256k1 and P256 curves have been added for
interoperability with Bitcoin and Hardware Security Modules (*HSMs*)
mostly. Unless your use case requires those, you should probably use
*Ed25519*. We use a verified library for Ed25519, and it is generally
recommended over other curves by the crypto community, for performance
and security reasons.

Make sure to make a back-up of the client data directory and that the
password protecting your secret keys is properly managed (if you
stored them encrypted).

For more advanced key management we offer ledger support and a remote
signer.


Get Free Test Tokens
--------------------

To test the networks and help users get familiar with the system, on
test networks you can obtain free tokens from a faucet. Transfer some
to Alice's address.


Transfers and Receipts
----------------------

To fund our newly created account for Bob, we need to transfer some
tez using the *transfer* operation. Every operation returns a
*receipt* that recapitulates all the effects of the operation on the
blockchain. A useful option for any operation is "--dry-run", which
instructs the client to simulate the operation without actually
sending it to the network, so that we can inspect its receipt.

Let's try:

   octez-client transfer 1 from alice to bob --dry-run

   Fatal error:
     The operation will burn 0.257 tez which is higher than the configured burn cap (0 tez).
      Use `--burn-cap 0.257` to emit this operation.

The client asks the node to validate the operation (without sending
it) and obtains an error. The reason is that when we fund a new
address we are also storing it on the blockchain. Any storage on chain
has a cost associated to it which should be accounted for either by
paying a fee to a baker or by destroying ("burning") some tez. This is
particularly important to protect the system from spam. Because
storing an address requires burning 0.257 tez and the client has a
default of 0, we need to explicitly set a cap on the amount that we
allow to burn:

   octez-client transfer 1 from alice to bob --dry-run --burn-cap 0.257

This should do it and you should see a rather long receipt being
produced, here's an excerpt:

   ...
   Simulation result:
     Manager signed operations:
       From: tz1RjtZUVeLhADFHDL8UwDZA6vjWWhojpu5w
       Fee to the baker: ꜩ0.001259
       ...
       Balance updates:
         tz1RjtZUVeLhADFHDL8UwDZA6vjWWhojpu5w ............ -ꜩ0.001259
         fees(tz1Ke2h7sDdakHJQh8WX4Z372du1KChsksyU,72) ... +ꜩ0.001259
       Revelation of manager public key:
         Contract: tz1RjtZUVeLhADFHDL8UwDZA6vjWWhojpu5w
         Key: edpkuK4o4ZGyNHKrQqAox7hELeKEceg5isH18CCYUaQ3tF7xZ8HW3X
         ...
     Manager signed operations:
       From: tz1RjtZUVeLhADFHDL8UwDZA6vjWWhojpu5w
       Fee to the baker: ꜩ0.001179
       ...
       Balance updates:
         tz1RjtZUVeLhADFHDL8UwDZA6vjWWhojpu5w ............ -ꜩ0.001179
         fees(tz1Ke2h7sDdakHJQh8WX4Z372du1KChsksyU,72) ... +ꜩ0.001179
       Transaction:
         Amount: ꜩ1
         From: tz1RjtZUVeLhADFHDL8UwDZA6vjWWhojpu5w
         To: tz1Rk5HA9SANn3bjo4qMXTZettPjjKMG14Ph
         ...
         Balance updates:
           tz1RjtZUVeLhADFHDL8UwDZA6vjWWhojpu5w ... -ꜩ1
           tz1Rk5HA9SANn3bjo4qMXTZettPjjKMG14Ph ... +ꜩ1
           tz1RjtZUVeLhADFHDL8UwDZA6vjWWhojpu5w ... -ꜩ0.257

The client does a bit of magic to simplify our life and here we see
that many details were automatically set for us. Surprisingly, our
transfer operation resulted in **two** operations, first a
*revelation*, and then a transfer. Alice's address, obtained from the
faucet, is already present on the blockchain, but only in the form of
a *public key hash* "tz1Rj...5w". To sign operations, Alice needs to
first reveal the *public key* "edpkuk...3X" behind the hash, so that
other users can verify her signatures. The client is kind enough to
prepend a reveal operation before the first transfer of a new address,
this has to be done only once, future transfers will consist of a
single operation as expected.

Another interesting thing we learn from the receipt is that there are
more costs being added on top of the transfer and the burn: *fees*. To
encourage a baker to include our operation, and in general to pay for
the cost of running the blockchain, each operation usually includes a
fee that goes to the baker. Fees are variable over time and depend on
many factors but the Octez client selects a default for us.

The last important bit of our receipt is the balance updates that
resume which address is being debited or credited a certain amount. We
see in this case that baker "tz1Ke...yU" is being credited one fee for
each operation, that Bob's address "tz1Rk...Ph" gets 1 tez and that
Alice pays the transfer, the burn, and the two fees.

Now that we have a clear picture of what we are going to pay we can
execute the transfer for real, without the "dry-run" option. You will
notice that the client hangs for a few seconds before producing the
receipt because after injecting the operation in your local node it is
waiting for it to be included by some baker on the network. Once it
receives a block with the operation inside it will return the receipt.

It is advisable to wait for several blocks to consider the transaction
as final. Please refer to the consensus algorithm documentation and
analysis to better understand block finality in Tezos. This page
provides concrete values for the number of blocks one should wait.

In the rare case when an operation is lost, how can we be sure that it
will not be included in any future block, and then we may re-emit it?
After 120 blocks a transaction is considered invalid and can't be
included anymore in a block. Furthermore each operation has a counter
that prevents replays so it is usually safe to re-emit an operation
that seems lost.


Block Explorers
---------------

Once your transaction is included in a block, you can retrieve it in
one of the public block explorers, which list the whole history of the
different Tezos networks (mainnet or test networks).


User Accounts and Smart Contracts
---------------------------------

In Tezos there are two kinds of accounts: *user accounts* (also called
implicit accounts) and *smart contracts* (also called originated
accounts), see Accounts and addresses for more details.

* Addresses with a *tz* prefix, like the *tz1* public key hashes used
  above, represent user accounts. They are created with a transfer
  operation to the account's public key hash.

* Smart contracts have addresses starting with *KT1* and are created
  with an origination operation. They don't have a corresponding
  secret key and they run Michelson code each time they receive a
  transaction.

Let's originate our first contract and call it *id*:

   octez-client originate contract id transferring 1 from alice \
                running ./michelson_test_scripts/attic/id.tz \
                --init '"hello"' --burn-cap 0.4

The initial balance is 1 tez, generously provided by user account
*alice*. The contract stores a Michelson program "id.tz" (found in
file michelson_test_scripts/attic/id.tz), with Michelson value
""hello"" as initial storage (the extra quotes are needed to avoid
shell expansion). The parameter "--burn-cap" specifies the maximal fee
the user is willing to pay for this operation, while the actual fee is
determined by the system.

A Michelson contract is expressed as a pure function, mapping a pair
"(parameter, storage)" to a pair "(list_of_operations, storage)".
However, when this pure function is applied to the blockchain state,
it can be seen as an object with a single method taking one parameter
("parameter"), and with a single attribute ("storage"). The method
updates the state (the storage), and submits operations as a side
effect.

For the sake of this example, here is the "id.tz" contract:

   parameter string;
   storage string;
   code {CAR; NIL operation; PAIR};

It specifies the types for the parameter and storage, and implements a
function which updates the storage with the value passed as a
parameter and returns this new storage together with an empty list of
operations.


Gas and Storage Costs
---------------------

A quick look at the balance updates on the receipt shows that on top
of funding the contract with 1 tez, *alice* was also charged an extra
cost that is burnt. This cost comes from the *storage* and is shown in
the line "Paid storage size diff: 46 bytes", 41 for the contract and 5
for the string ""hello"". Given that a contract saves its data on the
public blockchain that every node stores, it is necessary to charge a
fee per byte to avoid abuse and encourage lean programs.

Let's see what calling a program with a new argument would look like
with the "--dry-run" option:

   octez-client transfer 0 from alice to id --arg '"world"' --dry-run

The transaction would successfully update the storage but this time it
wouldn't cost us anything more than the fee, the reason is that the
storage for ""world"" is the same as for ""hello"", which has already
been paid for. To store more we'll need to pay more, you can try by
passing a longer string.

The other cost associated with running contracts is the *gas*, which
measures *how long* a program takes to compute. Contrary to storage
there is no cost per gas unit, a transfer can require as much gas as
it wants, however a baker that has to choose among several
transactions is much more likely to include a low gas one because it's
cheaper to run and validate. At the same time, bakers also give
priority to high fee transactions. This means that there is an
implicit cost for gas that is related to the fee offered versus the
gas and fees of other transactions.

If you are happy with the gas and storage of your transaction you can
run it for real, however it is always a good idea to set an explicit
limit for both. The transaction fails if any of the two limits are
passed. Note that the storage limit sets an upper bound to the storage
size *difference*, so in our case, it may be 0 because our new value
does not increase at all the storage size.

   octez-client transfer 0 from alice to id --arg '"world"' \
                                            --gas-limit 11375 \
                                            --storage-limit 0

A baker is more likely to include an operation with lower gas and
storage limits because it takes fewer resources to execute so it is in
the best interest of the user to pick limits that are as close as
possible to the actual use. In this case, you may have to specify some
fees (using option "--fee") as the baker is expecting some for the
resource usage. Otherwise, you can force a low fee operation using the
"--force-low-fee", with the risk that no baker will include it.

More Michelson test scripts can be found in directory
michelson_test_scripts/. Advanced documentation of the smart contract
language is available here.


Validation
----------

The node allows validating an operation before submitting it to the
network by simply simulating the application of the operation to the
current context. Without this mechanism, if you just send an invalid
operation (e.g. sending more tokens than you own), the node would
broadcast it and when it is included in a block you would have to pay
the usual fee even if it won't have an effect on the context. To avoid
this case the client first asks the node to validate the transaction
and only then sends it.

The same validation is used when you pass the option "--dry-run": the
receipt that you see is actually a simulated one. The only difference
is that, when this option is supplied, the transaction is not sent
even if it proves to be valid.

Another important use of validation is to determine gas and storage
limits. The node first simulates the execution of a Michelson program
and tracks the amount of gas and storage that has been consumed. Then
the client sends the transaction with the right limits for gas and
storage based on those indicated by the node. This is why we were able
to submit transactions without specifying these limits: they were
computed for us.

More information on validation can be found here.


It's RPCs all the Way Down
--------------------------

The client communicates with the node uniquely through RPC calls so
make sure that the node is listening on the right ports and that the
ports are open. For example the "get timestamp" command above is a
shortcut for:

   octez-client rpc get /chains/main/blocks/head/header/shell

The client tries to simplify common tasks as much as possible, however
if you want to query the node for more specific information you'll
have to resort to RPCs.

For example to check the value of important constants in Tezos, which
may differ between Mainnet and other test networks, you can use:

   octez-client rpc get /chains/main/blocks/head/context/constants | jq
   {
     "proof_of_work_nonce_size": 8,
     "nonce_length": 32,
     ...
   }

Another interesting use of RPCs is to inspect the receipts of the
operations of a block:

   octez-client rpc get /chains/main/blocks/head/operations

It is also possible to review the receipt of the whole block:

   octez-client rpc get /chains/main/blocks/head/metadata

An interesting block receipt is the one produced at the end of a cycle
as many delegates receive back part of their unfrozen accounts.

You can find more info on RPCs in the RPCs' page.


Other binaries
==============

In this short tutorial we will not use some other binaries, but let's
briefly review their roles.


Admin Client
------------

The admin client enables you to interact with the peer-to-peer layer
in order to:

* check the status of the connections

* force connections to known peers

* ban/unban peers

A complete manual page of admin client is available here.

A useful command to debug a node that is not syncing is:

   octez-admin-client p2p stat

The admin client uses the same format of configuration file as the
client (see Client configuration file).


Codec
-----

The Octez codec ("octez-codec") is a utility that:

* provides documentation for all the encodings used in the "octez-
  node" (and other binaries), and

* allows to convert from JSON to binary and vice-versa for all these
  encodings.

It is meant to be used by developers for tests, for generating
documentation when writing libraries that share data with the node,
for light scripting, etc. For more details on its usage, refer to its
online manual and to Encodings.


Protocol compiler
-----------------

The protocol compiler ("octez-protocol-compiler") can compile
protocols within the limited environment that the shell provides. This
environment is limited to a restricted set of libraries in order to
constrain the possible behavior of the protocols.

It is meant to be used:

* by developers to compile the protocol under development,

* by the packaging process to compile protocols that are pre-linked in
  the binaries,

* by the Octez node when there is an on-chain update to a protocol
  that is not pre-linked with the binary.


Summary
=======

In this tutorial, you have learned:

* to start an Octez node and set up its basic configuration;

* to use the Octez client to create user accounts and do transfers
  between them;

* to deploy and interact with a simple predefined smart contract;

* to distinguish between the various costs associated to transactions
  such as burnt tez, fees, storage costs, and gas consumption;

* some further concepts such as transaction validation and the RPC
  interface;

* the role of other binaries, less frequently used than the client and
  the node.

You may now explore Tezos further, and enjoy using it!

Running Octez
*************

In this section, we discuss how to take part in the protocol that runs
the network. There are three main ways to participate: delegating your
coins, staking, and running a delegate. The main advantage of
delegating your coins is simplicity. Staking your coins is also as
simple, but significantly increases the potential rewards modulo some
slightly increased risks. The third way allows to participate more
actively in the protocol, by baking blocks and voting, but is more
demanding; however, the extra effort is compensated by more rewards in
tez.

To learn more about the protocol refer to this page.

No matter how you decide to run Octez, your node must have an accurate
time source and be properly synchronized to it, e.g. by configuring an
NTP daemon. This is especially important for bakers, as baking nodes
desynchronized from the correct time of day have caused operational
problems in the past by "baking in the future".


Delegating your coins
=====================

If you don't want to deal with the complexity of running your own
delegate, you can always take part in the protocol by delegating your
coins to one.

Both user accounts and smart contracts can have a delegate. Setting or
resetting the delegate of a user account is achieved by the following
command:

   octez-client set delegate for <user_account> to <delegate>

where "<user_account>" is the address or alias of the user account to
delegate and "<delegate>" is the address or alias of the delegate
(which has to be registered).

To stop a delegation, the following command can be used:

   octez-client withdraw delegate from <user_account>

Smart contract can also delegate the tokens they hold to registered
delegates. The initial delegate of a smart contract can be set at
origination using the "--delegate" option:

   octez-client originate contract <contract_alias> transferring <initial_balance> from <originator> running <script> --delegate <delegate> --burn-cap <cap>

Once the contract is originated, the only way to stop or modify its
delegation is by using the "SET_DELEGATE" Michelson instruction (see
the Michelson documentation for more details).

Notice that only user accounts can be delegates, so your delegate must
be a *tz* address.

Funds in user accounts which are not registered as delegates do not
participate in baking.

Note that delegating coins doesn't mean that a delegate can spend
them, they only add to its delegated balance. In turn, delegators can
freely spend their own funds in spite of the active delegation (they
are not locked, like in other PoS algorithms). Technically, delegation
is a link between a delegator account and a delegate account, meaning
that *all* the funds of the former are delegated to the latter, until
the delegation is withdrawn. When a delegator spends their tokens, the
delegated balance of their delegate decreases; conversely, when they
receive tokens the delegated balance of their delegate increases.

Since the activation of the new staking mechanism, 33% of your total
funds count towards your delegator's baking power (before the staking
mechanism, 100% of your total funds counted towards your delegator
baking power). The delegated funds still count for 100% for the voting
power, just like before the activation of the new staking mechanism.


Staking your coins
==================

Since the activation of the new staking mechanism, if you don't want
to deal with the complexity of running your own delegate, you can also
take part in the protocol by staking part of your coins to accrue to
your delegate’s staking power.

Only user accounts (not smart contracts) can stake funds.

Funds can only be staked to your current delegate. Therefore, staking
requires previously declaring a delegate. Setting or resetting the
delegate of a user account is done as for delegating, see above.

The delegate has no control over your stake: cannot spend it nor
unstake it, so your deposit remains yours and under your control.
Staked coins cannot be spent by yourself, either; but you can unstake
them later. Your staked funds add to your delegate's staking balance
(so staked funds count twice as much as delegated funds). In turn,
your staked funds are frozen and subject to slashing in case of
misbehavior of your delegate such as double signing, similarly to the
delegate's own staked funds. In compensation of this risk assumed by
stakers, they may receive some part of the delegate's rewards, the
exact proportion being settable by the delegate.

As opposed to delegation (where you always delegate all your funds),
as a staker you can choose a specific amount from your balance to
stake, as follows:

   octez-client stake <amount> for <staker>

Later on, you can unstake part or all of your staked funds:

   octez-client unstake <amount|"everything"> for <staker>

The requested amount will immediately be unstaked, however it will
remain frozen for some period (4 cycles), and only after that the
funds are said to be both unstaked and finalizable. At that point, the
staker can make them spendable again as follows:

   octez-client finalize unstake for <staker>

For more details on the staking interface, see Staked funds
management.


Running a delegate
==================

A delegate is responsible for baking blocks, attesting blocks and
accusing other delegates in case they try to double bake or double
attest. A delegate is also responsible for taking part in the
governance process.

Rights for baking and attesting are randomly assigned to delegates
proportionally to their baking power, which usually is their own
staked funds plus the funds staked by external stakers, plus the third
of their total delegation.

Delegates are required to freeze some of their funds into a security
deposit. This can be done via the same commands used by external
stakers in the previous section.


Register and check your rights
------------------------------

To run a delegate, you first need to register as one using your user
account:

   octez-client register key mybaker as delegate

Note:

  It is possible to also register a consensus key and/or a companion
  key at this step.

You also need to stake some tez, at least "MINIMAL_FROZEN_STAKE" (see
Proof-of-stake parameters), and to have at least "MINIMAL_STAKE =
6000" baking power, taking into account your own and all your
delegators' staked balances, as well as their delegated balances with
a lesser weight.

On testnets, when you obtain coins from a faucet, if you are lucky to
obtain more than the minimum required to be a delegate, you can stake
the minimum amount by yourself and keep the rest for fees. Otherwise,
you need to ask the faucet for more accounts and delegate them to the
first.

Most commonly, you would stake (at least) the needed amount from your
own tez, without waiting for delegators:

   octez-client stake 6000 for mybaker

Once you registered and staked tez, you need to wait the end of the
current cycle plus "CONSENSUS_RIGHTS_DELAY = 2" cycles, for your
rights to be considered.

There is an RPC that can be used to check your rights for every cycle,
up to 2 cycles in the future.

   octez-client rpc get /chains/main/blocks/head/helpers/baking_rights\?cycle=300\&delegate=tz1_xxxxxxxxxxx\&max_round=2

Sometimes there is no consensus at a round, so it is worth considering
also baking rights at higher rounds, like 2 in the example above.


Configure the staking policy
----------------------------

A delegate may choose to accept (or not) staked funds from external
stakers. Both the delegate's own stake and the stake from external
stakers can be slashed (that is, partially lost), when the delegate
misbehaves (see Accuser).

Delegates can set two parameters by configuring their staking policy:

* the maximum ratio of external stake over their own stake: a factor
  between 0 and 9, by default 0, which means that:

  * for any factor *f*, the delegate accepts *f* times its own stake
    from external stakers

  * by default, delegates don't allow external staking

* the proportion of rewards kept by the delegator (the rest being paid
  to external stakers): a factor between 0 and 1, by default 1.

These parameters are configured as follows:

   octez-client set delegate parameters for  <delegate> --limit-of-staking-over-baking <value> --edge-of-baking-over-staking <value>

If the delegated funds exceed 9 times the delegate’s own stake, the
delegate is *overdelegated*. If the staked funds from external stakers
exceed the proportion defined by the delegate, the delegate is
*overstaked*. See details and consequences in Staking policy
configuration.


Baker
-----

The baker is a daemon that executes Tezos' consensus algorithm. The
baker runs on behalf of one or more specified accounts or, if none is
specified, on behalf of all accounts whose secret keys are known.

To learn more about how the Octez baker works see The Octez Baker.
Also, a complete manual page of the baker executable is available
here.

During its run, the baker bakes blocks (by selecting transactions from
the mempool and arranging them in a new block) and emits consensus
operations like attestations. It does so whenever the associated
accounts have the necessary rights. In our case, we already set up and
funded a baker account called "mybaker".

For its operation, the baker daemon needs to communicate with an Octez
node using RPCs. Therefore, you must indicate a node endpoint whose
RPC interface is available. The most common way is to run a node
locally, ensuring its RPC port is open for local calls (e.g. "--rpc-
addr 127.0.0.1").

Bakers are also supposed to run a DAL node before running the baker
daemon, to expand the amount of data that Tezos can distribute without
causing congestion on layer 1. Therefore, first start a DAL node.

Typically, you run an attester DAL node as follows:

   octez-dal-node run --endpoint <node-endpoint> --attester-profiles=<baker-addr>

Once the DAL node runs, we can launch the baker daemon pointing to the
standard node directory and baking for user "mybaker":

   octez-baker-<PROTO_HASH> run with local node ~/.tezos-node mybaker --liquidity-baking-toggle-vote pass

where "PROTO_HASH" is the short hash of the current protocol of the
network you want to bake on.

Note that:

* the baker needs direct access to the node data directory for
  performance reasons (to reduce the number of RPC calls to the node).

* since version 13.0, option "--liquidity-baking-toggle-vote" is
  mandatory, see the changelog.

* option "--liquidity-baking-toggle-vote" must be placed **after**
  "run" on the command-line.

* starting with version 22.0 it is recommended to run the Octez node
  with a DAL node (opting out requires the explicit option "--without-
  dal").

* starting with version 23.0, and the activation of protocol S, when
  running a baker with the DAL activated, and a tz4 consensus key, it
  is required to have also a tz4 companion key registered for the
  delegate to produce DAL attestations. Otherwise the baker will only
  produce regular attestations, without any DAL information, and the
  baker will be seen as not participating in the DAL.


Quickstart baker
----------------

Putting together all the above instructions, you may want to quickly
start a baker daemon on a testnet such as ghostnet as follows (after
starting an Octez node):

   octez-client gen keys mybaker
   BAKER_ADDRESS=$(octez-client show address mybaker | grep Hash | cut -d' ' -f2)
   # Fund BAKER_ADDRESS with > 6000 tez, e.g. at https://faucet.ghostnet.teztnets.com
   octez-client register key mybaker as delegate
   octez-client stake 6000 for mybaker
   octez-dal-node run --endpoint http://127.0.0.1:8732 --attester-profiles=$BAKER_ADDRESS
   octez-baker-PsRiotum run with local node $HOME/.tezos-node mybaker --liquidity-baking-toggle-vote pass --dal-node http://127.0.0.1:10732

Warning:

  **Remember that having two bakers running connected to the same
  account could lead to double baking/attesting and the loss of all
  your bonds.** If you are worried about the availability of your node
  when it is its turn to bake/attest, there are other ways than
  duplicating your credentials (see the discussion in section Inactive
  delegates). **Never** use the same account on two daemons.

However, it is safe (and actually necessary) to temporarily run two
bakers just before a protocol activation: the baker for the protocol
being replaced and the baker for the protocol to be activated.

Note:

  It is possible to bake and attest using a dedicated Consensus key
  instead of the delegate's key.

The baker uses the same format of configuration file as the client
(see Client configuration file).

Warning:

  When running a baker, it is recommended to carefully save the nonces
  generated by the baker as part of the consensus protocol, to be able
  to reveal the nonces before the end of the cycle even if the baker
  is restarted (e.g., on another machine), so as to avoid losing
  attestation rewards.


Inactive delegates
------------------

If a delegate doesn't show any sign of activity for
TOLERATED_INACTIVITY_PERIOD cycles, it is marked **inactive** and its
rights are removed. This mechanism is important to remove inactive
delegates and reallocate their rights to the active ones so that the
network is always working smoothly. Normally even a baker with the
minimal stake should perform enough operations during this period to
remain active. If for some reason your delegate is marked inactive you
can reactivate it simply by re-registering again like above.

To avoid your Tezos delegate being marked inactive while pausing it
for maintenance work, it is advised to check the schedule of future
baking and attesting slots assigned to it, using a Tezos block
explorer. Alternatively, you may use the baking rights RPC and the
attesting rights RPC (see RPCs - OpenAPI reference), which is able to
return a list of baking/attesting slots for a given delegate (see
example).


Accuser
-------

The accuser is a daemon that monitors all blocks received on all
chains and looks for:

* bakers who signed two blocks at the same level and the same round

* bakers who injected more than one pre-attestations or attestation
  operation for the same level and round (more details here)

Upon finding such irregularity, it will emit respectively a double-
baking, double-pre-attesting, or double-attesting denunciation
operation, which will cause the offender to be slashed, that is, to
lose part of its security deposit.

   octez-accuser-alpha run

The accuser uses the same format of configuration file as the client
(see Client configuration file). A complete manual page of the accuser
is available here.


DAL node
--------

To add a Data Availability Layer (DAL) node to support data
transmission across the network, see DAL layer.


Docker
------

If you are running the baker Docker image, you can watch the baker
logs with "docker logs". First, find the name of your container with:

   docker ps

If your container is running, its name will appear in the last column.
For instance, if the name is "mainnet_baker-PsQuebec", you can view
recent logs with:

   docker logs mainnet_baker-PsQuebec

If you want to keep watching logs, use "-f":

   docker logs mainnet_baker-PsQuebec -f

This allows you to know if you baked. You should see lines such as:

   Injected block BLxzbB7PBW1axq for bootstrap5 after BLSrg4dXzL2aqq  (level 1381, slot 0, fitness 00::0000000000005441, operations 21)

Setting up Octez Services
*************************

The Octez suite consists of several executables, some to be run
interactively, while others are to be run as daemons.

Previous tutorials in this section showed how to get started with the
different executables, and different options for participating to
Tezos such as delegating, staking, and baking. However, in these
tutorials, daemons are just run in background or left running in
another terminal.

This page shows how Octez daemons can be safely run from the official
binary packages, as Unix services, which can ensure that they are
started automatically and restarted in case of failures.

This is a intermediate-level baking howto assuming some familiarity
with the previous tutorials mentioned above, on installing, on setting
up Octez, and on the notion of baking. For a more accessible and
detailed step-by-step tutorial on running an Octez baker, including
the complete set up of a baking account and thorough manual
verifications upon each step, see Run a Tezos node in 5 steps.


Installing Octez
================

First of all, you must install Octez from binary packages, as
explained in the installation tutorial.

To make things concrete, we are assuming here that you installed Octez
Debian packages, but most commands should apply to other
distributions.


Verify Installation
-------------------

After installation, verify that Octez components are installed
correctly:

   octez-node --version
   octez-client --version

These binaries can be used by any user. However, to run Octez in
production our packages set up a special user named "tezos" to run all
daemons via "systemd" and without direct user intervention.


Configuring the node
====================

Octez packages provide a very basic configuration in
"/etc/default/octez-*" files. These are to be read as configuration
files for the "systemd" services. We took a non opinionated approach
regarding the basic configuration provided by default. We do not make
assumptions on the setup of the machine or try to imagine all possible
use cases. To create more advanced configurations, the user can
directly configure each daemon using commands such as:

   sudo su tezos -c "octez-node config ..."

For more details on configuring the Octez node, see Setting up the
node.

The default for the octez-node is to connect to "mainnet". You may
want to connect to a test network if your goal is learning,
developing, or testing.


Running the Octez node
----------------------

Once the node is configured, we can use "systemd" to start the daemon:

   sudo systemctl start octez-node

If configured to do so during package installation, this will
automatically download a snapshot and import it before starting the
"octez-node" service (starting with v22 packages). In this case, the
command can take quite a few minutes to execute, for downloading and
importing the snapshot.

We can check the status of the daemon in the logs of the node that are
stored by default in "/var/log/tezos/node.log". Logs are automatically
rotated using "logrotate".

If you need to relocate the datadir of the octez-node, you would first
need to change the home directory of the "tezos" user with the command
"usermod -m -d /custom tezos" and then configure the "DATADIR"
variable in "/etc/default/octez-node" accordingly.


Setting up baking keys
======================

The most important preliminary step for running a baker is setting up
a baker account having enough baking power (typically, possessing more
than 6000 tez). We don't cover here the optional set up of an
associated consensus key and/or companion key.

If you intend to bake on a testnet, you can simply create a key as
follows, fund it, and configure the DAL node to use it:

   $ sudo su tezos -c "octez-client gen keys mybaker"
   $ sudo su tezos -c "octez-client show address mybaker"
   Hash: tz1Ti8WHvfp3XKsTCKWLFv4TrER2HDofXG39
   Public Key: edpkuGRKH8oZDP2PH2EULw9PGzJHdf5g2zCiHnaYG7tapeePUNAWeC

   $ ... # Fund mybaker with > 6000 tez, e.g. at https://faucet.ghostnet.teztnets.com
   $ sudo su tezos -c "octez-client register key mybaker as delegate"
   $ sudo su tezos -c "octez-client stake 6000 for mybaker"
   $ sudo su tezos -c "octez-dal-node config init --endpoint http://127.0.0.1:8732 --attester-profiles=tz1Ti8WHvfp3XKsTCKWLFv4TrER2HDofXG39"

The baker will use the baking key automatically. Indeed, the baker
bakes for all the keys for which it has the private key. If you want
to avoid this behavior, you can specify a specific baking key by
editing the file /etc/default/octez-baker and assigning a value to
variable BAKER_KEY.

When baking on a testnet, you may skip the following section on
configuring the signer, and directly start the baker.


Configuring the signer
----------------------

If you envision to bake on mainnet some day, we highly recommend to
train yourself using the Octez signer, because of the sensitive nature
of the private keys needed by the baker to function (note that we
still show the instructions for a testnet in this training).

To configure the octez-signer, first, logged as the user chosen to run
the signer, we must create a set of keys. These are the private keys
that will be entrusted to the signer to actually sign operations on
behalf of the baker. The signer will run in a different process
(possibly on a separate host), and ideally using a hardware enclave
such as a hardware ledger. For the sake of brevity, in this example,
the keys will be simply stored on the disk, but **this is not a
recommended setting for a production baker**.

We create an authentication key that is going to be used to
authenticate the baker with the signer, and a signing key to sign the
operations.

The signer secret key is stored in the current user directory and we
will configure the baker using the "tz1" address for this key.

The signer authentication key is stored in the "tezos" user space and
we will configure the signer using the public key associated to the
auth key.

   # create a signing key ( as current user )
   $ octez-signer gen keys mybaker

   # create an authentication key for signer authorization
   $ sudo su tezos -c "octez-client gen keys auth"

   $ sudo su tezos -c "octez-client show address auth"
   Hash: tz1V7TgBR52wAjjqsh24w8y9CymFGdegt9qs
   Public Key: edpk123456789....

   # add the auth key to the octez-signer. This is the default
   # options set in the octez-signer service file
   $ octez-signer add authorized key edpk123456789... --name auth

Now we need to configure the "octez-signer" service. We use again
"systemd" and we run it as a user service. The "octez-signer" service
file can be customized by the user if needed to allow for more complex
and secure scenarios.

   # customize the octez-signer service if needed
   $ mkdir -p ~/.config/systemd/user/
   $ cp /usr/share/doc/octez-signer/octez-signer.service \
        ~/.config/systemd/user/

   # start the octez-signer service
   $ systemctl --user start octez-signer

   # examine the logs
   $ journalctl --user-unit octez-signer

For more advanced configurations, see the signer guide.

Now that the signer is running, we need to fund the baking address and
configure the baker and the DAL node to use it. Since the baker runs
as the user "tezos", we use "sudo su tezos -c" to wrap the
configuration commands below:

   # Get the tz1 address of our signing key
   $ octez-signer show address mybaker
   Hash: tz1V7TgBR52wAjjqsh24w8y9CymFGdegt9qs
   Public Key: edpkvGAz71r8SZomcvF7LGajXT3AnhYX9CrmK3JWgA2xk8rf8CudY8

   # Configure the baker to use the remote signer
   $ sudo su tezos -c "octez-client -R tcp://localhost:7732 \
      import secret key mybaker remote:tz1V7TgBR52wAjjqsh24w8y9CymFGdegt9qs"
   $ ... # Fund mybaker with > 6000 tez, e.g. at https://faucet.ghostnet.teztnets.com
   $ sudo su tezos -c "octez-client -R tcp://localhost:7732 register key mybaker as delegate"
   $ sudo su tezos -c "octez-client -R tcp://localhost:7732 stake 6000 for mybaker"
   $ sudo su tezos -c "octez-dal-node config init --endpoint http://127.0.0.1:8732 --attester-profiles=tz1V7TgBR52wAjjqsh24w8y9CymFGdegt9qs"


Starting the baker
==================

Now that everything is in place, we can start the Octez baker.

   sudo systemctl start octez-baker

This service will automatically start all accusers and bakers for all
protocols shipped with the package. It will also start the DAL node
alongside the baker, which is the recommended setting for all bakers.
The DAL node is run in a default configuration, that is, in controller
mode without any profile; you may edit file "/etc/default/octez-dal-
node" or use commands such as "sudo su tezos -c "octez-dal-node config
..."" to set up another configuration, refer to page The DAL node.

The logs of the baker are available in "/var/log/tezos/baker-<protocol
name>.log".


Upgrading Octez
===============

To upgrade Octez to the latest version, see the corresponding section
in the installation guide.

In our case, we can simply proceed as follows:

   sudo apt-get update
   sudo apt-get upgrade octez-node octez-client octez-baker

When necessary, the upgrade scripts will make the user aware of
breaking changes and required actions such as new configuration
parameters or changes in governance.

Mind reloading the new services using "sudo systemctl daemon-reload"
and then restarting the running services using "sudo systemctl restart
<service>". When doing so:

* Always make sure that all binaries or packages are on the same
  version of Octez.

* The restart order should follow the dependency order: (i) Octez
  node, (ii) DAL node, (iii) baker, (iv) accuser (if running one).

Migrating from Serokell's to Nomadic Labs' packages
***************************************************

This guide explains how to migrate your Octez setup from the legacy
packages provided by Serokell to the packages provided by Nomadic Labs
(abbreviated as NL packages hereafter).


Why Backups Are Crucial
=======================

The migration involves removing old packages, relocating data, and
reconfiguring services. Mistakes during this process can result in
irreversible data loss. **Always back up your node and client data
before proceeding.**


How This Guide is Structured
============================

1. **Step-by-Step Procedure**: A simple guide for users with standard
   setups. Follow this if you just want instructions.

2. **Context and Explanations**: Additional details to understand the
   migration process or adapt it to customized setups.


Step-by-Step Procedure
======================


1. Backup Data
--------------

Save a complete backup of your node and client data:

   sudo cp -a /var/lib/tezos /backup/tezos-data


2. Remove Serokell Packages
---------------------------

Uninstall Serokell packages while preserving data:

   sudo apt autoremove tezos-client tezos-node tezos-baking

If you were using RPM packages, use "dnf" instead of "apt".


3. Add NL Repository
--------------------

**If using Debian**, add the NL package repository and install the
"octez-node" package:

   export distribution=debian
   export release=bookworm

   sudo apt-get install -y gpg curl


     curl -s "https://packages.nomadic-labs.com/$distribution/octez.asc" |
       sudo gpg --dearmor -o /etc/apt/keyrings/octez.gpg
     echo "deb [signed-by=/etc/apt/keyrings/octez.gpg] https://packages.nomadic-labs.com/$distribution $release main" |
       sudo tee /etc/apt/sources.list.d/octez.list
     sudo apt-get update


4. Install NL Packages
----------------------

**If using Debian packages**, install them so:

   sudo apt install octez-node

When prompted, answer **Yes** to skip automatic configuration.

If needed, install the Dialog interface with "sudo apt install
dialog".

**If using RPM packages** rather then Debian ones, refer to RPM Octez
packages.


5. Relocate Data
----------------

The default data directory for NL packages is "/var/tezos" . To use a
different location, change the value of the "DATADIR" variable in
"/etc/default/octez-node", and relocate the home of the "tezos" user
using the command "usermod -m -d /custom tezos". We recommend to keep
the default.

You can copy your node and client data to the new directory structure:

   sudo mkdir -p /var/tezos
   sudo cp -a /var/lib/tezos/<network> /var/tezos/.tezos-node
   sudo cp -a /var/lib/tezos/.tezos-client /var/tezos/

where "<network>" is the network you choose for running Octez.


6. Update User Configuration
----------------------------

Change the "tezos" user’s home directory and shell:

   sudo usermod -d /var/tezos tezos
   sudo chsh -s /bin/bash tezos


7. Start Node Service
---------------------

First, check the configuration file inherited from the old packages:

   sudo -u tezos octez-node config show

If the RPC port is not defined (no field "listen-addrs"), open it,
e.g.:

   sudo -u tezos octez-node config update --rpc-addr=127.0.0.1:8732

Now enable and start the "octez-node" service:

   sudo systemctl enable octez-node
   sudo systemctl start octez-node

You may then inspect its trace with:

   sudo tail -f /var/log/tezos/node.log


8. Migrate the Baker
--------------------

Install the baker:

   sudo apt install octez-baker

If you want to install RPM packages rather then Debian ones, refer to
RPM Octez packages.

Then enable and start the baking service:

   sudo systemctl enable octez-baker-active
   sudo systemctl start octez-baker-active

You may then inspect its trace with:

   sudo tail -f /var/log/tezos/baker-active.log


Context and Explanations
========================

The migration transitions from Serokell packages, which use custom
scripts and configurations, to NL packages, which adhere to standard
tools such as "systemd". This change simplifies configuration and
improves maintainability but requires adapting your existing setup.

Understanding the differences between the legacy packages and the new
packages helps adjusting the procedure for edge cases, such as multi-
network setups.


Relocating Data
---------------

Serokell stores node data in network-specific subdirectories, while NL
uses ".octez-node" for all networks:

* **Serokell**: Custom scripts support multiple nodes and networks per
  machine. Data resides in "/var/lib/tezos/<network>".

* **NL**: The unique directory for node data "/var/tezos" allows to
  run the node only on a single network at a time.

Copy your node data into this new structure.

Also copy the client data (which is presumably not network-dependent).


Baker Configuration
-------------------

The installation procedure of "octez-baker" will ask a few questions.
Since the configuration of the baker keys lives in the Octez client
data, we don't have to migrate these, but check that the keys are
still relevant.

The "octez-baker" package manages two services:

* "octez-baker-active": For the current protocol.

* "octez-baker-next": For the upcoming protocol.

The protocol associated with these two services can be changed by
reconfiguring the package:

   sudo dpkg-reconfigure octez-baker

or manually editing the files in "/etc/default/octez-baker*".

After the migration is complete, if you are using Debian packages
consider removing all leftover configuration files from the Serokell
packages, by doing "apt purge tezos-client tezos-node tezos-baking".

Octez & Protocol versioning
***************************

Due to the fact that the Tezos blockchain is self-amending, versioning
concerns both the Tezos protocol and several components of Octez. This
page explains the various corresponding versioning schemes, which are
mostly independent of each other. Though, some relations do exist
(e.g. when a new protocol is proposed, a new Octez release is usually
delivered embedding the new proposal, for convenience); this is
explained in some versioning schemes below.


Octez versions
==============

The whole Octez software is versioned by releases, see Release System.


Protocol versions
=================

The Tezos economic protocol is versioned at a different pace,
according to a voting process, see Protocol versioning.


Protocol environment versions
=============================

The economic protocol can interact with the rest of the Octez software
through a sandboxed API called a protocol environment. When new
features are needed by a proposed protocol, or if the protocol API is
changed, a new environment version is created, see Protocol
environment versions. The new environment is delivered as part of a
new Octez release.


RPC Versioning
==============

In Octez, RPCs can be versioned using a query parameter called
"version". This query parameter exists only for RPCs which have at
least two different versions. For example:

   ./octez-client rpc get /chains/main/mempool/pending_operations?version=0

If the RPC is called with a bad version number (a negative or an
unsupported version) the call fails with an error message like:

   Fatal error:
   Command failed: The RPC was called with version number '2' which is not supported. Version numbers accepted are '0, 1'.

For technical reasons, the default version number of an RPC cannot be
retrieved easily yet.


New Version
-----------

Whenever a new version of an RPC is added (see When to Add a New
Version to an RPC), there is a corresponding entry in the changelog in
the release of Octez which includes the new version.

For example:

   Added version 5 to RPC GET chains/main/mempool/pending_operations.
   It can be used by calling the RPC with the parameter ?version=5
   (default version is still 4).


New Default Version
-------------------

Whenever an Octez release changes the default version of an RPC, there
is a corresponding entry in the changelog.

For example:

   The default version for RPC GET chains/main/mempool/pending_operations
   is now 5 (previously 4). You can still use the previous version
   by calling the RPC with the parameter ?version=4.

As a general rule (that we may break exceptionally), changing the
default version number of an RPC always follows a deprecation period.


Deprecated Version
------------------

Whenever an Octez release deprecates an RPC version, there is a
corresponding entry in the changelog.

For example:

   The version 4 for RPC GET chains/main/mempool/pending_operations
   is deprecated and may be removed in the next major release of Octez.

Moreover, the following deprecation policy is applied.


Deprecation Policy
------------------

As part of our ongoing efforts to improve and streamline our services,
we have established a deprecation policy for RPC endpoints.

Deprecation: RPC endpoints can be marked as "DEPRECATED", which serves
as a fair warning that these endpoints will soon be turned off.
Developers are actively encouraged to upgrade their code to use
alternative endpoints at this stage.

Sunset: The sunset of an RPC endpoint indicates that it is no longer
available and supported. This can occur no earlier than "6" months
after the version in which the endpoint was deprecated. For example,
an RPC endpoint marked as deprecated in January (such as in Octez
version "19.x") could be removed no earlier than July (that could
correspond to an Octez version "21.x" -- depending on the release
pace).

The "DEPRECATED" flag is visible in the RPC endpoint description in
the RPC reference.

Deprecated RPCs that have passed the sunset date will be deleted. The
deleted RPCs will be advertised as breaking changes and in the
changelog of Octez versions.


Sunset Dates
~~~~~~~~~~~~

Currently, the following RPCs are subject to the deprecation policy.

* None

Breaking changes
****************

This section presents the breaking changes that users can encounter
between different Protocols or Octez versions. It complements the
"Breaking changes" sections in the development changelogs by providing
more context and/or less fragmented mentions.

For each change, there may be subsections "Deprecation" and "Breaking
changes". The former subsection will explain what changes can be made
during a deprecation phase to adapt smoothly to the new changes. The
latter subsection will present the changes that can not be done by the
deprecation mechanism and that may be breaking.

In the particular case of RPC changes, you may consult complementary
information on RPC versioning, covering how new versions are
introduced, the deprecation policy, and a concrete calendar of RPCs
planned to be removed.


Denunciation operations
=======================

Protocol S adds new operations and changes the encoding of some
existing operations, for instance by adding new fields. These changes
are related to the support for tz4 BLS addresses and their aggregated
signatures.


Backward compatibility
----------------------

Most of the changes in the encodings of existing operations are either
purely added operations (e.g. "update_companion_key") or optional
fields that should not break (e.g., for operations "reveal" and
"update_consensus_key").

However, tool providers who do not use encodings but rather p2p
message format, may experience some issues.


Breaking changes
----------------

Starting in protocol S, the "Double_preattestation_evidence" and
"Double_attestation_evidence" operations are replaced with a new
"Double_consensus_operation_evidence" operation, in order to enable
denunciations of aggregated consensus operations. This new operation
contains a denounced slot and two denounced consensus operations. For
the evidence to be valid, the denounced operations must both be
preattestations (each one may be aggregated or not) or both be
attestations. Moreover, both must involve the denounced slot, that is,
be either a standalone operation for this slot or an aggregate whose
committee includes this slot. The receipts for these operations have
also been reworked, see Operation receipts.

All existing tz4 addresses are being unrevealed when protocol S is
adopted, and they must provide a proof of possession to be revealed
again, see Breaking Changes.


Attestations
============

Support for deprecated attestation legacy name ("endorsement"), that
was still usable with RPCs version "0" will be removed in the protocol
proposal "Q" and Octez "v21.0".


Opam packages
=============

Starting from Octez v18, the Opam packages are being reworked as a new
set containing fewer packages. This allows easier installation and
maintenance.

These changes are transparent for users of the different kinds of
Octez distributions (static executables, Docker images, Opam-installed
binaries, etc.). They only impact software developers directly relying
on Opam packages within the Octez repository (i.e. using them as
dependencies).


New architecture
----------------

Some Octez libraries which used to be distributed as their own Opam
package have been aggregated into fewer and coarser Opam packages.

Each aggregate is related to a part of Octez.

Octez is now distributed as the following set of Opam packages:
   * octez-libs : Contains the base libraries for Octez (cryptography
     primitives, error management helpers, etc.).

   * octez-shell : Contains the libraries related to the Octez shell.

   * octez-proto-libs : Contains the libraries for the Tezos protocol.

   * octez-l2-libs : Contains the libraries related to layer 2.

   * For each protocol "P": - octez-protocol-P-libs : The protocol "P"
     dependent libraries. - "tezos-protocol-P": The Tezos protocol "P"
     itself.

To have a better understanding of the packages and the complete
description of them, you might want to follow the OCaml API
documentation.


Note on library renaming
~~~~~~~~~~~~~~~~~~~~~~~~

In aggregated packages, redundant suffixes and prefixes have been
removed. Specifically, all the sub-libraries prefixed with "tezos-" or
"octez-" are now renamed without the prefix. For instance, "tezos-
base", which is now a sub-library of "octez-libs", is now "octez-
libs.base".

The protocol name suffixes of the protocol libraries have also been
removed. For instance, "Tezos-client-017-PtNairob" is now the sub-
library "Client" of the package "Octez-017-PtNairob-libs".


Backward compatibility
----------------------

One can install the Octez suite directly by using the command:

   opam install octez

This process is the same as with the previous set of packages. The
only difference is the installed packages, but no compatibility issues
will be encountered.

Alternatively, each Octez package can be installed separately:

   opam install package-name


Breaking changes
----------------

Opam packages can be used as dependencies for software development.
Contrary to the previous section, the rework of the Octez Opam
packages will require you to adapt how your software declares Octez-
related Opam dependencies.

For each dependency:

* Search for the new package name in the API.

* Change the Opam "depends" to the package name.

* Update the "dune" files with the new name "package.sub-library".

* Change the module name in the "open" in the code to "Package.Sub-
  library".

For instance, if your software depends on "tezos-rpc" which is now a
sub-library of  octez-libs  and has been renamed to "rpc":

* Update the opam file content to rename the "tezos-rpc" dependency to
  "octez-libs". If "octez-libs" is already present, only remove the
  dependency on "tezos-rpc".

* Update the dune file to rename occurences of "tezos-rpc", e.g. in
  "libraries" clauses of "executable" stanzas to "octez-libs.rpc".

* In your code, update all references to the "Tezos_rpc" module (e.g.
  "open Tezos_rpc") to "Octez-libs.Rpc" (e.g. "open Octez-libs.Rpc").

The same method applies to each dependency that is now a sub-library
of a new package. Check the API to see the new packages.


Delegates rights vs stake snapshots
===================================

The selection of the delegates' participation rights in the proof-of-
stake consensus protocol is done based on their stake. This
computation is explained in generic terms in Proof-of-stake.

One detail of the rights computation has changed: which values are
considered for the delegates' stake in each cycle. Previously, the
considered values corresponded to a notion of stake snapshots,
recorderd regularly by the protocol.


Breaking changes
----------------

Since protocol Paris, there are no more stake snapshots, so the
protocol no longer relies on stake snapshots to compute the rights.

Instead:

* Rights originating from staked tez are computed from the value at
  the end of the cycle;

* Rights originating from delegated tez are computing using the
  minimum value over the cycle.


Timelocks: chest keys
=====================

Timelocks were temporarily disabled by the activation of the Lima
protocol. to address a critical vulnerability.

A new safer version of Timelocks was developed to address this issue,
and the feature was re-enabled in the Oxford protocol. However, the
new chest keys format could not be made backward compatible with the
previous one.

Fortunately, **no contracts using the legacy format of chest keys are
deployed on Tezos mainnet**.


Breaking changes
----------------

However, this change may impact old contracts on the Ghostnet test
network. For example, one (trivial) chest key demo contract was
originated a long time ago on Ghostnet and uses the legacy format for
chest keys.

As a consequence, inspecting this Ghostnet contract currently returns
an error response with status 500:

   Body:
   [
       {
           "kind": "permanent",
           "id": "proto.019-PtParisB.michelson_v1.ill_typed_data",
           "expected_type": {
               "prim": "chest_key"
           },
           "ill_typed_expression": {
               "bytes": "e4c38197..."
           }
       },
       ...
   ]


Upcoming changes
****************


Baker: Explicit choice on using DAL or not via the CLI
======================================================

Octez "v21.3" introduces the new "--without-dal" option for the baker
daemon. In Octez "v21.3", this option is not mandatory and will only
trigger a warning.

Starting from Octez "v22", launching a baker daemon requires an
explicit mention of the DAL. The recommended approach is to run a DAL
node and start the baker using the "--dal-node <uri>" option. If you
do not wish to use a DAL node, you can opt-out by using the "--
without-dal" option.

Setting up the client
*********************

A client in the Tezos network provides different configuration
possibilities:

* tune various parameters of the client using flexible combinations
  of: a configuration file, command-line options, and environment
  variables

* configure how keys are managed, with different security tradeoffs

* select different running modes, beside the default mode, which are
  intended to facilitate testing applications on smaller or fake
  networks: for example by executing some RPCs locally, without
  sending requests to a Tezos node.

These configuration possibilities are described in the following
pages.

* Client Configuration

  * Client configuration file

  * Environment variables for the client

* Key Management

  * Ledger support

  * Signer

  * Consensus Key

  * Companion Key

  * Getting keys for fundraiser accounts

* Sandboxed mode

  * Run a sandboxed node

  * Use the sandboxed client

  * Tune protocol Alpha parameters

  * Preserve data

  * Baking multiple blocks

* Mockup mode

  * Motivation

  * Overview

  * Run a mockup client without persistent state

  * Run a mockup client with persistent state

  * Tune mockup parameters

  * Setting protocol constants for the mockup mode

  * Baking

  * Run a mockup client with asynchronous state

  * Performing protocol migrations of persistent mockup states

  * See also

* Proxy mode

  * Executing commands in proxy mode

  * Examples with the sandbox

  * How to deploy to relieve nodes from some RPCs

* Light mode

  * Executing commands in light mode

  * Examples with the sandbox

  * How to deploy to relieve nodes from some RPCs

Client Configuration
********************

The Octez client can be configured in flexible ways to control various
aspects of its behavior, such as running in different running modes
(Sandboxed mode, Mockup mode, ...), connecting to a public Tezos node,
selecting the directory for storing data, and so on.

All these aspects can be customized by supplying **options** on the
command line when running the client. Refer to the client manual for
details.

A subset of these aspects can be customized by specifying parameters
in a **configuration file** for the client. These include, for
example:

* the address and port of a Tezos node to connect to, as an RPC
  endpoint (by default, the local node; but it can be another node,
  see a list of public RPC nodes)

* the directory where the client stores data

* the number of confirmation blocks needed before an operation is
  considered included

* the files defining bootstrap accounts and protocol constants, when
  running in mockup mode.

When the same parameter is set both in the configuration file and
using a command line option, the value on the command line is taken
into account (and the configuration file is not updated).

Finally, a few aspects of the client behavior can be customized by a
set of **environment variables**.


Client configuration file
=========================

Note:

  The format of the client configuration file (and the associated
  commands to manipulate it) is understood not only by the Octez
  client, but also by several other Octez executables, such as "octez-
  admin-client", the baker, and the accuser. For details, refer to the
  manual of each tool.

Parameters in the configuration file can be specified in two different
ways:

* by creating and updating the configuration file using the "config"
  command of "octez-client".

* by directly editing the configuration file.


The config command
------------------

   ./octez-client config init

This will initialize a configuration file for the client in the
default client "base-dir": "$HOME/.tezos-client/config", using default
values. For instance, it specifies that the client will use the local
node as an RPC endpoint.

The easiest way to amend this default configuration is to use commands
such as:

   # Update the config file:
   octez-client <options...> config update
   # Check your new values:
   octez-client config show
   # If you want to restart from an empty cfg file:
   octez-client config reset


Editing the configuration file
------------------------------

You may also edit the configuration file directly ("$HOME/.tezos-
client/config" by default).

To run the client in multiple configurations on the same machine, you
can duplicate and edit "$HOME/.tezos-client/config" while making sure
they don't share the same "base-dir". Then run your client with
"./octez-client run --base-dir=</path/to/alternate_cfg>".

The config file could also be created outside of the client "base-dir"
("$HOME/.tezos-client/" by default) with the "--config-file <path/to
/alternate-cfg>" option.


Environment variables for the client
====================================

The behavior of the client can be configured using the following
environment variables:

* "TEZOS_CLIENT_UNSAFE_DISABLE_DISCLAIMER": Setting this variable to
  "YES" (or: "yes", "Y", "y") disables the warning displayed by the
  client at startup when it is not launched on Mainnet.

* "TEZOS_CLIENT_DIR": This variable may be used to supply the client
  data directory (by default, "~/.tezos-client"). Its value is
  overridden by option "-d".

* "TEZOS_SIGNER_*": These variables are used for connecting the client
  to a remote signer (see there for details).

* "TEZOS_CLIENT_RPC_TIMEOUT_SECONDS": This variable controls how long
  (in seconds, as an integer) the client will wait for a response from
  the node, for each of the two RPC calls made during startup. If this
  variable is not set, or otherwise cannot be parsed as a positive
  integer, a default value of "10" seconds is used for each call. The
  two RPC calls this variable affects are queries that the client
  makes to the node in order to determine: (1) the protocol version of
  the node it connects to, and (2) the commands supported in that
  version.

* "TEZOS_CLIENT_REMOTE_OPERATIONS_POOL_HTTP_HEADERS": This variable
  specifies custom HTTP headers to use with the "--operations-pool"
  option. Only the Host header is supported as of now (see description
  in rfc2616, section 14.23

Key Management
**************

Securely managing keys is of utmost importance in any blockchain,
including Tezos, because keys are used to sign sensitive operations
such as transfers of valuable assets (tez, FA tokens, tickets, ...) or
baking operations.

The Octez tool suite offers several solutions to store your private
keys safely and use them securely for signing operations. However,
these solutions are **not** enabled by default, so you have to turn
them on, as explained in this tutorial.

Indeed, by default:

* Private keys are stored in "$OCTEZ_CLIENT_DIR/secret_keys". Keys
  generated with "octez-client" are stored encrypted by default for
  Tezos "mainnet", whereas they are unencrypted by default for test
  networks.

* The client uses these keys to sign user operations (e.g. transfers)
  by itself.

* The baker daemon uses these keys to automatically sign its blocks
  and operations (e.g. (pre-)attestations).

* The baker's own key is used to sign consensus operations and DAL
  attestations.

The solutions provided to strengthen the security of the default key
management and signing are the following:

* A hardware wallet (highly recommended) allows to:

  * store your private keys securely

  * sign user operations (e.g. transfers) interactively on the wallet

  * automatically sign baking operations, such as (pre-)attestations,
    more securely.

* Using option "--encrypted" to encrypt the keys when using a testnet
  (this is the default on mainnet, where you must use option "--
  unencrypted" to force unencrypted keys.)

* A separate signer daemon allows to decouple the client and baker
  from the signing process.

* Separate keys (known as "consensus key" and "companion key") can be
  used to sign consensus operations and/or DAL attestations,
  respectively.

  In particular, this allows executing the signer remotely (that is,
  on a different machine than the client and/or the baker), perhaps
  less exposed to attacks.

  As the keys only need to be accessible to the signer, they can also
  benefit from the lesser exposure. Even better (and recommended), a
  remote signer can be combined with a hardware wallet connected to
  the same machine as the signer.

These solutions are detailed in the rest of this page.


Ledger support
==============

It is possible and advised to use a hardware wallet to securely store
and manage your keys. The Octez client supports Ledger Nano devices
provided that they have two Tezos apps installed, Tezos Wallet app and
Tezos Baking app which are provided by Trilitech Ltd. They can be
installed from the Ledger Live store.

For performance reasons (see details), a Ledger Nano S Plus model is a
preferable choice with respect to a Ledger Nano.


Ledger Manager
--------------

The preferred way to set up your Ledger is to install Ledger Live. On
Linux make sure you correctly set up your "udev" rules as explained
here. Connect your Ledger, unlock it and go to the dashboard. In
Ledger Live install "Tezos Wallet" from the applications list. To
install the "Tezos Baking" app, you need to enable Developer Mode in
the Ledger Live settings. Then you can install the app from the
Manager tab in the Ledger Live.

Warning:

  Ledger does not support tz4 addresses. If you wish to use a tz4
  address, consider using an alternative method.


Tezos Wallet app
----------------

Connect the Ledger device and open the "Tezos Wallet" app on the
Ledger device. Now on the Octez client we can import the keys (make
sure the device is in the Tezos Wallet app):

   ./octez-client list connected ledgers

This will display some instructions to import the Ledger encrypted
private key, and you can choose between the root or a derived address.
We can follow the instructions and then confirm the addition by
listing known addresses:

   ./octez-client import secret key my_ledger ledger://XXXXXXXXXX
   ./octez-client list known addresses

Optional: we can check that our Ledger signs correctly using the
following command and confirming on the device:

   octez-client show ledger ledger://XXXXXXXXXX --test-sign

The address can now be used as any other with the exception that
during an operation the device will prompt you to confirm when it's
time to sign an operation.


Tezos Baking app
----------------

The "Tezos Baking" app allows a delegate to sign automatically (i.e.,
there is no need to manually sign every block or (pre-)attestation).
Of course, the application is restricted to only sign baking
operations; it never signs a transfer, for example. To provide
additional protection from double signing the blocks/attestations, it
is recommended to use "octez-signer" along with the Ledger baking app.

Setup the Ledger baking app on any Tezos network using the following
command:

   octez-client setup ledger to bake for my_ledger

More details on setting up baking with the Ledger can be found on the
Tezos Baking app readme.

For details about baking using a Ledger device, see a dedicated
tutorial.


Signer
======

A solution to decouple the client and the baker from the signing
process is to use a *remote signer*.

A complete manual page of the signer is available here.

In this configuration, the client sends signing requests over a
communication channel towards "octez-signer", which can run on a
different machine that stores the secret key.

There are several *signing schemes* supported by the client,
corresponding to different communication channels, such as "unix",
"tcp", "http" and "https". We can list the available schemes with:

   octez-client list signing schemes

We now explain how this remote signer configuration works based on
signing requests, how can it be set up, and how the connection to the
signer can be secured (as by default it is not secure).


Signer requests
---------------

The "octez-signer" handles signing requests with the following format:

   <magic_byte><data>

In the case of blocks or consensus operations for example, this format
is instantiated as follows:

   <magic_byte><chain_id><block|consensus_operation>

Starting with Octez v12 (supporting the Ithaca protocol), consensus
operations also include preattestations. The magic byte distinguishes
pre-Ithaca messages from (post-)Ithaca messages, as follows:

+---------------------------------------------------------+---------------------------+
| Message type                                            | Magic byte                |
|=========================================================|===========================|
| Legacy block                                            | 0x01                      |
+---------------------------------------------------------+---------------------------+
| Legacy endorsement                                      | 0x02                      |
+---------------------------------------------------------+---------------------------+
| Transfer                                                | 0x03                      |
+---------------------------------------------------------+---------------------------+
| Authenticated signing request                           | 0x04                      |
+---------------------------------------------------------+---------------------------+
| Michelson data                                          | 0x05                      |
+---------------------------------------------------------+---------------------------+
| Block                                                   | 0x11                      |
+---------------------------------------------------------+---------------------------+
| Pre-attestation                                         | 0x12                      |
+---------------------------------------------------------+---------------------------+
| Attestation                                             | 0x13                      |
+---------------------------------------------------------+---------------------------+

The magic byte values to be used by the signer can be restricted using
its option "--magic-bytes", as explained in the signer's manual.


Signer configuration
--------------------

In our home server we can generate a new key pair (or import one from
a Ledger) and launch a signer that signs operations using these keys.
To select the "tcp" signing scheme, one has to launch "octez-signer"
with the "socket" argument, as shown below. The new keys are stored by
the signer in "$HOME/.octez-signer" in the same format as "octez-
client". On our internet-facing virtual private server, called "vps"
here, we can then import a key with the address of the signer.

   home~$ octez-signer gen keys alice
   home~$ cat ~/.octez-signer/public_key_hashs
   [ { "name": "alice", "value": "tz1abc..." } ]
   home~$ octez-signer launch socket signer -a home

   vps~$ octez-client import secret key alice tcp://home:7732/tz1abc...
   vps~$ octez-client sign bytes 0x03 for alice

Every time the client on *vps* needs to sign an operation for *alice*,
it sends a signature request to the remote signer on *home*.

However, with the above method, the address of the signer is hard-
coded into the remote key value. Consequently, if we ever have to move
the signer to another machine or access it using another protocol, we
will have to change all the remote keys. A more flexible method is to
only register a key as being remote, and separately supply the address
of the signer using the "-R" option:

   vps~$ octez-client -R 'tcp://home:7732' import secret key alice remote:tz1abc...
   vps~$ octez-client -R 'tcp://home:7732' sign bytes 0x03 for alice

Alternatively, the address of the signer can be recorded in
environment variables:

   vps~$ export TEZOS_SIGNER_TCP_HOST=home
   vps~$ export TEZOS_SIGNER_TCP_PORT=7732
   vps~$ octez-client import secret key alice remote:tz1abc...
   vps~$ octez-client sign bytes 0x03 for alice

All the above methods can also be used with the other signing schemes,
for instance, "http":

   home~$ octez-signer launch http signer -a home

   vps~$ octez-client import secret key alice http://home:7732/tz1abc...
   vps~$ octez-client sign bytes 0x03 for alice

   vps~$ octez-client -R 'http://home:7732' import secret key alice remote:tz1abc...
   vps~$ octez-client -R 'http://home:7732' sign bytes 0x03 for alice

   vps~$ export TEZOS_SIGNER_HTTP_HOST=home
   vps~$ export TEZOS_SIGNER_HTTP_PORT=7732
   vps~$ octez-client import secret key alice remote:tz1abc...
   vps~$ octez-client sign bytes 0x03 for alice

The complete list of environment variables for connecting to the
remote signer is:

* "TEZOS_SIGNER_TCP_HOST"

* "TEZOS_SIGNER_TCP_PORT" (default: 7732)

* "TEZOS_SIGNER_HTTP_HOST"

* "TEZOS_SIGNER_HTTP_PORT" (default: 6732)

* "TEZOS_SIGNER_HTTPS_HOST"

* "TEZOS_SIGNER_HTTPS_PORT" (default: 443)

* "TEZOS_SIGNER_UNIX_PATH"

* "TEZOS_SIGNER_HTTP_HEADERS"


Secure the connection
---------------------

Note that the above setup alone is not secure, **the signer accepts
requests from anybody and happily signs any transaction!**

Improving the security of the communication channel can be done at the
system level by setting up a tunnel with "ssh" or "wireguard" between
*home* and *vps*.

The signer itself can also be configured to provide additional
protection. With the option "--require-authentication" the signer
requires the client to authenticate before signing any operation.

First we create a new key on the *vps* and then import it as an
authorized key on *home* where it is stored under ".octez-
signer/authorized_keys" (similarly to "ssh"). Note that this key is
only used to authenticate the client to the signer and it is not used
as a Tezos account.

   vps~$ octez-client gen keys vps
   vps~$ cat ~/.tezos-client/public_keys
   [ { "name": "vps",
       "value":
          "unencrypted:edpk123456789" } ]

   home~$ octez-signer add authorized key edpk123456789 --name vps
   home~$ octez-signer --require-authentication launch socket signer -a home-ip

All request are now signed with the *vps* key, guaranteeing their
authenticity and integrity. However, this setup **does not guarantee
confidentiality**: an eavesdropper can see the transactions that you
sign (on a public blockchain this may be less of a concern). In order
to avoid that, you can use the "https" scheme or a tunnel to encrypt
your traffic.


Consensus Key
=============

Note:

  The "consensus key" feature is available starting with the Tezos
  Lima protocol.

By default, the baker's key, also called manager key, is used to sign
in the consensus protocol, i.e. signing blocks while baking, and
signing consensus operations (preattestations and attestations).

A delegate may elect instead to choose a dedicated key: the *consensus
key*. It can then be changed without redelegation.

It also allows establishment of baking operations in an environment
where access is not ultimately guaranteed: for example, a cloud
platform providing hosted Key Management Systems (KMS) where the
private key is generated within the system and can never be downloaded
by the operator. The delegate can designate such a KMS key as its
consensus key. Shall they lose access to the cloud platform for any
reason, they can simply switch to a new key.

However, both the delegate key and the consensus key give total
control over the delegate's funds: indeed, the consensus key may sign
a "Drain_delegate" operation to transfer the delegate's spendable
balance to an arbitrary account. In relevant RPCs like
"/chains/main/blocks/head/helpers/baking_rights", both the delegate's
manager and consensus keys are listed. As a consequence, the consensus
key should be treated with equal care as the manager key.


Registering a Consensus Key
---------------------------

A consensus key can be changed at any point.

The operation is signed by the manager key and does not require the
consensus private key to be accessible by the client.

However the public key must be known by the client. It can be imported
with the command:

   octez-client import public key <consensus_key> unencrypted:edpk...

The command to update the consensus key is:

   octez-client set consensus key for <manager_key> to <consensus_key>

The update becomes active after "CONSENSUS_RIGHTS_DELAY + 1" cycles.
We therefore distinguish the active consensus key and the pending
consensus keys. The active consensus key is by default the delegate’s
manager key, which cannot change.

However, it is also possible to register as a delegate and immediately
set the consensus key:

   octez-client register key <mananger_key> as delegate with consensus key <consensus_key>

There can be multiple pending updates: it is possible to have multiple
pending consensus keys for multiple future cycles. A subsequent update
within the same cycle takes precedence over the initial one.

Note that registering a tz4 consensus key, just like revealing a tz4
public key, requires a proof of possession. This is the signature of
the consensus public key using the consensus private key, and it
ensures ownership of the key. This process is done automatically by
the client, and the proof is included in the receipt of the update
operation.


Baking With a Consensus Key
---------------------------

In your baker's command, replace the delegate's manager key alias with
the consensus key alias:

   octez-baker-Ptxxxxxx run with local node ~/.tezos-node <consensus_key_alias> --liquidity-baking-toggle-vote pass

While transitioning from the delegate's manager key, it is possible to
pass the alias for both delegate's manager key and consensus key. The
delegate will seamlessly keep baking when the transition happens:

   octez-baker-Ptxxxxxx run with local node ~/.tezos-node <consensus_key_alias> <delegate_key_alias> --liquidity-baking-toggle-vote pass


Draining a Manager's Account With its Consensus Key
---------------------------------------------------

This operation immediately transfers all the spendable balance of the
"baker_pkh"’s user account into the "destination_pkh" user account:

   octez-client drain delegate <baker_pkh> to <destination_pkh> with <consensus_pkh>

If the destination is the consensus key account, this can be
simplified to:

   octez-client drain delegate <baker_pkh> to <consensus_pkh>

The active consensus key is the signer for this operation, therefore
the private key associated to the consensus key must be available in
the wallet of the client typing the command. The delegate's private
key does not need to be present.

The drain operation has no effect on the frozen balance.

A fixed fraction of the drained delegate’s spendable balance is
transferred as fees to the baker that includes the operation, i.e. the
maximum between 1 tez or 1% of the spendable balance.


Companion Key
=============

Starting with protocol S, bakers will be able to register a second key
called the *companion key*. It is a tz4 key, whose purpose is to sign
DAL specific content in consensus operations. This key is required for
delegates with a tz4 consensus key that wish to participate in the
DAL.

More precisely, if a delegate has an active tz4 consensus key, but no
companion key is active, or if it is missing from the client set of
known keys, the baker will still be able to produce attestations, but
without any DAL attestations. In other words, even if the baker is
connected to a DAL node and receives attestable slots for the
delegate, since the companion key is not available, it will not be
able to include a DAL attestation in its consensus operation, and will
only send a regular attestation.

Any delegate, regardless of their kind of address, can register a
companion key, it will only be used when necessary. There is no
downside in doing so, because "drain delegate" only applies to
consensus keys, not companion keys.

The command to update the companion key is:

   octez-client set companion key for <manager_key> to <companion_key>

Since a companion key has to be a tz4, this command will also create a
proof of possession and include it in the operation.

A companion key takes the same amount of time as a consensus key to
become activated, which is up to "CONSENSUS_KEY_ACTIVATION_DELAY + 1"
cycles (see Consensus related protocol parameters).

Alternatively, it is possible to register a companion key when
registering as a delegate:

   octez-client register key <manager_key> as delegate --companion-key <companion_key>

It is even possible to register both a consensus key and a companion
key, with the following command:

   octez-client register key <manager_key> as delegate --consensus-key <consensus_key> --companion-key <companion_key>


Getting keys for fundraiser accounts
====================================

If you took part in the fundraiser but didn't yet activate your
account, it is still possible to activate your Mainnet account on
https://check.tezos.com/. This feature is also included in some
wallets. If you have any questions or issues, refer to that page or to
the Tezos Foundation for support.

You may also use "octez-client" to activate your account, but **be
warned that you should have a very good understanding of key
management in Tezos and be familiar with the command-line.** The first
step is to recover your private key using the following command which
will ask for:

* the email address used during the fundraiser

* the 14 words mnemonic of your paper wallet

* the password used to protect the paper wallet

   octez-client import fundraiser key alice

Once you insert all the required information, the client computes your
secret key and it asks you to create a new password in order to store
your secret key on disk encrypted.

If you haven't already activated your account on the website, you can
use this command with the activation code obtained from the Tezos
foundation.

   octez-client activate fundraiser account alice with <code>

Check the balance with:

   octez-client get balance for alice

As explained above, your keys are stored under "~/.tezos-client". We
strongly advise you to first **make a backup** and then transfer your
tokens to a new pair of keys imported from a Ledger (see Ledger
support).

Sandboxed mode
**************

To run a ‘localhost-only’ instance of a Tezos network, we provide two
helper scripts:

* "./src/bin_node/octez-sandboxed-node.sh"

* "./src/bin_client/octez-init-sandboxed-client.sh"

For the moment these scripts are expected to be run on the "master"
branch (see Build from sources; in particular, use "git checkout
master" instead of "git checkout latest-release").


Run a sandboxed node
====================

For instance, if you want to run a local network with two nodes, in
the first terminal, the following command will initialize a node
listening for peers on port "19731" and listening for RPC on port
"18731".

   ./src/bin_node/octez-sandboxed-node.sh 1 --connections 1

This node will store its data in a temporary directory "/tmp/octez-
node.xxxxxxxx" which will be removed when the node is stopped.

The option "--connections" specifies the ideal number of peers the
node tries to connect to. Lowering the number of expected connections
removes the spurious “Too few connections” warnings. Set it to "1" for
our two-nodes network (and you would set it to "0" for a single-node
network).

More informations can be found in the api page of the octez node
config : or by simply calling

   ./src/bin_node/octez-sandboxed-node.sh 1 --connections 1 --help

To launch the second node, run the following command in another
terminal, and it will listen on port "19739" and "18739":

   ./src/bin_node/octez-sandboxed-node.sh 9

You might replace "1" or "9" by any number in between if you want to
run more than two nodes.


Use the sandboxed client
========================

Once your node is running, open a new terminal and initialize the
“sandboxed” client data in a temporary directory:

   eval `./src/bin_client/octez-init-sandboxed-client.sh 1`

It will also define in the current shell session an alias "octez-
client" preconfigured for communicating with the same-numbered node.

When you bootstrap a new network, the network is initialized with a
dummy economic protocol, called *genesis*. If you want to run the
whole implemented protocol, "init-sandboxed-client" also defines an
alias "octez-activate-alpha", that you need to execute once for
activating the whole network. For instance:

   $ octez-client rpc get /chains/main/blocks/head/metadata
     { "protocol": "PrihK96nBAFSxVL1GLJTVhu9YnzkMFiBeuJRPA8NwuZVZCE1L6i",
       "next_protocol": "ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im",
       ... }
   $ octez-activate-alpha
     Injected BMV9KnSPE1yw
   $ octez-client rpc get /chains/main/blocks/head/metadata
     { "protocol": "ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im",
       "next_protocol": "ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK",
       ... }

We now have the possibility to send transactions to the sandboxed
network. As the genesis block used to initialize the sandboxed network
differs from the one used in test networks, it is not possible to
activate accounts obtained from the faucet. However, we can use the
preconfigured accounts which can be listed with:

   $ octez-client list known addresses

     activator: tz1TGu6TN5GSez2ndXXeDX6LgUDvLzPLqgYV (unencrypted sk known)
     bootstrap5: tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv (unencrypted sk known)
     bootstrap4: tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv (unencrypted sk known)
     bootstrap3: tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU (unencrypted sk known)
     bootstrap2: tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN (unencrypted sk known)
     bootstrap1: tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx (unencrypted sk known)

We can run the following command to transfer some Tez from one account
to another:

   $ octez-client transfer 42 from bootstrap1 to bootstrap2 &
   ...
   Waiting for the operation to be included...

You will notice that this command doesn't terminate (hence the "&"),
as usual it is waiting for the network to include the transaction in a
block. Given that we are in a sandbox we need to bake a block
ourselves and we can do so with the following command:

   $ octez-client bake for --minimal-timestamp

If the previous transaction is valid, the operation is included in the
chain and the transfer terminates returning the usual receipt. Note
that the "bake for" command of the client is exclusively for testing
purposes, all baking should be done using the "octez-baker" binary.

We can now observe the transaction with:

   $ octez-client rpc get /chains/main/blocks/head
     { "protocol": "ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK",
       ...
       "header":
         { "level": 2,
           ... },
       "operations":
         [ ...
           [ { ...
               "contents":
                 [ { "kind": "transaction",
                     "source": "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx",
                     "fee": "268", "counter": "2", "gas_limit": "169",
                     "storage_limit": "0", "amount": "42000000",
                     "destination": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN",
                     ... } ] } ] ]


Tune protocol Alpha parameters
==============================

The "octez-activate-alpha" alias uses parameters from
"src/proto_alpha/parameters/sandbox-parameters.json" to activate
protocol Alpha. It can be useful to tune these parameters when you
need to debug something, for example, change the number of blocks per
cycle, the time between blocks, etc.


Preserve data
=============

If you want to preserve data and configuration files at the end of
your run, you can use the "DATA_DIR" environment variable.

   mkdir /tmp/tz-data
   DATA_DIR='/tmp/tz-data' ./src/bin_node/octez-sandboxed-node.sh 1 --connections 1

You can even provide a custom "identity.json" and "config.json" to the
sandboxed node by placing them in the data directory.


Baking multiple blocks
======================

To bake multiple blocks in a single command the "-n
<number_of_blocks>" option can be used like

   $ octez-client bake for --minimal-timestamp -n 1_000

Once the current timestamp is caught up, blocks are produced every
second or every "minimal_block_delay" set in the parameters file. To
speed up the process the protocol can be activated in the past with

   $ octez-activate-alpha --timestamp "2024-01-01T00:00:00Z"

This increases the number of blocks needed to reach the current
timestamp and speeds up the blocks production.

Mockup mode
***********

By default the "octez-client" described in the sandboxed node needs a
node running. This page describes the *mockup* mode, a mode that works
without connecting to a node. For the moment, its features are more
limited than the default mode (see the proxy mode for an intermediate
mode between default and mockup).


Motivation
==========

The main motivation for implementing the mockup mode is to provide
developers of Tezos smart contracts with an easy local environment,
offering a fast development cycle, which uses only lightweight local
files for keeping the state, and which does not require a running
blockchain.


Overview
========

In mockup mode, the client uses some dummy values for the initial
parameters that are usually gathered from a node, such as the head of
the chain or the network identifier. Then the mockup client simulates
activation from genesis and runs local implementations of the RPCs
itself.


Modes of operation
------------------

Mockup mode can run in three ways:

* Stateless mode.

  In this mode, octez-client operates on its inputs and returns a
  value. Nothing is written to disk, and no state is preserved between
  calls to the client. This is the default.

* Stateful mode.

  In this mode, octez-client creates or manipulates a state on disk.
  The switch for this is "--base-dir <directory_name>".

* Stateful asynchronous mode.

  This mode adds baking. The command-line switch for this is "--base-
  dir <directory_name> --asynchronous".


Supported features
------------------

Currently, depending on the way it is run, the mockup mode can:

* typecheck, serialize, sign and evaluate a contract -- without a
  node.

  These features do not require a persistent state.

* perform transactions, originations, contract calls in a purely local
  fashion; mimicking the sandboxed mode but without a node.

  These features require a persistent state.

* perform some RPCs locally via the command line.

  These features work in stateful asynchronous mode.

We recommend that beginners use the persistent state, as it represents
a suitable tradeoff between simplicity and supported features.

Nevertheless, the modes of operation are shown in the order above.


Run a mockup client without persistent state
============================================

Without persistent state, the mockup mode is already able to typecheck
scripts. Let's typecheck for example the script
michelson_test_scripts/mini_scenarios/hardlimit.tz:

   $ octez-client --mode mockup typecheck script michelson_test_scripts/mini_scenarios/hardlimit.tz

The script can also be executed:

   $ octez-client --mode mockup run script <filename> on storage <storage> and input <input>

where "<storage>" and "<input>" are some Michelson expression
describing the contract's storage and script input respectively. A "--
trace-stack" option can be added in the end to output the state of the
stack after each step of script's execution.

For example:

   $ octez-client --mode mockup run script michelson_test_scripts/attic/id.tz on storage '"hello"' and input '"world"'
   # Ignore warnings about the missing/wrong base directory, they do not apply to "run script"
   storage
     "world"
   [...]


Run a mockup client with persistent state
=========================================

Setting up a mockup state requires to choose a protocol. To see the
list of supported protocols in mockup mode, issue the following
command:

   $ octez-client list mockup protocols

At any given time, it should return "Alpha" and at least the two
protocols before that.

To create the mockup client state, issue the following command:

   $ octez-client \
     --protocol ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK \
     --base-dir /tmp/mockup \
     --mode mockup \
     create mockup

Now that this command has been issued, the next calls below **all**
use "--mode mockup" and "--base-dir /tmp/mockup" arguments. This is
akin to doing a mockup *session*. To avoid mistakes, we advise to do
the following in the local shell running the session:

   $ alias mockup-client='octez-client --mode mockup --base-dir /tmp/mockup'

You can now use standard commands, such as:

   $ mockup-client list known addresses
   bootstrap5: tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv (unencrypted sk known)
   bootstrap4: tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv (unencrypted sk known)
   bootstrap3: tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU (unencrypted sk known)
   bootstrap2: tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN (unencrypted sk known)
   bootstrap1: tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx (unencrypted sk known)

   $ mockup-client transfer 100 from bootstrap1 to bootstrap2
   Node is bootstrapped, ready for injecting operations.
   Estimated gas: 10207 units (will add 100 for safety)
   Estimated storage: no bytes added
   Operation successfully injected in the node.
   Operation hash is 'ooMyN7FDmDGyNk8CLdSFwcdxcQea5KLXYqrgzu6CEYB7G2xYbth'
   NOT waiting for the operation to be included.
   Use command
     octez-client wait for ooMyN7FDmDGyNk8CLdSFwcdxcQea5KLXYqrgzu6CEYB7G2xYbth to be included --confirmations 30 --branch BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU
   and/or an external block explorer to make sure that it has been included.
   This sequence of operations was run:
     Manager signed operations:
       From: tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx
       Fee to the baker: ꜩ0.001282
       Expected counter: 2
       Gas limit: 10307
       Storage limit: 0 bytes
       Balance updates:
         tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx ........... -ꜩ0.001282
         fees(tz1Ke2h7sDdakHJQh8WX4Z372du1KChsksyU,0) ... +ꜩ0.001282
       Transaction:
         Amount: ꜩ100
         From: tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx
         To: tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN
         This transaction was successfully applied
         Consumed gas: 10207
         Balance updates:
           tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx ... -ꜩ100
           tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN ... +ꜩ100

   $ mockup-client get balance for bootstrap1
   3999898.997437 ꜩ

One can also originate contracts:

   $ mockup-client originate contract foo transferring 100 from bootstrap1 running 'parameter unit; storage unit; code { CAR; NIL operation; PAIR}' --burn-cap 10
   [...]
   New contract KT1DieU51jzXLerQx5AqMCiLC1SsCeM8yRat originated.

The client can be used to display the state of the contract, e.g. its
storage:

   $ mockup-client get contract storage for foo
   Unit

The RPC mechanism can also be conveniently used to access the state of
the contract in JSON format:

   $ mockup-client rpc get /chains/main/blocks/head/context/contracts/KT1DieU51jzXLerQx5AqMCiLC1SsCeM8yRat/storage
   { "prim": "Unit" }

The stateful mockup mode stores state data in a single "context.json"
file, located under the "mockup" subdirectory of the base directory.
In our running example, its absolute file name is
"/tmp/mockup/mockup/context.json".


Tune mockup parameters
======================

The examples so far have used mockup mode’s default settings. Some use
cases need a custom setup, so mockup mode lets us configure some
initial parameters.

For simplicity, the mockup mode - like the sandboxed mode - uses
default values for the protocol constants. Such values are visible as
follows (we recall that "mockup-client" is an alias for "octez-
client", see previous section):

   $ mockup-client config show
   Default value of --bootstrap-accounts:
   ...
   Default value of --protocol-constants:
   ...

To tune these values, we recommend to first generate the files
corresponding to the default values:

   $ mockup-client config init
   Written default --bootstrap-accounts file: /tmp/mockup/bootstrap-accounts.json
   Written default --protocol-constants file: /tmp/mockup/protocol-constants.json

You can now edit the files "bootstrap-accounts.json" and "protocol-
constants.json" to your liking then create a tuned mockup state.

   $ mv /tmp/mockup/{bootstrap-accounts,protocol-constants}.json /tmp/.
   $ rm /tmp/mockup -Rf
   $ mockup-client --protocol ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK \
     create mockup \
     --protocol-constants /tmp/protocol-constants.json \
     --bootstrap-accounts /tmp/bootstrap-accounts.json

You can check your custom parameters were taken into account:

   $ mockup-client config show
   Default value of --bootstrap-accounts:
   ...
   Default value of --protocol-constants:
   ...


Setting protocol constants for the mockup mode
==============================================

Let's look at the contents of the "protocol-constants.json" file as
produced by the "--mode mockup config init" and "--mode mockup config
show" commands. The following was generated:

    {
         "preserved_cycles": 2,
         "blocks_per_cycle": 8,
         "blocks_per_commitment": 4,
         "nonce_revelation_threshold": 4,
         "blocks_per_stake_snapshot": 4,
         "cycles_per_voting_period": 8,
         "hard_gas_limit_per_operation": "1040000",
         "hard_gas_limit_per_block": "2600000",
         "proof_of_work_threshold": "4611686018427387903",
         "minimal_stake": "6000000000",
         "vdf_difficulty": "50000",
         "origination_size": 257,
         "reward_weights": {
             "base_total_rewards_per_minute": "85007812",
             "baking_reward_fixed_portion_weight": 5120,
             "baking_reward_bonus_weight": 5120,
             "attesting_reward_weight": 10240,
             "liquidity_baking_subsidy_weight": 1280,
             "seed_nonce_revelation_tip_weight": 1,
             "vdf_revelation_tip_weight": 1
         },
         "cost_per_byte": "250",
         "hard_storage_limit_per_operation": "60000",
         "quorum_min": 2000,
         "quorum_max": 7000,
         "min_proposal_quorum": 500,
         "liquidity_baking_toggle_ema_threshold": 1000000000,
         "max_operations_time_to_live": 240,
         "minimal_block_delay": "1",
         "delay_increment_per_round": "1",
         "consensus_committee_size": 256,
         "consensus_threshold": 0,
         "minimal_participation_ratio": {
             "numerator": 2,
             "denominator": 3
         },
         "max_slashing_period": 2,
         "limit_of_delegation_over_baking": 19,
         "percentage_of_frozen_deposits_slashed_per_double_baking": 10,
         "percentage_of_frozen_deposits_slashed_per_double_attestation": 50,
         "cache_script_size": 100000000,
         "cache_stake_distribution_cycles": 8,
         "cache_sampler_state_cycles": 8,
         "dal_parametric": {
             "feature_enable": false,
             "number_of_slots": 16,
             "attestation_lag": 4,
             "attestation_threshold": 50,
             "blocks_per_epoch": 2,
             "redundancy_factor": 8,
             "page_size": 128,
             "slot_size": 32768,
             "number_of_shards": 64
         },
         "smart_rollup_enable": true,
         "smart_rollup_arith_pvm_enable": false,
         "smart_rollup_origination_size": 6314,
         "smart_rollup_challenge_window_in_blocks": 80640,
         "smart_rollup_stake_amount": "10000000000",
         "smart_rollup_commitment_period_in_blocks": 60,
         "smart_rollup_max_lookahead_in_blocks": 172800,
         "smart_rollup_max_active_outbox_levels": 80640,
         "smart_rollup_max_outbox_messages_per_level": 100,
         "smart_rollup_number_of_sections_in_dissection": 32,
         "smart_rollup_timeout_period_in_blocks": 40320,
         "smart_rollup_max_number_of_cemented_commitments": 5,
         "smart_rollup_max_number_of_parallel_games": 32,
         "smart_rollup_reveal_activation_level": {
             "raw_data": { "Blake2B": 0 },
             "metadata": 0,
             "dal_page": 0
         },
         "zk_rollup_enable": false,
         "zk_rollup_origination_size": 4000,
         "zk_rollup_min_pending_to_process": 10,
         "zk_rollup_max_ticket_payload_size": 2048,
         "global_limit_of_staking_over_baking": 5,
         "edge_of_staking_over_delegation": 2,
         "adaptive_issuance_launch_ema_threshold": 1600000000,
         "adaptive_rewards_params": {
             "reward_ratio_min": { "numerator": "1", "denominator": "200" },
             "reward_ratio_max": { "numerator": "1", "denominator": "10" },
             "max_bonus": "50000000000000",
             "growth_rate": "115740740",
             "center_dz": { "numerator": "1", "denominator": "2" },
             "radius_dz": { "numerator": "1", "denominator": "50" }
         },
         "chain_id": "NetXynUjJNZm7wi",
         "initial_timestamp": "1970-01-01T00:00:00Z"
   }

Besides usual protocol constants, there are 2 additional fields
supported in Mockup mode:

* "chain_id": Used to prevent replay of operations between chains. You
  can pick a chain id for your mockup environment using the following
  command:

   $ octez-client compute chain id from seed <string>

For instance, the following command:

   $ octez-client compute chain id from seed strudel

yields the chain id "NetXwWbjfCqBTLV".

* "initial_timestamp": The creation time of the first block of the
  chain. This date string follows the ISO-8601 standard format, which
  can be generated by "date --iso-8601=seconds".


Baking
======

Baking in mockup mode is more aptly named *fake baking*. Indeed, it
behaves somewhat differently than baking in the sandbox.

With fake baking, everything happens locally, keeping track on disk of
the context and the mempool. In addition, the mockup chain only ever
has *one* live block, its head, so that you cannot have competing
chains. In effect, it behaves as if the time-to-live of transactions
was 0.

As a result of only having one block, only transactions done on the
head can be baked in. Consequently, transactions refused during
successful baking will not be in position to be added at any point
down the road. Thus, after each successful baking, the mempool is
emptied from any outstanding operations, which are appended to a so-
called *trashpool* containing the list of all refused transactions at
any point.

Let us make that clearer with an example.


Run a mockup client with asynchronous state
===========================================

We will start by creating a mockup directory supporting *asynchronous*
transfers, i.e., where transfers do not immediately bake the block.

   $ rm /tmp/mockup -Rf # Was created by commands above
   $ mockup-client create mockup --asynchronous

This will create a fresh mockup directory.  Notice that, in addition
to the "mockup/context.json" file, you now also have a
"mockup/mempool.json", which is initially empty.

Now, let us add 2 transactions, that we will label respectively "t1"
and "t2", to the mempool.

   $ mockup-client transfer 1 from bootstrap1 to bootstrap2 --fee 1
   $ mockup-client transfer 2 from bootstrap2 to bootstrap3 --fee 0.5

You can check that it is indeed the case by visiting
"mockup/mempool.json". This should look like this

   [ { "shell_header":
         { "branch": "BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU" },
       "protocol_data":
         { "contents":
             [ { "kind": "transaction",
                 "source": "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx",
                 "fee": "1000000", "counter": "1", "gas_limit": "10307",
                 "storage_limit": "0", "amount": "1000000",
                 "destination": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN" } ],
           "signature":
             "siggZXnjqYnFMjMxfE1avK2PZdRmRekp5fr56F5uJcuQkfHPL23HNDdtz2iG1QeYtU8DGEniWXjqDh1RxGx6scVgMaK74CrF" } },
     { "shell_header":
         { "branch": "BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU" },
       "protocol_data":
         { "contents":
             [ { "kind": "transaction",
                 "source": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN",
                 "fee": "500000", "counter": "1", "gas_limit": "10307",
                 "storage_limit": "0", "amount": "2000000",
                 "destination": "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx" } ],
           "signature":
             "sigTBpkXw6tC72L2nJ2r2Jm5iB6uidTWqoMNd4oEawUbGBf5mHVfKawFYL8X8MJECpL73oBnfujyUZNLK2LQWD1FaCkYMP4j" } } ]

Now let's simulate a selective baker, like so

   $ mockup-client bake for bootstrap1 --minimal-fees 0.6

The effect of successfully baking the new head will be to include "t1"
but discard "t2". You can check that "t2" has been added to the file
"mockup/trashpool.json", since we know it cannot be added to further
blocks of the mockup chain.

   [ { "shell_header":
      { "branch": "BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU" },
    "protocol_data":
      { "contents":
          [ { "kind": "transaction",
              "source": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN",
              "fee": "500000", "counter": "1", "gas_limit": "10307",
              "storage_limit": "0", "amount": "2000000",
              "destination": "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx" } ],
        "signature":
          "sigTBpkXw6tC72L2nJ2r2Jm5iB6uidTWqoMNd4oEawUbGBf5mHVfKawFYL8X8MJECpL73oBnfujyUZNLK2LQWD1FaCkYMP4j" } } ]

If we repeat somewhat similar steps

   $ mockup-client transfer 1 from bootstrap4 to bootstrap5 --fee 1
   $ mockup-client transfer 2 from bootstrap2 to bootstrap3 --fee 0.5

And bake once more selectively

   $ mockup-client bake for bootstrap3 --minimal-fees 0.6

Then, once again, the first transaction, with a fee of 1, will make it
as part of the new head whereas the second will be appended to the
trashpool, which now looks like this

    [ { "shell_header":
       { "branch": "BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU" },
     "protocol_data":
       { "contents":
           [ { "kind": "transaction",
               "source": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN",
               "fee": "500000", "counter": "1", "gas_limit": "10307",
               "storage_limit": "0", "amount": "2000000",
               "destination": "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx" } ],
         "signature":
           "sigTBpkXw6tC72L2nJ2r2Jm5iB6uidTWqoMNd4oEawUbGBf5mHVfKawFYL8X8MJECpL73oBnfujyUZNLK2LQWD1FaCkYMP4j" } },
   { "shell_header":
       { "branch": "BKmdPRhxVBU4RCpHsLtU2FHNXRPCbcquMTpzK5QWvHG9C4TwMCj" },
     "protocol_data":
       { "contents":
           [ { "kind": "transaction",
               "source": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN",
               "fee": "500000", "counter": "1", "gas_limit": "10307",
               "storage_limit": "0", "amount": "2000000",
               "destination": "tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU" } ],
         "signature":
           "sigeFcabZTE8Y2LXv19Fe7TbRtkjzVpBy2qhABp263Xnj8TJtA6XpRRMfGeD5YxwCJiTr9r6ZFGBdLnpxL9Y9CG3bpbXmu7E" } } ]


Performing protocol migrations of persistent mockup states
==========================================================

The persistent state of the mockup mode is highly protocol-dependent.
But Tezos is self-amending: protocols regularly evolve from one to the
next. When a protocol switch happens on-chain, the protocol state is
automatically migrated to the format used by the new protocol.

A command is provided to do the same on the persistent mockup state:

   $ mockup-client migrate mockup to <protocol hash>

The protocol corresponding to the hash must know how to migrate from
the current protocol.

This is mostly useful for protocol developers, but also for other
developers, e.g., those wanting to check the robustness of their
application with respect to protocol changes, including new features
or breaking changes.


See also
========

This tutorial has also served as a base for a nice blog post, written
in a more casual way and intended for a larger audience of application
developers. Of course, some aspects may gradually become outdated in
the blog version.

Proxy mode
**********

The "octez-client", described in a dedicated tutorial, heavily relies
on node RPCs to implement its features. Thus, when a client need to
perform some computation which cannot be done entirely locally, but
which is implemented by a node RPC, it will simply call the
corresponding RPC.

The current page describes the *proxy* mode, an execution mode where
the client avoids some RPC calls to the node, especially computation-
intensive RPCs. Thus, for computations that cannot be done entirely
locally, the proxy client only requests the data it needs from the
node using RPCs (that are not computation-intensive), and uses this
data locally to perform computations by itself, whenever possible.

For an entirely local mode, which never calls node RPCs, see the
mockup mode.

Typical examples of potentially long computations the proxy mode
relieves the node of include estimating gas required by a smart
contract call or asking for baking rights.


Executing commands in proxy mode
================================

The CLI interface of the client in proxy mode (the *proxy client* in
short) is the same as the default client. To turn proxy mode ON,
simply pass option "--mode proxy" to "octez-client".

Because some computations usually done by the node are protocol-
dependent, the proxy mode has to be configured for a specific
protocol. However, the proxy mode does not support all protocols.
Execute "octez-client list proxy protocols" to see the supported
protocols. It is expected that, at any given time, the proxy mode
supports "Alpha", the current protocol of Mainnet, and the current
protocol proposal on Mainnet at the time of release, if any.


Examples with the sandbox
=========================

In this section, we show examples of using the proxy mode with a
sandboxed node. For convenience, we repeat instructions for the
sandboxed mode here, but we refer the reader to the sandboxed mode
page for further details. In a terminal, start a sandboxed node:

   $ ./src/bin_node/octez-sandboxed-node.sh 1 --connections 1

Leave that terminal running and, in another terminal, prepare the
appropriate environment for using the proxy client:

   $ eval `./src/bin_client/octez-init-sandboxed-client.sh 1`

Then upgrade the node to protocol alpha:

   $ octez-activate-alpha
   $ octez-client bake for bootstrap1

To avoid warnings being printed in upcoming commands (optional):

   $ export TEZOS_CLIENT_UNSAFE_DISABLE_DISCLAIMER=y

You're now ready to use the proxy client. For example, request baking
rights:

   $ octez-client --mode proxy rpc get /chains/main/blocks/head/helpers/baking_rights
   [ { "level": 3, "delegate": "tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv",
       "round": 0, "estimated_time": "2022-11-17T14:20:17Z",
       "consensus_key": "tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv" },
     { "level": 3, "delegate": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN",
       "round": 1, "estimated_time": "2022-11-17T14:20:18Z",
       "consensus_key": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN" },
     { "level": 3, "delegate": "tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv",
       "round": 2, "estimated_time": "2022-11-17T14:20:20Z",
       "consensus_key": "tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv" },
     { "level": 3, "delegate": "tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU",
       "round": 3, "estimated_time": "2022-11-17T14:20:23Z",
       "consensus_key": "tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU" },
     { "level": 3, "delegate": "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx",
       "round": 4, "estimated_time": "2022-11-17T14:20:27Z",
       "consensus_key": "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx" } ]

Well, that doesn't seem very different from what the default client
would return. Indeed, it's the same; that was the point! To see what
the proxy client is doing differently, you may set up debugging on the
following event sections (see Logging features):

* "proxy_rpc_ctxt" shows whether the proxy client is delegating RPCs
  to the node or doing them locally. This is the main debug section.

* "proxy_rpc" shows the RPCs that the proxy client is performing to
  retrieve up-to-date data from the node. This debug section is useful
  to understand if the proxy is performing slowly. It might be because
  it is performing a lot of such RPCs.

For convenience, let's define an alias before continuing, to save
keystrokes and the "protocol of proxy unspecified" warning:

   $ alias proxy-client="octez-client --mode proxy --protocol ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK"

Now configure "proxy_rpc_ctxt" to have more information:

   $ export TEZOS_LOG="proxy_rpc_ctxt->debug"

And redo the same RPC as before:

   $ proxy-client rpc get /chains/main/blocks/head/helpers/baking_rights
   Nov 17 15:21:19.959 - proxy_rpc_ctxt: delegating to http: GET call_service version
   Nov 17 15:21:19.967 - proxy_rpc_ctxt: locally done: GET call_service chains/<chain_id>/blocks/<block_id>/protocols
   Nov 17 15:21:19.969 - proxy_rpc_ctxt: locally done: GET call_service describe/<string>
   Nov 17 15:21:19.976 - proxy_rpc_ctxt: locally done generic media type call: GET
   Nov 17 15:21:19.976 - proxy_rpc_ctxt:   /chains/main/blocks/head/helpers/baking_rights
   [ { "level": 3, "delegate": "tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv",
       "round": 0, "estimated_time": "2022-11-17T14:20:17Z",
       "consensus_key": "tz1b7tUupMgCNw2cCLpKTkSD1NZzB5TkP2sv" },
     { "level": 3, "delegate": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN",
       "round": 1, "estimated_time": "2022-11-17T14:20:18Z",
       "consensus_key": "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN" },
     { "level": 3, "delegate": "tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv",
       "round": 2, "estimated_time": "2022-11-17T14:20:20Z",
       "consensus_key": "tz1ddb9NMYHZi5UzPdzTZMYQQZoMub195zgv" },
     { "level": 3, "delegate": "tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU",
       "round": 3, "estimated_time": "2022-11-17T14:20:23Z",
       "consensus_key": "tz1faswCTDciRzE4oJ9jn2Vm2dvjeyA9fUzU" },
     { "level": 3, "delegate": "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx",
       "round": 4, "estimated_time": "2022-11-17T14:20:27Z",
       "consensus_key": "tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx" } ]

In this case, the bulk of the computation is done locally.

If you also want to see the data requests to the node, do the
following before running your commands:

   $ export TEZOS_LOG="proxy_rpc_ctxt->debug; proxy_rpc->debug"


How to deploy to relieve nodes from some RPCs
=============================================

Using proxy clients can reduce the load of nodes, by having clients
perform more computations locally. For this to work, however,
deployment should be done in a specific manner, which this section
describes.

While the proxy mode has been designed to reduce the load of nodes, it
doesn't suffice on its own to achieve this goal. Because proxy clients
perform many calls to the
"/chains/<chain>/blocks/<block>/context/raw/bytes/" RPC (and because
the payload of doing an RPC call is a bit too high at the moment), the
node's load doesn't decrease. It pretty much stays the same: for
example, when benchmarking the time spent honoring "rpc get
/chains/main/blocks/head/helpers/baking_rights?&all=true", a node
serving proxy clients spends its time serving "../raw/bytes" instead
of serving "../baking_rights?&all=true".

To reduce the load of a node in presence of proxy clients, deployment
should be done as follows (we suppose there's a single node for
simplicity):

* Deploy the node as usual

* In front of the node, put multiple HTTP caches (let's avoid the term
  proxy here, to disambiguate with the proxy client) that cache the
  following RPCs:

  * "/chains/<chain>/blocks/<block_id>/context/raw/bytes/"

  * "/chains/<chain>/blocks/<block_id>/protocols"

  * "/chains/<chain>/blocks/<block_id>/header"

  Intercepting "../raw/bytes" is required because proxy clients call
  it a lot, as described above.

  Intercepting "../header" is recommended, because the proxy client
  calls this RPC when it starts honoring a request locally, i.e. when
  it starts performing a computation that would happen on the node
  with a regular client.

  Note that it is safe to cache these RPCs, because the corresponding
  data is immutable (if it's there it won't change in the future).

Regarding clients, either:

* Use proxy clients

* Or intercept requests of regular clients, and honor them by spawning
  proxy clients on the fly, in front of the setup described in the
  previous list.

We refer to the proxy mode's merge request for details regarding how
we did the benchmarks that led us to the conclusions of this section.

Light mode
**********

The proxy mode, described in a dedicated tutorial, is an execution
mode where the Octez client avoids some RPC calls to the node,
especially computation-intensive RPCs. It does so by requesting the
data it needs from the node using RPCs (that are not computation-
intensive), and uses this data locally to perform computations by
itself, whenever possible.

This page describes the *light* mode, a variant of the proxy mode,
where the client also performs RPCs locally whenever possible.
However, contrary to the proxy mode, the light mode provides a high
level of security. For that, it obtains its data from multiple
(hopefully unrelated) node endpoints, and makes sure that all
endpoints send the same data, by using *Merkle proofs*. Such proofs
make it very hard for unrelated endpoints to craft fake data.

This mode is akin to a light client or *thin client* in Bitcoin terms.

While the existing implementation of the light mode is entirely
functional, it still has room for improvement. For instance,
communications over the network can be reduced. Users are encouraged
to share their experience, by submitting issues here on GitLab.


Executing commands in light mode
================================

The CLI interface of the client in light mode (the *light client* for
short) is the same as the default client. To turn the light mode on,
you must pass two arguments to "octez-client":

* "--mode light", and

* "--sources sources.json".

The "sources.json" file contains:

* The list of endpoints to contact for retrieving data, as a list of
  URIs in the "uris" field. These endpoints are added to the main
  endpoint, either specified by option "--endpoint", if given, or to
  the default endpoint ("localhost:8732"), otherwise. To avoid being
  surprised by the automatic addition of the default endpoint, it is
  recommended to supply option "--endpoint" whenever "--sources" is
  supplied, valued as one member of the "uris" field in "--sources"
  (e.g., the first member).

* An optional "min_agreement" field, which must be a float from "0.0"
  (excluded) to "1.0" (included). This field specifies the ratio of
  endpoints that must agree for data to be accepted. The default value
  is "1.0", which means that all endpoints must agree for data to be
  considered valid (no rogue endpoint is tolerated).

Here is an example of a valid "--sources" file:

   { "min_agreement": 1.0,
     "uris": [
       "http://localhost:19733",
       "http://localhost:19735"
     ]
   }

Because computations done locally are protocol dependent, the light
mode has to be configured for a specific protocol. However, the light
mode does not support all protocols. Execute "octez-client list light
protocols" to see the supported protocols. It is expected that, at any
given time, it should support "Alpha", the current protocol of
Mainnet, and the current protocol proposal on Mainnet at the time of
release, if any.


Examples with the sandbox
=========================

In this section, we show examples of usage of the light mode when
using the sandboxed node. For convenience we repeat instructions for
the sandboxed mode here, but refer the reader to the sandboxed mode
page for further details.

In a terminal, start a sandboxed node:

   $ ./src/bin_node/octez-sandboxed-node.sh 1 --connections 1
     # This node listens to p2p events on localhost:19731
     # RPC server of the node is reachable at localhost:18731

Leave that terminal running. In a second terminal, start another node:

   $ ./src/bin_node/octez-sandboxed-node.sh 2 --connections 1
     # This node listens to p2p events on localhost:19732
     # RPC server of the node is reachable at localhost:18732

Leave that terminal running. In a third terminal, prepare the
appropriate environment for using the light client (from now on, all
commands happen in this terminal):

   $ eval `./src/bin_client/octez-init-sandboxed-client.sh 1`

Then upgrade the node to protocol alpha:

   $ octez-activate-alpha  # Triggers output in terminal of first node
   $ octez-client bake for bootstrap1  # Triggers output in terminal of first node

To avoid warnings being printed in upcoming commands (optional):

   $ export TEZOS_CLIENT_UNSAFE_DISABLE_DISCLAIMER=y

The last step before being able to use the light client is to prepare
the JSON file passed to "--sources". In our scenario, this file
specifies the two endpoints to use:

   $ echo '{ "uris": [ "http://localhost:18731", "http://localhost:18732" ] }' > sources.json

You're now ready to use the light client. For example, bake a block
(note that this command may take up to a few minutes to complete):

   $ octez-client --endpoint http://localhost:18731 --mode light --sources sources.json bake for bootstrap1
   Apr  8 16:42:24.202 - alpha.baking.forge: found 0 valid operations (0 refused) for timestamp 2021-04-08T14:42:24.000-00:00 (fitness 01::0000000000000004)
   Injected block BMAHozsNCos2

As you may have noticed, the block is baked when the "boostrap1" baker
has rights to bake, which explains the possible delay of up to a few
minutes. If you are in a hurry, you may want to bake for any baker, by
replacing "bake for bootstrap1" with "bake for --minimal-timestamp".
We will use this option for the next times.

Well, that doesn't seem very different from what the default client
would return. Indeed, it's the same; that was the point! To see what
the light client is doing differently, you may use the environment
variable "TEZOS_LOG". Set it as follows:

   $ export TEZOS_LOG="light_mode->debug"

Variable "light_mode" shows how the light mode is obtaining data from
the different endpoints.

For convenience, let's define an alias before continuing, to save
keystrokes and the "protocol of light mode unspecified" warning:

   $ alias light-client="octez-client --endpoint http://localhost:18731 --mode light --sources sources.json"

And then bake a new block:

   $ light-client bake for --minimal-timestamp
   Apr  8 16:49:28.172 - light_mode: light mode's core created for chain main and block head
   Apr  8 16:49:28.173 - light_mode: API call: do_rpc v1
   Apr  8 16:49:28.175 - light_mode: integrated data for key v1 from one endpoint, about to validate from 1 other
   Apr  8 16:49:28.175 - light_mode:   endpoints
   Apr  8 16:49:28.177 - light_mode: API call: get v1;constants
   Apr  8 16:49:28.177 - light_mode: API call: get v1;first_level
   Apr  8 16:49:28.177 - light_mode: API call: do_rpc pending_migration_balance_updates
   Apr  8 16:49:28.179 - light_mode: integrated data for key pending_migration_balance_updates from one endpoint,
   Apr  8 16:49:28.179 - light_mode:   about to validate from 1 other endpoints
   Apr  8 16:49:28.180 - light_mode: API call: get pending_migration_balance_updates
   ...
   ... A lot of output prefixed with light_mode: ...
   ...
   Injected block BMdbKufTymQJ

Here is the meaning of these lines:

* Line "light mode's core created" indicates that the light mode was
  initialized. It should be printed once per block being inspected.

* Line "API call: do_rpc v1" indicates that the light mode needs the
  data associated to the low-level storage's key "v1"

* Line "integrated data for key v1 ..." indicates that the light mode
  obtained data for "v1" from a single endpoint and that it is about
  to fetch Merkle proofs for this key from other endpoints.

* Lines "API call: get ..." indicate that "octez-client" is requesting
  data from the light mode's cache. In this snippet, after the light
  mode gathered data for key "v1", the client is requesting data for
  the children keys "v1;constants" and "v1;first_level" (the ";"
  indicates  nesting). This example shows how the light mode sometimes
  batches requests, to avoid querying many keys in a row. Here it did
  a single request for "v1" instead of doing one request for
  "v1;constants" and a second one for "v1;first_level".

To see that computations are done locally by the light client, we
refer to the proxy mode's documentation. Debug variables of the proxy
mode apply to the light mode, as internally, the light mode is a more
complex instance of the proxy mode.


How to deploy to relieve nodes from some RPCs
=============================================

Again, we refer to the corresponding section in the page of the proxy
mode. The exact same recommendations apply for the light mode.

Setting up the node
*******************

For running a node in the Tezos network with a basic setup, see the
introductory section on running the node.

This section describes different configuration possibilities provided
by the Octez node:

* tune various parameters of the node using flexible combinations of:
  a configuration file, command-line options, and environment
  variables

* specify the Tezos network to connect to, which can be the Mainnet or
  different test networks.

* configure the amount of history kept by the node according to
  different tradeoffs, using history modes

* rapidly catch up with a given (main or test) network by loading
  network snapshots

* set up infrastructure for continuosly monitoring the node to help
  spotting operational or efficiency issues

* set up infrastructure for continuosly monitoring the chain consensus
  spotting operational or efficiency issues

These possibilities are described in the following pages.

* Node Configuration

  * Node configuration file

  * RPC parameters

  * P2P parameters

  * Shell parameters

* Connecting to a Network

  * Test Networks

  * Built-In Networks

  * Custom Networks

  * Alias Versus Explicit Configuration

* History modes

  * History modes in a nutshell

  * Setting up a node in full mode

  * Setting up a node in rolling mode

  * Setting up a node in archive mode

  * Keeping additional cycles

  * Switching between node's modes

* Snapshots

  * Importing a snapshot

  * Exporting a snapshot

* Monitoring an Octez Node

  * Starting a node with monitoring

  * Gathering data

  * Dashboards

* Monitoring the consensus protocol

  * Run a Tezos node

  * Run a Teztale server

  * Run a Teztale archiver

  * Visualize your data

Node Configuration
******************

The Octez node can be configured in flexible ways to control various
aspects of its behavior, such as RPC, P2P, or shell parameters, the
directory for storing data, logging, and so on. These aspects can be
customized in two different ways:

* by supplying **options** on the command line when running the node

* by specifying **parameters** in a configuration file for the node

When the same parameter is set both in the configuration file and
using a command line option, the value on the command line is taken
into account (and the configuration file is not updated).

The list of configurable options on the command line interface (CLI)
can be obtained using the following command:

   octez-node run --help


Node configuration file
=======================

Parameters in the configuration file can be specified in two different
ways:

* by creating and updating the configuration file using the "config"
  command of "octez-node". This covers a subset of the CLI of the
  "run" command of "octez-node" mentioned above.

  The list of parameters configurable via the "config" command can be
  obtained using the following command:

     octez-node config update --help

* by directly editing the configuration file. This allows to specify
  all the available configuration parameters, including some that
  cannot be set using the options of the "config" and "run" commands
  of "octez-node", for example network parameters such as connection
  or authentication timeouts.

  The list of configurable parameters in the configuration file can be
  obtained using the following command:

     octez-node config --help

  This command also explains the role of each option and parameter and
  the range of possible values.


The config command
------------------

   ./octez-node config init

This will initialize a configuration file for the node in "$HOME
/.tezos-node/config.json", using default values. It only specifies
that the node will listen to incoming connections on socket address
"[::]:9732".

Warning:

  In most of the cases, the default configuration file is sufficient
  as is, or slightly adjusted. The following instructions detail how
  to adjust the node parameters, but you are advised to only change
  parameters that you fully understand. Don't blindly use the values
  in the examples below; they are just examples!

The easiest way to amend this default configuration is to use

   # Update the config file:
   octez-node config update <…>
   # Check your new values:
   octez-node config show
   # If you want to restart from an empty cfg file:
   octez-node config reset <…>

However, note that the "network" configuration parameter, needed to
run the node on a network other than the default one (Mainnet), can
only be defined when the configuration file is initialized (using
"init"), and cannot be updated later (using "update"). See the
instructions for running the node in test networks.

For example, the following script initializes and fills a
configuration file using several command-line options:

   # [remove config file if exists]
   rm -f tmp/config.json
   # [initialize config file]
   ./octez-node config init --config-file=tmp/config.json --network=sandbox
   # [update config file]
   ./octez-node config update --config-file=tmp/config.json --data-dir=tmp \
     --peer="[::]:10732" --peer="192.168.1.3:9733" \
     --private-mode \
     --rpc-addr="localhost:8733" --net-addr="1.2.3.4" \
     --connections=25 \
     --log-output="octez-node.log" \
     --history-mode=full
   # [show config file]
   ./octez-node config show --data-dir=tmp


Editing the configuration file
------------------------------

All blockchain data is stored under "$HOME/.tezos-node/" by default.
You can change this by doing "./octez-node config update --data-dir
</somewhere/in/your/disk>".

To run multiple nodes on the same machine, you can duplicate and edit
"$HOME/.tezos-node/config.json" while making sure they don't share the
same "data-dir". Then run your node with "./octez-node run --config-
file=</path/to/alternate_cfg>".

Here is an example configuration file with several parameters
specified. Incidentally, it is exactly the one resulting from the
command sequence above.

   { "data-dir": "tmp", "rpc": { "listen-addrs": [ "localhost:8733" ] },
     "p2p":
       { "bootstrap-peers": [ "[::]:10732", "192.168.1.3:9733" ],
         "listen-addr": "1.2.3.4", "private-mode": true,
         "limits":
           { "connection-timeout": 10, "min-connections": 12,
             "expected-connections": 25, "max-connections": 37,
             "max_known_points": [ 200, 150 ],
             "max_known_peer_ids": [ 200, 150 ] } },
     "log": { "output": "octez-node.log", "advertises_level": true },
     "shell":
       { "chain_validator": { "synchronisation_threshold": 6 },
         "history_mode": "full" }, "network": "sandbox" }

Thus, you can start with the initial configuration file, and
eventually modify it using "config update" commands and/or using a
text editor, to define any set of node parameters that are suitable
for you (recall that the former covers a subset of parameters, while
the latter covers the full set).

The rest of this page provides more details about the use of some
selected configuration options/parameters.


RPC parameters
==============

RPC parameters allow to customize the JSON/RPC interface, by defining
for instance network addresses (corresponding to node's network
interfaces) to listen for RPC requests on, or a certificate/key file
necessary when TLS is used.


Access Control Lists
--------------------

Access to RPC endpoints can be restricted using an Access Control
List. The default policy grants full access to clients connecting from
the local machine, but only allows *safe* endpoints if accessed
remotely (see Default ACL for RPC for details). One important
exception to this rule is when the node listens on the special address
"0.0.0.0", which means roughly "listen to requests addressed to any
address" and hence all requests coming to such a node are treated as
coming from a remote host. It is worth remembering that it's
unimportant where the request actually comes from. The policy is
defined for an address on which the node is configured to listen to
incoming RPC requests. A custom list can be put in the configuration
file. In the "rpc" object a key "acl" can be included, containing a
list of policies for different listening addresses:

   {
      "rpc": {
        "acl": [
          {
            "address": "localhost:8732",
            "blacklist": ["GET /chains/**", "GET /network/points/*/unban"]
          }
        ]
      }
   }

The "address" field specifies a binding address this rule applies to.
Port can be omitted, in which case the rule applies to any port on
that address. Note that this does not automatically enable RPC on that
address, to do that the address must be included in "listen-addrs" or
passed by command-line argument "--rpc-addr" when starting the node.

Note:

  Both listening addresses and ACL addresses are resolved at the
  node's startup (they're not re-resolved at any point after that),
  before matching. This means that only IP addresses are ever matched,
  so if e.g. "localhost" is written in the policy and listening
  address is set "127.0.0.1", it *does* match. The converse is also
  true.

Warning:

  Note that "0.0.0.0" is a specific address, distinct from any other.
  It is usually treated specially by operating systems to mean "any
  address". However, the ACL configuration **does not** follow this
  rule, because matching addresses to rules would have become
  difficult and potentially very confusing. Therefore if a node is
  configured to listen on address "0.0.0.0", a policy may be defined
  for "0.0.0.0" specifically (otherwise, the default policy will apply
  to it). If one wishes to apply different policies to different
  addresses, multiple "rpc-addr" switches can be used to listen on
  those specific addresses and then separate policies can be
  configured for them.

Additionally either the "whitelist" **or** the "blacklist" field must
be specified (but not both), containing a list of paths which should
be black-listed or white-listed. Each element in the list is an API-
endpoint (that can be passed to e.g. the "octez-client rpc" command).
It may be preceded by a capitalized HTTP method name. In the absence
of the method, the rule applies to all methods. The path can include
an "*" character, which stands for any whole path segment (i.e. it's
not allowed to mix "*" with other characters in a path segment). A
"**" stands for any possible path suffix.

Additionally "--allow-all-rpc" CLI option to "octez-node" can be used
to simply allow all RPC endpoints on a given address. When passed to
"octez-node config" command, this option modifies the "config.json"
file, putting an appropriate ACL there. When passed to "octez-node
run", it overrides the settings of the "config.json" for this
particular run, without modifying the file.

Danger:

  Exposing all RPCs over the public network is extremely dangerous and
  strongly advised against. It opens the door widely to DoS attacks
  and allows anyone to manipulate the node's configuration, inject
  blocks and in general harm the node's state. Even more discouraged
  is exposing all RPCs on the special "0.0.0.0" address.

Warning:

  Rules are always searched from the beginning of the list to the end
  and the first matching address is returned. Therefore if one wants
  to put one rule on a specific port on a given host and another rule
  for all other ports on the same host, then more specific rules
  should always be written *first*. Otherwise they'll be shadowed by
  the more general rule.


Examples
~~~~~~~~

   $ octez-node run --rpc-addr localhost:8732

In this case the RPC is only available for requests coming from
"localhost" (i.e. "127.0.0.1"). There's no need configure the ACL, as
an allow-all policy is applied to the local host by default.

   $ octez-node run --rpc-addr localhost:8732 --rpc-addr 192.168.0.3:8732

In this example the RPC is available to both "localhost" and to the
local network (assuming the node does have address "192.168.0.3" in
that network). However, different policies apply to each address. For
"localhost" an allow-all policy will be selected as before, but
requests addressed to "192.168.0.3" will be filtered by the default
ACL (see below).

In this last case, to listen on both "localhost" and local network, it
might be tempting to listen on the special "0.0.0.0" address:

   $ octez-node run --rpc-addr 0.0.0.0:8732

"0.0.0.0" is a special address, not attached to any particular
networking interface. Instead it tells the OS to route messages
incoming to **all** interfaces to the node. However, the ACL mechanism
does not attach such special significance to this address. It will
apply to this listening address a policy written for it specifically
and in the absence thereof – a default policy for **remote
addresses**. Thus, even if a request is coming through a local
interface, it does not matter, it'll still be treated as if it came
from a remote address.

A common situation is when one wants to accept only safe RPC requests
coming from remote hosts, but enable all RPCs for localhost (which is
for instance necessary to perform baking and attesting). Since all
RPCs are available to localhost by default, it is sufficient to open
another listening address:

   $ octez-node run --rpc-addr localhost --rpc-addr 0.0.0.0

The "--allow-all-rpc" switch can be used to open all RPCs on a
specific address:

   $ octez-node run --rpc-addr 192.168.0.3 --allow-all-rpc 192.168.0.3

Note that the addresses of both "--rpc-addr" and "--allow-all-rpc"
switches should match. In particular remember that "0.0.0.0" is a
specific address and won't match anything else except for itself, even
though the underlying OS might treat it differently. Also be advised
that using this option is discouraged as dangerous, especially when
applied to the special "0.0.0.0" address.

Both "--rpc-addr" and "--allow-all-rpc" switches can be used multiple
times in order to accommodate each specific setup.


Default ACL for RPC
~~~~~~~~~~~~~~~~~~~

The default ACL for RPC depends on the listening address that the node
is using.

If the listening address resolves to the loopback network interface,
then full access to all endpoints is granted. Note that it does not
matter from which machine the client is really making a request, but
only what the listening address is. This guarantees that insecure
endpoints can only be accessed from "localhost".

If the listening address is a network address, then a more restrictive
policy applies. Its main purpose is to protect nodes from attacks.
These attacks can take two main forms:

* spamming the node with costly requests (denial of service attack)

* breaking the node by forcing it to perform a risky operation

Thus all costly or risky endpoints are blocked by default. This can be
relaxed or tightened by modifying the configuration file. It's worth
noting that this default policy among other things disallows baking
and attesting by bakers running on remote servers, because endpoints
such as "/injection/block" are not open remotely. Rather than opening
them remotely, the recommended practice for baking is to run a node
locally listening to "localhost", with the default ACL policy.

The following is the default ACL policy for the node, hard-coded in
src/lib_rpc_http/RPC_server.ml (remember to replace
"any.public.address" with an IP address or a domain name that you'll
be actually listening on):

   [ { "address": "127.0.0.1", "blacklist": [] },
     { "address": "any.public.address",
       "whitelist":
         [ "GET/chains/*/blocks", "GET/chains/*/blocks/*",
           "GET/chains/*/chain_id", "GET/chains/*/checkpoint",
           "GET/chains/*/blocks/*/context/adaptive_issuance_launch_cycle",
           "GET/chains/*/blocks/*/context/big_maps/*/*",
           "GET/chains/*/blocks/*/context/cache/**",
           "GET/chains/*/blocks/*/context/constants",
           "GET/chains/*/blocks/*/context/contracts/**",
           "GET/chains/*/blocks/*/context/delegates/**",
           "GET/chains/*/blocks/*/context/denunciations",
           "GET/chains/*/blocks/*/context/issuance",
           "GET/chains/*/blocks/*/context/issuance/*",
           "GET/chains/*/blocks/*/context/liquidity_baking/*",
           "GET/chains/*/blocks/*/context/merkle_tree/**",
           "GET/chains/*/blocks/*/context/merkle_tree_v2/**",
           "GET/chains/*/blocks/*/context/nonces/*",
           "GET/chains/*/blocks/*/context/sapling/**",
           "GET/chains/*/blocks/*/context/seed_computation",
           "GET/chains/*/blocks/*/context/selected_snapshot",
           "GET/chains/*/blocks/*/context/total_frozen_stake",
           "GET/chains/*/blocks/*/context/total_supply",
           "GET/chains/*/blocks/*/hash", "GET/chains/*/blocks/*/header",
           "GET/chains/*/blocks/*/header/**",
           "GET/chains/*/blocks/*/helpers/current_level",
           "GET/chains/*/blocks/*/live_blocks",
           "GET/chains/*/blocks/*/metadata",
           "GET/chains/*/blocks/*/metadata_hash",
           "GET/chains/*/blocks/*/minimal_valid_time",
           "GET/chains/*/blocks/*/operation_hashes",
           "GET/chains/*/blocks/*/operation_hashes/**",
           "GET/chains/*/blocks/*/operation_metadata_hashes",
           "GET/chains/*/blocks/*/operations",
           "GET/chains/*/blocks/*/operations/**",
           "GET/chains/*/blocks/*/operations_metadata_hash",
           "GET/chains/*/blocks/*/protocols",
           "GET/chains/*/blocks/*/resulting_context_hash",
           "GET/chains/*/blocks/*/votes/**", "GET/chains/*/invalid_blocks",
           "GET/chains/*/invalid_blocks/*", "GET/chains/*/is_bootstrapped",
           "GET/chains/*/levels/*", "GET/chains/*/mempool/filter",
           "GET/chains/*/mempool/pending_operations", "GET/config/history_mode",
           "GET/config/network/user_activated_protocol_overrides",
           "GET/config/network/user_activated_upgrades",
           "GET/config/network/dal", "GET/network/stat", "GET/network/version",
           "GET/network/versions", "GET/protocols",
           "GET/protocols/*/environment", "GET/version",
           "POST/chains/*/blocks/*/context/contracts/*/big_map_get",
           "POST/chains/*/blocks/*/context/seed", "POST/injection/operation" ] } ]


HTTP Caching Headers
--------------------

It is possible to enable http caching headers in the RPC responses
with the "--enable-http-cache-headers" option. This feature is
disabled by default.

When enabled, the RPC server will support max-age header. The header
"Cache-control: public, max-age: <duration>" will be included in the
response headers of head related queries ("/chains/main/blocks/head*")
for responses that are cacheable. This also works on paths that are
relative to "head" such as "head-n" and "head~n". The response is
cacheable throughout the "<duration>" of the head block's consensus
round where "<duration>" is the remaining time until the estimated end
time of the consensus round. If a response should not be cached, the
RPC server will not include any cache control headers.

The RPC server will also support the conditional request header If-
None-Match and include the "ETag" field in every head related query.
The value of the "ETag" will be set to the block hash that the query
is related to. If the client sends a request with "If-None-Match:
<comma-separated list of ETag values>" header and the block hash is
included in the list of etag values, the RPC server will respond with
a "304 Not Modified" status code with an empty body.

This feature is useful when running the RPC server behind a reverse
proxy that supports automatic content caching (eg. NGINX's proxy_cache
setting). Beware that enabling this feature adds a non-negligible
performance overhead (up to 10-15% slower) to every head related query
as the RPC server needs to perform additional checks and calculations.
Consequently, it is advised to enable thie feature only when operating
the RPC server behind appropriate caching infrastructure.


P2P parameters
==============

P2P parameters allow to customize aspects related to the peer-to-peer
network layer, such as:

* defining the bootstrap peers (and ports),

* defining the ports listening to peers,

* setting the greylist timeout,

* running the node in private mode.


Listening ports
---------------

By default, the node listens to incoming connections from peers on
port "9732", on any of the available network interfaces on the node's
host. This behavior can be changed by passing an IP address and
optionally a port number ("addr:port"), using either the "--net-addr"
configuration option or the P2P "listen-addr" configuration parameter.
Note that the IP address is only used for selecting an active network
interface (technically, it is only passed to the "bind()" function of
the socket API). It is also possible to advertise to peers a different
port than the binding port using "--advertised-net-port" configuration
option or the P2P "advertised-net-port" configuration parameter.
Currently it is only possible to specify an advertised port, but not
an IP address.

Note:

  If the node is run on a machine "M" not disposing of a public IP
  address, subject to NAT, a port forwarding rule has to be added on
  the NAT server "S" on the listening port towards machine "M": "S:p1
  -> M:p2", where "p1" is "advertised-net-port" and "p2" is the port
  specified by "listen-addr". Alternatively, if the "advertised-net-
  port" is not configured, "p1" must be the same as "p2".As a
  consequence, if a second node has to be run behind the same server
  on a machine "M'" (possibly the same as "M"), it should be
  configured to listen on a different port "p2'", to allow defining
  another forwarding rule for it: "S:p1' -> M':p2'", where "p1'" is
  "advertised-net-port" configured for the second node "M'".Many
  routers can be configured with UPnP to open ports dynamically, so
  the port forwarding can be initiated by the internal host without
  any manual modification on the router. This is not possible for
  corporate networks with UPnP disabled, but is typically handy for
  home routers, or other networks where this option is available.


Private node
------------

The node can be set in private mode with the option "--private-mode"
so that:

* it doesn't connect to any peer other than those provided with "--
  peer" or in "bootstrap-peers"

* the peers connected to a private node don't advertise it in the list
  of peers sent to their neighborhood

This feature is especially useful to hide a sensitive node that signs
operations.

For example we could have a set up with two nodes, a private one
connected uniquely with a public one. The public node runs on a VPS,
connects normally to the network and keeps an up to date state of the
network while the private node runs at your home and is in charge of
injecting and signing operations with a hardware wallet.

   octez-node run --rpc-addr [::] --private-mode \
                                  --no-bootstrap-peers \
                                  --synchronisation-threshold=1 \
                                  --connections 1 \
                                  --peer <public-node-ip>


Shell parameters
================

Configuration options/parameters for the shell allow tuning the
working of the validation subsystem.

In particular, the synchronization heuristics implemented by the chain
validator can be controlled using parameters such as the
synchronization threshold or the latency, described in the
documentation of the synchronization heuristics.

Configuration parameters for the context's storage can also be done
through environment variables, see Context.

Connecting to a Network
***********************

Tezos is run on several networks, such as Mainnet (the main network)
and various test Networks. Some users may also want to run their own
networks for various reasons. Networks differ in various ways:

* they start from their own genesis block;

* they have different names so that each node can choose the network
  to connect to;

* they may run different protocols (or upgrade protocols at different
  moments);

* protocols may run with different constants (for instance, test
  networks move faster);

* they have different bootstrap peers (nodes that new nodes connect to
  initially);

* some networks may change the protocol without going through the
  regular voting process, via user-activated upgrades or user-
  activated protocol overrides.

The Octez node can be configured to connect to a given network when it
is started. By default, the node connects to Mainnet. To connect to
other networks, you can either use one of the Built-In Networks or
configure the node to connect to Custom Networks. See also Alias
Versus Explicit Configuration for a discussion regarding what happens
when you update your node, in each case.


Test Networks
=============

Mainnet is the main Tezos network, but is not appropriate for testing.
A number of test networks are available to this end. Test networks
usually run with different constants to speed up the chain.

Each test network listed there also indicates a **faucet** delivering
test tokens. Enter the public key hash of any test account on the
corresponding website to receive test tokens.

The faucet interface is meant for interactive use. If you are looking
for a scriptable alternative, check the programmatic faucet frontend
which includes the "get-tez" tool.


Built-In Networks
=================

The simplest way to select the network to connect to is to use the "--
network" option for selecting a test network when you initialize your
node configuration.

For instance, to run on Ghostnet:

   octez-node config init --data-dir ~/tezos-ghostnet --network ghostnet
   octez-node identity generate --data-dir ~/tezos-ghostnet
   octez-node run --data-dir ~/tezos-ghostnet

Note:

  Once initialized, the node remembers its network settings on
  subsequent runs and reconnects to the same network every time you
  run it. If you specify a different network when running the node
  again, it will refuse to start. In order to switch to a different
  network you need to either reinitialize it with a different data
  directory using the "--data-dir" option or remove everything from
  the existing data directory, which defaults to "~/.tezos-node" (and
  also initialize again).

The "--network" option is not case-sensitive and can be used with the
following built-in networks:

* "mainnet" (this is the default)

* "sandbox"

* "ghostnet"

If you did not initialize your node configuration, or if your
configuration file contains no "network" field, the node assumes you
want to run Mainnet. You can use the "--network" option with "octez-
node run" to make sure your node runs on the expected network. For
instance, to make sure that it runs on Ghostnet:

   octez-node run --data-dir ~/tezos-ghostnet --network ghostnet

This command will fail with an error if the configured network is not
Ghostnet. The node also displays the chain name (such as
"TEZOS_MAINNET") when it starts. Also mind opening the RPC interface
as appropriate.

The list of built-in networks is in
src/lib_node_config/config_file.ml. Octez developers edit the
"builtin_blockchain_networks_with_tags" variable in this file to add
or remove built-in networks.


Custom Networks
===============

If the network you want to connect to is not in the list of built-in
networks, you need a corresponding network configuration file. There
are several ways to set that up. If you have an appropriate file, you
can specify it with the "--network" argument when you initialize your
node configuration (see above), and the node will load it. If you know
a URL from which the file can be downloaded, you can also specify it
with "--network". The node will then download the config
automatically. The network configuration should be in JSON format,
containing an object matching the contents of the "network" field in
"config.json" (see below for an example, and the node configuration
documentation for reference).

Note:

  The contents of the network configuration file will be saved in your
  node configuration file "config.json", so it won't be downloaded
  again on subsequent runs of the node.

Finally you can manually edit the main configuration file of the node
("config.json"). Here is an example configuration file for Mainnet:

   {
     "p2p": {},
     "network": {
       "genesis": {
         "timestamp": "2018-06-30T16:07:32Z",
         "block": "BLockGenesisGenesisGenesisGenesisGenesisf79b5d1CoW2",
         "protocol": "Ps9mPmXaRzmzk35gbAYNCAw6UXdE2qoABTHbN2oEEc1qM7CwT9P"
       },
       "chain_name": "TEZOS_MAINNET",
       "old_chain_name": "TEZOS_BETANET_2018-06-30T16:07:32Z",
       "incompatible_chain_name": "INCOMPATIBLE",
       "sandboxed_chain_name": "SANDBOXED_TEZOS_MAINNET",
       "user_activated_upgrades": [
         {
           "level": 28082,
           "replacement_protocol": "PsYLVpVvgbLhAhoqAkMFUo6gudkJ9weNXhUYCiLDzcUpFpkk8Wt"
         },
         {
           "level": 204761,
           "replacement_protocol": "PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP"
         }
       ],
       "user_activated_protocol_overrides": [
         {
           "replaced_protocol": "PsBABY5HQTSkA4297zNHfsZNKtxULfL18y95qb3m53QJiXGmrbU",
           "replacement_protocol": "PsBabyM1eUXZseaJdmXFApDSBqj8YBfwELoxZHHW77EMcAbbwAS"
         },
         {
           "replaced_protocol": "PtEdoTezd3RHSC31mpxxo1npxFjoWWcFgQtxapi51Z8TLu6v6Uq",
           "replacement_protocol": "PtEdo2ZkT9oKpimTah6x2embF25oss54njMuPzkJTEi5RqfdZFA"
         },
         {
           "replaced_protocol": "PtHangzHogokSuiMHemCuowEavgYTP8J5qQ9fQS793MHYFpCY3r",
           "replacement_protocol": "PtHangz2aRngywmSRGGvrcTyMbbdpWdpFKuS4uMWxg2RaH9i1qx"
         }
       ],
       "default_bootstrap_peers":
         [
          "boot.tzinit.org",
          "boot.tzboot.net",
          "boot.tzbeta.net"
        ]
     }
   }

This is equivalent to doing "config init --network mainnet", or using
""network": "Mainnet"" in the configuration file (or to doing nothing,
as Mainnet is the default), except that you will not automatically get
updates to the list of bootstrap peers and user-activated upgrades
(see Alias Versus Explicit Configuration).

* "genesis" is the description of the genesis block, i.e. the first
  block of the chain. Inspect the genesis block using "octez-client
  rpc get /chains/main/blocks/0" to find these values.

* "chain_name" is the name of the network (nodes only talk to other
  nodes which use the same network name).

* "old_chain_name" is usually the same as "chain_name", except for
  networks that were renamed.

* "incompatible_chain_name" is a name which must be different from
  "chain_name" and "old_chain_name". It is thus ensured to be
  incompatible. It is used for testing purposes.

* "sandboxed_chain_name" is the name of the network in sandbox mode.
  It can be the same as "chain_name" but it is safer to pick a
  different name.

* "user_activated_upgrades" is the list of past user-activated
  upgrades. Each item has a field "level", which is the level at which
  the protocol must be changed, and a field "replacement_protocol",
  which is the hash of the protocol to switch to.

* "user_activated_protocol_overrides" is the list of past user-
  activated protocol overrides. Each item has a field
  "replaced_protocol" and a field "replacement_protocol". Both are
  protocol hashes. If "replaced_protocol" is to be activated using on-
  chain voting, "replacement_protocol" is activated instead.

* "default_bootstrap_peers" is the list of addresses of default
  bootstrap peers. They are only used if "p2p.bootstrap_peers" is not
  present in the configuration file, and "--no-bootstrap-peers" is not
  given on the command-line.


Genesis Parameters
------------------

In addition to the above fields, you can also specify custom genesis
parameters. That is, you can additionally specify the activation key:

   {
     "p2p": {},
     "network": {
       "genesis": { ... },
       "genesis_parameters": {
         "context_key": "sandbox_parameter",
         "values": {
           "genesis_pubkey": "edpk..."
         }
       },
       ...
     }
   }

The "genesis_parameters" object contains:

* "context_key", the name of the key in the context part of the
  storage, whose value must be modified (if omitted, the default
  context key is "sandbox_parameter");

* "values", which contains the protocol parameters.

In the above example, we set the "genesis_pubkey" parameter of
"proto_genesis".

Note that the genesis parameters that you specify in the configuration
file can be overridden by the "--sandbox" parameter on the command-
line. Similarly, if you are using a built-in network and if this
built-in network comes with genesis parameters, you can override them
with "--sandbox".


Alias Versus Explicit Configuration
===================================

The previous sections explained two different ways to configure the
network a node is connecting to:

* alias configuration: using the name (also called the "alias") of an
  existing, built-in network

* explicit configuration: explicitly specifying the parameters of the
  network, which can be an existing or a custom network.

When connecting to existing networks, both options may apply, so here
are some useful explanations to inform your choice.

If you use alias configuration, the configuration file stores the name
of the network to connect to. For instance, if you configured it to
connect to Ghostnet, it will contain something like:

   {
     "p2p": {},
     "network": "ghostnet"
   }

For Mainnet, it would contain "mainnet", or nothing as this is
actually the default.

When you update your node to new versions, built-in network parameters
may change. For instance, the list of bootstrap peers may be updated
with new addresses; new user-activated upgrades or user-activated
protocol overrides may be added. Because the configuration file only
contains the name of the network and not its parameters, it will
automatically use the updated values.

However, if you use explicit configuration, the configuration file
will no longer contain an alias such as "mainnet" or "ghostnet".
Instead, it will explicitly contain the list of bootstrap peers, user-
activated upgrades and user-activated protocol overrides that you
specify. This means that when you update your node, the updated values
will not be used.

As a consequence, if you use explicit configuration, you need to
update its parameters yourself when you update your node, unless you
wish to keep the old network parameters.

History modes
*************

History modes allow a node to require less disk storage. Indeed,
depending on the chosen history mode, some parts of the complete chain
history can be deleted as they are not required anymore.

Three history modes are provided:

* **Full**

  The node stores the minimal data since the genesis required to
  reconstruct (or 'replay') the complete chain's ledger state.

  * Upsides:

    * Can synchronize using a snapshot.

    * Keep all necessary information in order to reconstruct all the
      chain's ledger state (balances, contracts, etc..) since the
      genesis block.

    * Requires little disk storage.

    * Suitable for bakers as you can still query any block information
      or operation at any level.

    * Help other nodes to bootstrap and synchronize with the chain.

  * Downsides:

    * The node is not able to query the balances or staking rights
      before the current savepoint.

    * Disk storage slowly increases as the node keeps the history.

  See how to set up a full node.

* **Rolling** (default mode with 1 additional cycle)

  This is the lightest mode as it only maintains a minimal rolling
  fragment of the chain data so the node can still validate new blocks
  and synchronize with the head.

  * Upsides

    * Only requires a minimal and bounded disk storage.

    * Can run on low resources architectures.

    * Can be bootstrapped within minutes, thanks to a rolling snapshot
      import.

  * Downsides:

    * The node is not able to query block information of balances and
      staking rights before the current savepoint.

    * The node does not help other nodes to bootstrap as it is not
      able to send the whole chain history.

  See how to set up a rolling node.

* **Archive**

  This is the heaviest mode as it keeps the whole chain data to be
  able to query any information stored on the chain since the genesis.
  It is particularly suitable for indexers or block explorers.

  * Upsides

    * The whole chain data is available.

  * Downsides

    * Consume an increasing and rather large amount and data storage.

  See how to set up an archive node.


History modes in a nutshell
===========================

+-----------+------------------+-----------------------+----------------------+
|           | Storage amount   | Suitable for bakers   | Operations history   |
|===========|==================|=======================|======================|
| Archive   | High             | Yes                   | Complete             |
+-----------+------------------+-----------------------+----------------------+
| Full      | Limited          | Yes                   | Complete             |
+-----------+------------------+-----------------------+----------------------+
| Rolling   | Low              | Yes*                  | Yes*                 |
+-----------+------------------+-----------------------+----------------------+

(*) Suitable for delegation services if the number of additional kept
cycles is high enough to allow the computation of rewards. See Keeping
additional cycles for details.

History modes use some markers which are used to describe the state of
the storage:

* *checkpoint*: the most recently finalized block of the chain,

* *savepoint*: the lowest level block that contains context data and
  metadata,

* *caboose*: the lowest level known block.

For more details about what data are stored in each mode, refer to The
storage layer.


Setting up a node in full mode
==============================

To run a "full" node you can either use the command line arguments:

   octez-node run --history-mode full

or use your configuration file as described in here:

   { "shell": {
       "history_mode": "full"
   }}

You can then verify that your history mode is set to full by using the
following RPC.

   octez-client rpc get /config/history_mode

   { "history_mode": { "full": { "additional_cycles": 1 } } }

In full mode, the savepoint is the last block which contains its
context and metadata. The caboose is the last known block which is
pruned (that contains partial data).


Setting up a node in rolling mode
=================================

To run a "rolling" node you can either use the command line arguments:

   octez-node run --history-mode rolling

or use your configuration file as described in here:

   { "shell": {
       "history_mode": "rolling"
   }}

Note that, since the rolling mode is the default one, this
configuration is optional.

In "rolling" mode, the caboose is the genesis at its early state, and
then, it is updated to the last known block of the rolling window. The
savepoint is moved in accordance to the number of configured
additional cycles:

   octez-client rpc get /chains/main/levels/caboose
   octez-client rpc get /chains/main/levels/savepoint


Setting up a node in archive mode
=================================

To run an "archive" node you can use the command line arguments: "$
octez-node run --history-mode archive"

Or the configuration file: "{ "shell": {"history_mode": "archive"} }"

If you want to start an "archive" node, it is now mandatory to pass
this argument the first time you launch your node. Indeed, there are
some restrictions when switching from one mode to another.

In "archive" mode, both the savepoint and caboose are located down to
the genesis.


Keeping additional cycles
=========================

When running a node in "full" or "rolling" mode, you have a full
access to the block information history in a sliding window of a given
number of block levels. Indeed, at each new cycle, a garbage
collection phase removes the ledger state and the block metadata
(operation receipts, rewards updates, etc.) of blocks outside the
offset of this sliding window. The number of cycles kept (beyond the
current cycle) is the "blocks_preservation_cycles" protocol parameter
(set to 1 cycle on mainnet) plus a number of additional cycles which
is configurable.

By default, 1 additional cycle is kept for both "full" and "rolling"
nodes. It is possible to increase this parameter to keep more history
or, on the contrary, decrease it to reduce the storage size. For
example, it is possible to run a node with *5* additional cycles. On
mainnet, this would total *6 cycles* of complete history
(approximately four weeks), as we keep 5 cycles beyond the minimal
number of cycles, that is *1 + 5 = 6*.

When initializing your node on an empty storage, you may specify the
history mode and number of additional cycles using "--history-mode
<HISTORY_MODE>:<NB_CYCLES>" when running it. For example, running a
node with "--history-mode rolling:5" would allow full RPC queries of
the 10 previous cycles.

It is also possible to modify the number of additional cycles kept of
a previously configured node, see Switch mode restrictions. When
updating the number of additional cycles to keep on an already
configured node, one must consider that the change may require time.
If the update aims to shrink the number of additional cycles to keep,
it is just a matter of deleting some data, and the operation is
performed instantaneously. If the update aims to increase the number
of additional cycles to keep, one must consider that the switch will
be complete only after waiting for the end of the aforementioned
number of cycles. Indeed, the "cycles window" will be expanded as
blocks/cycles are processed by the node and it won't fetch blocks
prior to the previous limit. For example, switching from "5" to "7"
cycles requires to wait "2" complete cycles to reach the target number
of cycles to keep. On the contrary, switching from "7" to "5" is
instantaneous.


Switching between node's modes
==============================

It is possible to switch between history modes and/or to modify the
number of additional cycles. To do so, it is necessary to restart the
node with the desired history mode and add the flag "--force-history-
mode-switch". This flag is required to prevent erroneous history
switches. Indeed, changing from one history mode to an other can
irremediably remove data from the storage. The history mode switches
must be manipulated with care.

However, as the different modes rely on different storage schemes,
there are some restrictions when switching from one mode to another.

+-----------+-----------+--------+-----------+
| From/to   | Archive   | Full   | Rolling   |
|===========|===========|========|===========|
| Archive   | X         | Yes    | Yes       |
+-----------+-----------+--------+-----------+
| Full      | Yes*      | Yes    | Yes       |
+-----------+-----------+--------+-----------+
| Rolling   | No        | No     | Yes       |
+-----------+-----------+--------+-----------+

(*) Switching from a "full" node to an "archive" one is possible using
the "reconstruct" feature. To do so, run "octez-node reconstruct" on
your node. Note that the storage reconstruction is a long process
that, on the main network, may require more than a week to complete.
Reconstruction also requires a machine with at least 16GB of memory
(for the flattening of the context storage operation) and takes up
about 1TB of storage once completed.

Snapshots
*********

When a node in the Tezos network is started, it has to catch up the
current state of the blockchain (a process known as "bootstrapping"),
making sure that this state is coherent with the whole history of the
ledger. As the chain invariably grows every day, retrieving a full
chain from the peer-to-peer network can be a very long process.
Thanks to the implementation of history modes, it is now possible to
propose an import/export feature: snapshots.  This procedure allows to
gather all the data necessary to bootstrap a node from a single file
within few minutes.


Importing a snapshot
====================

When bootstrapping from a snapshot, the first thing that you want to
be sure of is that the loaded state is not fake.

The snapshot format does not (and cannot) provide any evidence that
the imported block is actually a real point in the history of the
Tezos network blockchain. To avoid to be fooled by a fake chain, it is
**necessary** to carefully check that the block hash of the imported
block is included in the main chain. (Note that all the websites
providing snapshots mention the hash corresponding to the head block
in each snapshot.) This check can be done by comparing the hash to one
provided by another node under the user’s control, or by relying on
social cues to obtain a hash from a large number of trusted parties
which are unlikely to be colluding.

For instance, one may use one or more Tezos block explorers to check
whether this block hash matches an existing block.

After that careful selection and verification of the imported block
hash, you can trust the node with the rest of the procedure. In
particular, you do not need to trust the source of the file. The
snapshot format contains everything necessary for the node to detect
any inconsistency, malicious or not.

This safety comes from the fact that block headers are designed to
make sure that applying a block has the same result for everyone in
the network. To achieve this, a block header includes hashes of its
operations and predecessor, as well as the resulting chain state. The
import process does the same checks, recomputing and checking all the
hashes it encounters in the snapshot.

To bootstrap an Octez node from a file to an empty Tezos node
directory (running this command from an already synchronised node will
not work), run:

   octez-node snapshot import <FILE> --block <BLOCK_HASH> [--data-dir <NODE_DIR>]

The "--block <BLOCK_HASH>" option argument aims to verify that the
block contained in the snapshot is the one that you are expecting to
import.

Warning:

  While importing a snapshot, many checks are performed to ensure the
  consistency of the imported data. In order to speed up the process
  and only if the snapshot's source is highly trusted (or exported by
  yourself), it is possible to disable some checks. The validity of
  the target block will be, of course, ensured. However, the rest of
  the data will be copied directly, without additional consistency
  checks. To do so, use the "--no-check" option.


Snapshot information
--------------------

When retrieving a snapshot, it can be useful to check the actual
content of the snapshot. To do so, the node's "snapshot info" command
can be used to display snapshot's information such as:

   * snapshot's version

   * chain name

   * history mode

   * targeted block hash, level and timestamp

   * ...

This information is displayed by the following command:

   octez-node snapshot info <FILE>

As can be seen in the snapshot information, a snapshot contains
historical data corresponding to a given history mode, which can be:
Full, Archive, or Rolling (see history modes).


Storage reconstruction from a snapshot
--------------------------------------

When importing a "full" snapshot you can **optionally** trigger an
"archive" storage reconstruction using the "--reconstruct" option.
After importing the snapshot's data, all the chain data will be
recomputed. This operation may take a couple of days to complete.


Exporting a snapshot
====================

To export a snapshot, we first select a block hash which will
represent the point in history at which consumers of this snapshot
will start bootstrapping. By default, if no block hash is provided, we
automatically choose a block corresponding to the last checkpoint.
This is important as nodes bootstrapped from this snapshot will not be
able to reorganise their chain below this block (they will set their
checkpoint to this block).

Depending on the snapshot export option, additional history may also
be put in the snapshot file.  By default, the snapshot export command
will create a "full" snapshot. Such a snapshot will contain all the
blocks from a given block hash back to the genesis. Thus, the whole
chain will be exported into the snapshot, from the beginning to the
selected point. This kind of snapshot can only be created from a
"full" or an "archive" node.

   octez-node snapshot export --block <BLOCK>

The "<BLOCK>" hint can be given as a *block hash*, a *block level*, an
alias (*head*, *savepoint* or *checkpoint*) and a relative block
target using the "~", "-" or "+" notation (such as "head~42").

If no "--block <BLOCK>" option is given, the checkpoint level will be
chosen as the default block to export.

By default, the snapshot will be exported into a file with a name
following this pattern
"<NETWORK>-<BLOCK_HASH>-<BLOCK_LEVEL>.<SNAPSHOT_KIND>". A specific
snapshot file name can be given as an additional argument. For
example:

   octez-node snapshot export recent_head_snapshot.full --block head

Note:

  If your node's current storage was created with Octez v13.0 (or
  earlier versions), it is not compatible with the context pruning
  storage layout, and therefore it is no longer possible to export a
  storage snapshot.In order to benefit from recent storage features
  and optimizations, it is strongly recommended to restart your node
  with the latest Octez version and with a fresh, up-to-date storage
  -- either by importing a recent snapshot (that is, exported by an
  Octez node version >= v13.0), or by bootstrapping from scratch if
  this is not possible.


Rolling export
--------------

Rolling snapshots are useful to quickly deploy a node or for testing
and experimentation purposes (such as in a classroom setting), as they
are much smaller. However, to bootstrap a long running node on the
network, we recommend using "full" snapshots to participate into the
network-wide preservation and sharing of chain history.

   octez-node snapshot export <FILE>.rolling --block <BLOCK_HASH> --rolling


Snapshot file format and IPFS
-----------------------------

By default, the snapshot is exported as an archive file (in the ".tar"
format). Such a single archive file is suitable for applying
compression mechanisms. However, the compression of a snapshot file is
not handled by the node.

If one prefers not to export the snapshot as a single archive file, it
is possible to add the flag "--export-format raw" to the export
command. The snapshot is then exported as a folder containing all the
necessary files. As the structure of the snapshot follows the storage
representation which is based on the network's cycles, the major part
of the "raw" snapshot's data is canonical. The block history is thus
represented as *cemented cycles* and will stay as it is for ever. Only
the rest of the file contains data relative to the snapshot's target
block, such as the current incomplete cycle and the block's associated
ledger state. This canonical representation is well suited for
distributing snapshots through IPFS.


Export capabilities
-------------------

The following table recapitulates the different kinds of snapshots
that can be exported from a given history mode node.

+-----------+-----------------+-------------------+
| From/To   | Full snapshot   | Rolling snapshot  |
|===========|=================|===================|
| Archive   | Yes             | Yes               |
+-----------+-----------------+-------------------+
| Full      | Yes             | Yes               |
+-----------+-----------------+-------------------+
| Rolling   | No              | Yes               |
+-----------+-----------------+-------------------+


Snapshot providers
------------------

There are several services providing node snapshots. They create
snapshots of their nodes on a regular basis (usually daily) and make
them available for download. These include:

* Tzinit snapshots

* Lambs on acid

Monitoring an Octez Node
************************

Monitoring the behavior of an Octez node can be partially achieved by
exploring the logs or, more efficiently, through the RPC server. The
use of RPCs is detailed in the RPC documentation and the RPC
references.

Most practically, however, is to use Octez Metrics to gather
information and statistics, which has been integrated directly into
the node since Octez version 14. Users are now able to get metrics
without using an external tool, such as tezos-metrics (which is now
deprecated). The node now includes a server that registers the
implemented metrics and outputs them for each received "/metrics" http
request. So now you can configure and launch your node with a metrics
exporter.


Starting a node with monitoring
===============================

Metrics for the node are always available on the "/metrics" endpoint
of the address and port of the "--rpc-addr" (or "--external-rpc-addr")
argument. One can query this using:

   curl http://<rpc_addr>:<rpc_port>/metrics

Alternatively, the user has an option to gather metrics when no RPC
server is setup at all, or when they want a dedicated server for this
process.


Starting metrics server manually
--------------------------------

The node can be started with its metrics exporter with the option "--
metrics-addr" which takes as a parameter "<ADDR>:<PORT>" or "<ADDR>"
or ":<PORT>".

"<ADDR>" and "<PORT>" are respectively the address and the port on
which to expose the metrics. By default, "<ADDR>" is "localhost" and
"<PORT>" is "9932".

   octez-node run --metrics-addr=<ADDR>:<PORT> …

Note that it is possible to serve metrics on several addresses by
using the option more than once.

You can also add this configuration to your persistent configuration
file through the command line:

   octez-node config init --metrics-addr=<ADDR>:<PORT> ...

   #Or if the configuration file already exists
   octez-node config update --metrics-addr=<ADDR>:<PORT> ...

See the documentation of the node configuration for more information.

A correct setup should write an entry in the logs similar to:

   <date> - node.main: starting metrics server on <addr>:<port>

As some metrics (i.e. node internal metrics) are issued by the node
only, some metrics accessible on the external RPC process metrics
server may lack some data. It is therefore the duty of the user to
aggregate both metrics if needed.


Gathering data
==============


Scraping Octez Metrics
----------------------

Once your node is correctly set up to export metrics, you can scrape
them by querying the metrics server of your node with the request
"/metrics".

Ex.:

   curl http://<node_addr>:<port>/metrics

You will be presented with the list of defined and computed metrics as
follows:

   #HELP metric description
   #TYPE metric type
   octez_metric_name{label_name=label_value} x.x

The metrics that can be exposed by the node can be listed with the
command:

   octez-node dump-metrics

Version 14 of Octez exports metrics from various components of the
node, namely:

* The p2p layer

* The store

* The prevalidator

* The chain validator

* The block validator

* The peer validator

* The distributed database

* The RPC server

* The node version

Each exported metric has the following form:

   octez_subsystem_metric{label_name=label_value;...} value

Each metric name starts with "octez" as its namespace, followed by the
a subsystem name, which is the section of the node described by the
metric. It follows the OpenMetrics specification described here

A metric may provide labeled parameters which allow for different
instances of the metric, with different label values. For instance,
the metric "octez_distributed_db_requester_table_length" has a label
name "requester_kind" which allows this metric to have one value for
each kind of requester.

   octez_distributed_db_requester_table_length{requester_kind="block_header"} x
   octez_distributed_db_requester_table_length{requester_kind="protocol"} y
   ...

Metrics provide information about the node in the form of a gauge that
can increase or decrease (like the number of connections), a counter
that can only increase (like the head level), or a histogram used to
track the size of events and how long they usually take (e.g., the
time taken by an RPC call).

The label value is sometimes used to store information that can't be
described by the metric value (which can only be a float). This is
used for example by the "octez_version" metric that provides the
version within the labels.

For the list of metrics provided by the node, and a typical monitoring
setup, see the following page:

* Supported Open Metrics

  * Example

  * Collecting metrics

Note:

  Most of the metrics are computed when scraped from the node. As
  there is no rate limiter, you should consider scraping wisely and
  adding a proxy for a public endpoint, to limit the impact on
  performance.


Prometheus
----------

Scraping metrics gives you instant values of the metrics. For a more
effective monitoring, you should create a time series of these
metrics.

We suggest using Prometheus for that purpose.

First, you need to install Prometheus from your favourite package
manager, e.g.:

   sudo apt install prometheus

Once installed, you need to add the scraping job to the configuration
file (e.g. "/etc/prometheus/prometheus.yml"):

   - job_name: 'octez-exporter'
     scrape_interval: <interval> s
     metrics_path: "/metrics"
     static_configs:
       - targets: ['<addr>:<port>']

For instance, a typical target address can be "localhost:9091".

If you plan to also use Netdata (as described below), you should also
add:

   - job_name: 'netdata-scrape'
     metrics_path: "/api/v1/allmetrics"
     params:
       format: [prometheus]
     static_configs:
       - targets: ['<addr>:<port>']

For instance, a typical target address may be "localhost:19999".

Note that Prometheus can also scrape metrics from several nodes.

Prometheus is a service, so you need to start it:

   sudo systemctl start prometheus

You may visualize the log in real time to make sure the service works
correctly:

   sudo journalctl --follow --unit=prometheus.service


History size
~~~~~~~~~~~~

By default, Prometheus keeps 15 days of data. It may be useful to
increase the history window to keep wider history. To do so, update
the Prometheus service execution argument (typically, in
"/etc/systemd/system/multi-user.target.wants/prometheus.service"). To
increase the history to 30 days, simply update the following line:

   ExecStart=/usr/bin/prometheus $ARGS --storage.tsdb.retention 30d

It is also possible to limit the size of the history by adding "--
storage.tsdb.retention.size 5GB". The first limit reached will trigger
the cleanup. You can also add "--storage.tsdb.path <path>" to change
the path were Prometheus stores data.

There are RPCs to trigger an early garbage collection of the
Prometheus data but this requires to start the Prometheus RPC server
and tweak some rights. Alternatively, the easiest way to delete the
whole history is to remove the output directory (typically,
"/var/lib/prometheus/metrics2/").

Mind restarting the Prometheus service after updating its parameters.


Hardware metrics
----------------

In addition to node metrics, you may want to gather other information
and statistics for effective monitoring, such as hardware metrics.

For that purpose, we suggest using Netdata.

To install Netdata:

   bash <(curl -Ss https://my-netdata.io/kickstart.sh)

Add the following at the end of "/etc/netdata/apps_groups.conf"

   octez: octez-node octez-validator

Optionally, you can enable storage monitoring with "filecheck".

To do so, create a "filecheck.conf" file in "/etc/netdata/go.d/" and
add:

   jobs:
     - name: octez-data-dir-size
       discovery_every: 30s
       dirs:
         collect_dir_size: yes
         include:
           - '/path/to/data/dir'

     - name: octez-context-size
       discovery_every: 30s
       dirs:
         collect_dir_size: yes
         include:
           - '/path/to/data/dir/context'

     - name: octez-store-size
       discovery_every: 30s
       dirs:
         collect_dir_size: yes
         include:
           - '/path/to/data/dir/store'

Then, you need to make sure that the "netdata" user has the correct
read/write/execute permissions. This can be achieved by adding this
user to your user's group, or by defining custom rules.

To check that the setup is correct:

   #Log as netdata user
   sudo -u netdata -s

   #Go to the plugin directory
   cd /usr/libexec/netdata/plugins.d/

   #Run the debugger
   ./go.d.plugin -d -m filecheck

With a correct install, you should see lines such as:

   BEGIN 'filecheck_octez-data-dir-size.dir_size' 9999945
   SET '/path/to/data/dir/' = 48585735837
   END

Note, if you use filecheck for storage monitoring, you need to
configure your dashboards accordingly. More details in the Grafazos
documentation.


Logs
----

Eventually, you may want to gather the logs from the different Octez
executables. To do so, we suggest to use Loki and Promtail. Promtail
is used to gather the logs from each executable of Octez and pushes
them to a Loki instance, for indexing metadata about the logs.

You first need to install both tools, following their installation
instructions.

A configuration file will be required, which can be downloaded with:

   wget https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml
   wget https://raw.githubusercontent.com/grafana/loki/main/clients/cmd/promtail/promtail-local-config.yaml

The config file for Loki, "loki-local-config.yml", can be left
untouched. However, the Promtail config file, "promtail-local-
config.yml", requires to be adapted to get the logs needed. For each
Octez executable you want the logs from, you need to add a new job to
the "scrape_configs" part of the config file. For instance, to gather
the logs from the node, you would add:

   - job_name: octez-node
   static_configs:
   - targets:
       - localhost
     labels:
       job: octez-node
       __path__: /path/to/file/node-logs.log

Note that it requires to redirect the logs from your node into a log
file, "/path/to/file/node-logs.log" in this example. To do so, you can
follow the guidelines from the logging documentation.

You can now run both tools with their config files:

   ./loki-linux-amd64 -config.file=loki-local-config.yaml
   ./promtail-linux-amd64 -config.file=promtail-local-config.yaml


Dashboards
==========

Dashboards will take your node monitoring to the next level, allowing
you to visualize the raw data collected with pretty, colorful graphs.


Grafana
-------

Dashboards can be created and visualized with Grafana. Grafana can be
installed by following these instructions.

Once installed and running, you should be able to reach the interface
on port "3000" (you can change the port on the Grafana config file).

Then you need to add the configured Prometheus server (see Prometheus)
as a data source in "Configuration/Data sources".

If you want to have logs on your dashboards, as described in the logs
part, you also need to add Loki as a data source.


Grafazos
--------

You can interactively create your own dashboards to monitor your node,
using the Grafana GUI. Alternatively Grafana allows you to import
dashboards from JSON files.

Grafazos  generates JSON files that you can import into the Grafana
interface.

For more details on how to use it, check Grafazos documentation.

Monitoring the consensus protocol
*********************************

Monitoring the consensus protocol on a Tezos network (either Mainnet
or a test network, see Connecting to a Network) can be useful to
measure the overall health of the network -- how fast consensus is
achieved for each new block --, or the performance of individual
bakers, in order to spot and fix eventual performance issues.

Teztale is a tool for monitoring the Tezos consensus protocol, named
Tenderbake. Teztale is regularly used by network maintainers to
prevent and manage network incidents, but can also be used by bakers
or anyone interested in understanding and inspecting the process of
creating new blocks.

This page explains how to set up Teztale in a simple configuration,
running on a single machine.

Teztale is composed by:

* 2 backend components: the server and the archiver, respectively
  gathering and storing data, and

* a data visualization interface, displaying the data as an
  interactive dashboard (e.g., https://nomadic-labs.gitlab.io/teztale-
  dataviz/)

A Teztale archiver is connected to an Octez node, and feeds one or
many Teztale server(s) with information extracted from this node.

A Teztale server gathers data from one or many archiver(s), but only
one network. Another way to say that is that if you feed a Teztale
server with multiple archivers, all of them must be connected to nodes
that participate to the same network.

The data visualization tool connects to a Teztale server through its
public Tetzale RPC API.

   [image]

Let's see how to set up and connect these components in order to
visually monitor the consensus protocol with Teztale.


Run a Tezos node
================

First, you need to setup an "octez-node". The rest of the tutorial
will assume that the node uses the "--rpc-addr 127.0.0.1:8732" option.
If you want to use a different address/port, adapt the rest
accordingly.


Run a Teztale server
====================

The first thing to set up is the Teztale server. The server is the one
that will:

1. Receive updates from Teztale archiver(s)

2. Return data via its "GET" API

Download the "octez-teztale-server" binary from gitlab. In this
example, we will put it in "/home/user/bin/".

Create a configuration file for your server (e.g. "/home/user
/.teztale-server/config.json")

   {
       "db": "sqlite3:/home/user/.teztale-server/mainnet.sqlite",
       "interfaces": [
           {
               "address": "127.0.0.1",
               "port": 8880
           }
       ],
       "users": [
           {
               "login": "TEZTALE_LOGIN",
               "password": "TEZTALE_PASSWORD"
           },
       "admins": [
           {
               "login": "TEZTALE_ADMIN_LOGIN",
               "password": "TEZTALE_ADMIN_PASSWORD"
           }
       ]
   }

Teztale currently support two DBMS: SQLite and PostgreSQL. Using
PostgreSQL looks like something like
"postgres://DB_USER:DB_PASSWORD@DB_HOST:DB_PORT/DB_BASE".

Create a service description like this "/etc/systemd/system/teztale-
server.service" file, and use it to launch "teztale-server":

   [Unit]
   Description      = Teztale Server Service
   Documentation    = https://gitlab.com/nomadic-labs/teztale
   Wants            = network-online.target
   After            = network-online.target

   [Service]
   User             = user
   Group            = user
   WorkingDirectory = /home/user/
   ExecStart        = /home/user/bin/octez-teztale-server /home/user/.teztale-server/config.json
   Restart          = on-failure

   [Install]
   WantedBy         = multi-user.target

   $ sudo systemctl enable teztale-server.service
   $ sudo systemctl start teztale-server.service


Run a Teztale archiver
======================

Download the "octez-teztale-archiver" binary. In this example, we will
put it in "/home/user/bin/".

Create the "teztale-archiver" service:

   [Unit]
   Description      = Teztale Archiver Service
   Documentation    = https://gitlab.com/nomadic-labs/teztale

   After            = octez-node.target
   Wants            = octez-node.target
   PartOf           = octez-node.target

   [Service]
   User             = user
   Group            = user
   WorkingDirectory = /home/user/
   ExecStart        = /home/user/bin/octez-teztale-archiver --endpoint http://127.0.0.1:8732 feed http://TEZTALE_LOGIN:TEZTALE_PASSWORD@127.0.0.1:8880
   Restart          = always
   RestartSec       = 10s

   [Install]
   WantedBy         = multi-user.target

Enable and start the service:

   $ sudo systemctl enable teztale-archiver.service
   $ sudo systemctl start teztale-archiver.service


Visualize your data
===================

Once you have Teztale server and archiver up and running, you may want
to also host the data visualization interface.

Download the zip file from the release page of the teztale-dataviz
repository and extract it somewhere accessible by the Teztale server,
for instance in our case: "/home/user/teztale-dataviz".

After that, you will need to edit your server configuration in order
to tell "teztale-server" to serve static files from your directory.

   {
       "db": "sqlite3:/home/user/.teztale-server/mainnet.sqlite",
       "interfaces": [
           {
               "address": "127.0.0.1",
               "port": 8880
           }
       ],
       "users": [
           {
               "login": "TEZTALE_LOGIN",
               "password": "TEZTALE_PASSWORD"
           }
   -     ]
   +     ],
   +     "public_directory": "/home/user/teztale-dataviz/"
   }

You can restart your server, and visit:
"<ADDRESS_OF_YOUR_SERVER>:8880/index.html"

You can find a quick presentation of the "teztale-dataviz" interface
in this blog post: https://research-development.nomadic-labs.com
/introducing-teztale.html

Built-in multisig contracts
***************************

A multi-signed account, or *multisig* for short, is a way to share the
ownership of an account (and its associated balance) between several
participants.

To act on a multisig, a fraction of the participants must agree on the
action by signing it with their private keys. The minimal number of
participants that need to agree for the action to be approved is
called the multisig *threshold*.

Tezos supports developing and deploying multisig smart contracts. Such
a multisig contract has built-in support in the "octez-client" and has
been formally verified using the Mi-Cho-Coq framework.


Interacting with a multisig contract using "octez-client"
=========================================================

The recommended way to create and use a multisig contract is via the
"octez-client" built-in commands for the multisig contract. The
command "octez-client man multisig" gives a complete list of multisig-
related commands with details about the syntax of each command.


Originating a new multisig contract
-----------------------------------

To originate a new generic multisig contract, use the "octez-client
deploy multisig" command. It is similar to "octez-client originate
contract" with the following differences:

* no script is given because the script of the generic multisig
  contract is fixed

* instead of giving the initial storage with the "--init" option, the
  threshold and the public keys of the participants are given on the
  command line.

For example, the following commands can be used to generate three
pairs of keys named "alice", "bob", and "charlie" and originate a
multisig contract named "msig" that can be actioned by any two of
them; the initial balance of this contract and of each of the signers
is ꜩ100 generously offered by the first bootstrap account:

   $ octez-client gen keys alice
   $ octez-client gen keys bob
   $ octez-client gen keys charlie
   $ octez-client transfer 100 from bootstrap1 to alice --burn-cap 1
   $ octez-client transfer 100 from bootstrap1 to bob --burn-cap 1
   $ octez-client transfer 100 from bootstrap1 to charlie --burn-cap 1
   $ octez-client deploy multisig msig transferring 100 from bootstrap1 with threshold 2 on public keys alice bob charlie --burn-cap 1


Preparing a transaction
-----------------------

The "octez-client prepare multisig transaction" commands are used to
obtain the byte sequence corresponding to a possible action and that
needs to be signed.

To avoid writing Michelson lambdas, special cases for a single
transfer or delegate change have their own commands.

By default the "octez-client prepare multisig transaction" commands
display not only the byte sequence to sign but also a cryptographic
hash (this can be useful when signing with a hardware signer), the
threshold and the participant public keys. To obtain the byte sequence
only, these commands accept a "--bytes-only" option. For example, if
Alice and Charlie want to send ꜩ10 from the multisig to Bob they will
need to sign a transaction. They can call

   $ octez-client prepare multisig transaction on msig transferring 10 to bob

This command will give them the byte sequence they need to sign, its
cryptographic hash, the threshold (which is 2 in this case), and the
public keys of Alice, Bob, and Charlie.


Signing an action
-----------------

There are two equivalent ways to sign an action with "octez-client":

* preparing the action with one of the "octez-client prepare multisig
  transaction" commands and then signing it using the "octez-client
  sign bytes" command,

* or directly using one of the "octez-client sign multisig
  transaction" commands that combine these two steps.

For example, Alice can sign the transfer to Bob using

   $ TO_SIGN=$(octez-client prepare multisig transaction on msig transferring 10 to bob --bytes-only)
   $ ALICE_S_SIGNATURE=$(octez-client sign bytes "$TO_SIGN" for alice | cut -d ' ' -f 2)

and Charlie can sign the same transfer using

   $ CHARLIE_S_SIGNATURE=$(octez-client sign multisig transaction on msig transferring 10 to bob using secret key charlie)


Acting on the multisig contract
-------------------------------

Once a user has gathered enough signatures to act on the multisig
contract, there are again two equivalent ways of sending the
signatures:

* preparing the action with one of the "octez-client prepare multisig
  transaction" commands and then using the produced byte sequence in
  the "octez-client run transaction" command,

* or directly using one of the following commands depending on the
  action:

  * "octez-client from multisig contract <multisig> transfer"

  * "octez-client from multisig contract <multisig> run lambda"

  * "octez-client set delegate of multisig contract"

  * "octez-client withdraw delegate of multisig contract"

  * "octez-client set threshold of multisig contract"

For example, if Alice sends her signature to Charlie, he can perform
the multi-signed transfer of ꜩ10 to Bob using either:

   $ octez-client run transaction "$TO_SIGN" on multisig contract msig on behalf of charlie with signatures "$ALICE_S_SIGNATURE" "$CHARLIE_S_SIGNATURE"

or

   $ octez-client from multisig contract msig transfer 10 to bob on behalf of charlie with signatures "$ALICE_S_SIGNATURE" "$CHARLIE_S_SIGNATURE"


Supported versions of the multisig contract
===========================================

Two main versions of the multisig contract are supported by "octez-
client". They are called the generic multisig contract and the legacy
multisig contract.


The generic multisig contract
-----------------------------

The generic multisig contract is the multisig contract that is
currently recommended. It has the following features:

* it can receive tokens from unauthenticated sources on its default
  entrypoint of type "unit"

* the possible authenticated actions on the contract are:

  * atomically execute an arbitrary list of operations (of type
    "lambda unit (list operation)" in Michelson)

  * update the contract storage to change both the threshold and the
    participant public keys


The legacy multisig contract
----------------------------

The "octez-client" also supports a legacy version of the multisig
contract which has the following limitations:

* it cannot receive tokens from unauthenticated sources, sending
  tokens to the contract is only possible as a side effect of an
  authenticated action

* the possible authenticated actions on the contract are:

  * transfer without parameter to a user account or to a smart
    contract with an entrypoint of type "unit"

  * set the delegate of the contract

  * remove the delegate of the contract

  * update the contract storage to change both the threshold and the
    participant public keys

In particular, the legacy multisig contract does not support executing
several operations atomically, calling smart contracts with
parameters, and originating new contracts. In contrast, all the
features of the legacy multisig contract are supported by the generic
multisig contract.


Listing supported hashes
------------------------

For security reasons, "octez-client" will not interact with unknown
scripts even if their interface matches one of the supported multisig
contracts. To check if a script is one of the supported ones, it
stores a list of script hashes that can be printed by "octez-client
show supported multisig hashes". The script originated by the "octez-
client deploy multisig" command is always one of the supported
multisig contracts.


Interacting with a multisig contract directly
=============================================

The following subsections describe in detail the low-level API of a
built-in multisig contract, allowing one to originate and use in
situations where "octez-client" cannot be used e.g., when interacting
with the chain from a web browser or in a mobile application. In
particular, this interface is typically useful when developing
multisig support in another Tezos wallet.


Anti-replay protection
----------------------

A replay attack consists in authenticating as someone else by reusing
a signature emitted in a different context. Examples of replay attacks
include reusing a signature sent in a previous transaction, to another
multisig contract, or to the same contract on another chain.

To protect against replay attack, signed data of a multisig contract
needs to contain not only the action to perform but also:

* the address of the multisig contract to avoid replaying signatures
  meant for another multisig contract,

* the chain identifier of the current chain to avoid replaying
  signatures between the test chain forked during the testing period
  of the voting procedure and the main chain,

* an always-increasing anti-replay counter to avoid replaying past
  transactions on the same multisig contract.

The anti-replay counter is stored in the multisig contract storage and
incremented at each successful call of the multisig contract.


Multisig contract storage
-------------------------

Both the generic and the legacy multisig contracts have a storage of
type "(pair (nat %stored_counter) (pair (nat %threshold) (list %keys
key)))" so the storage of the multisig contract is of the form "Pair
<stored_counter> (Pair <threshold> { <first_public_key>;
<second_public_key>; ...; <last_public_key> })" where
"<stored_counter>" and "<threshold>" are Micheline integers
representing respectively the anti-replay counter and the threshold
and each public key is either a Micheline byte sequence or a Micheline
string depending on the mode used to unparse the storage.


Multisig contract actions
-------------------------

The type of actions for the generic multisig is "(or :action (lambda
%operation unit (list operation)) (pair %change_keys (nat %threshold)
(list %keys key)))" so a valid action is either of the form "Left
{<code>}" where "code" is of type "lambda unit (list operation)" for
executing the given lambda and sending the produced operations or
"Right (Pair <new_threshold> {<new_first_public_key>; ...;
<new_last_public_key>})" for changing the threshold and participant
public keys.

The type of actions for the legacy multisig is "(or :action (pair
:transfer (mutez %amount) (contract %dest unit)) (or (option %delegate
key_hash) (pair %change_keys (nat %threshold) (list %keys key))))" so
a valid action is either of the form "Left (Pair <amount>
<destination>)" for a transfer, "Right (Left None)" for withdrawing
the delegate, "Right (Left (Some <new_delegate>))" for changing the
delegate, or "Right (Right (Pair <new_threshold>
{<new_first_public_key>; ...; <new_last_public_key>}))" for changing
the threshold and participant public keys.


Multisig contract sign data
---------------------------

The data to sign for a given action is the binary serialisation (using
the "PACK" Michelson instruction) of an expression of type "pair (pair
chain_id address) (pair :payload (nat %counter) <action>)" where the
"<chain_id>" is the chain id of the current chain as returned by the
"CHAIN_ID" instruction, the address is the one of the multisig
contract as returned by "SELF; ADDRESS", the "nat" counter must match
exactly the stored counter.


Multisig contract parameter
---------------------------

The generic contract has two entrypoints:

* "default" of type "unit" used to receive tokens from unauthenticated
  sources

* "main" of type "pair (pair :payload (nat %counter) <action>) (list
  %sigs (option signature))" used to perform a multi-signed action.

The legacy contract has only one entrypoint that is unnamed and whose
type corresponds to the second above.

The "nat" counter must exactly match the stored counter and the list
of optional signatures must be of the same length and given in the
same order as the stored public keys; "None" can be used to skip a
signature, the number of provided signatures must be greater or equal
to the stored threshold.


Formal verification
===================

See here for a formal specification and a correctness proof of the
generic multisig script written in Coq using the Mi-Cho-Coq framework.

FA1.2 support
*************

Financial application 1.2, or **FA1.2** for short, is a standard that
describes the interface of smart contracts that implement ledger with
balances that is described by tzip-007. We refer hereafter to FA1.2
assets by their common name: tokens.

This page does not present FA1.2 tokens in general, but their
Michelson interface and their support in the Octez client.

For a more general presentation of token standards such as FA1.2 from
a Tezos developer perspective, using higher-level smart contract
languages, see Token standards in Tezos.


Michelson interface
===================

An FA1.2 contract implements multiple entrypoints that describe how
users can transfer tokens, approve others to withdraw tokens from
their account, and retrieve some information such as balances,
withdrawal allowance and total supply. The Michelson interface is then
described by the following entrypoints:

* "transfer": "(pair (address :from) (pair (address :to) (nat
  :amount)))"

* "approve": "(pair (address :spender) (nat :value))"

* "getBalance": "(pair (address :owner) (contract nat))"

* "getAllowance": "(pair (pair (address :owner) (address :spender))
  (contract nat))"

* "getTotalSupply": "(pair unit (contract nat))"

Multiple implementations of such a standard exist within the
ecosystem. For example, one is provided with the TZIP-007
specification. Edukera and camlCase both provide a contract that
implements the standard, and both have been verified using the Mi-Cho-
Coq framework (see related merge request ).

The "octez-client" supports this standard with specific commands that
allow the user to avoid calling FA1.2 contracts using entrypoints and
forging Michelson arguments. Moreover, it supports calling views (the
"get*" entrypoints) offchain, without doing an explicit transaction
that returns a value on a given contract.

"octez-client man fa1.2" gives a complete list of the built-in
commands to interact with FA1.2-compatible contracts, with details
about the syntax of each one.


Client commands
===============


Checking whether a contract is FA1.2 compatible
-----------------------------------------------

To check whether a contract has an FA1.2-compatible interface, use
"octez-client check contract <alias or KT1> implements fa1.2". The
underlying function is actually used before invoking any FA1.2
command.


Managing tokens
---------------

An FA1.2 contract features two main functionalities: transferring
tokens from an address to another, and allowing another account to
transfer an amount of tokens from its own address to any third party.

* "octez-client from fa1.2 contract <fa1.2> transfer <amount> from
  <sender> to <receiver>" transfers a given "amount" of tokens from
  "sender" to "receiver". In that case "sender" is the caller of the
  contract. If "--as <operator>" is given to the command, operator
  becomes the caller of the contract (and must have allowance on
  "sender"'s account, as explained).

* "octez-client from fa1.2 contract <fa1.2> approve <amount> as
  <sender> from <operator>" allows operator to transfer a given
  "amount" of tokens from "sender" to any other account: this is the
  allowance described above.

For example, let's assume two accounts: Alice and Bob, and a contract
token "tk". If Alice gives an allowance of 10 to Bob, then Bob can
transfer up to 10 tokens of "tk" from Alice to any other account.


Batch transfer of tokens
------------------------

As for tez, "octez-client" supports batch transfer of tokens from a
single source to multiple recipients on multiple FA1.2 contracts,
using the command "octez-client multiple fa1.2 transfers from
<account> using <json>".

To be on par with batch transactions, only one account can be the
source of the transfer. However, thanks to the approval mechanism,
this command takes an optional argument "--as <operator>", allowing an
approved account operator to make transfers from the source to any
other accounts. The JSON format for the transfers is the following:

   [{ "token_contract": <string>, // address or alias of the FA1.2 contract
      "destination": <string>, // address or alias of the recipient of the transfer
      "amount": <string>, // amount of tokens to transfer as a string
      "tez_amount": <string>, // (optional) amount of tez to send with transaction as a string
      "fee": <string>, // (optional) custom fees for the transaction as a string
      "gas_limit": <string>, // (optional) gas limit for the transaction as a string
      "storage_limit": <string>, // (optional) storage limit the transaction can use as a string
    },
    ..
   ]

The complete schema can be inspected via "octez-codec describe
<protocol_name>.fa1.2.token_transfer json schema" (where
"<protocol_name>" can be replaced with e.g. "alpha" or
"019-PsQuebec").


View information
----------------

An FA1.2 contract implements three entrypoints that allow a user to
check any account's balance, or allowance between an account and an
operator, and get the total supply of tokens of the contract. These
entrypoints are implemented as offline views: they give back the
requested information as a transaction on a given contract, assuming
its parameter is compatible.

"octez-client" supports calling these entrypoints completely offchain.
One can get a balance using "octez-client from fa1.2 contract
<contract> get balance for <account>", or the allowance using "octez-
client from fa1.2 contract <contract> get allowance on <owner> as
<operator>".

Logging features
****************

Logging features in Octez allow to monitor the execution of Octez
binaries, informing in real time about events of interest, such as
errors, completion of certain steps, etc. This is why various software
components emit *events* throughout the codebase (see Using The Event
Logging Framework), the logging framework dispatches them to an
arbitrary number of (active) *sinks* which can filter print, store, or
otherwise handle events.

Note:

  Log events should not be confused with contract events, which are
  emitted by smart contracts using specific support in the Michelson
  language.

The logging framework can be configured with environment variables,
which specify how events are mapped to sinks. Some Octez binaries
provide additional configurations means.


Events
======

Events have:

* a *name*

* a *section*; a hierarchical event classification mechanism, i.e. a
  path-like list of strings, sometimes the name is appended to make a
  fully-qualified name,

* a *level* quantifying the relative importance of the event (Debug,
  Info, Notice, Warning, Error, and Fatal — inspired from the Syslog
  standard severity levels),

* and, as *contents*, arbitrary structured data. They can be
  serialized to any format supported by the data-encoding library
  (e.g. JSON) and pretty-printed for human readers.

Events are by default “lazy”: if no active sink accepts their section
and level, they are not emitted at all. This means that outputting
*more* events implies more CPU and memory usage.

*“Legacy events”* are the events that use the old & deprecated logging
API; their contents all have the same structure: they are just human-
readable text. The API makes that these events are evaluated even if
they are not consumed by a sink; i.e. they are not lazily evaluated.
The codebase is in the process of getting rid of them.


Sink Configuration
==================

The logging framework refers to sinks with URIs (a.k.a.  structured
strings of the form "schema://path?query"): the schema of the URI is
used to find the appropriate sink-implementation, the rest of the URI
is used to configure the particular instance.


File-Descriptor Sinks
---------------------

File-descriptor sinks allow to configure loggers that output events to
regular Unix file-descriptors. They are duplicatable. They actually
consist in a *family* of four URI-schemes:

* "file-descriptor-path://" outputs to a file

* "file-descriptor-stdout://" outputs to "stdout" / "1"

* "file-descriptor-stderr://" outputs to "stderr" / "2"

* "file-descriptor-syslog://" outputs to a "syslog" facility

*Note:* "-stdout" and "-stderr" schemes are there for convenience and
for making the API consistent; most Unix-ish systems nowadays have
"/dev/std{out,err}" and "/dev/fd/[0-9]+" special files which can be
used with "file-descriptor-path" (with some precautions).

The path of the URI is used by "file-descriptor-path" to choose the
path to write to and ignored for the other two.

The query of the URI is used to further configure the sink instance.

Common options:

* "level-at-least=<loglevel>" the minimal log-level that the sink will
  output.

* "section-prefix" can be given many times and defines a list of pairs
  "<section-prefix>:<level-threshold>" which can be used to setup more
  precise filters. "level-at-least=info" can be understood as
  "section-prefix=:info", the empty section prefix matches all
  sections. To exclude a specific section use the "none" filter, for
  example "section-prefix=p2p:none". To define a filter only for one
  specific section "section-prefix=p2p:debug&section-prefix=:none"

* "format=<value>" the output format used. Note that syslog output
  will ignore this option and use the syslog formatting. Possible
  values are:

  * "one-per-line" (the default): output JSON objects, one per line,

  * "netstring": use the Netstring format (cf. Wikipedia) to separate
    JSON records,

  * "pp" or "pp-rfc5424" to output the events pretty-printed, one per
    line, using a format compatible with RFC-5424 (or Syslog).

  * "pp-short" to output the events pretty-printed in a shorter and
    more user-friendly fashion.

* "color=true" enables logs coloring. It only works on ttys, in
  conjunction with "format=pp-short".

* "advertise-levels=true" enables the advertisement of the log-level
  of each events outputted in the sink.

Options available only for the "file-descriptor-path://" case:

* "with-pid=<bool>" when "true" adds the current process-id to the
  file path provided (for instance, useful for the node when not
  running in "--singleprocess" mode).

* "fresh=<bool>" when "true" smashes the content of the file if it
  already exists instead of appending to it.

* "chmod=<int>" sets the access-rights of the file at creation time
  (default is "0o600", provided Umask allows it).

* "daily-logs=<int>" sets up a rotation for log files, creating a file
  for each day. The parameter is the number of days these logs should
  be kept, and hence the number of files. The day of the year with
  format ["yyyymmdd"] is added as a suffix to the filename, prefixed
  by a dash. We recomend not to put your tezos logs in "/var/log" if
  you use this option, as you system would use "logrotate"
  automatically.

* "create-dirs=<bool>" when "true" allows to create the directory
  where the log files are stored and all its parents recursively if
  they don't exist.

Option available only for "file-descriptor-syslog://" case:

* "facility=<facility>" is the targeted syslog output. The possible
  values are: "auth, authpriv, cron, daemon, ftp, kernel, local0,
  local1, local2, local3, local4, local5, local6, local7, lpr, mail,
  news, syslog, user, uucp, ntp, security, console". "user" is the
  default facility if no value is provided. See RFC-3164 for more
  information.

  Examples:

* "file-descriptor-path:///the/path/to/write.log?format=one-per-line
  &level-at-least=notice&with-pid=true&chmod=0o640" → Executables will
  write all log events of level at least "Notice" to a file
  "/the/path/to/write-XXXX.log" where "XXXX" is the PID, the file will
  be also readable by the user’s group ("0o640").

* "file-descriptor-stderr://?format=netstring" → Executables will
  write to "stderr" JSON blobs *“packetized” as* Netstrings.

* "file-descriptor-path:///dev/fd/4?section-prefix=rpc:debug" →
  Executables will write to the file-descriptor "4" likely opened by a
  parent monitoring process. The reader will only receive the logs
  from the section "rpc" (but all of them including "Debug").

* "file-descriptor-path:///the/path/to/write.log?section-
  prefix=rpc:debug&section-prefix=validator:debug&section-
  prefix=:none"" → Write only sections validator and rpc at debug
  level but exclude all other sections from the stream.

* ""file-descriptor-path:///tmp/node-logs/node.log?daily-logs=5
  &create-dirs=true&section-prefix=:info"" sets up daily log files
  with a history of up to 5 days and verbosity level "info" for all
  logs. Files will be named "node-19700101.log" in an example of a
  file produced in 1970, January, the 1st. The log directory "node-
  logs" will be automatically created if it doesn't exist.

The format of the events is (usually minified):

   {"fd-sink-item.v0":
     {"hostname": <host-name>,
      "time_stamp": <float-seconds-since-epoch>,
      "section":[ <list-of-strings> ],
      "event":
        <event-specific-json> } }

Additionally, the ""hostname"" field can be customized with
environment variable "TEZOS_EVENT_HOSTNAME"; Its default value is the
hostname of the device the node is running on.

To store rotated logs, there is the "daily-logs" option to create logs
files on a daily basis. However, it is also possible to use
"logrotate" by putting the log file in "/var/log/tezos/sink.log", for
exemple. The following configuration can then be put in
"/etc/logrotate.d/tezos/sink.log":

   /var/log/tezos/sink.log {
           daily
           copytruncate
           rotate 4
           compress
   }


File-Tree Sink
--------------

This is a sink that dumps events as JSON files (same format as above)
in a directory structure guided by the section of the events. It can
be useful for testing the logging framework itself, or for off-line
post-mortem analysis for instance.

The URI scheme is "unix-files", the path is the top-level directory in
which the JSON files will be written.

The query of the URI allows one to filter the events early on.

* "level-at-least=<loglevel>" the minimal log-level that the sink will
  output.

* "name-matches=<regexps>" comma-separated-list of POSIX regular
  expressions on the name of the events.

* "name=<names>" comma-separated-list of event names matched
  *exactly*.

* "section=<sections>" comma-separated-list of event sections matched
  *exactly*.

* "no-section=<bool>" when true only catch the events that have an
  empty section.

Example: "unix-files:///the/path/to/write?level-at-least=info" (the
path should be inexistent or already a directory).

The directory structure is as follows: "<section-dirname>/<event-
name>/<YYYYMMDD>/<HHMMSS-MMMMMM>/<YYYYMMDD-HHMMSS-MMMMMM-xxxx.json>"
where "<section-dirname>" is either "no-section" or "section-<section-
name>".


Global Defaults
===============

By default, the Octez binaries generate **user logs** as follows:

* "file-descriptor-stdout://" sink is activated by default and
  configured to output events of level at least "Notice" to stdout.

The node and the baker additionally generate by default more detailed
**internal logs** as follows:

* A file-descriptor-sink is activated to store logs from last seven
  days with an "Info" level. For the node, the path is "<node-data-dir
  >/daily-logs/". For other tools, see the corresponding sections in
  this page.


JSON Configuration Format
=========================

A configuration JSON blob, is an object with one field
""active_sinks"" which contains a list of URIs:

   {
     "active_sinks": [ <list-of-sink-URIs> ]
   }

The URIs are discriminated among the sink implementations above using
their schemes and activated.

It is used in various places: node configuration file, logging-
configuration RPC, etc.


Environment Variables
=====================

The logging framework can be configured with environment variables
before starting an Octez executable (e.g., the node). Those variables
work on all the code using the "tezos-stdlib-unix" library as long as
"Internal_event_unix.init" is called; this should include *all* the
regular "octez-*" binaries.

* "TEZOS_EVENTS_CONFIG" must be a whitespace-separated list of URIs:

  * URIs that have a schema are activated.

  * URIs without a schema, i.e. simple paths, are understood as paths
    to configuration JSON files (format above) to load (which
    themselves activate sinks).

* "TEZOS_LOG" and "LWT_LOG" (deprecated and has a lower priority)
  contain “rules” to configure the default "file-descriptor-stdout"
  sink. The rules are expressed with a DSL:

  * rules are separated by semi-colons ";",

  * each rule has the form "pattern -> level",

  * a pattern is a minimalist glob-expression on the "section.name" of
    the event, e.g. "rpc*" for all events whose section.name starts
    with "rpc",

  * rules are ordered, i.e., the first pattern that matches, from left
    to right, fires the corresponding rule.

* "TEZOS_EVENT_HOSTNAME" is used by the file-descriptor-sink to tweak
  the JSON output (see above).

As the Irmin context backend uses an internal and specific logging
system, it is possible to configure it through the "TEZOS_CONTEXT"
environment variable, see Context.


Node-Specific Configuration
===========================

The node supports some additional means to configure logging, besides
environment variables.


Configuration File
------------------

See "octez-node config --help" for the full schema of the node’s JSON
configuration file.

In particular the fields:

* ""internal-events"" contains a configuration of the sinks (format
  above).

* ""log"" is an object which defines the configuration of the default
  "file-descriptor-stdout" sink; one can redirect the output to a
  file, set the rules, and change the formatting template. The goal of
  this configuration field is to be simpler to express that "internal-
  events" for simpler changes.

Note that "log" is ignored if "internal-events" is present.


Command Line Options
--------------------

See "octez-node run --help", the default "file-descriptor-stdout://"
sink configuration can be also be changed with the following options:

* "-v" / "-vv": set the global log level to "Info" or "Debug"
  respectively.

* "--log-output": set the output file.

* "--log-coloring=<bool>": enable or disable colors in the default
  stdout logs. The default value is "true".


RPC "/config/logging"
---------------------

The node exposes an administrative "PUT" endpoint: "/config/logging".

The input schema is the JSON configuration of the sinks. It
deactivates all current sinks and activates the ones provided
**except** the "file-descriptor-stdout://" sink that is left
untouched.

Example: (assuming the "file-descriptor-stdout://" is active not to
miss other events) this call adds a sink to suddenly start pretty-
printing all "rpc" events to a "/tmp/rpclogs" file:

   octez-client rpc put /config/logging with \
     '{ "active_sinks": [ "file-descriptor-path:///tmp/rpclogs?section-prefix=rpc:debug&format=pp&fresh=true" ] }'


Client and baker configuration
==============================

Both "octez-client" and "octez-{baker,accuser}-*" can be configured
either using environment variables or using "internal-events" in the
client configuration file, with the file-descriptor sinks described
above.

There is also one common option "--log-requests" which can be used to
trace all the interactions with the node (but it does *not* use the
logging framework).

By default, the baker also generates internal logs, which are stored
at "<client-base-dir>/logs/octez-baker-<protocol-name>/*". Hence,
running two bakers (for two different accounts) using the same
protocol with the same base directory is not recommended.


Processing Structured Events
============================

This is work-in-progress, see:

* "octez-admin-client show event-logging" outputs the configuration
  currently understood by "octez-admin-client" (hence through the
  "TEZOS_EVENTS_CONFIG" variable) and lists all the events it knows
  about.

* "octez-admin-client output schema of <Event-Name> to <File-path>"
  get the JSON-Schema for an event.

Example: "octez-admin-client output schema of block-seen-alpha to
block-seen-alpha.json"

Exit codes and signals
**********************

This page documents the exit codes possibly returned by the Octez
binaries, as well as the signals they handle.


Exit codes
==========

The meaning of exit codes is presented in the following table. The
action column indicates a recommended course of action.

+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+
| exit code     | meaning                                                                            | action                                                             |
+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+
| 0             | the process exited successfully                                                    | nothing                                                            |
+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+
| 1–125         | something went unexpectedly                                                        | check output/log to see if you forgot an argument or some such     |
+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+
| 126           | an exception was not handled                                                       | report a bug                                                       |
+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+
| 127           | the process received a signal (e.g., via Ctrl-C)                                   | nothing                                                            |
+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+
| 128           | the process was about to exit successfully but an error occurred during exit       | check output/logs, clean-up leftover files, open a bug report      |
+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+
| 129–253       | like 1–125 and an error occurred during exit                                       | check output/logs, clean-up leftover files, open a bug report      |
+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+
| 254           | like 126 and an error and an error occurred during exit                            | check output/logs, clean-up leftover files, open a bug report      |
+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+
| 255           | like 127 but an error and an error occurred during exit (e.g., "kill -9")          | check output/logs, clean-up leftover files                         |
+---------------+------------------------------------------------------------------------------------+--------------------------------------------------------------------+


Signals
=======

Upon receiving "SIGINT" (e.g., via Ctrl+C in an interactive session)
or "SIGTERM" (e.g., via "systemctl stop") the process will exit with
code 127 if the cleanup happens normally. However, sending the same
signal a second time (after a one (1) second grace period) will
terminate the process immediately, interrupting the normal clean-up
functions, and in this case the exit code will be 255.

Octez Software Architecture
***************************

This page presents the software architecture of the Octez suite,
focusing on the most important tool in a Tezos network: the Tezos
node, implemented in our case as the Octez node executable ("octez-
node"). The Octez node software architecture faithfully implements the
Tezos software architecture principles:

* It is separated into a shell and a protocol, to support protocol
  evolution.

* It implements a client/server architecture, to allow composition
  with many other tools in a safe way.

This page contains two sections. The first section, which should be
readable by anyone, describes the main elements of a Tezos node from a
distance. It abstracts from all plumbing and both internal and system
dependencies to give a simple view of the main components, their
responsibilities, and interactions. The second part is written for
developers and is at the level of OPAM packages.

Although the description is focused on the Octez node, some of these
components are reused to generate other tools in the Octez suite, see
The Final Executables.


The Big Picture
===============

The diagram below shows a very coarse-grained architecture of the
Tezos node.

[image: Tezos architecture diagram][image]

The characteristic that makes Tezos unique is its self-amending
property. The part that amends itself is called the economic protocol
(the green brain of the octopus), sometimes abbreviated by protocol or
even proto in the source code. The rest of a Tezos node is what we
call the *shell* (the blue octopus).

The protocol is responsible for interpreting the transactions and
other administrative operations. It also has the responsibility to
detect erroneous blocks.

An important thing to notice is that the protocol always sees only one
block chain, that is a linear sequence of blocks since the genesis. It
does not know that it lives in an open network where nodes can propose
alternative heads.

Only the shell knows about the multiple heads. It is responsible for
choosing between the various chain proposals that come from the bakers
(the programs that cook new blocks) of the network. The shell has the
responsibility of selecting and downloading alternative chains, feed
them to the protocol, which in turn has the responsibility to check
them for errors, and give them an absolute score. The shell then
simply selects the valid head of the highest absolute score. This part
of the shell is called the validator.

The rest of the shell includes the peer-to-peer layer, the disk
storage of blocks, the operations to allow the node to transmit the
chain data to new nodes and the versioned state of the ledger. In-
between the validator, the peer-to-peer layer, and the storage sits a
component called the distributed database, that abstracts the fetching
and replication of new chain data to the validator.

Protocols are compiled using a tweaked OCaml compiler (green part on
the left of the picture) that does two things. First, it checks that
the protocol’s main module has the right type. A good analogy is to
see protocol as plug-ins, and in this case, it means that it respects
the common plugin interface. Then, it restricts the typing environment
of the protocol’s code so that it only calls authorized modules and
functions. Seeing protocols as plug-ins, it means that the code only
called primitives from the plug-in API. It is a form of statically
enforced sandboxing (see Protocol Environment).

Note that the economic protocol on the chain is subject to an
amendment procedure. On-chain operations can be used to switch from
one protocol to another. The procedure is described in more details in
the protocol's voting procedure documentation.

Finally, the RPC layer (in yellow on the right in the picture) is an
important part of the node. It is how the client, third-party
applications and daemons can interact with the node and introspect its
state. This component uses the mainstream JSON format and HTTP
protocol.  It uses the library "resto". It is fully interoperable, and
auto descriptive, using JSON schema.


Software Architecture and Packages Relationship
===============================================

The diagram below shows the main OPAM packages present in the source
code of Tezos, and their dependencies. The "tezos-" or "octez-"
prefixes have been dropped for clarity.

[image: Tezos source packages diagram][image]

In green at the bottom are binaries. Highlighted in yellow are the
OPAM packages (sometimes with shortened names). Black arrows show
direct dependencies. Orange arrows show other indirect relationships
(code generation, interface sharing), explained below. The part
circled in blue, contains modules that bear no dependency to Unix, and
can thus be compiled to JavaScript. External dependencies are not
shown in this illustration.

Note that many packages described below are grouped in a single one:
octez-libs. This includes packages as diverse as tezos-stdlib , tezos-
base , tezos-shell-services , tezos-context , etc.)


Base and below
--------------

At the center, the tezos-base  package is where the blockchain
specific code starts. Above it in the figure (but below in terms of
abstraction level) is the set of libraries that are used everywhere
for basic operations.

* tezos-stdlib  contains a few extensions over the OCaml standard
  library (a few string primitives, an "Option" module, etc.), a few
  "Lwt" utilities, and a "Compare" module that implements monomorphic
  comparison operators.

* tezos-error-monad  is an in-house monadic interface to the OCaml
  "('a, 'b) result" type, that fixes the "'b" to an extensible type
  "error" (actually a list, to hold an error trace). When extending
  the type, programmers must also call the "register_error" function
  that registers a pretty printer and an encoding for serialization. A
  tutorial is available for this library.

* tezos-rpc  provides the basics of Tezos' RPC service mechanism. It
  provides combinators for building service hierarchies à la
  Ocsigen/Eliom, registering, and calling services. This module is
  based on resto, that allows for automatic generation of machine and
  human-readable descriptions of the hierarchy of services, including:
  the structure of URLs and the expected formats for input and output
  bodies, via the use of "data_encoding".

* tezos-crypto  wraps the external cryptography libraries that we use.
  We try to use minimal reference implementations, with as thin as
  possible bindings, and rely on libraries from the HACL* project,
  written and verified in the F* programming language, and extracted
  to C.

* tezos-micheline  is the concrete syntax used by Michelson, the
  language of smart contracts. It mostly contains the generic, untyped
  AST, a printer, and a parser.

* tezos-base  wraps all these modules in a common foundation for all
  the other components of Tezos, and introduces the data structures of
  the blockchain (e.g. "Block_hash", "Block_header", "Block_locator",
  "Fitness", "P2p_identity") that are shared between the shell,
  economic protocol, client, daemons and third-party software. It also
  rewraps some modules from "crypto" as functors that build all-in-one
  modules for a given type (for instance, the module for block hashes
  contains all possible converters, command line and RPC argument
  builders, pretty printers, an autocompleter, etc.). This package
  also contains the "cli_entries" module that we use to handle
  command-line parsing in some executables.


The Shell
---------

The shell is the part of the node responsible for all communications,
peer-to-peer and RPC, acting as a cocoon around the economic
protocols.

* tezos-shell-services  contains the definition of the node's service
  hierarchy, and calling functions to use in the client (or any third
  party software). As this library is linked into the client to call
  the services in a type-safe way, only the description of services is
  done here. The registration of handlers is done in the rest of the
  node's implementation.

* tezos-rpc-http-client  and tezos-rpc-http-server  use cohttp to
  implement the RPC over HTTP server and client, allowing to make
  actual use of services declared using tezos-rpc .

* tezos-p2p  is the in-house peer-to-peer layer.

* tezos-store  is the chain-data store that handles on-disk block
  storage, snapshots exporting/importing and chain reconstruction.

* tezos-context  contains the raw versioned key-value store used for
  storing the ledger's context (one version per block). This is
  implemented using irmin.

* tezos-protocol-updater  maintains the table of available protocol
  versions, embedded or dynamically linked.

* tezos-shell  implements the scheduling of block validations, the
  mempool management, and the distributed database. A description is
  available in this document.


The Economic Protocol Environment and Compiler
----------------------------------------------

Economic protocols are compiled by a specific version of the OCaml
compiler. This compiler restricts the set of modules available to the
economic protocol, as a form of static sandboxing. It also generates a
functorized version of the protocol, to make the execution of the
protocol in an alternative environment possible.

* tezos-protocol-environment  contains the protocol generic
  environment, that is the API of the modules that are available to
  the economic protocol and the exported protocol API. A review of
  this sandbox is available here. These modules include a stripped-
  down standard library, and interfaces to the crypto APIs, RPC
  definitions, and a key-value store. It also defines two different
  context instances: one that simulates the key-value store in memory
  for testing, and one whose context function are dummy ones which can
  be used when only the types and noncontextual functions of the
  protocol are needed.

* octez-protocol-compiler is the compiler for economic protocols: an
  alternative driver to the OCaml ocaml-compiler-libs that typechecks
  within the protocol environment, and performs some more checks on
  the protocol code.

* "tezos-protocol-xxx" is produced by the protocol compiler and
  contains a functorized version of protocol "xxx" that takes its
  standard library as a parameter. This parameter can be filled with
  any of the implementations described in the two points below.

* tezos-shell-context  implements a context representation that is
  accepted by the protocol environment. The node uses this instance to
  read and write data on disk.

* "tezos-embedded-protocol-xxx" contains a version of the protocol
  "xxx" whose standard library is pre-instantiated to the shell's
  implementation, these are the ones that are linked into the node. It
  also contains a module that registers the protocol in the node's
  protocol table.


The Embedded Economic Protocols
-------------------------------

Three kinds of economic protocols are included in the main Octez
repository.

* The genesis protocol. tezos-protocol-genesis (tezos-embedded-
  protocol-genesis ) is the protocol of the genesis block. It accepts
  a single block, signed by an activator whose public key is
  hardcoded, and whose single action is to switch to a new protocol
  chosen by the activator. The "master" branch contains additional
  variants of the genesis protocol, one for each of the existing test
  networks.

* The active protocols. "tezos-protocol-nnn-hhhhhhhh" ("tezos-
  embedded-protocol-nnn-hhhhhhhh") is either the current protocol on
  Mainnet or a protocol that has been active on Mainnet at some point,
  where "nnn" is a counter starting at 0 and "hhhhhhhh" is a prefix of
  the hash of the protocol code. Also, tezos-protocol-alpha (tezos-
  embedded-protocol-alpha ) on the "master" branch is the protocol
  currently under development (see here for more details). A tutorial
  is available to start reading the protocol's code.

* Demo protocols. tezos-embedded-protocol-demo-noops  is just a demo
  protocol that does nothing interesting but has the right shape.
  tezos-embedded-protocol-demo-counter  is another demo protocol in
  which blocks can contain simple operations.


The Client Library
------------------

The client is split into many packages, to enforce three separation
lines: shell vs economic protocol, Unix dependent vs JavaScript
compatible, and library vs command-line interface.

* tezos-client-base  defines the client context, which is an object
  whose methods allow for: accessing a wallet of keys, interacting via
  the user, making RPC calls, and signing data using signer plug-ins.
  Most of them, including RPC calling functions from tezos-shell-
  services  and tezos-protocol-alpha, are abstracted over this object
  type. That way, it is possible to use the same code for different
  platforms or toolkits.

* tezos-client-alpha  provides some functions to perform the
  operations of protocol Alpha using the wallet and signers from the
  client context.

* tezos-client-genesis contains the basic activator commands available
  on the genesis protocol.

* tezos-client-base-unix  implements configuration file and wallet
  storage in Unix files, user interaction via the Unix console, and
  terminal based signer plug-ins.


Tests Packages
--------------

The tests are split into various packages, testing more and more
elements while following the dependency chain. Use "make test" to run
them.

* tezt/: end-to-end tests as Tezt tests that e.g. launch local
  sandboxed nodes and performs various tasks using the client

* tezos-p2p  (in directory src/lib_p2p/test/): tests of the peer-to-
  peer layer, independently of the Tezos gossip protocol (establishing
  connections, propagating peers, etc.)

* tezos-protocol-environment  (in directory
  src/lib_protocol_environment/test/): tests for the in-memory context
  implementation.

* tezos-shell  (in directory src/lib_shell/test/): tests for the chain
  data storage.

* tezos-stdlib  (in directory src/lib_stdlib/test/): tests for the
  basic data structures.

* tezos-context  (in directory src/lib_context/test/): tests for the
  versioned key-value context.

* tezos-store  (in directory src/lib_store/unix/test/): tests for the
  on-disk store.

* tezos-protocol-alpha (in directory
  src/proto_alpha/lib_protocol/test/): tests of the Alpha protocol
  (without launching a node).

* tezos-crypto  (in directory src/lib_crypto/test/): tests for the in-
  house merkle trees.


The Final Executables
---------------------

The Octez executables are generated from packages such as the
following ones (for the complete list of binaries, see The Binaries):

* octez-node provides the node launcher binary "octez-node". All the
  algorithmic being implemented in the shell, this package only
  implements the node's CLI. It also provides the sandboxed node shell
  script launcher (see the main readme).

* octez-client provides the "octez-client" and "octez-admin-client"
  binaries. The former contains a small command line wallet, the
  latter an administration tool for the node. It also provides a shell
  script that configures a shell environment to interact with a
  sandboxed node.

* octez-baker-alpha provides the "octez-baker-alpha" binary.

* octez-accuser-alpha provides the "octez-accuser-alpha" binary.

* octez-protocol-compiler provides the "octez-protocol-compiler"
  binary that is used by the node to compile new protocols on the fly,
  and that can be used for developing new protocols.

The Octez Shell
***************

As explained in the Octez Software Architecture, the *shell* is the
rest of an Octez node when excluding the protocol; in other terms, the
"blue octopus".

The following pages describe the principles underlying the shell
design and implementation.

* The validation subsystem

  * Concepts

  * Validator

  * Prevalidator

  * Distributed DB

* The Prevalidator

  * Prevalidator filtering mechanisms

  * Pending operations ordering and prioritization

  * Operations classification

  * One manager operation per manager per block

  * Replacing by fee

  * Banning operations

* The storage layer

  * Store

  * Context

* Synchronisation heuristic

  * Synchronisation heuristic status

  * Bootstrapped

  * Basic description of the heuristic

  * Acceptable values for parameters

  * Formal description of the heuristic

  * If the heuristic fails

* The peer-to-peer layer

  * General operation

  * Welcome worker & connect handler

  * Maintenance worker

* Protocol Environment

  * Motivation

  * Environment contents

  * Environment versions

* Micheline

  * Syntax

  * Serialization

  * Other tools and resources

The validation subsystem
************************

This document explains the inner workings of the validation subsystem
of the Tezos shell, that sits between the peer-to-peer layer and the
economic protocol. This part is in charge of validating chains, blocks
and operations that come from the network, and deciding whether they
are worthy to propagate. It is composed of three main parts: the
validator, the prevalidator, and the distributed DB.

[image: Tezos validation diagram][image]


Concepts
========

Before presenting these three components, we define some basic
concepts.


Block
-----

The Tezos blockchain is a linked list of blocks (or actually, a tree
when several competing branches exist). Blocks conceptually contain a
header and a list of operations. In the implementation, the list of
operations in the block is represented as the hash of the Merkle tree
containing them.

The header itself decomposes into a shell header (common to all
protocols) and a protocol-specific header.


Fitness
-------

To each block, we associate a measure of fitness which determines the
quality of the chain leading to that block. This measure is computed
by the consensus protocol. The shell changes the head of the chain to
the valid block that has the highest fitness.

The fitness belongs to the shell part of the block header. The shell
does not know the exact representation of the fitness, except that it
has a total order on it.


Shell header
------------

The shell header contains:

* "level": the height of the block, from the genesis block.

* "proto": number of protocol changes since genesis (modulo 256).

* "predecessor": the hash of the preceding block.

* "timestamp": the timestamp at which the block is claimed to have
  been created.

* "validation_pass": number of validation passes. Indeed, operations
  included in a block may be validated in several passes. This enables
  some kind of operations (e.g., consensus operations) to be validated
  in priority.

* "operations_hash": Hash of the list of lists (actually root hashes
  of Merkle trees) of operations included in the block. There is one
  list of operations per validation pass.

* "fitness": a sequence of sequences of unsigned bytes, shortlex-
  ordered (by length and then lexicographically). It represents the
  claimed fitness of the chain ending in this block.

* "context": the hash of the state of the context after application of
  this block.

The rest of this page presents the three components of the validation
subsystem.


Validator
=========

The validator is the component responsible for checking that blocks
coming from the network or a baker are valid, w.r.t. the rules defined
by the economic protocol, and for selecting the block that it
considers to be the current head of the blockchain, based on its
fitness.

The validator is written as a collection of workers: local event loops
communicating with each other via message passing. Workers are spawned
and killed dynamically, according to connected peers, incoming blocks
to validate, and active (test)chains.

A *chain validator* worker is launched by the validator for each
*chain* that it considers alive. Each chain validator is responsible
for handling blocks that belong to this chain, and select the best
head for this chain. A chain validator is spawned for the main chain
that starts at the genesis, a second one when there is an active test
chain. Forking a chain is decided from within the economic protocol.
In protocol Alpha, this is only used to try new protocols before self
amending the main chain.

The chain validator spawns one *peer validator* worker per connected
peer. The set of peer validators is updated, grown, or shrunk on the
fly, according to the connections and disconnections signals from the
peer-to-peer component. Each peer validator will treat new head
proposals from the associated peer, one at a time, in a loop. In the
simple case, when a peer receives a new head proposal that is a direct
successor of the current local head, it launches a simple *head
increment* task: it retrieves all the operations and triggers a
validation of the block. When the difference between the current head
and the examined proposal is more than one block, mostly during the
initial bootstrap phase, the peer worker launches a *bootstrap
pipeline* task.

Peer validators interact with the *distributed database* to retrieve
block headers and operations. The database functions as a cache,
returning data instantly if available or querying linked peers when
it's missing. Once a block is complete, peer validators submit it to
the *block validator*.

The block validator processes blocks sequentially, assuming all
necessary data has already been retrieved from the peer-to-peer
network. Blocks are submitted through a two-step process:

* validation performs a light check doing the minimal computation
  required to assess with certitude whether a block is well-formed and
  can be applied in the current context.

* application computes the resulting context by executing manager
  operations and, particularly, running smart contracts, which can be
  computationally expensive.

The block validator advertises a block to peers immediately after
validation (if successfull). This optimization ensures that blocks are
broadcast as quickly as possible while still guaranteeing that only
valid blocks are shared across the network.

Once the block is successfully applied, the chain validator is
notified and may update its head. The chain validator will then
propagate this information to its associated prevalidator.

The validator relies on the protocol for both validation and
application functions.


Prevalidator
============

Each chain validator is associated to a *prevalidator* that is
responsible for determining which operations to propagate for this
chain over the peer-to-peer network. The page The Prevalidator gives a
detailed description of the prevalidator component.

The prevalidator also interacts with the protocol in order to
determine valid operations in the mempool to propagate in the gossip
network.


Distributed DB
==============

The gathering of resources needed for validation is centralized in the
*distributed db*. This component allocates a slot per requested
resource, whose priority depends on the number of peer validators
requesting it.

The Prevalidator
****************

The *prevalidator* is responsible for determining which operations to
propagate over the gossip network. It relies on the economic protocol
to classify operations and decide whether they should be propagated or
not.

The baker uses the prevalidator too, via the "monitor_operations" RPC,
for selecting operations that can be included in the block being
crafted (if any), and for observing consensus operations/quorums (in
particular for the Tenderbake protocol).

The prevalidator aims to protect the network against DDoS attacks and
must decide which operations to broadcast and which to keep in its
bounded memory. To do so, it maintains a set of operations along with
their associated classifications in a data structure called the
(prevalidator) *mempool*. Operations deemed valid are broadcast and
made available for peer requests. Each time the validation system
switches to a new head, the operations classification is reconsidered
through a process known as *flushing*, as previous classification may
become obsolete.

The classification maintained by the prevalidator ensures a strong and
usefull property: the mempool's validated operations are co-validable.
Any subset of these operations can be included in the next block in
any order, with confidence that they can be applied to the current
ledger without causing errors.


Prevalidator filtering mechanisms
=================================

The prevalidator implements several filtering mechanisms that limit to
some extent the risk of flooding the network. Since the filtering must
be done by looking at the content of an operation, which the
prevalidator cannot achieve solely, it relies on a protocol-specific
plugin, that follows the shell signatures from the shell-libs., and on
a protocol-side mempool data-structure.

An operation is first submitted through the pre_filter provided by the
plugin. If it passes without being classified as an error, the
prevalidator queries the protocol-side mempool data-structure to
assess the possibility of adding the operation to the current set of
validated operations. If the operation is not classified as an error,
then it is considered valid and broadcasted to peers.

The "pre_filter" is executed for each operation received from the
network and every time an operation needs to be reclassified (after
receiving a new block for example). This function should be quick to
execute and static: rejection is done solely based on the content of
the operation, not taking into account the state of the ledger. A
detailed description is given in the dedicated plugin page for the
corresponding economic protocol (e.g., Protocol Plugins for the active
protocol).

The protocol-side mempool scrutinizes the operation in greater detail.
It ensures that the operation does not conflict with any already
validated operations to maintain co-validity. In case of a conflict,
the protocol may either reject the new operation or accept it while
reclassifying previously validated operations out of the validated set
(see Mempool).

The number of valid operations in the prevalidator mempool is limited
to a maximum of "max_operations", and the total byte size of these
operations cannot exceed "max_total_bytes". If the mempool is full,
the prevalidator may prioritize a new operation over an already
validated one based on their fee-to-gas ratio. These values can be
accessed and configured via RPCs.


Pending operations ordering and prioritization
==============================================

The prevalidator leverages additional information provided by the
"pre_filter" to sort operations. This results in a finer-grained
ordering, where consensus and freshly injected operations are
prioritized over others.


Operations classification
=========================

The Octez prevalidator classifies an operation based on its validity
in the prevalidator's current state:

* "Valid": the operation was successfully validated by the protocol. A
  validated operation is guaranteed to be applied successfully on the
  current context even though the operation has not been applied.

* "Branch_delayed": the operation could not be applied on the current
  context, but could be in the future.

* "Branch_refused": the operation could not be applied on the current
  context, but could be if a reorganisation happens.

* "Outdated": this operation will never be applicable in the future
  but was valid in the past.

* "Refused": There is no block on the current economic protocol that
  would accept this operation.

When a node switches to a new head, its prevalidator state is reset
accordingly. Operations that were previously classified as "Valid" are
marked as "pending" awaiting to be reclassified. Depending on the
relationship between the current block and the previous one,
"Branch_delayed" or "Branch_refused" operations might also be
reconsidered and marked as "pending" (i.e. "Branch_delayed" are
reclassified when the new head is a successor of the previous one,
while "Branch_refused" operations are reclassified if the current head
is on a different branch).

Operations living in the prevalidator (already classified or not) can
be accessed via the "pending_operations" RPC.

The journey of an operation can be outlined as follows:

* When advertised from peers, an operation is first known by its hash
  before its content is retrieved from the peer if necessary. Once an
  operation content is fetched, it is marked as "arrived" and is
  submitted to the "pre_filter". If prefiltered without errors, the
  operation is assigned a priority based on its validation passes and
  fee/gas ratio. It is then marked as "pending" and await to be
  classified with help of the protocol-side mempool. After
  classification, it is marked as classified and sorted accordingly,
  remaining there until the next prevalidator flush. If classified as
  "Valid", the operation is advertised to peers.

* On injection, an operation bypasses prefiltering and is submitted
  directly to the protocol-side mempool. As for operations received
  from peers, they will be broadcast to the network if classified as
  "Valid". Otherwise, unlike operations received form the network,
  they will not be classified, and an error will be output to inform
  the user.

The following diagram illustrates the lifecycle of an operation
through the prevalidator, from its arrival or injection to its
advertising and classification.

   [image: prevalidator diagram][image]Lifecycle of an operation,
   including entry points, states, and processing steps


One manager operation per manager per block
===========================================

The prevalidator ensures that at most one manager operation per
manager is classified as "Valid" at any given time. All other
operations received from the same manager will be classified as
"Branch_delayed" and will not be propagated until the "Valid"
operation is cleared. This restriction simplifies maintaining the co-
validity property and thereby speeds up the prevalidator.


Replacing by fee
================

Alternatively, a user can inject an operation with the same manager
and the same counter, but with a higher fee to replace an already
existing operation in the mempool. Only one of the two operations will
be eventually included in a block. To be able to replace the first
operation, the fee and the "fee/gas limit" ratio of the second one is
supposed to be higher than the first's by a factor. This factor can be
configured via the "replace_by_fee" (see filter RPCs) parameter of the
filter configuration of your node. In case of successful replacement,
the old operation is re-classified as "Outdated".

Concretely, a user can replace a successfully classified manager
operation in the mempool, with the help of "octez-client", using two
methods :

* manually provide a higher fee to bump the "fee/gas limit" ratio by
  at least 5% for the new operation,

* via option "--replace": In this case, "octez-client" will
  automatically compute the minimal amount of fee for the second
  operation to be able to replace the one in the mempool.


Banning operations
==================

To remove an injected operation that has not yet been included in a
block, one can ban an operation via the "ban_operation" RPC. The
operation won't be considered again until unbanned via the
"unban_operation" or "unban_all_operations" RPCs. The prevalidator
will reconsider other operations from the same manager that could have
been blocked by the banned one, and can lead to the advertisement of
one of them. Note that banning is local to your node, meaning that
peers won't be informed in any way. In particular, if the banned
operation has already been advertised to peers, and that your local
node advertises a new manager operation that is conflicting with the
former, peers will classify the latest "Branch_delayed" if they hold
the former in their mempool.

The storage layer
*****************

This document explains the inner workings of the storage layer of the
Octez shell. The storage layer is responsible for aggregating blocks
(along with their respective ledger state and metadata) and operations
within blocks. It is composed of two main components: a store
component providing storage abstractions for blockchain data such as
blocks and operations; and the context component providing storage
abstractions for ledger states (also called contexts).


Store
=====

The store component is the Tezos_store  module implemented in the
src/lib_store directory. It handles the on-disk storage of static
objects such as blocks, operations, block's metadata, protocols and
chain data. The store also handles the chain's current state: current
head, invalid blocks, active test chains, etc. The store component is
designed to handle concurrent accesses to the data. Both a mutex and a
lock file are present to prevent concurrent access to critical
sections. The store also provides an accessor to the context and
handles its initialization, but it is not responsible to commit
contexts on disk. This is done by the validation toolchain.

The store is initialized using a history mode that can be either
*Archive*, *Full* or *Rolling*.

In Archive mode, the storage keeps the complete history of all the
blocks, up to the genesis block. The history of each block includes
the block itself, the context (ledger state) in which the block was
applied, and metadata such as the changes resulting of the block
application.

Depending on the chosen history mode, some data will be pruned while
the chain is growing. In *Full* mode, all blocks that are part of the
chain are kept but their associated context and metadata below a
certain threshold level are discarded. In *Rolling* mode, blocks under
a certain threshold level are discarded entirely. The thresholds of
*Full* and *Rolling* modes may be varied by specifying a number of
additional cycles to keep.

The moments when data may be pruned are when cycles complete. When
this happens, the store performs three operations. First, the block
history is linearized by trimming branches in the completed cycle.
Secondly, the cycle that just became un-reorganizable (with
Tenderbake, this is the predecessor of the completed cycle) is
archived, aka *cemented*. This process is called a **merge** and is
performed asynchronously. Thirdly, blocks in the oldest cemented cycle
(or just their context and metadata), can be pruned, according to the
history mode. Trimming and pruning are further explained next.


Trimming
--------

The protocol indicates to the shell, through some metadata present in
the block application result, how much history is relevant to keep in
order to preserve useful informations. If too much history is present,
the storage layer triggers a clean-up mechanism which trims the
chain's outdated history. Only the linear history that is part of the
finalized chain will remain, discarding all the unreachable forks in
the process. The resulting sequential interval of blocks that is
returned represents a *cycle*.


Pruning
-------

Depending on which history mode is ran and on the amount of additional
cycles, blocks in cemented cycles and possibly their associated
context and metadata may or may not be preserved. For instance, if the
history mode is *Archive*, every block is preserved, with its context
and metadata. If it is *Full* with 5 additional cycles, no cemented
block is pruned, but context and metadata will only be kept for the
blocks in the 6 most recent cemented cycles (see details at Keeping
additional cycles). Older contexts and metadata are pruned.

*Context pruning* has been introduced with Octez v15.0, before that
only metadata was pruned, see Context Pruning Requirements and
Optimizations.


Other features
--------------

It is possible to export a canonical representation of the chain for a
given block, also known as a snapshot, if that block is stored as a
non-pruned one (that is a block from which we can read its header,
metadata and associated context).

Another notable feature is that after pruning the metadata and context
of some blocks, the store has the capability to reconstruct them by
replaying every block and operation present and repopulating the
context. Hence, it is possible to transform a "Full" store into an
"Archive" one (see also Switching between node's modes).

The store also writes on disk the sources of protocols no longer
active. This allows to recompile them or even share them on the
network if needed.


Store variables
---------------

The store maintains two specific variables related to the pruned data,
whose values depend on the history mode:

* The *caboose*, which represents the oldest block known by the store.
  The latter block may or may not have its metadata and context in the
  store. In *Archive* and *Full* mode, this would always be the
  genesis block.

* The *savepoint* which indicates the lowest block known by the store
  that possesses metadata and context.

The *checkpoint* is another variable maintained by the store, that
indicates one block that must be part of the chain. This special block
may be in the future. Setting a future checkpoint on a fresh node
before bootstrapping adds protection in case of eclipse attacks where
a set of malicious peers will advertise a wrong chain. When the store
reaches the level of a manually defined checkpoint, it will make sure
that this is indeed the expected block or it will stop the bootstrap.
When the checkpoint is unspecified by the user, the store sets it to
the value provided by the protocol consensus.

While the node is running, it is possible to call the following RPCs
to access the values of all these variables:

* the checkpoint: GET /chains/<chain_id>/levels/checkpoint

* the savepoint GET /chains/<chain_id>/levels/savepoint

* the caboose: GET /chains/<chain_id>/levels/caboose

* the history mode: GET /config/history_mode


Files hierarchy
---------------

The Store maintains data on disk in the "store" subdirectory of the
node's "<data-dir>", organized as follows:

* "<data-dir>/store/protocols/" the directory containing stored
  protocols.

* "<data-dir>/store/protocols/<protocol_hash_b58>*" files containing
  the stored encoded protocol.

* "<data-dir>/store/<chain_id_b58>/" the *chain_store_dir* directory
  containing the main chain store.

* "<data-dir>/store/<chain_id_b58>/lock" the lock file.

* "<data-dir>/store/<chain_id_b58>/config.json" the chain store's
  configuration as a JSON file.

* "<data-dir>/store/<chain_id_b58>/cemented/" contains the cemented
  cycles and index tables.

* "<data-dir>/store/<chain_id_b58>/cemented/metadata" contains the
  cemented cycles' compressed metadata (using *zip* format).

* "<data-dir>/store/<chain_id_b58>/{ro,rw}_floating_blocks" contains
  the most recent blocks in the chain not yet ready to be archived and
  potential branches.

* "<data-dir>/store/<chain_id_b58>/<stored_data>*" files containing
  encoded simple data structures such as: genesis block, checkpoint,
  savepoint, caboose, protocol levels, forked chains, alternate heads,
  invalid blocks, etc.

* "<data-dir>/store/<chain_id_b58>/testchain/<chain_id_b58>*/"
  contains the stores for every encountered test chains throughout the
  network. The underlying hierarchy follows the same format as the
  *chain_store_dir* directory containing the main chain store,
  described above.


Context
=======

The context component is the the tezos-context  package, implemented
in the src/lib_context library. It is a versioned key/value store that
associates to each block a view of its ledger state. The on-disk
context API  exports versioning concepts similar to Git. The current
implementation is using Irmin as a backend.

The API provides generic accessors/modifiers for manipulating a
concrete context object and git-like commands: "commit", "checkout" to
manipulate different context branches.

The Tezos context comes with a specific context hash function that
cannot be changed. Otherwise, the replicated consistency would not be
maintained. In particular, the resulting hash of the application of a
block is stored in its header. When validated, a block's announced
"context hash" is checked against our local validation result. If the
two context hashes are different, the block is considered invalid.

The context of a block can be accessed using the protocols' RPCs such
as GET ../<block_id>, and more specifically by RPCs under the path
"../<block_id>/context".

The context of the blockchain is only modified by blocks applications.
Only the contexts resulting from the application of valid blocks is
committed on disk, by the validation toolchain.

It is possible to export to a file a concrete context associated to a
specific block's ledger state. This feature dumps a canonical
representation of this ledger state that may be incorporated in a
snapshot, exposing a minimal storage state.

Note that it is possible to enable logging for the context backend
using the "TEZOS_CONTEXT" environment variable. There are two possible
values for this variable: "v" for "Info" logging and "vv" for "Debug"
logging (warning: the "Debug" mode is very talkative). Additionally,
this environment variable allows to tweak, with care, the following
context parameters (using the standard
"TEZOS_CONTEXT="variable=value"" pattern, separating the items with
commas such as "TEZOS_CONTEXT="v, variable=value""):

* "index-log-size": number of entries stored in the Irmin's index
  (default "2_500_000")

* "auto-flush": number of tree mutations allowed before a disk flush
  (default "10_000")

* "lru-size": number of entries stored in the Irmin's LRU cache
  (default "5_000")

Synchronisation heuristic
*************************

When a new node joins the network, it must **bootstrap**: fetch and
validate the chain before starting to bake or attest new blocks. A
bootstrapping node cannot bake or attest new blocks, so for efficiency
it should not bother to track a **mempool**: a pool of active
operations.

Knowing whether a node is bootstrapped is challenging because a node
cannot trust (in general) its neighbours. We tackle this problem using
a **synchronisation heuristic**.


Synchronisation heuristic status
================================

A synchronisation heuristic is used to determine whether the node is
synchronised with respect to its peers.

The current synchronisation heuristic uses a **synchronisation
status** as follows:

* "Unsynced": not synchronised

* "Synced": synchronised and the chain is not stuck

* "Stuck": synchronised and the chain is stuck


Bootstrapped
============

We consider a node bootstrapped if the heuristic's status has been
"Synced" or "Stuck" at least once.

Once the node is bootstrapped, the synchronisation status is still
updated (at least after each change of head) and is accessible via the
RPC "/chains/<chain>/is_bootstrapped". This RPC returns a pair where
the first component indicates if the node is bootstrapped and the
second component is the current status.


Basic description of the heuristic
==================================

The synchronisation heuristic relies on a notion of **candidate**: a
pair of a block's timestamp and a peer. The block from which the
timestamp is taken must have been validated locally and it must be the
most-recent such block advertised by the peer.

The heuristic is parameterised by two values (see node configuration):

* a "threshold" (configuration parameter "synchronisation_threshold"):
  the number of candidates kept by the heuristic;

* a "latency" (configuration parameter "latency"): a delay in seconds
  to control possible forks and the latency of the network (see
  Acceptable values for parameters).

The heuristic maintains the most recent "threshold" candidates it is
aware of, with the constraint that there is at most one candidate per
peer.

The heuristic status is "Synced" if the "threshold" kept candidates
are all recent enough: their age is less than "latency" seconds (the
age is counted with respect to the current local time). Otherwise, the
heuristic status is "Stuck" if the "threshold" kept candidates all
have the same timestamp. Otherwise, the status is "Unsynced".


Acceptable values for parameters
================================

The heuristic accepts any value for the "threshold", but values of "1"
or less are mainly used for testing and debugging:

* If "threshold" is negative, then the status is always "Unsynced".

* If "threshold" is "0", then the status is always "Synced".

* If "threshold" is "1", then the status cannot be "Stuck".

Other values are acceptable for "threshold", but a small "threshold"
(between "2" and "10"; the default being "4") is probably best:
performances and accuracy may degrade for values much higher than
"10".

The default value for "latency" is "50" seconds. Whatever you change
it to, it should be at least long enough to include a few blocks'
worth of the protocol's baking rate plus some network latency, but not
so long that the node considers itself bootstrapped even though it is
many blocks away from the chain's head.

A good value for "latency" is "2" to "5" times the time between
blocks, plus a small delta for network delays. At the time of writing,
the time between two consecutive blocks is "10" seconds when the chain
is healthy (see the consensus algorithm).

A shorter "latency" might give false negatives: delays from a few
neighbours might result in the node considering itself not
synchronised yet, whilst a longer "latency" might give false
positives: the node considers itself synchronised whilst it still has
several blocks to catch-up on.


Formal description of the heuristic
===================================

In the normal case of the synchronisation heuristic, when the
"threshold" parameter is greater than "1", the status is computed as
follows:

* The status is "Synced" if there are more than "threshold" candidates
  that are more recent than "latency".

* The status is "Stuck" if the "threshold" most recent candidates have
  the same timestamp that is older than "latency".

* The status is "Unsynced" otherwise.


If the heuristic fails
======================

The heuristic may fail and declare a node not synchronised, thus not
bootstrapped, when actually it should be.  The administrator of the
node can use the RPC "patch /chains/main {"bootstrapped": true}" to
force the node bootstrapped state, but this should be used carefully.
If you see an issue with the current heuristic, please report it.

The peer-to-peer layer
**********************

This document explains the concepts and internal architecture of the
peer-to-peer (or P2P) layer of the Tezos shell. This part is in charge
of establishing and maintaining network connections with other nodes
(gossip).

The exposition is kept high-level so as to be useful for Tezos users
needing to know the global working of this system and the meaning of
some parameters that can be configured.

The P2P layer is instantiated by the node. It is parameterized by the
type of messages that are exchanged over the network (to allow
different P2P protocol versions/extensions), and the type of metadata
associated with each peer. The latter is useful to compute a score for
each peer that reflects the level of trust we have in it. Different
policies can be used when communicating with peers with different
score values.

The P2P layer is comprised of a pool of connections, a set of
operations on those connections, and a set of workers following the
worker pattern pervasively used in the codebase.

The P2P layer is packaged in module Tezos-p2p , which has
documentation for all the constituent modules.


General operation
=================

The main entry point of the P2P layer is the module P2p .

The type P2p_conn.t  defined by the P2P layer represents a connection
to a peer with some associated metadata. It wraps a P2p_socket.t  type
that is basically a UNIX socket upgraded with I/O scheduling, peer
metadata, cryptographic keys, and two message queues operated by
dedicated workers which operate on those queues.


I/O Scheduling
--------------

The P2P layer uses a scheduling mechanism to control its bandwidth
usage as well as implementing different policies (e.g. read/write
quotas) to different peers. For now, each peer is granted a fair share
of the global allocated bandwidth, but it is planned for the
individual allocated bandwidth to each peer to be a function of the
peer's score.


Message queues
--------------

On top of basic I/O scheduling, two finite-size typed message queues
are used to store incoming (resp. outgoing) messages for each peer, as
implemented in P2p_socket.t . This further restricts the speed at
which communication is possible with a peer; when a queue is full, it
is not possible to read (resp. write) an additional message.


Encryption
----------

The connection between each peer is encrypted using "NaCl"
authenticated-encryption API. This is done to provide an additional
level of security and tamper-proof guarantees in the communication
between peers.


Pool of connections
-------------------

All the above modules are used in P2p_pool , which constitutes the
core of the P2P layer, together with the worker processes described
below. It comprises various tables of connections as well as methods
to query them, also connections are extended with another message
queue where lower-level messages (like responses to ping) are filtered
out and only application-level messages are kept.

See below for a description of workers acting onto the P2P layer.


Welcome worker & connect handler
================================

The welcome worker & the connect handler are responsible for accepting
incoming connections and register them into the pool of connections
managed by the P2P layer. They basically run the "accept(2)" syscall
and build the stack of types that compose a connection. They decide
how this new connection must be handled.


Peers and addresses
-------------------

A peer (node) has 2 different ways to be identified, both are useful
to identify an active connection or to contact a peer.

First, a peer is identified by a cryptographic identifier. This **peer
identifier** is often named "peer_id" but sometimes it is simply named
peer by abuse of notation.

Secondly, a peer can be identified by its **point**, which is an IP
address plus a port (several peers can run on a same IP address, on
different ports). Points are sometimes called (peer) addresses, by
abuse of notation.

A peer may change of point and on the other hand, at two different
times, a point may lead to different peers.


{Black, White, Grey}lists
-------------------------

The welcome worker takes care of filtering all incoming connections
using two static lists of addresses configurable using "octez-admin-
client" and a system table that is handled automatically by the P2P
layer. The node administrator can block (blacklist) or enable
(whitelist) individual IP addresses, while the P2P layer is in charge
of temporarily banning (greylisting) IP addresses and peers who
misbehave.

IP addresses are automatically removed from the greylist table after
some timeout, which is defined by the configuration variable
"greylist_timeout". On the other hand, peers that are greylisted are
maintained in a limited-size list, managed according to a LRU policy
(dropping least-recently used items).

The node administrator can also:

* flush greylist tables using "octez-admin-client";

* unban peers and IP addresses that were manually banned.

Indeed, as a peer has 2 different ways to be identified, it is
important to be able to ban/unban both peers and IP addresses.

Beware that banning/unbanning a peer does not ensure that its IP
address is also banned/unbanned. On the other hand, banning an IP
address will ban all currently connected peers from that IP address
(and disable incoming connections). Therefore, if you want to unban a
peer you should unban its IP address and/or the peer itself, depending
on how it was banned.


Maintenance worker
==================

The maintenance worker is in charge of establishing an appropriate
number of connections with other nodes to guarantee a realistic view
of the state of the blockchain. It is created with a set of targets to
reach regarding the desired amount of peers it needs to keep an active
connection to.

At the pool level, the minimum (resp. maximum) acceptable number of
connections is defined.

At the maintenance worker level, two other sets of thresholds are
defined: "target" (min and max) and "threshold" (min and max).

Given these bounds, the maintenance worker:

* Will be triggered periodically (e.g., every two minutes), when asked
  by the shell, and when the minimum or maximum number of acceptable
  connections is reached.

* Will perform the following actions when triggered: if the number of
  connections is above "max_threshold", it will kill connections
  randomly until it reaches "max_target" connections. If the number of
  connections is below "min_threshold", it will attempt to connect to
  peers until it reaches at least "min_target" connections (and never
  more than "max_target" connections).

The maintenance worker is also in charge of periodically running the
greylists GC functions to unban IP addresses from the greylist.

Protocol Environment
********************

The protocol environment is a restricted API that may be used by the
economic protocol's code, on one hand, and the rest of the Octez code,
on the other hand, to interact with each other. Firstly, this allows
protocols to only call authorized modules and functions. Secondly, it
allows the node (i.e., the main binary built around the shell) as well
as the other binaries (e.g., the baker) to call a subset of the
protocol functions.


Motivation
==========

In the currently active protocol, as in any sound future protocols,
updates are approved by voting. That way, the responsibility of
switching to a new protocol code is the responsibility of voters, and
one could argue that it is up to them to check that the code does not
call, for instance, unsafe array access functions.

Yet, we decided to introduce a minimum level of machine checks, by
compiling the protocol with a specific compiler that checks that no
known-unsafe function is used in its code. This static form of
sandboxing is performed by the OCaml typechecker: we simply compile
protocols in a restricted set of modules with restricted interfaces
that hide any unsafe, non wanted feature.

Another goal of that specific environment is maintaining a stable
OCaml API for protocol development. Imagine that at some point, the
OCaml standard library changes (a function is added or removed, a type
is changed), then we will be able to upgrade to the new OCaml while
still remaining compatible with past protocols, by providing an
adapter layer.


Environment contents
====================

Here is a quick description of each file in this environment, all
located under src/lib_protocol_environment/:

* Files in the "sigs/vX/" directories declare the interface that the
  protocols can be compiled against. These interfaces are repeats of
  other library interfaces with some unsafe functions removed. These
  interfaces are frozen and cannot be changed: changing them can break
  previous protocols.

  * Special cases: the files "context.mli", "fitness.mli", and
    "updater.mli" are interfaces that the *protocol* exposes to the
    shell (rather than the other way around).

  * All the files in "sigs/vX/" are assembled into a single interface
    file "VX.mli" by a helper program located in "s_packer/".

* Files in the "structs/vX/" directory declare compatibility layers
  for old protocol environments. E.g.,
  "structs/V0/error_monad_traversors.ml" contain the code of error-
  monad-compatible list traversors that used to be part of the Error
  Monad at the time of the environment V0. These implementations are
  meant to evolve alongside the ecosystem: when an incompatibility is
  introduced in a library a compatibility layer needs to be
  implemented or amended in order to provide a backwards compatible
  overlay to the protocol. These implementations are assembled into a
  single implementation file "VX.ml" by a helper program located in
  "s_packer/".

The API can be found in tezos-protocol-environment


Environment versions
====================

An environment interface includes a frozen, immutable interface
provided to the protocol. And so when a new protocol needs new
functions, types, or values, this protocol must use a new environment.
This is why the environments are versioned.

A protocol's manifest (e.g.,
src/proto_alpha/lib_protocol/TEZOS_PROTOCOL) includes a field named
"expected_env_version". This field specifies the environment used to
compile the protocol.

Protocol environments can only ever increase in version. Specifically,
from within a protocol P1 built on environment version X, the
activation of a protocol P2 built on environment Y fails unless X ≤ Y.

Some examples of prior environment switches can be found within past
protocol changelogs:

* Edo - environment V1

* Hangzhou - environment V3

* Ithaca - environment V4

* Jakarta - environment V5

* Kathmandu - environment V6

For how to prepare and implement an environment switch, see Adding a
new protocol environment.

Micheline
*********

Micheline is a data format comparable to JSON, XML, S-expressions, and
YAML. Its main purpose is to serve as the concrete syntax for the
Michelson language.


Syntax
======

The structure of a Micheline node is very simple: a node can only be
one of the five following constructs:

1. An integer in decimal notation.

2. A character string delimited by the double quotation character """.

3. A byte sequence in hexadecimal notation prefixed by "0x".

4. The application of a primitive to a whitespace-delimited list of
   nodes and annotations.

5. A sequence of nodes delimited by curly braces ("{" and "}") and
   separated by semi-colons (";").


Primitives
----------

Micheline primitives are character strings starting by an alphabetic
or underscore ("_") characters and containing only alphanumeric or
underscore ("_") characters. Additional restrictions on primitives can
also be put to achieve the equivalent of JSON and XML schemas; for
example each protocol specifies a fixed list of Michelson primitives.
This restriction impacts the way Micheline nodes are serialized to
binary format (see Binary serialization).


Annotations
-----------

Micheline annotations start by one of the special characters "@", ":",
"$", "&", "%", "!", and "?" followed by any number of alphanumeric,
underscore ("_"), period ("."), percent sign ("%"), and at sign ("@")
characters.


Character strings
-----------------

The following usual escape sequences are allowed inside strings: "\n",
"\\", "\"". Unescaped line-breaks (both "\n" and "\r") cannot appear
in Micheline strings. Since Micheline strings are delimited by double
quotes, the double quote character cannot appear unescaped inside a
string.


Primitive applications
----------------------

A primitive application is a primitive followed by zero, one, or
several arguments (either nodes or annotations):

   prim arg1 arg2

When a primitive application with more than 0 arguments is the
argument of another primitive application, it must be wrapped with
parentheses. This wrapping is also allowed for applications without
arguments.

   prim (prim1 arg11 arg12) (prim2 arg21 arg22)


Sequences
---------

Successive nodes can be grouped as a single sequence node using curly
braces as delimiters and semicolon as separators.

   { expr1 ; expr2 ; expr3 ; expr4 }

A sequence can be passed as argument to a primitive.

   prim arg1 arg2 { arg3_expr1 ; arg3_expr2 }

Primitive applications right inside a sequence cannot be wrapped.

   { (prim arg1 arg2) } # is not ok


Indentation
-----------

To remove ambiguities for human readers, the Micheline parser enforces
some indentation rules.

* For sequences:

  * All nodes in a sequence must be aligned on the same column.

  * An exception is made when consecutive nodes fit on the same line,
    as long as the first of them is correctly aligned.

  * All nodes in a sequence must be indented to the right of the
    opening curly brace by at least one column.

  * The closing curly brace cannot be on the left of the opening one.

* For primitive applications:

  * All arguments in an application must be aligned on the same
    column.

  * An exception is made when consecutive arguments fit on the same
    line, as long as the first of them is correctly aligned.

  * All arguments in a sequence must be indented to the right of the
    primitive name by at least one column.


Comments
--------

A hash sign ("#") anywhere outside of a string literal will make the
rest of the line (and itself) completely ignored, as in the following
example.

   { PUSH nat 1 ; # pushes 1
     PUSH nat 2 ; # pushes 2
     ADD }        # computes 2 + 1

Comments that span on multiple lines or that stop before the end of
the line can also be written, using C-like delimiters ("/* ... */").


File encoding
-------------

Micheline files must be encoded in UTF-8. Non-ASCII characters can
only appear in comments and strings.


BNF grammar
-----------

Formally, Micheline nodes have the following BNF grammar:

   <natural> ::= [0-9]+
   <char> ::= \" | \r | \n | \t | \b | \\ | [^"\]
   <hexadecimal> ::= [0-9a-fA-F][0-9a-fA-F]
   <primitive> ::= [a-zA-Z_][a-zA-Z_0-9]*
   <annotation> ::= [@:$&%!?][_0-9a-zA-Z\.%@]*

   <node> ::= <integer> | <string> | <bytes> | <primitive application> | <sequence>
   <integer> ::= <natural> | - <natural>
   <string> ::= " <char>* "
   <bytes> ::= 0x<hexadecimal>*
   <primitive application> ::= <primitive> <argument>*
   <argument> ::= <integer> | <string> | <bytes> | <primitive> | ( <primitive application> ) | <sequence> | <annotation>
   <sequence> ::= { nodes }
   <nodes> ::= | <node> | <node> ; | <node> ; <nodes>


Serialization
=============

Micheline nodes can be converted to JSON and binary formats. The JSON
conversion is useful for example to interact with a Tezos node using
RPCs. The binary encoding is used to store Micheline nodes in blocks.

Both forms of serialization are achieved using The data_encoding
library.


Locations
---------

Locations are used in error messages to indicate which part of a
Micheline file is faulty. Micheline file locations are composed of two
points; a starting point and a stopping point. Each point is a pair of
a line number (counted from 1) and a column number (counted from 0).

Locations only make sense when we have access to the source file and
they are lost when serializing Micheline nodes to either JSON or
binary. During translation, locations are replaced by a *canonical*
representation of locations. A canonical location is a single integer;
the root of the node has a canonical location of zero, and each node
adds one in the order of infix traversal.

A mapping between file locations and canonical locations is produced
during the serialization so that errors reported by tools working on
the serialized form can be printed in a meaningful way to users.


Conversion to JSON
------------------

The JSON conversion is very straightforward:

* integers are encoded as JSON objects with a single ""int"" field
  whose value is a JSON string;

* strings are encoded as JSON objects with a single ""string"" field
  whose value is either a JSON string or an array of bytes if the
  string to encode is not valid in JSON;

* byte sequences are encoded as JSON objects with a single "bytes"
  field whose value is a JSON string containing the hexadecimal
  representation of the byte sequence without the "0x" prefix.

* sequences are encoded as JSON arrays;

* primitive applications are encoded as JSON objects with three fields
  named ""prim"", ""args"", and ""annots"" containing respectively the
  primitive name, the JSON array of argument nodes, and the JSON array
  of annotations. Both ""args"" and ""annots"" are optional and
  default to the empty array.


Binary serialization
--------------------

The binary serialization of Micheline is more complex because it
contains some space optimisations for the very common cases of
primitive applications with few arguments. Moreover, the binary
encoding depends on the way primitives are encoded.

The documentation for the binary encoding specialized to Michelson
primitives of the Alpha protocol can be generated by the command
"octez-codec describe alpha.script.expr binary schema".


Usage of the OCaml Micheline library
------------------------------------

In the Octez codebase, Micheline nodes are handled by a small library
named "lib_micheline". This library is used in the Octez client
(responsible for parsing the Michelson files, expanding macros, and
encoding the result to JSON) and in the Tezos protocol (responsible
for decoding from JSON, type checking, and serializing to binary
Michelson scripts and values).

The library is composed of three modules: "Micheline",
"Micheline_parser", and "Micheline_printer". Only the "Micheline"
module is needed for the protocol so its signature is the only one
exported in the protocol environment.

The "Micheline" module defines the type of Micheline nodes. This type
is parameterised by the type of locations "'l" and the type of
primitives "'p".

   type annot = string list

   type ('l, 'p) node =
     | Int of 'l * Z.t
     | String of 'l * string
     | Bytes of 'l * Bytes.t
     | Prim of 'l * 'p * ('l, 'p) node list * annot
     | Seq of 'l * ('l, 'p) node list

Since printing and parsing are provided by the two other modules of
the library, the only way to produce or consume Micheline nodes from
this module is through the encodings.  To encode a Micheline node, we
first canonicalize its locations using the "extract_locations"
function that takes a "('l, 'p) node" as argument and returns both a
"'p canonical" and an association list between "canonical_location"s
and "'l" locations. The type "'p canonical" is an abstract version of
"(canonical_location, 'p) node" that ensures the invariant on the
numbering of canonical locations presented in the Locations section
and "canonical_location" is an alias of "int".

If we know an encoding for the "'p" primitives, we can derive an
encoding for "'p canonical" using the "canonical_encoding" function
and then produce JSON and binary outputs using the functions from The
data_encoding library.

In the other direction, from a "'p canonical" produced by an encoding,
either we know how to map its canonical locations to file locations
and we can use the "inject_locations" function to do so or we want to
keep the locations canonical and use the "root" function to obtain a
plain node.

The "Micheline_printer" module exports two pretty-printing functions
based on the "Format" standard OCaml module: "print_expr" and
"print_expr_unwrapped". The difference is that "print_expr" wraps its
output in parentheses if it is a primitive application with at least
one argument. The input of these printing functions can be produced
from a "'p canonical" by the "printable" function.

The "Micheline_parser" module can be used to produce Micheline nodes
from strings and files. It is done in two stages: tokenization (also
known as lexing) and parsing. Tokenization is done by the "tokenize"
function that uses the "uutf" OCaml library to decode UTF-8
characters. For the parsing phase, two functions "parse_expression"
and "parse_toplevel" are exported; the former produces a single
Micheline node while the second parses a list of expressions separated
by semicolons. They both take an optional boolean parameter named
"check"; setting it to "false" disables indentation checking.


Other tools and resources
=========================

The following links are not part of the Octez OCaml code base but are
reimplementations of parts of "lib_micheline" in other tools and
languages:

* https://medium.com/the-cryptonomic-aperiodical/the-magic-and-
  mystery-of-the-micheline-binary-format-33bf85699bef describes the
  binary serialization of Micheline nodes. This documents the
  ConseilJS library but assumes no familiarity with it.

* A Micheline parser written using the Menhir parser generator is
  available as part of the Mi-Cho-Coq framework: https://gitlab.com
  /nomadic-labs/mi-cho-
  coq/-/blob/master/src/michocoq/micheline_parser.vy it is incomplete
  as it does not support annotations.

* An online converter between Micheline files and their JSON encoding
  (improperly called "Michelson") developed as part of Cryptonomic
  hackathon and documented here: https://scalac.io/blog/we-hakked-
  tezos-and-created-a-micheline-michelson-translator/

The Octez Baker
***************

The Octez baker is an executable responsible for participating in the
Tezos blockchain consensus by producing blocks and creating consensus
operations on behalf of a set of delegated accounts.


Consensus Operations
====================

The baker plays a central role in the consensus protocol. When a new
block is validated by the node, the baker receives this proposal and
initiates the consensus mechanism. The baker may have injected the
block itself, if it was its turn to do so.

First, the baker ensures that the proposal is valid — meaning that the
proposed block has the correct level and round, and meets other
protocol criteria. Once validated, the baker begins the voting
process.

Tezos uses a two-phase voting system for consensus:


Preattestation Phase
--------------------

The baker produces preattestation operations for the proposal on
behalf of each delegate it manages. These preattestations are injected
into the node and propagated through the network. At the same time,
the baker monitors the node's mempool for preattestations from other
bakers.

For each valid preattestation received, the baker retrieves the
delegate’s baking power and updates a prequorum state. Once the baker
observes preattestations representing at least 66% of the total baking
power, a prequorum is considered reached, and the second voting phase
begins.


Attestation Phase
-----------------

Before attesting, the baker queries the DAL node to determine whether
any DAL slots should be attested. It then crafts an attestation
operation, optionally including this DAL data. Like with
preattestations, the baker monitors attestations from other bakers.

If attestations representing at least 66% of the total baking power
are observed, the proposal block is elected. This means that a
sufficient number of delegates have confirmed the block, and the chain
can proceed to the next level.

For more details, see The consensus algorithm.


Block Production
================

Blocks are produced at regular intervals, defined by the block_delay
parameter. The baker is responsible for producing a block when one of
its delegates has baking rights for that level and round. (See the
Baking power documentation for details.)

When the previous proposal block is elected, the baker can construct a
new block at the next level. This block must include a proof of
consensus for the elected block — essentially, a set of attestations
supporting it.

Next, the baker selects operations from the node’s mempool. These
operations include:

* Voting operations (e.g., protocol proposals and ballots)

* Anonymous operations (e.g., nonce revelations, denunciations)

* Manager operations (e.g., token transfers, smart contract calls)

Since a block has limits — such as the maximum allowed gas — the baker
must solve an optimization problem (akin to the Knapsack Problem) to
choose which operations to include.

Once the operations are selected, the baker creates a block header
containing metadata such as the block’s level, operation hashes,
fitness, and more (see the Shell header documentation and protocol-
specific block details). After the header is signed, the block is
ready to be injected into the node.

To ensure timely block injection, the baker performs all of these
steps ahead of time, so the block can be submitted exactly at the
scheduled time.

Once injected, the block is validated by the node and propagated
through the network. Other bakers then begin the attestation process
for a new block, continuing the cycle of consensus.


Support for "tz4" (BLS) Keys
============================

Tezos supports the use of tz4 keys using BLS cryptography (starting
with protocol Seoul) as consensus keys for delegates. This enables
(pre)attestation aggregation, a feature that significantly improves
efficiency by allowing many (pre)attestations to be compressed into a
single operation within blocks.

BLS (Boneh–Lynn–Shacham) signatures have the unique property that
multiple signatures over the same payload can be aggregated into a
single signature. This makes them ideal for consensus, where many
bakers attest to the same block. However, for this to work
efficiently, all (pre)attestations must have identical payloads.

This becomes a challenge when DAL (Data Availability Layer) data is
involved. DAL attestations are included as part of the regular Layer 1
attestation operation, and the set of DAL slots attested to is
different for each delegate. This means the payloads differ, which
prevents simple BLS aggregation.


The Role of the Companion Key
-----------------------------

To enable aggregation even when DAL slot data is included, a separate
Companion Key is used (introduced in protocol Seoul):

* This is a second BLS key, separate from the consensus key.

* The consensus key signs the common part of the attestation (the
  block reference and quorum vote).

* The companion key signs the delegate-specific DAL payload (the set
  of DAL slots being attested).

Both signatures, one from the consensus key and one from the companion
key, are aggregated and included in the attestation. When a baker
produces the next block, they aggregate all "tz4" attestations
(including both already aggregated signatures) into a single
aggregated attestation.

Data Availability Layer
***********************

The Data-Availability Layer (or DAL) is a software component of Octez
which allows users to publish data outside of the Tezos Layer 1
blocks, with no compromises on decentralization principles. Its
primary use case today is to scale the data bandwidth of smart
rollups, but other applications could be envisioned in the future.

The following pages provide an overview of the DAL and cover its
different aspects and features.

* DAL overview

  * DAL slots

  * The DAL's P2P protocol

* The DAL node

  * Concepts and features

  * Operational aspects

* Running a DAL attester node

  * Connecting to other nodes

  * Running the DAL node

  * Troubleshooting

* Bakers & the DAL

See also:

* Protocol support for the DAL

DAL overview
************

The *Data-Availability Layer* (DAL) enables users to publish blobs of
data outside of the Tezos Layer 1 (L1) blocks. A blob (for “binary
large object”) is a piece of data in binary form. Currently, the
primary use case for these blobs is to represent Layer 2 (L2)
operations for Tezos smart rollups, but the DAL is more generic and
could be used for other use cases in the future.

In practice, the DAL is implemented by a network of *DAL nodes*,
constituting a *DAL network*. The DAL network employs a P2P protocol
to publish blobs (distinct from the P2P layer used by the Octez node),
enabling a better bandwidth sharing between the peers (further
elaborated in the P2P section below). The DAL currently supports a
data bandwidth of 0.5MiB/s, a stark contrast to the current situation
on L1, which has an approximate bandwidth of 32 KiB/s. This highlights
the significant boost in bandwidth provided by the DAL. Furthermore,
the DAL aims to put back by several orders of magnitudes the data
bandwidth constraints imposed by the L1 block size, up to 100MiB/s.
This translates into reduced fees for users when posting L2 operations
without compromising the fundamental principle of decentralization.

The amount of data that can be transmitted over the DAL network is,
however, still controlled by the economic protocol. The economic
protocol also plays a crucial role in determining the availability of
those data through the participation of bakers.

Similarly to the Tezos L1, the DAL is permissionless, enabling any
user to effectively contribute data to it, and allowing any smart
rollup kernel to access this data.

[image: DAL overview][image]

The figure illustrates the standard process for interacting with the
DAL as follows:

The diagram depicts a scenario where a user intends to upload data for
a dedicated rollup.

Anyone engaging with the DAL must utilize a DAL node, implemented by a
binary named "octez-dal-node". When a user decides to provide a new
blob to the DAL (depicted as step 1 in the diagram), the user
transmits the data to the DAL node to calculate a commitment to the
data. This commitment is then communicated to L1 via a dedicated
operation (indicated as step 2). Following L1’s approval of this
operation (step 3), the DAL expands this data for redundancy,
partitions the expanded data into segments called shards, and
disseminates the shards across the DAL network (shown in step 4).
Shards are assigned to bakers proportionally to their stake by the
economic protocol. Hence, the bakers, also connected to the DAL
network, retrieve these assigned shards (step 5). The bakers must
download and verify these shards within a specific timeframe,
precisely defined by the economic protocol as the "attestation lag"
period (see DAL-related protocol constants). At the end of the
attestation lag period, bakers declare using an optional field of the
consensus attestation operation, whether they were able to download
the shards effectively (illustrated in step 6). The economic protocol
collates these attestations and, if a sufficient number of bakers have
successfully obtained the shards, the blob is declared as available
(step 7). Only when the blob is labeled as available can the rollup
utilize it (represented as step 8).

The rationale for having the attestation lag parameter is to give
bakers sufficient time to download their assigned shards, while
guaranteeing that the latency of publishing blobs stays within
acceptable limits (around one minute).


DAL slots
=========

There is a limited number of blobs, called *DAL slots* (or slots for
short), that can be published at a L1 level. A slot is identified by
the L1 level at which it is published, an index, and its commitment.
Depending on the context, a “slot” may refer to the data blob or to
the placeholder (the slot index) of that blob. Each slot is treated
independently of other slots. Presently, all slots are of the same
size, though this may change with future updates. To publish a slot,
it is first expanded to create redundancy through erasure coding,
increasing its size by a factor known as the *redundancy factor*. This
expanded slot is then divided into a set of smaller data pieces of
equal size, called *shards*. Finally, shards are distributed across
the DAL P2P network.

The advantage of using erasure coding is that one only needs a subset
of the shards, specifically, “number of shards / redundancy factor” to
recover the full original slot data. A key feature for the shards is
that each shard can be verified to ensure it corresponds to a
particular commitment, which safeguards against spam in the DAL P2P
network.

For each slot index, multiple commitments can be included in a block,
but only one will be acknowledged, the first one appearing in the
block ordering. Because bakers typically prioritize operations that
offer higher fees, the commitment with the higher fee is usually
chosen for inclusion, although this outcome is not guaranteed if a
baker uses a different selection criterion.

For smart rollups, it is necessary to divide the slot into smaller
segments called *pages* (see Populating the reveal channel). The
economic protocol specifies the size of each page as a constant, with
4KiB being a practical size. While it might seem feasible to choose
shards fitting this page limit and directly feed shards to the rollup,
accessing the original data from shards involves complex cryptographic
computations. We prefer to avoid performing such computations within
the smart rollups. Therefore, even though the DAL network uses shards
internally, the DAL node can serve individual pages as requested by
the smart-rollup node. These pages can then be imported individually
by the smart rollup node to the kernel on demand. To reconstruct the
entire slot, one simply needs to arrange these pages in the correct
sequence. For developers working with smart-rollup kernels, the
technicalities of reconstructing the entire slot out of pages should
ideally be handled by the SDK they use, simplifying the process even
more.

[image: DAL slot][image]

When the slot producer (user publishing a slot) posts the commitment
onto the L1, it also posts a proof that the slot does not exceed the
slot size. This prevents malicious users from producing and posting
data larger than the expected size.


The DAL's P2P protocol
======================

The peer-to-peer (P2P) protocol for the DAL is made out of two
components:

* A gossipsub algorithm, instantiated as detailed below.

* A transport layer for handling connections with peers. We reuse for
  that the P2P protocol used by the Octez node.

The gossip algorithm used for the DAL is an in-house version of the
gossipsub v1.1 P2P protocol defined by the lib-p2p project. A detailed
overview of this protocol is available here and an informal English
specification can be found here. This gossip algorithm allows to
partition the network into virtual subnetworks, each identified by a
**topic**. The topic also determines the valid data that can be
exchanged over the corresponding virtual subnetwork, as any exchanged
message has exactly one associated topic. Each peer subscribes to
topics of interest to him. This protocol enhances the network's
scalability compared to traditional gossip algorithms.

For each message, there is a **message id** that uniquely identifies
this message. When a message is pushed, it comes with its message id.
When the message is pulled, it is done via the message id. For every
topic a node subscribes to, it maintains a virtual subnetwork, or
**mesh**, of peers also subscribed to that topic. When a node has a
new message to share (originating from the application layer) or needs
to relay a received message, it does so to all peers in the
corresponding topic's mesh. Moreover, the node broadcasts the ids of
the last received messages to a random selection of peers outside the
mesh. Peers receiving these teasers can request the full message if
they are interested in it. For the DAL instantiation of gossipsub, a
message is defined as a 3-tuple: a shard, the shard’s index, and the
shard’s proof proving that the shard corresponds to the commitment
given by the message id. The associated message id consists of the
shard index and the associated slot index, (published) level, slot
commitment, and attestor’s public key hash.

A topic is defined as a pair "(slot_index, public_key_hash)". The
first component identifies the slot associated to any shard published
under this topic, while the second component identifies the baker
assigned to this shard. Such a set of topics ensures that the
bandwidth of bakers and slot producers is bounded (for valid messages)
over a cycle.

A slot producer should subscribe to all relevant topics associated
with their slot index. This includes every topic where a baker is
assigned at least one shard for that slot index. On the other hand, a
baker should subscribe to all topics that feature their public key
address.

Gossipsub also defines a notion of score which is used to only connect
to peers with a good score.

Regarding peer discovery, the current implementation of the DAL relies
on gossipsub v1.1 peer exchanges. In particular, DAL nodes can be
configured in bootstrap mode to facilitate peer discovery.

Note:

  The current topic structure in the DAL for Tezos may be revised in a
  future update. Presently, topics include the bakers’ address (public
  key hash), which leads to a potentially unbounded number of topics
  over time. Another approach under consideration involves using a
  "(slot_index, shard_index)" pair, offering a more scalable solution
  in the long run, when the number of attesters surpasses the number
  of slots.

Warning:

  Attention must be paid to the security implications for bakers in
  the DAL network. Since a baker's bandwidth is proportional to their
  stake, it can become relatively straightforward to identify the IP
  address of their DAL node, particularly for those with substantial
  stakes. To mitigate this risk, bakers are advised to operate their
  DAL node using an IP address different from their L1 node. This
  separation helps in preventing the unintentional exposure of the L1
  node's IP address.Plans are underway to address these concerns. One
  proposed solution is to enable bakers to divide their bandwidth
  across multiple DAL nodes, enhancing both security and operational
  flexibility.

The DAL node
************

A DAL node is an executable (called "octez-dal-node") whose main roles
are to publish, store, and exchange data for the DAL layer.
Interacting with a DAL node is done through the command-line interface
(CLI) and through RPCs. In what follows, we describe the main
components and features of a DAL node, and then present some
operational aspects on configuring and running a DAL node.


Concepts and features
=====================


DAL P2P network
---------------

The "octez-dal-node" executable runs a node in the DAL’s P2P network.
Recall that the DAL's P2P protocol is based on a gossipsub algorithm
for distributing shards, running on top of a networking layer using
the same P2P library as L1 nodes.

Actors with various roles run DAL nodes, but they need different views
of the network depending on their role:

* Operators of Smart Rollups care about data on the slots that their
  Smart Rollups use, but not about the slots used by other rollups.

* Bakers only care about the shards that they are assigned to attest
  to, but they must watch all slots because the assigned shards can be
  in any slot.

* Bootstrap node operators care about other nodes connecting to peers
  that are interested in the same messages via the P2P network but
  don't care about the content of the messages on the network or any
  specific slot or shard more than any other.

Non-bootstrap DAL nodes distinguish themselves only in the topics they
subscribe to:

* Operators subscribe to all topics containing some specified slot
  indexes.

* Bakers subscribe to all topics containing the attester identities
  they run for (for all possible slot indexes).

Bootstrap nodes are DAL network entry points that are used for network
discovery. A bootstrap node remains connected to a large number of
peers and is subscribed to all topics. When a DAL node starts, it gets
the URLs of the bootstrap nodes from its layer 1 node and uses these
bootstrap nodes to connect to peers. When a DAL node does not have the
necessary connections to the P2P network, bootstrap nodes provide
connection points with the relevant topics.


Modes
~~~~~

The DAL node has two modes:

* **Controller mode** is appropriate for most nodes; it is for nodes
  that post, share, or attest data. Nodes in this mode run in one or
  more profiles to determine what data they post, share, or attest, as
  described below.

* **Bootstrap mode** is for nodes that help other nodes connect to
  peers. Bootstrap nodes are already running on most Tezos networks,
  so in most cases, you don't need to run a node in bootstrap mode
  unless you are starting a new Tezos network.

  To run a DAL node in bootstrap mode, pass the "--bootstrap-profile"
  argument, as in this example:

     octez-dal-node run --endpoint http://127.0.0.1:8732 --bootstrap-profile --data-dir $DATA_DIR

  The configuration file for a DAL node running in bootstrap mode
  shows only the base information about the node, as in this example:

     {
       "data-dir": "dal-node/",
       "endpoint": "http://127.0.0.1:8732",
       "profiles": {
         "kind": "bootstrap"
       }
     }


Profiles
~~~~~~~~

As described above, different actors are interested in different roles
when running DAL nodes. Some want to produce data, others want to
share data, and others want to attest data. For this reason, DAL
controller nodes run in different profiles; these profiles control
which data the nodes post, share, and attest. You can set these
profiles in the node's configuration file, as CLI arguments to the
node's commands, or via RPC calls.

Currently, a DAL node in controller mode can use any combination of
profiles. However, for future-proofing and for production
installations, run a DAL controller node in a single profile.

A DAL node in controller mode can run in these profiles:

* The "operator" profile (formerly the "producer" profile) is for
  users who are running a Smart Rollup as an operator and want to post
  data. To run a DAL node with the "operator" profile, pass the "--
  operator-profiles" argument with the indexes of the slots to accept
  data for, as in this example:

        octez-dal-node run --endpoint http://127.0.0.1:8732 --operator-profiles=0,1 --data-dir $DATA_DIR

* The "attester" profile is for bakers who want to attest to data. DAL
  nodes in this profile must pass the "--attester-profiles" argument
  with the public key hashes of the bakers to attest data for (their
  delegate or manager keys, not their consensus keys), as in this
  example:

        octez-dal-node run --endpoint http://127.0.0.1:8732 --attester-profiles=tz1QCVQinE8iVj1H2fckqx6oiM85CNJSK9Sx --data-dir $DATA_DIR

* The "observer" profile contributes to the resilience of network by
  helping distribute data in the specified slots. To run a DAL node
  with the "observer" profile, pass the "--observer-profiles" argument
  with the indexes of the slots to monitor or an empty string (as in "
  --observer-profiles ''") to use a random index, as in this example:

        octez-dal-node run --endpoint http://127.0.0.1:8732 --observer-profiles=0,1 --data-dir $DATA_DIR

The configuration file for a DAL controller node shows its active
profiles. For example, this configuration file shows that it is
running in the "operator" profile and accepting data for slots 0 and
1:

      {
        "data-dir": "dal-node/",
        "endpoint": "http://127.0.0.1:8732",
        "profiles": {
          "kind": "controller",
          "controller_profiles": {
            "operators": [ 0, 1 ],
            "observers": [],
            "attesters": []
          }
        }
      }

The configuration file for a DAL node running with the "attester"
profile shows the public key hashes of the associated bakers, as in
this example:

      {
        "data-dir": "dal-node/",
        "endpoint": "http://127.0.0.1:8732",
        "profiles": {
          "kind": "controller",
          "controller_profiles": {
            "operators": [],
            "observers": [],
            "attesters": [ "tz1QCVQinE8iVj1H2fckqx6oiM85CNJSK9Sx" ]
          }
        }
      }

By default, the DAL node runs in controller mode without any profile.

When a baker starts with the "--dal-node" argument, it checks the DAL
node's configuration. If the DAL node is not in bootstrap mode and not
already set up with the "attester" profile, the baker configures the
DAL node to use the attester profile associated with the keys that it
is using.


Routing
~~~~~~~

DAL nodes must be able to initiate connections outside their local
network and accept connections from outside their local network. By
default, the DAL node accepts P2P connections on port 11732, but you
can change the address and port that the node listens on by setting
the "--net-addr" argument. In simple setups with a single DAL node,
routing configuration is usually not necessary. However, if you are
using a load balancer or running multiple DAL nodes, you may need to
configure port forwarding on your router and the ports and addresses
that the DAL nodes use.

The "--net-addr" argument sets the address and port that the node
listens for incoming connections on. For example, setting "--net-addr
0.0.0.0:11733" makes the node listen on port 11733 on all network
interfaces.

The "--public-addr" argument sets the address and port at which other
nodes can reach the node, such as if you are using a load balancer,
firewall rule, or router to forward traffic to the node from another
port. The node does not actually bind to this port or listen on it.

If you change the defaults with the "--net-addr" or "--public-addr"
arguments, you must configure your firewall to permit incoming and
outgoing TCP connections on the ports that you specify in the
arguments.


Storage
-------

The DAL node essentially stores slots and shards. Slots are injected
into the node through an RPC (see details at Lifetime of slots and
shards), at which moment the corresponding commitment is computed and
stored. Shards and their proofs are computed and stored via another
RPC. It is important to also compute the shards’ proofs, because
shards can be exchanged over the P2P network only if they are
accompanied by their proof. Shards received over the P2P network are
also stored. The node also tracks and stores the status of commitments
by monitoring the L1 chain, connecting to this end to an L1 node
specified at startup.

The amount of storage space a DAL node needs depends on how long it
keeps the data, and different profiles keep the data for different
amounts of time:

* Bootstrap nodes store no DAL data and therefore require negligible
  storage.

* Attester and observer nodes store data in memory for a few blocks
  after the attestation delay by default.

* Operator nodes store data on disk for the slots registered with this
  profile for 3 months by default because the data may be needed for
  the Smart Rollup refutation game.

You can set how long the node stores data with the "--history-mode"
option.


L1 monitoring
-------------

The DAL node tracks the L1 chain, in particular it monitors the heads
of the L1 chain via an RPC. Each time a block becomes final, it
retrieves and stores the commitments published in that block, and if
its storage contains shards for these commitments, it publishes them
on the P2P network. The DAL node also retrieves the slot attestation
status from the block’s metadata. Finally, based on this information
it updates the status of the stored commitments, where the status of a
commitment can be one of the following:

* *waiting for attestation*: The commitment was included and applied
  in a finalized L1 block but the corresponding slot remains to be
  attested.

* *attested*: The commitment was included in an L1 block and the
  corresponding slot is attested.

* *unattested*: The commitment was included in an L1 block but the
  corresponding slot was not timely attested.

* *not selected*: The commitment was included in an L1 block but was
  not selected for that slot index. This can happen if there is
  another commitment in the same block for the same slot index that
  was placed before the commitment in question (in operation order) by
  the baker.

* *unseen or not finalized*: The commitment was not seen in a final L1
  block. For instance, this could happen if the RPC "PATCH
  /commitments/<commitment>" was called but the corresponding
  commitment was never included into a block; or the commitment was
  included in a non-final block. This means that the publish operation
  was not sent (yet) to L1, or sent but not included (yet) in a block,
  or included in a not (yet) final block.


RPC server
----------

The DAL node incorporates an RPC server which answers to RPC queries
to update or retrieve information about the node’s state. For
instance, one can post slots, ask the node to compute and store
shards, to update profiles, or to connect to or disconnect from peers.

The default listening port for RPCs is 10732, but can be changed using
option "--rpc-addr". The RPC server is started by default, even if
this option is not given.


Operational aspects
===================


DAL node commands
-----------------

The DAL node has two commands "config init" and "run".

The command "init config" creates a new configuration file in the
specified data directory or in the default location (ie "~/.tezos-dal-
node") with the parameters provided on the command-line by the
corresponding arguments, in case no configuration file exists already.
If such a file already exists, it overrides it with the provided
parameters (old parameters are lost).

The command "run" runs the DAL node. The CLI arguments take precedence
over the configuration file arguments, except for the list of
bootstrap peers and of profiles, which are considered in addition to
the ones from the configuration file. The configuration file is
however not overridden with the new values of the node’s parameters.
However, at the end of the execution, the node’s profiles, which may
have been given as arguments or set via RPCs, are written to the
configuration file.

Both commands have the same arguments, which can be seen by executing,
e.g., "octez-dal-node config init --help":

   OPTIONS
          --attester-profiles=PKH1,PKH2,..., --attester=PKH1,PKH2,...
              The Octez DAL node attester profiles for given public key hashes.

          --bootstrap-profile, --bootstrap
              The Octez DAL node bootstrap node profile. Note that a bootstrap
              node cannot also be an attester or a slot producer

          --config-file=FILE
              The configuration file of the octez DAL node.

          -d DIR, --data-dir=DIR
              The directory where the Octez DAL node will store all its data.
              Parent directories are created if necessary.

          --disable-amplification
              Disables shard amplification. Default value is false.

          --disable-shard-validation
              Disable the shard verification. This is used conjoyintly with the
              `TEZOS_DISABLE_SHARD_VERIFICATION_I_KNOW_WHAT_I_AM_DOING`
              environment variable. To actually disable the shard verification
              this option must be used and the environment variable must be set.
              If only the environment variable is set, the DAL node will refuse
              to start. 

          -E URI, --endpoint=URI
              The endpoint (an URI) of the Tezos node that the DAL node should
              connect to. The default endpoint is 'http://localhost:8732'.

          --expected-pow=FLOAT
              The expected proof-of-work difficulty level for the peers'
              identity.

          --fetch-trusted-setup=true | false
              Should the DAL node fetch the trusted setup when it needs it. By
              default, it does so.

          --history-mode=VAL
              The duration for the shards to be kept in the node storage. Either
              a number, the string "full" or the string "auto". A number is
              interpreted as the number of blocks the shards should be kept; the
              string "full" means no shard deletion, the string "auto" means the
              default of the profile: 3 months for an operator, twice the
              attestation lag for an attester and other profiles.

          --ignore-l1-config-peers
              Ignore the boot(strap) peers provided by L1 config.

          --ignore-topics=PKH1,PKH2,...
              The producer Octez DAL node will not publish shards for the
              provided pkhs. This argument is for testing purposes only.

          --metrics-addr=ADDR[:PORT]
              The TCP address and optionally the port of the node's metrics
              server. The default address is 0.0.0.0. The default port is 11733.

          --net-addr=ADDR[:PORT]
              The TCP address and optionally the port bound by the DAL node. If
              --public-addr is not provided, this is also the address and port
              at which this instance can be reached by other P2P nodes. The
              default address is 0.0.0.0. The default port is 11732.

          --observer-profiles=INDEX1,INDEX2,..., --observer=INDEX1,INDEX2,...
              The Octez DAL node observer profiles for given slot indexes.

          --peers=ADDR:PORT,...
              An additional list of peers (bootstrap or not) to connect to,
              expanding the one from the DAL node's configuration parameter
              'peers' and the one from the Octez node's configuration parameter
              'dal_config.bootstrap_peers'.

          --producer-profiles=INDEX1,INDEX2,..., --producer=INDEX1,INDEX2,..., --operator-profiles=INDEX1,INDEX2,..., --operator=INDEX1,INDEX2,...
              The Octez DAL node operator profiles for given slot indexes. These
              were previously known as producer profiles, however this name now
              refers to both operator and observer profiles.

          --public-addr=ADDR[:PORT]
              The TCP address and optionally the port at which this instance can
              be reached by other P2P nodes. By default, the point is
              '127.0.0.1:11732'. You can override the port using the syntax
              ':2222'. If the IP address is detected as a special address (such
              as a localhost one) it won't be advertised, only the port will.

          --rpc-addr=ADDR[:PORT]
              The TCP address and optionally the port at which the RPC server of
              this instance can be reached. The default address is 0.0.0.0. The
              default port is 10732.

          --service-name=VAL (absent OTEL_SERVICE_NAME env)
              A name that can be used to identify this node. This name can
              appear in observability data such as traces.

          --service-namespace=VAL (absent OTEL_SERVICE_NAMESPACE env)
              A namespace associated with the node. This namespace can appear in
              observability data such as traces.

          --slots-backup-uri=URI
              List of base URIs to fetch missing DAL slots if they are
              unavailable locally or cannot be reconstructed from shards.
              Supported URI schemes include 'http://', 'https://', and
              'file://'. The option accepts a list of fallback sources separated
              with commas.

          --trust-slots-backup-uris
              If set, skip cryptographic verification of slots downloaded from
              the backup URIs provided via --slots-backup-uri. This can speed up
              slot retrieval when replaying history or for debugging purposes.
              Use with caution during normal operation or when data integrity is
              critical, unless the backup source is fully trusted.

          --verbose
              Controls the verbosity of some emitted events. Default value is
              false.


See the DAL node manual for more details.

The concrete operational steps for participating in the DAL network
are described in page Running a DAL attester node.


DAL configuration of the L1 node
--------------------------------

All operator and observer DAL nodes should use the same initialization
parameters of the cryptographic primitives used by the DAL, see
section Install DAL trusted setup (**NB:** just that section, not the
rest of the page such as compiling sources, etc.).

Also, in order for the nodes to be able to join the P2P network, a set
of bootstrap nodes can be provided using the
"network.dal_config.bootstrap_peers" configuration parameter of the L1
node (thus using the same mechanism as for L1 nodes, see Connecting to
a Network and P2P parameters).


System requirements
-------------------

The system requirements for the DAL node depend on whether the node is
being run by a baker for the purpose of attesting DAL data or by a
slot producer for the purpose of publishing data to the DAL. The
following sections provide system requirements for these cases based
on experimentation. For more information about the experimentation,
see Hardware and bandwidth requirements for the Tezos DAL.

Note:

  These requirements are for DAL nodes that run independently of any
  other Octez binary. The requirements are higher if they are running
  on the same system as other Octez binaries.


DAL attesters
~~~~~~~~~~~~~

The amount of data that a DAL node must attest to depends on how much
baking power the associated baker has. The larger the baking power,
the more data the DAL sends to the DAL node to attest and the more
system resources the DAL node needs.

This table shows the system requirements for a DAL node (independent
of any other Octez binary) that attests 100% of the data assigned to
it. The specifications in this table are an estimate based on
experimentation with Google Cloud Platform compute instances; you must
monitor your DAL node to ensure that it attests all or nearly all of
the data that it is assigned to attest.

+-------------------------+-------------------------+-------------------------+-------------------------+-------------------------+
| Baking power            | 0.5% of total           | 1% of total             | 2% of total             | 5% of total             |
|=========================|=========================|=========================|=========================|=========================|
| Machine type            | e2-small (ssd)          | e2-small (ssd)          | e2-small (ssd)          | e2-medium (ssd)         |
+-------------------------+-------------------------+-------------------------+-------------------------+-------------------------+
| CPU clock               | 2.25 GHz                | 2.25 GHz                | 2.25 GHz                | 2.25 GHz                |
+-------------------------+-------------------------+-------------------------+-------------------------+-------------------------+
| RAM                     | 2 GiB                   | 2 GiB                   | 2 GiB                   | 4 GiB                   |
+-------------------------+-------------------------+-------------------------+-------------------------+-------------------------+
| Disk space              | 20 GiB                  | 20 GiB                  | 20 GiB                  | 20 GiB                  |
+-------------------------+-------------------------+-------------------------+-------------------------+-------------------------+
| Bandwidth (upload)      | 250 KiB/s               | 250 KiB/s               | 250 KiB/s               | 250 KiB/s               |
+-------------------------+-------------------------+-------------------------+-------------------------+-------------------------+
| Bandwidth (download)    | 250 KiB/s               | 350 KiB/s               | 400 KiB/s               | 600 KiB/s               |
+-------------------------+-------------------------+-------------------------+-------------------------+-------------------------+


DAL producer node
~~~~~~~~~~~~~~~~~

The system resources that a DAL producer node needs depends on how
much data it needs to publish to the DAL. This example is an estimate
based on experimentation with a DAL producer node that publishes data
into one slot in each block:

+-------------------------+-----------------+
| CPU                     | n2-standard-2   |
+-------------------------+-----------------+
| RAM                     | 4 GiB           |
+-------------------------+-----------------+
| Disk space              | 500 GiB         |
+-------------------------+-----------------+
| Bandwidth (upload)      | 2.5 MiB/s       |
+-------------------------+-----------------+
| Bandwidth (download)    | 0.5 MiB/s       |
+-------------------------+-----------------+


Monitoring
----------

As for L1 nodes, to monitor the DAL node, one can inspect the node’s
logs, query the node through RPCs, or use the incorporated Octez
Metrics server. The usage of metrics is analogous to that for the L1
nodes, and we therefore refer to the L1 node’s documentation for
details.


The bootstrap mechanism
-----------------------

The gossipsub algorithm establishes upper bounds on the size of a
topic mesh. Whenever a peer B would like to join the mesh of a peer A,
but the peer A cannot add B to its mesh due to the upper bound, peer A
responds with a random list of peers it knows are subscribed to the
same topic. Peer B will then try to connect to these advertised peers.
This peer exchange mechanism is used to bootstrap the P2P network: for
bootstrap DAL nodes, the upper bound is set to 0, and thus, while they
do not participate in exchanging shards, they keep advertising known
peers so that new peers can connect to them and thus enter the P2P
network.

If a node is behind a NAT, it is important to make the distinction
between its "public address" and its "network address". It is the
public address that is advertised, and this should be the node’s
external address, while the network address should be the node’s
internal address. In other words, the public address behaves as the "
--advertised-net-port" option of the L1 node, see Listening ports.


Lifetime of slots and shards
----------------------------

The life cycle of slots and shards is described by the following
steps:

1. The operator posts the slot data to some DAL node of its choice.
   The node computes the corresponding commitment and adds the
   association commitment - slot to the store. This is done via the
   RPC "POST /commitments/<slot_data>", which returns the
   corresponding commitment.

2. The operator instructs the DAL node to compute and save the shards
   of the slot associated with the given commitment. It is important
   to set the query flag "with_proof" to true to be able to publish
   the shards on the P2P network. This is done via the RPC "PUT
   /commitments/<commitment>/shards".

3. The operator instructs the DAL node to compute the proof associated
   with the commitment. This is done via the RPC "GET
   /commitments/<commitment>/proof", which returns the corresponding
   commitment proof.

4. The operator selects a slot index for its slot, and posts the
   commitment to L1, via the "publish_commitment" operation. This can
   be done via RPCs for injecting an operation into L1, or using the
   Octez client, via the following command:

      octez-client publish dal commitment <commitment> from <pkh> for slot <slot_index> with proof <proof>

5. Once the operation is included in a final block (that is, there are
   at least two blocks on top of the one including the operation), and
   the slot is considered published (see DAL overview), all DAL nodes
   exchange the slot’s shards they have in their store on the P2P
   network, depending on their profile (see The DAL's P2P protocol),
   and they store previously unknown shards.

6. Attesters check, for all published slots, the availability of the
   shards they are assigned by interrogating their DAL node, via the
   RPC "GET /profiles/<pkh>/attested_levels/<level>/attestable_slots",
   where level is the level at which the slot was published plus
   "attestation_lag", and "pkh" is the attester’s public key hash.
   (See also Bakers & the DAL)

7. Attesters attach a DAL payload containing the information received
   at step 6 to their attestation operation, via their baker binary.
   (See also Bakers & the DAL)

8. The protocol aggregates the received attestations, and declares
   each published slot as available or unavailable, depending on
   whether some threshold is reached, via the blocks metadata.

9. Rollups and other users can request stored pages or shards for an
   attested slot from any DAL node via the RPCs "GET
   /slot/pages/<commitment>" or "GET /shards/<commitment>"
   respectively. Only nodes that store enough shards to reconstruct
   the slot can provide the requested pages.

Step 2 can be done in parallel with steps 3-4, but before step 5.

Running a DAL attester node
***************************

To attest the data published on the DAL, you need an active baker and
node. While you can run a DAL node without an active baker, when you
do so, your account is not assigned DAL attestation rights and
therefore your node can distribute data among other DAL nodes but
cannot attest that it is available. Adding a DAL node to a baker's
setup improves the Tezos ecosystem because the DAL greatly expands the
amount of data that Tezos can distribute without causing congestion on
layer 1.

Therefore, we assume here you have installed Octez, see Installing
Octez. Make also sure that you have an instance of the "octez-node"
binary running, as explained in Setting up the node.


Connecting to other nodes
=========================

The DAL node needs the address of at least one already running DAL
node to connect to.

If the "octez-node" binary was started with the "--network <network>"
argument, the DAL node can get the addresses of bootstrap nodes on the
specified network from the node's configuration via the
"/config/network/dal" RPC endpoint. These bootstrap nodes do not
propagate DAL data, but instead provide entry points for other DAL
nodes to become part of the network.

If the "octez-node" binary was not started with the "--network
<network>" argument, the bootstrap nodes are not running, or you are
setting up a closed network, you may need to run your own DAL node in
the "bootstrap" profile or manually connect the new DAL node to an
existing DAL node via the "--peers" argument before you can run a DAL
node in the "attester" profile with an active baker.


Running the DAL node
====================

Before anything else, make sure your machine satisfies the hardware
requirements at The DAL node.

Follow these steps to run a DAL node along with a layer 1 node and a
baker.

1. **If** you plan to run the DAL node in **operator or observer**
   profiles, install the DAL trusted setup as described in section
   Install DAL trusted setup (**NB:** just that section, not the rest
   of the page such as compiling sources, etc.).

2. Initialize the DAL node by running its "config init" command,
   passing the address of a local "octez-node" instance and your
   attester's address. For example, this command initializes the DAL
   node with the address of a local "octez-node" instance on port 8732
   and stores data in the default DAL node directory ("~/.tezos-dal-
   node"):

      octez-dal-node config init --endpoint http://127.0.0.1:8732 --attester-profiles="$MY_ADDRESS"

   where "$MY_ADDRESS" is the baking key address (not the alias
   registered with "octez-client").

   You can specify a custom directory to store the DAL node data in by
   using the "--data-dir" argument:

      octez-dal-node config init --endpoint http://127.0.0.1:8732 --attester-profiles="$MY_ADDRESS" --data-dir my-attester-tezos-dal-node

   You can specify parameters such as the RPC node in the "config
   init" command or in the "run" command. These commands have the same
   parameters. For information about them, run "octez-dal-node config
   init --help" or see DAL node commands.

   At minimum, you must specify with the "--endpoint" parameter the
   URL of an RPC node that the DAL node can use.

   Check the section above on connecting to other nodes to make sure
   that your new DAL node can connect to other peers on the DAL P2P
   network. If you need to explicitly connect your node to running DAL
   nodes, pass the "--peers" argument with a comma-separated list of
   DAL node host names and ports. The default P2P port is 11732, so
   the argument might look like this: "--
   peers=host1.example.com:11732,host2.example.com:11732".

3. Ensure that the P2P port that the DAL node runs on is accessible
   from outside its local network.

   DAL nodes must be able to initiate connections outside their local
   network and accept connections from outside their local network. By
   default, the DAL node accepts P2P connections on port 11732, but
   you can change the address and port that the node listens on by
   setting the "--net-addr" argument. In simple setups with a single
   DAL node, routing configuration is usually not necessary.

   Depending on your network, you may need to adapt your firewall
   rules or set up network address translation (NAT) to direct
   external traffic to the DAL node. For example, if you are running
   more than one DAL node, each node needs a unique socket
   (combination of IP address and port) to listen on. In this case,
   use the "--net-addr" argument to set a different IP address and
   port for one of the nodes to listen on, as in "--net-addr
   0.0.0.0:11733" to make the node listen on port 11733 on all network
   interfaces. Then you must configure your firewall to permit
   incoming and outgoing TCP connections on port 11733.

   If a firewall rule directs traffic from an external port that is
   different from the port that you set in the "--net-addr" argument,
   use the "--public-addr" argument to set the port from which the
   node can be reached by other nodes. You may also need to set "--
   public-addr" if you are directing traffic from a load balancer to
   the DAL node.

4. Start the DAL node by running its "run" command, passing the
   directory that you set in the "config init" command if you changed
   the default. You can also pass any other parameters that you did
   not set in that command:

      octez-dal-node run --data-dir .tezos-dal-node

   Leave the DAL node process running.

5. In a new terminal window, start or restart a baking daemon as
   usual, but tell it to connect to the DAL node by passing the "--
   dal-node" argument with the host name and RPC port of the DAL node.
   The DAL node accepts RPC calls on port 10732 by default, so the
   command might look like this example, where "<PROTO_HASH>" is the
   short hash of the current protocol of the network:

      octez-baker-<PROTO_HASH> run with local node "$HOME/.tezos-node" bob --liquidity-baking-toggle-vote pass --dal-node http://127.0.0.1:10732

   The baker daemon connects to the DAL node and attests to the
   availability of DAL data as well as its usual layer 1 baking
   function.

6. In a new terminal window, verify that your baking daemon has
   attestation rights allocated for the current cycle, by running:

      octez-client rpc get "/chains/main/blocks/head" | jq '.metadata.level_info.cycle'
      octez-client rpc get "/chains/main/blocks/head/helpers/attestation_rights?delegate=$MY_ADDRESS&cycle=<current-cycle>"

   Be aware that the last command may take several minutes to execute
   if it returns a long list of rights. In turn, if the previous
   command reports no attestation rights ("[]"), you may have to
   register as a delegate or re-activate your delegate and wait for a
   few cycles to get some rights (see Register and check your rights).

7. Verify that the DAL node is running properly:

   1. Verify that the node is connected to other DAL nodes by running
      this command:

         curl http://localhost:10732/p2p/points/info?connected

      This command should show a list of other DAL nodes that the node
      is connected to, as in this example:

         [
           {
             "point": "46.137.127.32:11732",
             "info": {
               "trusted": true,
               "state": {
                 "event_kind": "running",
                 "p2p_peer_id": "idrpUzezw7VJ4NU6phQYuxh88RiU1t"
               },
               "p2p_peer_id": "idrpUzezw7VJ4NU6phQYuxh88RiU1t",
               "last_established_connection": [
                 "idrpUzezw7VJ4NU6phQYuxh88RiU1t",
                 "2024-10-24T15:02:31.549-00:00"
               ],
               "last_seen": [
                 "idrpUzezw7VJ4NU6phQYuxh88RiU1t",
                 "2024-10-24T15:02:31.549-00:00"
               ]
             }
           },
           {
             "point": "52.31.26.230:11732",
             "info": {
               "trusted": true,
               "state": {
                 "event_kind": "running",
                 "p2p_peer_id": "idqrcQybXbKwWk42bn1XjeZ33xgduC"
               },
               "p2p_peer_id": "idqrcQybXbKwWk42bn1XjeZ33xgduC",
               "last_established_connection": [
                 "idqrcQybXbKwWk42bn1XjeZ33xgduC",
                 "2024-10-24T15:02:31.666-00:00"
               ],
               "last_seen": [
                 "idqrcQybXbKwWk42bn1XjeZ33xgduC",
                 "2024-10-24T15:02:31.666-00:00"
               ]
             }
           }
         ]

   2. Verify that the node is connected to topics by running this
      command:

         curl http://localhost:10732/p2p/gossipsub/topics

      This command should return topics in the form
      "{"slot_index":<index>,"pkh":"<ADDRESS OF BAKER>"}" to represent
      the topics that the node is subscribed to on the DAL peer-to-
      peer network.

   3. Verify that your baker is assigned to attest DAL shards by
      running this command and using the address of the baker:

         octez-client rpc get /chains/main/blocks/head/context/dal/shards?delegates=$MY_ADDRESS

      The response should show the address and the indexes of multiple
      shards that the baker is assigned to attest.

      For more information about topics and shards, see DAL overview.

8. Verify that the node is connected to the network by going to the
   Explorus block explorer, selecting the network, going to the
   Consensus Ops tab, and looking up your baker.

   The **DAL attested/total published slots** column shows information
   about bakers' attestations.

   If the column for your baker shows a symbol that looks like a
   missing image or empty page, then either the baker did not have
   shards assign to it at the corresponding level or the DAL node is
   not connected to the network. If you have sufficient stake and you
   always see the missing image symbol, check the steps above and make
   sure that the DAL node is running correctly.

   If the baker is connected to the network, the column shows the
   ratio of slots attested by the baker to published slots.

   * If the numbers are equal, as in "2/2", the baker is attesting all
     published slots (the DAL node has seen all shards that are
     assigned to it).

   * If the first number is always 0, as in "0/2", the DAL node and
     baker may not be configured correctly. See the troubleshooting
     section below.

   * If the baker attests all published slots sometimes and other
     times attests zero or fewer slots, the DAL node may be running
     too slowly or failing to fetch the data in time. Verify that the
     node hardware is sufficient, using these Hardware and bandwidth
     requirements for the Tezos DAL.

Now the DAL node is running and subscribed to the relevant topics.


Troubleshooting
===============


Troubleshooting connections
---------------------------

Follow these steps if the DAL node is running but not connected to the
network:

1. Upgrade your installation of Octez to the latest version. The Octez
   DAL node version 20.3 has some issues that are resolved in later
   versions.

2. Check if the "config.json" file of the "octez-node" daemon that the
   DAL node is connected to has a field named "network". If there is a
   "network" field, update the node's configuration by running this
   command:

      octez-node config update --network <network>

   Use "mainnet", "ghostnet", or "sandbox" as the value for the "--
   network" argument.

3. Verify that the node is connected to a bootstrap peer by running
   this command with the address and RPC port of your DAL node:

      octez-client --endpoint http://127.0.0.1:10732 rpc get /p2p/gossipsub/connections | jq ".[].connection.bootstrap"

   At least one entry in the output should show "true" to indicate
   that the peer is a bootstrap node. If not, run the command a few
   more times over a one-minute interval. If you still see no entries
   that say "true", restart the DAL node.

4. Verify that the baker can get information about the slots that are
   available to attest by running this command, where "$MY_ADDRESS" is
   your baker's account address and "<level>" is a recent level:

      curl -v "http://127.0.0.1:10732/profiles/$MY_ADDRESS/attested_levels/<level>/attestable_slots"

   The baker uses this request to get a list of the attestable slots
   from the DAL node, that is, the slots for which the node has all
   the shards assigned to the baker. If the response is an HTTP error,
   the error code may help you determine why the baker cannot get the
   necessary information from the DAL node.

   For example, the following response shows that the DAL node is
   providing information about which slots are attestable, even though
   none of the slots at this particular level are attestable.

      {"kind":"attestable_slots_set","attestable_slots_set":[false,false,false,false,false,false,false,false,false,false,true,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false,false],"published_level":9290818}

   If the baker is not currently assigned any shards, the response may
   be "{"kind":"not_in_committee"}", which also means that the baker
   can get information from the DAL node.

5. If the problem persists, contact Octez developers on the tezos-dev
   Slack or the Tezos Discord.


Troubleshooting firewall/NAT issues
-----------------------------------

By default, the P2P port for a DAL node is 11732, but you can change
it with the "--net-addr" argument.

If you want to use an external port different from the one specified
in "--net-addr", use the "--public-addr" argument. Currently, there is
a limitation requiring you to know your public IP address to do this,
though we plan to improve this in the future.

For both producers and bakers, it is essential to maintain good
connectivity by ensuring that your node can receive connections:

* If you're behind a NAT, you must implement a forwarding rule.

* If you're behind a firewall, you must configure it to allow both
  incoming and outgoing connections on the P2P port.

* If you're not using the "--public-addr" argument, ensure that the
  NAT forwarding rule uses the same external and internal ports.

* If you are running two DAL nodes on a network that uses NAT, set up
  the NAT rules to forward the P2P ports of the DAL nodes to different
  public ports.


DAL and the Ledger
------------------

If you are using a Ledger device, make sure it runs a version of the
Tezos baking app that supports the DAL, such as version 2.5.0 or
later.

Bakers & the DAL
****************

This page documents some aspects that are most useful to know for
bakers when participating to the DAL network (baking the DAL is a
serious matter!). We assume the reader is familiar with the DAL basics
and terminology introduced in DAL overview and DAL integration.

Bakers play a crucial role in Tezos through the Tenderbake protocol,
by validating transactions and securing the network. This
participation involves producing new blocks when assigned the rights
by the protocol, attesting proposed valid blocks to make the chain
progress, and (indirectly) slashing misbehavior that would put the
chain at risk. Due to their strategic role in Tezos, entrusting them
with verifying and attesting data availability using the DAL is
natural.

From a baker's point of view, DAL attestation is carried out in three
main steps.

First, the baker is attributed DAL attestation rights (automatically,
based on its consensus attestation rights). Similar to Tenderbake's
baking and attestation rights, the L1 protocol assigns a range of DAL
shards to bakers. For a given level, the list of shards attributed to
a baker is the same for every DAL slot.

Second, Tezos bakers should launch DAL nodes and join the DAL P2P
network subscribing to the appropriate topics. The topics of interest
for a baker are those of the form "(tz1TheBakerSPublicKeyHash,
slot_index)" where the first component is the baker's public key hash
and the slot index ranges from "0" to "number of DAL slots - 1". Note
that the baker binary automatically subscribes to those topics when
launched with "--dal-node <dal-node-RPC-endpoint>". Once launched and
connected to the baker's binary, the DAL node is responsible for
downloading the shards assigned to the corresponding public key
hash(es) for every slot via the DAL P2P network, assuming the slot
commitment operation has been accepted by L1 and the slot's shards are
published on the DAL network (see steps 3 and 4 in DAL overview).

Finally, bakers attach a DAL payload (via the optional field
"dal_content") to their consensus attestation operations. This field
informs the protocol, for each slot, whether they managed to download
all assigned shards or not. Concretely, this DAL payload consists of a
bitset, where the bit at position "n" is set to "1" if and only if the
DAL node connected to the baker managed to download all the shards
assigned to that baker for the slot number "n".

Based on the DAL payloads of all the attestations included in a block,
the protocol decides which slots are available and announces them in
that block's metadata.

It's important for bakers to note the following:

* No DAL payload is attached if there is no slot to attest (i.e. if
  the bitset is zero).

* If a slot commitment is included in an L1 block at some level "PL"
  (a.k.a "published_level"), the attestation operations (via the
  optional DAL payload which attests the baker's shards) are injected
  on top of the block(s) at level "PL + attestation_lag - 1" and
  target inclusion at level "PL + attestation_lag".

* Theoretically, if a baker is assigned more than "number_of_shards /
  redundancy_factor" shards, it could declare the shards as available
  if it succeeds in downloading "number_of_shards / redundancy_factor"
  shards at least. The missing shards could be calculated by
  reconstructing the whole slot (but this is not implemented yet).

* As the number of shards is quite small compared to the number of
  Tenderbake slots, DAL and Tenderbake committees might not always
  coincide in practice: there might be bakers in the Tenderbake's
  committee that are not part of the DAL's for some levels. However,
  any baker in the DAL committee is part of the Tenderbake committee.

* Like Tenderbake attestations, bakers receive a number of shards to
  attest proportional to their staking balance, directly impacting the
  required bandwidth to download shards' content at each level. A
  baker with 1% of the stake will need a bandwidth between
  0.5-1.0MiB/s. This is why we plan to give the opportunity to bakers
  to run multiple DAL nodes so that this bandwidth can be spread
  between those DAL nodes.

* With the DAL, it becomes easier to associate the baker identity with
  an IP address. Besides, the more stake a baker has, the easier it
  is. Fortunately, bakers could run their DAL nodes on distinct
  machines to mitigate possible DDoS attacks on their bakery
  infrastructure.

* If bakers use a tz4 consensus key (or don't have a consensus key and
  use a tz4 delegate key), they need to register and use a companion
  key in order to include their DAL payload in their attestations.

Smart rollup node
*****************

This page describes the Octez rollup node, the main executable
supporting Smart Optimistic Rollups.

The Octez rollup node is used by a rollup operator to deploy a rollup.
The rollup node is responsible for making the rollup progress by
publishing commitments and by playing refutation games.

A complete manual page of the rollup node is available here.

Just like the Octez node, the Octez rollup node provides an RPC
interface. The services of this interface can be called directly with
HTTP requests.

We first cover the operation of the rollup node and the corresponding
workflow, using some predefined rollup logic (called kernel), and then
we explain how the logic of a rollup can be defined by developing a
custom rollup kernel.


Prerequisites
=============

To experiment with the commands described in this section, we use the
Weeklynet. In this section, we assume that "${OPERATOR_ADDR}" is a
valid user account on Weeklynet owned by the reader.

Notice that you need a specific development version of Octez to
participate to Weeklynet. This version is either available from docker
images or can be compiled from sources. Please refer to the Weeklynet
website for installation details.

An Octez rollup node needs an Octez node to run. It is recommended
that the two nodes run on the same machine. If this is the case, there
is no additional configuration required of the Octez node. If they are
on different network interfaces, the Octez node needs to allow the
rollup node to make specific RPCs. To achieve this, one can add the
following to the Octez node configuration file, where
"<ip.address:port>" is the address at which the rollup node can
contact the Octez node.

   {
      "rpc": {
        "acl": [
          {
            "address": "<ip.address:port>",
            "blacklist": []
          }
        ]
      }
   }

Warning:

  Configuring a public facing Octez node this way exposes it to DoS
  attacks. However one can allow all RPCs on the Octez node to be
  accessed locally while still keeping sane defaults for outside
  accesses by specifying an additional RPC server with, *e.g.*, "--
  rpc-addr 127.0.0.1 --rpc-addr 0.0.0.0".

We assume that an Octez node has been launched locally, typically by
issuing:

   octez-node config init --data-dir "${ONODE_DIR}" --network "${NETWORK}"
   octez-node run --data-dir "${ONODE_DIR}" --network "${NETWORK}" --rpc-addr 127.0.0.1

in a terminal where "${NETWORK}" is of the form "https://teztnets.com
/weeklynet-YYYY-MM-DD" and "${ONODE_DIR}" is a path for the Octez node
store, by default "~/.tezos-node".

The commands will only work when the node is completely bootstrapped,
and therefore the current protocol on the target network is activated.
This can be checked by:

   octez-client bootstrapped
   octez-client rpc get /chains/main/blocks/head/protocols

In case you do not already have a user account, you can generate one
with:

   octez-client gen keys "${ACCOUNT_NAME}"
   octez-client show address "${ACCOUNT_NAME}"

Then, the "${OPERATOR_ADDR}" can be set to the hash value ("tz1...")
returned.

Finally, you need to check that your balance is greater than 10,000
tez to make sure that staking is possible. In case your balance is not
sufficient, you can get test tokens for the "tz1" address from a
faucet, after your node gets synchronized with Weeklynet.

   octez-client get balance for "${OPERATOR_ADDR}"


Origination
===========

Anyone can originate a smart rollup with the following invocation of
the Octez client:

   octez-client originate smart rollup "${SR_ALIAS}" \
     from "${OPERATOR_ADDR}" \
     of kind wasm_2_0_0 \
     of type bytes \
     with kernel "${KERNEL}" \
     --burn-cap 999

where "${SR_ALIAS}" is an alias to memorize the smart rollup address
in the client. This alias can be used in any command where a smart
rollup address is expected. "${KERNEL}" is a hex representation of a
WebAssembly bytecode serving as an initial program to boot on. From a
WASM bytecode file named "kernel.wasm", such representation can be
obtained through

   xxd -ps -c 0 <kernel.wasm> | tr -d '\n'

To experiment, we propose that you use the value "${KERNEL}" defined
in the "given file".

   source sr_boot_kernel.sh # defines shell variable KERNEL

If everything went well, the origination command results in:

   This sequence of operations was run:
     Manager signed operations:
       From: tz1fp5ncDmqYwYC568fREYz9iwQTgGQuKZqX
       Fee to the baker: ꜩ0.000357
       Expected counter: 36
       Gas limit: 1000
       Storage limit: 0 bytes
       Balance updates:
         tz1fp5ncDmqYwYC568fREYz9iwQTgGQuKZqX ... -ꜩ0.000357
         payload fees(the block proposer) ....... +ꜩ0.000357
       Revelation of manager public key:
         Contract: tz1fp5ncDmqYwYC568fREYz9iwQTgGQuKZqX
         Key: edpkukxtw4fHmffj4wtZohVKwNwUZvYm6HMog5QMe9EyYK3QwRwBjp
         This revelation was successfully applied
         Consumed gas: 1000
     Manager signed operations:
       From: tz1fp5ncDmqYwYC568fREYz9iwQTgGQuKZqX
       Fee to the baker: ꜩ0.000956
       Expected counter: 37
       Gas limit: 2849
       Storage limit: 6572 bytes
       Balance updates:
         tz1fp5ncDmqYwYC568fREYz9iwQTgGQuKZqX ... -ꜩ0.000956
         payload fees(the block proposer) ....... +ꜩ0.000956
       Smart rollup origination:
         Kind: wasm_2_0_0
         Parameter type: bytes
         Kernel Blake2B hash: '24df9e3c520dd9a9c49b447766e8a604d31138c1aacb4a67532499c6a8b348cc'
         This smart rollup origination was successfully applied
         Consumed gas: 2748.269
         Storage size: 6552 bytes
         Address: sr1RYurGZtN8KNSpkMcCt9CgWeUaNkzsAfXf
         Genesis commitment hash: src13wCGc2nMVfN7rD1rgeG3g1q7oXYX2m5MJY5ZRooVhLt7JwKXwX
         Balance updates:
           tz1fp5ncDmqYwYC568fREYz9iwQTgGQuKZqX ... -ꜩ1.638
           storage fees ........................... +ꜩ1.638

The address "sr1RYurGZtN8KNSpkMcCt9CgWeUaNkzsAfXf" is the smart rollup
address. Let's write it "${SR_ADDR}" from now on.


Deploying a rollup node
=======================

Now that the rollup is originated, anyone can make it progress by
deploying a rollup node.

First, we need to decide on a mode the rollup node will run:

1. "operator" activates a full-fledged rollup node. This means that
   the rollup node will do everything needed to make the rollup
   progress. This includes following the Layer 1 chain, reconstructing
   inboxes, updating the states, publishing and cementing commitments
   regularly, and playing the refutation games. In this mode, the
   rollup node will accept transactions in its queue and batch them on
   the Layer 1.

2. "batcher" means that the rollup node will accept transactions in
   its queue and batch them on the Layer 1. In this mode, the rollup
   node follows the Layer 1 chain, but it does not update its state
   and does not reconstruct inboxes. Consequently, it does not publish
   commitments nor play refutation games.

3. "observer" means that the rollup node follows the Layer 1 chain to
   reconstruct inboxes, to update its state. However, it will neither
   publish commitments, nor play a refutation game. It does not
   include the message batching service either.

4. "maintenance" is the same as the operator mode except that it does
   not include the message batching service.

5. "accuser" follows the layer1-chain and computes commitments but
   does not publish them. Only when a conflicting commitment
   (published by another committer) is detected will the "accuser
   node" publish a commitment and participate in the subsequent
   refutation game.

6. "bailout" mode is designed to assist committers in recovering their
   bonds. It functions as a slightly modified version of "Accuser",
   differing in that it does not post any new commitments but instead
   focuses on defending the ones that have been previously submitted.
   When the operator is no longer staking on any commitment, the node
   runs the recover bond operation. If the node detects that this
   operation has been successful, it exits.

7. "custom" mode refers to a mode where the users individually select
   which kinds of operations the rollup node injects. It provides
   tailored control and flexibility customized to specific
   requirements, and is mostly used for tests.

To each mode corresponds a set of purposes where each purpose is a set
of L1 operations which are injected by the rollup node.

The following table links each purpose to its corresponding L1
operations.

+---------------------+-------------------------------------------------------------------+
| Operating           | smart_rollup_publish, smart_rollup_refute, smart_rollup_timeout   |
+---------------------+-------------------------------------------------------------------+
| Batching            | smart_rollup_add_messages                                         |
+---------------------+-------------------------------------------------------------------+
| Cementing           | smart_rollup_cement                                               |
+---------------------+-------------------------------------------------------------------+
| Recovering          | smart_rollup_recover                                              |
+---------------------+-------------------------------------------------------------------+
| Executing_outbox    | smart_rollup_execute_outbox_message                               |
+---------------------+-------------------------------------------------------------------+

The table below summarises the modes and their associated purposes:

+---------------+--------------+------------+--------------+--------------+--------------------+
|               | Operating    | Batching   | Cementing    | Recovering   | Executing_outbox   |
|===============|==============|============|==============|==============|====================|
| Operator      | Yes          | Yes        | Yes          | No           | Yes [1]            |
+---------------+--------------+------------+--------------+--------------+--------------------+
| Maintenance   | Yes          | No         | Yes          | No           | Yes [1]            |
+---------------+--------------+------------+--------------+--------------+--------------------+
| Bailout       | Yes [2]      | No         | Yes          | Yes          | No                 |
+---------------+--------------+------------+--------------+--------------+--------------------+
| Accuser       | Yes [3]      | No         | No           | No           | No                 |
+---------------+--------------+------------+--------------+--------------+--------------------+
| Batcher       | No           | Yes        | No           | No           | No                 |
+---------------+--------------+------------+--------------+--------------+--------------------+
| Observer      | No           | No         | No           | No           | No                 |
+---------------+--------------+------------+--------------+--------------+--------------------+

[1] If and only it's a private rollup. In that case, only the
    whitelist update outbox message are injected.

[2] A rollup node in bailout mode won't publish any new commitments
    but only defends the one published by the operator if they are
    refuted.

[3] An accuser node will publish commitments only when it detects
    conflicts; for such cases it must make a deposit of 10,000 tez.

Then to run the rollup node, use the following command:

   octez-smart-rollup-node --base-dir "${OCLIENT_DIR}" \
                    run "${ROLLUP_NODE_MODE}" \
                    for "${SR_ALIAS_OR_ADDR}" \
                    with operators "${OPERATOR_ADDR}" \
                    --data-dir "${ROLLUP_NODE_DIR}"

where "${OCLIENT_DIR}" is the data directory of the Octez client, by
default "~/.tezos-client", and "${ROLLUP_NODE_DIR}" is the data
directory of the Octez smart rollup node, by default "~/.tezos-smart-
rollup-node".

The log should show that the rollup node follows the Layer 1 chain and
processes the inbox of each level.

Distinct Layer 1 signers can be used for each purpose of the mode by
either editing the configuration file or by listing multiple operators
on the command line.

For example for the "operator" mode we can replace "${OPERATOR_ADDR}"
by "default:${OPERATOR_ADDR1} batching:${OPERATOR_ADDR2}".  Where the
rollup node will use "${OPERATOR_ADDR2}" for the batching purpose and
"${OPERATOR_ADDR1}" for everything else.

The L1 chain has a limitation of one manager operation per key per
block (see One manager operation per manager per block). In the case
of a high throughput rollup, this limitation could slow down the
rollup by capping the number of L2 messages that the rollup node's
batcher purpose can inject per block to the maximum size of one L1
operation's maximal size (e.g., 32kb on mainnet).

To bypass that limitation and inject multiple
"smart_rollup_add_messages" L1 operations within a single L1 block, it
is possible to provide multiple keys for the batcher purpose of a
rollup node. At each block, the rollup node will use as many keys as
possible to inject a corresponding number of queued L2 messages into
the L1 rollup inbox. The order of the batches of L2 messages is not
guaranteed to be preserved by the rollup node nor by the octez node
mempool.

The way to provide multiple batcher keys on the command line is:

   octez-smart-rollup-node run ${ROLLUP_NODE_MODE} for "${SR_ALIAS_OR_ADDR}" \
                    with operators default:${DEFAULT_ADDR} \
                    batching:${BATCHER_ADDR1} \
                    batching:${BATCHER_ADDR2} ...


Configuration file
------------------

The rollup node can also be configured via one configuration file
stored in its own data directory, with the following command that uses
the same arguments as the "run" command:

   octez-smart-rollup-node --base-dir "${OCLIENT_DIR}" \
                    init "${ROLLUP_NODE_MODE}" config \
                    for "${SR_ALIAS_OR_ADDR}" \
                    with operators "${OPERATOR_ADDR}" \
                    --data-dir "${ROLLUP_NODE_DIR}"

where "${OCLIENT_DIR}" must be the directory of the client, containing
all the keys used by the rollup node, i.e. "${OPERATOR_ADDR}".

This creates a smart rollup node configuration file:

   Smart rollup node configuration written in ${ROLLUP_NODE_DIR}/config.json

Here is the content of the file:

   {
     "smart-rollup-address": "${SR_ALIAS_OR_ADDR}",
     "smart-rollup-node-operator":
     {
       "operating": "${OPERATOR_ADDR}",
       "batching": [ "${OPERATOR_ADDR}" ],
       "cementing": "${OPERATOR_ADDR}",
       "executing_outbox": "${OPERATOR_ADDR}"
     },
     "fee-parameters": {},
     "mode": "operator"
   }

The rollup node can now be run with just:

   octez-smart-rollup-node --base-dir "${OCLIENT_DIR}" run --data-dir ${ROLLUP_NODE_DIR}

The configuration will be read from "${ROLLUP_NODE_DIR}/config.json".


Monitoring a smart-rollup node
------------------------------

The smart-rollup node offers the same monitoring features as the Octez
node.

For the list of metrics provided by the rollup node see the following
page:

* Supported Open Metrics


Rollup node in a sandbox
------------------------

The node can also be tested locally with a sandbox environment. (See
sandbox documentation.)

Once you initialized the "sandboxed" client data with
"./src/bin_client/octez-init-sandboxed-client.sh", you can run a
sandboxed rollup node with "octez-smart-rollup-node run".

A temporary directory "/tmp/tezos-smart-rollup-node.xxxxxxxx" will be
used. However, a specific data directory can be set with the
environment variable "SCORU_DATA_DIR".


History modes
=============

The rollup node can be configured (1) to remove data on disk that is
not needed anymore for the correct operation of a rollup node (i.e. to
still be able to play all refutation games that could occur) or (2) to
keep the full history of the rollup and the L2 chain since the rollup
genesis.

The history mode can be set on the command line with "--history-mode
<mode>" or in the configuration file with:

   {
     "history-mode" : "<mode>"
   }


Full mode
---------

The *full* history mode makes the rollup node keep its history since
the last cemented commitment (LCC). Everything before the LCC (both
the context containing the PVM state and the rollup node store
containing the L2 chain) is automatically deleted periodically by a
*garbage collection* phase.


Archive mode
------------

When configured in *archive* mode, a rollup node will keep all history
since the origination of the rollup. This mode can be useful for
applications that require to regularly access historical data before
the LCC, i.e. for applications that need more than two weeks of
history.

This mode can be chosen e.g. on the command line with "--history-mode
archive".

Note that an archive node can be converted to a full node but not the
other way around. The conversion will happen automatically if the
history mode is changed in the configuration file or command line.

This is the default history mode.


Snapshots
=========

Smart rollup node snapshots are a way to bootstrap a rollup node
without having to replay the whole L2 chain since the rollup genesis.
Without this snapshot mechanism, one would need an archive L1 node to
bootstrap a rollup node or to catch up (if the rollup node data is
more than a few days old) because it needs access to metadata about L1
operations on the chain.

Snapshots for a particular rollup must be obtained from an off-chain
source (for instance a rollup snapshot provider service which
regularly publishes snapshots online) and imported into an existing,
or empty, rollup node to get started quickly.


Format of snapshots
-------------------

A smart rollup node snapshot is a binary file which contains a header
part and a data part. The data part is a tar archive of the non-local
storage files of the rollup node while the metadata header exposes
information to quickly validate or discard a snapshot.


Snapshot format
^^^^^^^^^^^^^^^

+---------------------------+---------------------------+----------------------------------------------------+
| Name                      | Size                      | Contents                                           |
|===========================|===========================|====================================================|
| Snapshot version          | 1 byte                    | "0" (the only version so far)                      |
+---------------------------+---------------------------+----------------------------------------------------+
| History mode              | 1 byte                    | "0" for archive, "1" for full                      |
+---------------------------+---------------------------+----------------------------------------------------+
| Address                   | 20 bytes                  | Address of the smart rollup                        |
+---------------------------+---------------------------+----------------------------------------------------+
| Head level                | 4 bytes                   | Level of the last seen L1 block of the rollup      |
|                           |                           | (int32)                                            |
+---------------------------+---------------------------+----------------------------------------------------+
| Last commitment hash      | 32 bytes                  | Hash of last commitment in the L2 chain            |
+---------------------------+---------------------------+----------------------------------------------------+
| Data                      | Variable                  | Tar archive of rollup node data                    |
+---------------------------+---------------------------+----------------------------------------------------+

The snapshots can be imported (and exported) as either compressed
(with gzip) or uncompressed files.


Importing a snapshot
--------------------

A snapshot "${SNAPSHOT_FILE}" can be imported by issuing the following
command:

   octez-smart-rollup-node -E ${L1_NODE_ENDPOINT} \
     snapshot import ${SNAPSHOT_FILE} \
     [--data-dir ${ROLLUP_NODE_DIR}] \
     [--force]

where "${ROLLUP_NODE_DIR}" is the data directory of the rollup node in
which we want to import the snapshot, and "${L1_NODE_ENDPOINT}" is the
RPC endpoint of an L1 node, needed to verify the snapshot.

Option "--force" allows to import a snapshot in an already populated
data directory of a rollup node.

Warning:

  When using the "--force" option, it is recommended to run the
  snapshot info command and to first import the snapshot in an empty
  directory to run all the checks.

While importing a snapshot, many checks are performed to ensure the
consistency of the imported data. In order to speed up the process,
but only if the snapshot's source is highly trusted (or exported by
yourself), it is possible to disable some checks. Some rudimentary
checks will still be performed. However, most of the data will be
copied directly, without additional consistency checks. To do so, use
the "--no-check" option.

Warning:

  The snapshot importing mechanism checks that the chain of
  commitments from the LCC (last cemented commitment) to the last
  commitment is published on the L1 chain but this does not prevent a
  malicious provider of snapshots from providing snapshots with
  inaccurate data about the L2 state, as soon as she or he is willing
  to also forfeit her/his deposit (these commitments were exposed on
  L1 to eventual refutation games). The check described above gives
  some acceptable level of assurance without having to recompute the
  whole chain from the LCC (which can be costly depending on the
  rollup).


List of checks performed for import
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* Metadata checks:

  * Rollup address matches (*)

  * History mode matches (*)

  * Snapshot head is fresher than the one on disk (*)

  * Last commitment is published on L1

* Metadata commitment matches the snapshot one

* LCC on L1 is a valid commitment in the snapshot

* Ensure the chain of commitments goes back to LCC

* For each L2 block:

  * The commitment, if any, must be for the PVM state of this block

  * Hashes are for the correct content (for state hash, commitment
    hash, inbox hash)

(*) Marks the rudimentary checks that are performed on import with
option "--no-checks".


Snapshot information
--------------------

When retrieving a snapshot, it can be useful to check its actual
content, such as:

* snapshot format

* history mode

* smart rollup address

* head level

* last commitment

This information is displayed by the following command:

   octez-smart-rollup-node snapshot info ${SNAPSHOT_FILE}

which will essentially decode and display the metadata header of the
snapshot file.


Exporting a snapshot
--------------------

Exporting a snapshot for a currently running rollup node will
temporarily stop its progression (during the time the data is
initially copied). The export creates a file with a chosen name
"${SNAPSHOT_FILE}" or one which is automatically generated of the form
"snapshot-<address>-<level>.<history_mode>" and is achieved by running
the following command (the rollup node does not need to be stopped):

   octez-smart-rollup-node snapshot export [${SNAPSHOT_FILE}] \
     [--data-dir ${ROLLUP_NODE_DIR}] \
     [--dest ${DEST_DIR}]

The export has three phases:

1. Initial export of the data (blocking)

2. Compression of snapshot (non-blocking)

3. Integrity check of snapshot (non-blocking)

The checks for the export are less thorough than the ones for an
import but ensure that the snapshot is not corrupted and can be
imported by other users.

Note:

  It is also possible to use the "--no-check" option to disable the
  integrity checks during the export (i.e., phase 3), which will speed
  up the process.

Note:

  Snapshots produced with the "--compact" option will be significantly
  smaller (by a factor of 3) than otherwise as they contain a single
  commit of the context for the first available block (instead of the
  full context history). They take a comparable amount of time to be
  exported but take longer to be imported because the context needs to
  be reconstructed.

Warning:

  Snapshots exported with "--compact" for *archive* rollup nodes will
  need a significant time to import because the context will need to
  be reconstructed from the rollup genesis.


Workflows
=========


Sending an external inbox message
---------------------------------

The Octez client can be used to send an external message into the
rollup inbox. Assuming that "${EMESSAGE}" is the hexadecimal
representation of the message payload, one can do:

   octez-client -d "${OCLIENT_DIR}" -p ${PROTO_HASH} \
    send smart rollup message "hex:[ \"${EMESSAGE}\" ]" \
    from "${OPERATOR_ADDR}"

to inject such an external message,  where "${PROTO_HASH}" is the hash
of your protocol (e.g. "ProtoALphaAL" for Alpha; see how to obtain
it). So let us focus now on producing viable content for
"${EMESSAGE}".

The kernel used previously in our running example is a simple "echo"
kernel that copies its input as a new message to its outbox.
Therefore, the input must be a valid binary encoding of an outbox
message to make this work. Specifically, assuming that we have
originated a Layer 1 smart contract as follows:

   octez-client -d "${OCLIENT_DIR}" -p ${PROTO_HASH} \
     originate contract go transferring 1 from "${OPERATOR_ADDR}" \
     running 'parameter string; storage string; code {CAR; NIL operation; PAIR};' \
     --init '""' --burn-cap 0.4

and that this contract is identified by an address "${CONTRACT}" (a
"KT1..." address), then one can encode an outbox transaction using the
"octez-codec" as follows:

   MESSAGE='{
       "transactions": [
         {
           "parameters": {"int": "37"},
           "destination": "KT1VD4SdQF2ruNNTCE1aTWErmGz9tN4Mg8F5",
           "entrypoint": "%default"
         }
       ],
       "kind": "untyped"
     }'

   EMESSAGE=$(octez-codec encode alpha.smart_rollup.outbox.message from "${MESSAGE}")


Triggering the execution of an outbox message
---------------------------------------------

Once an outbox message has been pushed to the outbox by the kernel at
some level "${L}", it becomes executable when the commitment that
includes this level is cemented. On Weeklynet, the cementation process
of a non-disputed commitment is 40 blocks long while on Mainnet, it is
2 weeks long.


Executing outbox messages automatically
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The rollup node can be configured to execute outbox messages
automatically as soon as they become executable (it will not do so by
default). For this, it is strongly recommended to declare a distinct
signing key for this purpose (see operators and purposes).

The configuration file must contain a key for the purpose
"executing_outbox" and potentially some "fee parameters" to control
what the rollup node accepts to pay as fees for injecting these
operations on L1.

   {
     "smart-rollup-node-operator": {
       "executing_outbox": "tz1fp5ncDmqYwYC568fREYz9iwQTgGQuKZqX"
     },
     "fee-parameters" : {
       "execute_outbox_message" : {
         "fee-cap" : "1000000",
         "burn-cap" : "1000000"
       }
     }
   }

In this snippet, the rollup node will accept to inject outbox message
execution operations which do not consume more than 1 tez in fees and
1 tez in storage burn (these are the default values). They can be
tweaked depending on the use case (e.g. increased if the outbox
messages call smart contracts on L1 which are known to be costly, or
decreased to limit abuse and better control funds allocated for this
purpose).

To decide which outbox messages the rollup node should execute, the
configuration file must contain a filter. An example is given below:

   {
      "execute-outbox-messages-filter": [
        {
          "transaction": {
            "destination" : ["KT1Wj8SUGmnEPFqyahHAcjcNQwe6YGhEXJb5"],
            "entrypoint" : ["mint", "burn"]
          }
        },
        {
          "transaction": {
            "destination" : [
              "tz1QQAPf4v6ApPx9AteMa3ZV2kTiBT6x16Ps",
              "tz2WtKth6KCGDT79gQkVovQiDFDa5wFPzmYq"
            ],
            "entrypoint" : "any"
          }
        },
        {
          "transaction": {
            "destination" : "any",
            "entrypoint" : ["default"]
          }
        }
      ]
   }

In this case, the rollup node will execute outbox messages which are
transactions that either:

1. Call the entry points "mint" or "burn" on the smart contract
   "KT1Wj8SUGmnEPFqyahHAcjcNQwe6YGhEXJb5".

2. Call any entry point on the destinations
   "tz1QQAPf4v6ApPx9AteMa3ZV2kTiBT6x16Ps" or
   "tz2WtKth6KCGDT79gQkVovQiDFDa5wFPzmYq"

3. Call the entry point "default" on any destination.

It is advised to define filters with both restricted sets of entry
points and destinations to prevent misuse of this feature. In some
cases (in test sandboxes, or if outbox messages are
restricted/controlled by the kernel), it may be desirable to allow the
rollup node to execute every outbox messages it sees.

The following entry in the configuration file makes the rollup node
execute all outbox messages for its rollup indiscriminately.

   {
      "execute-outbox-messages-filter": [
        {
          "transaction": { "destination" : "any", "entrypoint" : "any"}
        }
      ]
   }


Executing outbox messages manually
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If the rollup node is not configured to execute outbox messages
automatically, messages have to be executed manually by users (anyone
can execute an outbox message once it becomes executable).

When the commitment that includes level "${L}" is cemented, one can
observe that the outbox is populated as follows:

   curl -i "${ROLLUP_NODE_ENDPOINT}/global/block/cemented/outbox/${L}/messages"

where:

* ${ROLLUP_NODE_ENDPOINT} represents the address of the Rollup node
  server. It can be set to a specific server address such as
  "http://localhost:36149".

* ${L} denotes the block level for which one wants to retrieve
  information from the Rollup node.

Here is the output for this command:

   [ { "outbox_level": ${L}, "message_index": "0",
    "message":
      { "transactions":
          [ { "parameters": { "int": "37" },
              "destination": "${CONTRACT}",
              "entrypoint": "%default" } ] } } ]

At this point, the actual execution of a given outbox message can be
triggered. This requires precomputing a proof that this outbox message
is indeed in the outbox. In the case of our running example, this
proof is retrieved as follows:

   PROOF=$(curl -i "${ROLLUP_NODE_ENDPOINT}/global/block/head/helpers/\
              proofs/outbox/${L}/messages?index=0)"

Finally, the execution of the outbox message is done as follows:

   "${TEZOS_PATH}/octez-client" -d "${OCLIENT_DIR}" -p ${PROTO_HASH} \
           execute outbox message of smart rollup "${SR_ALIAS_OR_ADDR}" \
           from "${OPERATOR_ADDR}" for commitment hash "${LCC}" \
           and output proof "${PROOF}"

where "${LCC}" is the hash of the latest cemented commitment. Notice
that anyone can trigger the execution of an outbox message (not only
an operator as in this example).

One can check in the receipt that the contract has indeed been called
with the parameter ""37"" through an internal operation. More complex
parameters, typically containing assets represented as tickets, can be
used as long as they match the type of the entrypoint of the
destination smart contract.


Sending an internal inbox message
---------------------------------

A smart contract can push an internal message in the rollup inbox
using the Michelson "TRANSFER_TOKENS" instruction targeting a specific
rollup address. The parameter of this transfer must be a value of the
Michelson type declared at the origination of this rollup.

Remember that our running example rollup has been originated with:

   octez-client originate smart rollup "${SR_ALIAS}" \
     from "${OPERATOR_ADDR}" \
     of kind wasm_2_0_0 \
     of type bytes \
     booting with "${KERNEL}" \
     -burn-cap 999

The fragment "of type bytes" of this command declares that the rollup
is expecting values of type "bytes". (Notice any Michelson type could
have been used instead. To transfer tickets to a rollup, this type
must mention tickets.)

Here is an example of a Michelson script that sends an internal
message to the rollup of our running example. The payload of the
internal message is the value passed as parameter of type "bytes" to
the rollup.

   parameter bytes;
   storage unit;
   code
     {
       UNPAIR;
       PUSH address "${SR_ADDR}";
       CONTRACT bytes;
       IF_NONE { PUSH string "Invalid address"; FAILWITH } {};
       PUSH mutez 0;
       DIG 2;
       TRANSFER_TOKENS;
       NIL operation;
       SWAP;
       CONS;
       PAIR;
     }


Populating the reveal channel
-----------------------------

It is the responsibility of rollup node operators to get the data
passed through the reveal data channel when the rollup requests it.

To answer a request for a page of hash "H", the rollup node tries to
read the content of a file "H" named "${ROLLUP_NODE_DIR}/wasm_2_0_0".

Notice that a page cannot exceed 4KB. Hence, larger pieces of data
must be represented with multiple pages that reference each other
through hashes. It is up to the kernel to decide how to implement
this. For instance, one can classify pages into two categories: index
pages that are hashes for other pages and leaf pages that contain
actual payloads.

In addition to data stored locally on disk in the data directory, the
rollup node can also fetch missing pages to be revealed from an
external source. Data fetched from a remote pre-images service will be
cached on disk (in the "${ROLLUP_NODE_DIR}/wasm_2_0_0").

To configure a remote source (whose server can be contacted over HTTP
at "${PRE_IMAGES_URL}") for pre-images, one can either pass the option
"--pre-images-endpoint ${PRE_IMAGES_URL}" to the "run" command or add
the following in the configuration file
"${ROLLUP_NODE_DIR}/config.json":

   {
     "pre-images-endpoint" : "${PRE_IMAGES_URL}"
   }

Note:

  One does not need to trust the provider service for pre-images
  because the rollup node will ensure that the content matches the
  requested hash before using it.


Configure WebAssembly fast execution
====================================

When the rollup node advances its internal rollup state under normal
operation, it does so using the fast execution engine.

This engine uses Wasmer for running WebAssembly code. You may
configure the compiler used for compiling WebAssembly code, via the
"OCTEZ_WASMER_COMPILER" environment variable.

The choice of a compiler primarily affects the performance of the
WebAssembly code execution *vs* the compilation time. Some compilers
offer certain security guarantees in a blockchain context, such as
compiling in linear time to avoid JIT bombs.

The available options are:


Wasmer compiler options
^^^^^^^^^^^^^^^^^^^^^^^

+---------------------------+---------------------------+----------------------------------------------------+
| Compiler                  | "OCTEZ_WASMER_COMPILER"   | Description                                        |
|                           | value                     |                                                    |
|===========================|===========================|====================================================|
| Singlepass                | "singlepass"              | When to use Singlepass                             |
+---------------------------+---------------------------+----------------------------------------------------+
| Cranelift                 | "cranelift"               | When to use Cranelift                              |
+---------------------------+---------------------------+----------------------------------------------------+

Note that while the rollup node is generally capable of using Wasmer's
LLVM-based compiler, Octez does not currently ship with it.

When the environment variable is undefined, Cranelift is used by
default.


Developing WASM Kernels
=======================

This page provides a first overview on writing a Wasm kernel for a
smart rollup. (See smart optimistic rollup)

A rollup is primarily characterized by the semantics it gives to the
input messages it processes. This semantics is provided at origination
time as a WASM program (in the case of the "wasm_2_0_0" kind) called a
*kernel*. More concretely, the kernel is a WASM module encoded in the
binary format defined by the WASM standard.

Except for necessary restrictions to ensure determinism (a key
requirement for any web3 technology), we support the full WASM
language.  More precisely, determinism is ensured by the following
restrictions:

1. Instructions and types related to floating-point arithmetic are not
   supported. This is because IEEE floats are not deterministic, as
   the standard includes undefined behavior operations.

2. The length of the call stack of the WASM kernel is bounded.

Modulo the limitations above, a valid kernel is a WASM module that
satisfies the following constraints:

1. It exports a function "kernel_run" that takes no argument and
   returns nothing.

2. It declares and exports exactly one memory.

3. It only imports the host functions exported by the (virtual) module
   "smart_rollup_core".

For instance, the mandatory example of a "hello, world!" kernel is the
following WASM program in text format.

   (module
     (import "smart_rollup_core" "write_debug"
        (func $write_debug (param i32 i32) (result i32)))
     (memory 1)
     (export "mem" (memory 0))
     (data (i32.const 100) "hello, world!")
     (func (export "kernel_run")
       (local $hello_address i32)
       (local $hello_length i32)
       (local.set $hello_address (i32.const 100))
       (local.set $hello_length (i32.const 13))
       (drop (call $write_debug (local.get $hello_address)
                                (local.get $hello_length)))))

This program can be compiled to the WASM binary format with general-
purpose tool like WABT.

   wat2wasm hello.wat -o hello.wasm

The contents of the resulting "hello.wasm" file is a valid WASM
kernel, though its relevance as a decentralized application is
debatable.

One of the benefits of choosing WASM as the programming language for
smart rollups is that WASM has gradually become a ubiquitous
compilation target over the years. Its popularity has grown to the
point where mainstream, industrial languages like Go or Rust now
natively compile to WASM. Thus, "cargo" —the official Rust package
manager— provides an official target to compile Rust to ".wasm" binary
files, which are valid WASM kernels. This means that, for this
particular example, one can build a WASM kernel while enjoying the
strengths and convenience of the Rust language and the Rust ecosystem.

The rest of the section proceeds as follows.

1. First, we explain the execution environment of a WASM kernel: when
   it is parsed, executed, etc.

2. Then, we explain in more detail the API at the disposal of WASM
   kernel developers.

3. Finally, we demonstrate how Rust in particular can be used to
   implement a WASM kernel.

Though Rust has become the primary language whose WASM backend has
been tested in the context of smart rollups, the WASM VM has not been
modified in any way to favor this language. We fully expect that other
mainstream languages such as Go are also good candidates for
implementing WASM kernels.


Execution Environment
---------------------

In a nutshell, the life cycle of a smart rollup is a never-ending loop
of fetching inputs from the Layer 1, and executing the "kernel_run"
function exposed by the WASM kernel.


State
-----

The smart rollup carries two states:

1. A transient state, that is reset after each call to the
   "kernel_run" function and is akin to RAM.

2. A persistent state, that is preserved across "kernel_run" calls.
   The persistent state consists in an *inbox* that is regularly
   populated with the inputs coming from the Layer 1, the *outbox*
   which the kernel can populate with contract calls targeting smart
   contracts in the Layer 1, and a durable storage which is akin to a
   file system.

The durable storage is a persistent tree, whose contents are addressed
by path-like keys. A path in the storage may contain: a value (also
called file) consisting of a sequence of raw bytes, and/or any number
of subtrees (also called directories), that is, the paths in the
storage prefixed by the current path. Thus, unlike most file systems,
a path in the durable storage may be at the same time a file and a
directory (a set of sub-paths).

The WASM kernel can write and read the raw bytes stored under a given
path (the file), but can also interact (delete, copy, move, etc.) with
subtrees (directories).

The values and subtrees under the key "/readonly" are not writable by
a kernel, but can be used by the PVM to give information to the
kernel.


WASM PVM Versioning
-------------------

One of Tezos distinguishing features is its native support for
upgrades. At its core, Tezos is a Layer 1 designed to evolve via a
self-updating mechanism, subject to an on-line governance process. The
self-updating mechanism is also implemented by the smart rollup
infrastructure.

The WASM PVM is versioned. Kernels can read the version of the
underlying WASM PVM (which is currently interpreting them) by reading
the contents of the file stored under the key "/readonly/wasm_version"
in their durable storage.

New WASM PVM versions are introduced by new Layer 1’s protocol
upgrades. The WASM PVM will upgrade itself when it reads the
"Protocol_migration" internal message.

+----------------+------------------+
| Protocol       | Version          |
|================|==================|
| Mumbai         | 2.0.0            |
+----------------+------------------+
| Nairobi        | 2.0.0-r1         |
+----------------+------------------+
| Oxford2        | 2.0.0-r2         |
+----------------+------------------+
| Paris          | 2.0.0-r4         |
+----------------+------------------+
| Quebec         | 2.0.0-r5         |
+----------------+------------------+
| Alpha          | 2.0.0-r5         |
+----------------+------------------+

The changes in each WASM PVM version can be found by searching for
string "PVM" in the corresponding protocol's changelog, section "Smart
Rollups" (e.g. this section for protocol Alpha).


Control Flow
------------

When a new block is published on Tezos, the inbox exposed to the smart
rollup is populated with all the inputs published on Tezos in this
block. It is important to keep in mind that all the smart rollups
which are originated on Tezos share the same inbox. As a consequence,
a WASM kernel has to filter the inputs that are relevant to its
purpose from the ones it does not need to process.

Once the inbox has been populated with the inputs of the Tezos block,
the "kernel_run" function is called, from a clean “transient” state.
More precisely, the WASM kernel is re-initialized, then "kernel_run"
is called.

By default, the WASM kernel yields when "kernel_run" returns. In this
case, the WASM kernel execution is put on hold while the inputs of the
next inbox are being loaded. The inputs that were not consumed by
"kernel_run" are dropped. "kernel_run" can prevent the WASM kernel
from yielding by writing arbitrary data under the path
"/kernel/env/reboot" in its durable storage. In such a case (known as
reboot), "kernel_run" is called again, without dropping unread inputs.
The value at "/kernel/env/reboot" is removed between each call of
"kernel_run", and the "kernel_run" function can postpone yielding at
most 1,000 reboots for each Tezos level.

A call to "kernel_run" cannot take an arbitrary amount of time to
complete, because diverging computations are not compatible with the
optimistic rollup infrastructure of Tezos. It is the responsibility of
the kernel developers to ensure the computation time necessary to
track their rollup is bounded and reasonable, for two reasons:

1. If a kernel requires more time than the time between two Tezos
   blocks, then a rollup node is doomed to lag behind the Layer 1
   chain it is tracking.

2. If a single "kernel_run" takes too much time to compute, then it
   becomes difficult to protect the resulting commitment. This is
   because the WASM PVM interpreter has not been optimized for speed
   but for producing small execution step proofs.

Kernel developers are expected to design their kernel such that it
addresses these two constraints.

The WASM PVM does enforce a limit on the number of ticks available per
"kernel_run", but it is arbitrary high enough (50 trillion) that it
becomes virtually impossible to exceed it. "octez-smart-rollup-wasm-
debugger" is probably the best tool available to verify the
"kernel_run" function does not take more ticks than authorized.

The direct consequence of this setup is that it might be necessary for
a WASM kernel to span a long computation across several calls to
"kernel_run", and therefore to serialize any data it needs in the
durable storage to avoid losing them.

Finally, the kernel can verify if the previous "kernel_run" invocation
was trapped by verifying if some data are stored under the path
"/kernel/env/stuck".


Host Functions
--------------

At its core, the WASM machine defined in the WASM standard is just a
very evolved arithmetic machine. It needs to be enriched with so-
called host functions in order to be used for greater purposes. The
host functions provide an API to the WASM program to interact with an
“outer world”.

As for smart rollups, the host functions exposed to a WASM kernel
allow it to interact with the components of persistent state:

"read_input"
   Loads the oldest input still present in the inbox of the smart
   rollup in the transient memory of the WASM kernel. This means that
   the input is lost at the next invocation of "kernel_run" if it is
   not written in the durable storage. Since version "2.0.0" of the
   WASM PVM.

"write_output"
   Writes an in-memory buffer to the outbox of the smart rollup. If
   the content of the buffer follows the expected encoding, it can be
   interpreted in the Layer 1 as a smart contract call, once a
   commitment acknowledging the call to this host function is
   cemented. Since version "2.0.0" of the WASM PVM.

"write_debug"
   Can be used by the WASM kernel to log events which can potentially
   be interpreted by an instrumented rollup node. Since version
   "2.0.0" of the WASM PVM.

"store_has"
   Returns the kind of data (if any) stored in the durable storage
   under a given path: a directory, a file, neither or both. Since
   version "2.0.0" of the WASM PVM.

"store_delete"
   Cuts both the value (if any) and any subdirectory under a given
   path out of the durable storage. Since version "2.0.0" of the WASM
   PVM.

"store_delete_value"
   Cuts the value under a given path out of the durable storage, but
   leaves the rest of the subtree untouched. Since version "2.0.0-r1"
   of the WASM PVM.

"store_copy"
   Copies the subtree under a given path to another key. Since the
   "2.0.0" version of the WASM PVM.

"store_move"
   Behaves as "store_copy", but also cuts the original subtree out of
   the tree. Since version "2.0.0" of the WASM PVM.

"store_read"
   Loads at most 4,096 bytes from a file of the durable storage to a
   buffer in the memory of the WASM kernel. Since version "2.0.0" of
   the WASM PVM.*

"store_write"
   Writes at most 2048 bytes from a buffer in the memory of the WASM
   kernel to a file of the durable storage, increasing its size if
   necessary. Note that files in the durable storage cannot exceed
   2^{31} - 1 bytes (i.e. 2GB - 1). Since the "2.0.0" version of the
   WASM PVM.

"store_create"
   Allocates a new file in the durable storage under a given key.
   Similarly to "store_write", "store_create" cannot create files
   larger than the durable storage limits, that is 2GB - 1. Since the
   "2.0.0-r1" of the WASM PVM.

"store_value_size"
   Returns the size (in bytes) of a file under a given key in the
   durable storage. Since version "2.0.0" of the WASM PVM.

"store_list_size"
   Returns the number of child objects (either directories or files)
   under a given key. Since version "2.0.0" of the WASM PVM.

"reveal_preimage"
   Loads in memory the preimage of a hash. The size of the hash in
   bytes must be specified as an input to the function. Since the
   "2.0.0" version of the WASM PVM.

"reveal_metadata"
   Loads in memory the address of the smart rollup (20 bytes), and the
   Tezos level of its origination (4 bytes). Since the "2.0.0" version
   of the WASM PVM.

These host functions use a "C-like" API. In particular, most of them
return a signed 32bit integer, where negative values are reserved for
conveying errors, as shown in the next table.

+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| Code    | Description                                                                                                                                |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -1      | Input is too large to be a valid key of the durable storage                                                                                |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -2      | Input cannot be parsed as a valid key of the durable storage                                                                               |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -3      | There is no file under the requested key                                                                                                   |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -4      | The host functions tried to read or write an invalid section (determined by an offset and a length) of the value stored under a given key  |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -5      | Cannot write a value beyond the 2GB size limit                                                                                             |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -6      | Invalid memory access (segmentation fault)                                                                                                 |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -7      | Tried to read from the inbox or write to the outbox more than 4,096 bytes                                                                  |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -8      | Unknown error due to an invalid access                                                                                                     |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -9      | Attempt to modify a readonly value                                                                                                         |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -10     | Key has no tree in the storage                                                                                                             |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -11     | Outbox is full, no new message can be appended                                                                                             |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+
| -13     | Key has already a value in the storage                                                                                                     |
+---------+--------------------------------------------------------------------------------------------------------------------------------------------+


Implementing a WASM Kernel in Rust
----------------------------------

Though WASM is a good fit for efficiently executing computation-
intensive, arbitrary programs, it is a low-level, stack-based, memory
unsafe language. Fortunately, it was designed to be a compilation
target, not a language in which developers would directly write their
programs.

Rust has several advantages that make it a good candidate for writing
the kernel of a smart rollup. Not only does the Rust compiler treat
WASM as a first class citizen when it comes to compilation targets,
but its approach to memory safety eliminates large classes of bugs and
vulnerabilities that arbitrary WASM programs may suffer from.

Additionally there is support for implementing kernels in Rust, in the
form of the Smart Rollup Kernel SDK.


Setting-up Rust
~~~~~~~~~~~~~~~

rustup is the standard way to get Rust. Once "rustup" is installed,
enabling WASM as a compilation target is as simple as running the
following command.

   rustup target add wasm32-unknown-unknown

Rust also proposes the "wasm64-unknown-unknown" compilation target.
This target is **not** compatible with Tezos smart rollups, which only
provide a 32bit address space.

Note:

  This document is not a tutorial about Rust, and familiarity with the
  language and its ecosystem (*e.g.*, how Rust crates are structured
  in particular) is assumed.

The simplest kernel one can implement in Rust (the one that returns
directly after being called, without doing anything particular) is the
following Rust file (by convention named "lib.rs" in Rust).

   #[no_mangle]
   pub extern "C" fn kernel_run() {
   }

This code can be easily computed with "cargo" with the following
"Cargo.toml".

   [package]
   name = 'noop'
   version = '0.1.0'
   edition = '2021'

   [lib]
   crate-type = ["cdylib"]

The key line to spot is the "crate-type" definition to "cdylib". As a
side note, when writing a library that will eventually be consumed by
a Kernel WASM crate, this line must be modified to

   crate-type = ["cdylib", "rlib"]

Compiling our “noop” kernel is done by calling "cargo" with the
correct argument.

   cargo build --target wasm32-unknown-unknown

It is also possible to use the "--release" CLI flag to tell "cargo" to
optimize the kernel.

To make the use of the "target" optional, it is possible to create a
".cargo/config.toml" file, containing the following line.

   [build]
   target = "wasm32-unknown-unknown"

   [rust]
   lld = true

The resulting project looks as follows.

   .
   ├── .cargo
   │   └── config.toml
   ├── Cargo.toml
   └── src
       └── lib.rs

and the kernel can be found in the "target/" directory, *e.g.*,
"./target/wasm32-unknown-unknown/release/noop.wasm".

By default, Rust binaries (including WASM binaries) contain a lot of
debugging information and possibly unused code that we do not want to
deploy in our rollup. For instance, our “noop” kernel weighs
1.7MBytes. We can use wasm-strip to reduce the size of the kernel
(down to 115 bytes in our case).


Host Functions in Rust
~~~~~~~~~~~~~~~~~~~~~~

The host functions exported by the WASM runtime to Rust programs are
exposed by the following API. The "link" pragma is used to specify the
module that exports them (in our case, "smart_rollup_core"). Define
these functions in the "host.rs" as follows:

   #[repr(C)]
   pub struct ReadInputMessageInfo {
       pub level: i32,
       pub id: i32,
   }

   #[link(wasm_import_module = "smart_rollup_core")]
   extern "C" {
       /// Returns the number of bytes written to `dst`, or an error code.
       pub fn read_input(
           message_info: *mut ReadInputMessageInfo,
           dst: *mut u8,
           max_bytes: usize,
       ) -> i32;

       /// Returns 0 in case of success, or an error code.
       pub fn write_output(src: *const u8, num_bytes: usize) -> i32;

       /// Does nothing. Does not check the correctness of its argument.
       pub fn write_debug(src: *const u8, num_bytes: usize);

       /// Returns
       /// - 0 the key is missing
       /// - 1 only a file is stored under the path
       /// - 2 only directories under the path
       /// - 3 both a file and directories
       pub fn store_has(path: *const u8, path_len: usize) -> i32;

       /// Returns 0 in case of success, or an error code
       pub fn store_delete(path: *const u8, path_len: usize) -> i32;

       /// Returns the number of children (file and directories) under a
       /// given key.
       pub fn store_list_size(path: *const u8, path_len: usize) -> i64;

       /// Returns 0 in case of success, or an error code.
       pub fn store_copy(
           src_path: *const u8,
           scr_path_len: usize,
           dst_path: *const u8,
           dst_path_len: usize,
       ) -> i32;

       /// Returns 0 in case of success, or an error code.
       pub fn store_move(
           src_path: *const u8,
           scr_path_len: usize,
           dst_path: *const u8,
           dst_path_len: usize,
       ) -> i32;

       /// Returns the number of bytes written to the durable storage
       /// (should be equal to `num_bytes`, or an error code).
       pub fn store_read(
           path: *const u8,
           path_len: usize,
           offset: usize,
           dst: *mut u8,
           num_bytes: usize,
       ) -> i32;

       /// Returns 0 in case of success, or an error code.
       pub fn store_write(
           path: *const u8,
           path_len: usize,
           offset: usize,
           src: *const u8,
           num_bytes: usize,
       ) -> i32;

       /// Returns the number of bytes written at `dst`, or an error
       /// code.
       pub fn reveal_metadata(
           dst: *mut u8,
           max_bytes: usize,
       ) -> i32;

       /// Returns the number of bytes written at `dst`, or an error
       /// code.
       pub fn reveal_preimage(
           hash_addr: *const u8,
           hash_size: u8,
           dst: *mut u8,
           max_bytes: usize,
       ) -> i32;
   }

These functions are marked as "unsafe" for Rust. It is possible to
provide a safe API on top of them. For instance, the "read_input" host
function can be used to declare a safe function which allocates a
fresh Rust Vector to receive the input.

Define these functions in the "lib.rs" as follows:

   // Assuming the host functions are defined in a module `host`.

   mod host;
   use crate::host::read_input;
   use crate::host::ReadInputMessageInfo;

   pub const MAX_MESSAGE_SIZE: u32 = 4096u32;

   pub struct Input {
       pub level: u32,
       pub id: u32,
       pub payload: Vec<u8>,
   }

   pub fn next_input() -> Option<Input> {
       let mut payload = Vec::with_capacity(MAX_MESSAGE_SIZE as usize);

       // Placeholder values
       let mut message_info = ReadInputMessageInfo { level: 0, id: 0 };

       let size = unsafe {
            read_input(
               &mut message_info,
               payload.as_mut_ptr(),
               MAX_MESSAGE_SIZE.try_into().unwrap(),
           )
       };

       if 0 < payload.len() {
           unsafe { payload.set_len(size as usize) };
           Some(Input {
               level: message_info.level as u32,
               id: message_info.id as u32,
               payload,
           })
       } else {
           None
       }
   }

Coupling "Vec::with_capacity" along with the "set_len" unsafe function
is a good approach to avoid initializing the 4,096 bytes of memory
every time you want to load data of arbitrary size into the WASM
memory.


Testing your Kernel
-------------------

Note:

  "octez-smart-rollup-wasm-debugger" is available in the Octez
  distribution starting with Version 16.1.

Testing a kernel without having to start a rollup node on a test
network is very convenient. We provide a debugger as a means to
evaluate the WASM PVM without relying on any node and network: "octez-
smart-rollup-wasm-debugger".

   octez-smart-rollup-wasm-debugger --kernel "${WASM_FILE}" --inputs "${JSON_INPUTS}" --rollup "${SR_ADDR}"

"octez-smart-rollup-wasm-debugger" takes the target WASM kernel to be
debugged as argument, either as a ".wasm" file (the binary
representation of WebAssembly modules) or as a ".wast" file (its
textual representation), and actually parses and typechecks the kernel
before giving it to the PVM.

Beside the kernel file, the debugger can optionally take an input file
containing inboxes and a rollup address. The expected contents of the
inboxes is a JSON value, with the following schema:

   [
     [ { "payload" : <Michelson data>,
         "sender" : <Smart contract sending to the rollup, optional>,
         "source" : <User account sending the message, optional>
         "destination" : <Smart rollup address, optional> }
       ..
       // or
       { "external" : <hexadecimal payload> }
       ..
     ]
   ]

The contents of the input file is a JSON array of arrays of inputs,
which encodes a sequence of inboxes, where an inbox is a set of
messages. These inboxes are read in the same order as they appear in
the JSON file. For example, here is a valid input file that defines
two inboxes: the first array encodes an inbox containing only an
external message, while the second array encodes an inbox containing
two messages:

   [
     [
       {
         "external":
         "0000000023030b01d1a37c088a1221b636bb5fccb35e05181038ba7c000000000764656661756c74"
       }
     ],
     [
       {
         "payload" : "0",
         "sender" : "KT1ThEdxfUcWUwqsdergy3QnbCWGHSUHeHJq",
         "source" : "tz1RjtZUVeLhADFHDL8UwDZA6vjWWhojpu5w",
         "destination" : "sr1RYurGZtN8KNSpkMcCt9CgWeUaNkzsAfXf"
       },
       { "payload" : "Pair Unit False" }
     ]
   ]

Note that the "sender", "source" and "destination" fields are optional
and will be given default values by the debugger, respectively
"KT18amZmM5W7qDWVt2pH6uj7sCEd3kbzLrHT",
"tz1Ke2h7sDdakHJQh8WX4Z372du1KChsksyU" and
"sr163Lv22CdE8QagCwf48PWDTquk6isQwv57". If no input file is given, the
inbox will be assumed empty. If the option "--rollup" is given, it
replaces the default value for the rollup address.

"octez-smart-rollup-wasm-debugger" is a debugger, as such it waits for
user inputs to continue its execution. Its initial state is exactly
the same as right after its origination. Its current state can be
inspected with the command "show status":

   > show status
   Status: Waiting for input
   Internal state: Collect

When started, the kernel is in collection mode internally. This means
that it is not executing any WASM code, and is waiting for inputs in
order to proceed. The command "load inputs" will load the first inbox
from the file given with the option "--inputs", putting
"Start_of_level" and "Info_per_level" before these inputs and
"End_of_level" after the inputs.

   > load inputs
   Loaded 1 inputs at level 0

   > show status
   Status: Evaluating
   Internal state: Snapshot

At this point, the internal input buffer can be inspected with the
command "show inbox".

   > show inbox
   Inbox has 4 messages:
   { raw_level: 0;
     counter: 0
     payload: Start_of_level }
   { raw_level: 0;
     counter: 1
     payload: Info_per_level {predecessor_timestamp = 1970-01-01T00:00:00-00:00; predecessor = BKiHLREqU3JkXfzEDYAkmmfX48gBDtYhMrpA98s7Aq4SzbUAB6M} }
   { raw_level: 0;
     counter: 2
     payload: 0000000023030b01d1a37c088a1221b636bb5fccb35e05181038ba7c000000000764656661756c74 }
   { raw_level: 0;
     counter: 3
     payload: End_of_level }

The first input of an inbox at the beginning of a level is
"Start_of_level", and is represented by the message "\000\001" on the
kernel side. We can now start a "kernel_run" evaluation:

   > step kernel_run
   Evaluation took 11000000000 ticks so far
   Status: Waiting for input
   Internal state: Collect

The memory of the interpreter is flushed between two "kernel_run"
calls (at the "Snapshot" and "Collect" internal states), however the
durable storage can be used as a persistent memory. Let's assume this
kernel wrote data at key "/store/key":

   > show key /store/key
   `<hexadecimal value of the key>`

Since the representation of values is decided by the kernel, the
debugger can only return its raw value. Please note that the command
"show keys <path>" will return the keys under the given path. This can
help navigate in the durable storage.

   > show keys /store
   /key
   /another_key
   ...

It is also possible to inspect the memory by stopping the PVM before
its snapshot internal state, with "step result", and inspect the
memory at pointer "n" and length "l", and finally evaluate until the
next "kernel_run":

   > step result
   Evaluation took 2500 ticks so far
   Status: Evaluating
   Internal state: Evaluation succeeded

   > show memory at p for l bytes
   `<hexadecimal value>`

   > step kernel_run
   Evaluation took 7500 ticks so far
   Status: Evaluating
   Internal state: Snapshot

Once again, note that values from the memory are output as is, since
the representation is internal to WASM.

Finally, it is possible to evaluate the whole inbox with "step inbox".
It will take care of the possible reboots asked by the kernel (through
the usage of the "/kernel/env/reboot_flag" flag) and stop at the next
collection phase.

   > step inbox
   Evaluation took 44000000000 ticks
   Status: Waiting for input
   Internal state: Collect

To obtain more information on the execution, the command "profile"
will also run the kernel on a full inbox, consume all inputs, run
until more inputs are required, and output some information about the
run.

   > profile
   Starting the profiling until new messages are expected. Please note that it will take some time and does not reflect a real computation time.
   Profiling result can be found in /tmp/wasm-debugger-profiling-2023-09-26T09:10:09.860-00:00.out
   ----------------------
   Detailed results for a `kernel_run`:
   %interpreter(decode): 35948 ticks (277ms)
   %interpreter(link): 6 ticks (3.605us)
   %interpreter(init): 201823 ticks (62.246ms)
   kernel_run: 22962 ticks (20.280ms)

   Full execution: 260739 ticks (359ms)
   ----------------------
   Detailed results for a `kernel_run`:
   %interpreter(decode): 35948 ticks (273ms)
   %interpreter(link): 6 ticks (7.287us)
   %interpreter(init): 201823 ticks (63.946ms)
   kernel_run: 29388 ticks (9.275ms)

   Full execution: 267165 ticks (346ms)
   ----------------------
   Full execution with padding: 22000000000 ticks

Each cycle is a call of the "kernel_run" function. For each cycle, the
number of *effective* ticks used is shown (ticks corresponding to
execution, and not used for padding), along with the duration in
seconds.

It is also possible to show the outbox for any given level ("show
outbox at level 0")

   > show outbox at level 0
   Outbox has N messages:
   { unparsed_parameters: ..;
     destination: ..;
     entrypoint: ..; }
   ..

The reveal channel described previously is available in the debugger,
either automatically or through specific commands. The debugger can
fill automatically preimages from files in a specific directory on the
disk, by default in the "preimage" subdirectory of the working
directory. It can be configured with the option "--preimage-dir
<directory>". In case there is no corresponding file found for the
requested preimage, the debugger will ask for the hexadecimal value of
the preimage:

   > step inbox
   Preimage for hash 0000[..] not found.
   > 48656c6c6f207468657265210a
   Hello there!
   ...

Metadata are automatically filled with level "0" as origination level
and the configured smart rollup address (or the default one).

Note that when stepping tick by tick (using the "step tick" command),
it is possible to end up in a situation where the evaluation stops on
"Waiting for reveal". If the expected value is a metadata, the command
"reveal metadata" will give the default metadata to the kernel. If the
value expected is the preimage of a given hash, there are two possible
solutions:

* "reveal preimage" to read the value from the disk. In that case, the
  debugger will look for a file of the same name as the expected hash
  in the "preimage" subdirectory.

* "reveal preimage of <hex encoded value>" can be used to feed a
  custom preimage hash.

The economic protocol
*********************

The economic protocol provides the rules for checking the validity of
the blocks and operations, and for updating the blockchain state
accordingly, by applying new valid blocks and operations on the
current blockchain state. These rules can be changed through voting.
Thus, the economic protocol represents the amendable part of Tezos.

The following pages provide the full documentation for this protocol.
The differences with respect to other protocols are summarized in the
protocol release page Protocol Rio.

* Economic protocol features

  * Overview of the economic protocol

  * Proof-of-stake

  * Randomness generation

  * Baking Power

  * The consensus algorithm

  * The Amendment (and Voting) Process

  * Accounts and addresses

  * Michelson: the language of Smart Contracts in Tezos

  * On-chain Views

  * Time-lock

  * Sapling support

  * Liquidity Baking

  * Global Constants

  * Michelson Anti-Patterns

  * Token transfers and balance updates

  * Mempool

  * Smart Optimistic Rollups

  * Protocol Plugins

  * Contract events

  * Blocks and Operations

  * Validation and Application

  * Tickets

  * Adaptive Issuance

  * Staking mechanism

  * Adaptive Slashing

  * DAL integration

The following pages provide a reference for this protocol:

* Glossary

  * Tezos terms

  * Protocol terms

* Command Line Interface

  * Client manual

  * Baker manual

  * Accuser manual

* Rio RPCs - Reference

  * Index

  * Full description

Economic protocol features
**************************

This page groups the documentation helping developers and users in
understanding the basic concepts of the economic protocol (Proof Of
Stake, consensus, voting, etc), its features (Michelson, Sapling,
etc), and some details about its implementation.

* Overview of the economic protocol

  * The role of the economic protocol

  * Shell-protocol interaction

  * Blocks, Operations and their Validation

  * Protocol constants

  * See also

* Proof-of-stake

  * Overview

  * Delegation and Staking

  * Delegates' rights selection

  * Proof-of-stake parameters

  * Further External Resources

* Randomness generation

  * Overview

  * Cryptographic primitives

  * Protocol

* Baking Power

  * Overview

  * Delegate, delegators, stakers

  * Staked tez

  * Delegated tez

  * Overstaking

  * Overdelegation

  * Minimal power and own staked requirements

* The consensus algorithm

  * History

  * Overview

  * Transaction and block finality

  * Time between blocks

  * Validator selection

  * Economic Incentives

  * Fitness

  * Consensus related protocol parameters

  * Further External Resources

* The Amendment (and Voting) Process

  * Periods

  * Activation

  * Voting Power

  * Super-majority and Quorum

  * The Hash and the Protocol

  * Operations

  * Client Commands

  * Further External Resources

* Accounts and addresses

  * User accounts

  * Smart contracts

* Michelson: the language of Smart Contracts in Tezos

  * Semantics of smart contracts and transactions

  * Language semantics

  * Introduction to the type system and notations

  * Types and instructions

  * Removed instructions and types

  * Macros

  * Concrete syntax

  * Annotations

  * Entrypoints

  * JSON syntax

  * Development tools

  * Examples

  * Full grammar

  * Reference implementation

  * TZT, a Syntax extension for writing unit tests

* On-chain Views

  * Homonyms

  * Execution flow

  * Declaration

  * Forbidden types

* Time-lock

  * Cryptographic design

  * Opcode and types

  * Implementation of the time-lock puzzle

  * Example

* Sapling support

  * Sapling

  * Sapling integration

* Liquidity Baking

  * Contracts

  * Subsidy

  * Toggle vote

* Global Constants

  * Global Constant Registration

  * Originating a Contract that uses Global Constants

  * Global Constants at Runtime

* Michelson Anti-Patterns

  * Storing unbounded data

  * Refunding to a list of contracts

  * Avoid batch operations when users can increase the size of the
    batch

  * Signatures alone do not prevent replay attacks

  * Storing/transferring private data

  * Not setting all state before a transfer

* Token transfers and balance updates

  * Overview

  * Source, container, and sink accounts

  * Token transfers and metadata

* Mempool

  * Merging Mempools

  * Conflict

* Smart Optimistic Rollups

  * Overview

  * Rollup kernel

  * User roles

  * Address

  * Inputs

  * Origination

  * Processing

  * Commitments

  * Bonds

  * Refutation

  * Private rollups

* Protocol Plugins

  * Prevalidator filters

  * Filters RPCs

* Contract events

  * Emitting events

  * Retrieving events

* Blocks and Operations

  * Protocol-specific block header

  * Operations

  * Validation Passes

  * Consensus Operations

  * Voting Operations

  * Anonymous Operations

  * Manager Operations

  * Failing_noop operation

* Validation and Application

  * Validation modes

  * Block Validation

  * Operation Validation and Application

* Tickets

  * Semantics

  * Operations on tickets

* Adaptive Issuance

  * Adaptive issuance rate

  * Adaptive rewards

* Staking mechanism

  * Staking policy configuration

  * Staked funds management

  * Change of delegate

* Adaptive Slashing

  * Overview

  * Adaptive Slashing Function

  * New definition for the forbidden period

* DAL integration

  * DAL publish commitment

  * DAL attestation payloads

  * DAL incentives scheme

  * Smart Rollups integration

  * DAL-related protocol constants

Overview of the economic protocol
*********************************

Tezos is a self-amending blockchain, in that a large part of Tezos can
be changed through a so-called amendment procedure. To this end, as
mentioned in the big picture, a Tezos node consists of two components:

* the shell, which comprises the network and storage layer, and embeds

* the economic protocol component, which is the part that can be
  changed through amendment.


The role of the economic protocol
=================================

At a very high level, a protocol must:

* implement protocol-specific types, such as the type of operations or
  protocol-specific block header data (in addition to the shell
  generic header),

* define under which conditions a block is a valid extension of the
  current blockchain, and define an ordering on blocks to arbitrate
  between concurrent extensions.

Validity conditions are implemented in the "apply" function which is
called whenever the node processes a block---see the dedicated
protocol validation and operation entry for further detail into the
validation and application process for blocks and their operations.


Shell-protocol interaction
==========================

In the Tezos architecture, the economic protocol and the shell
interact in order to ensure that the blocks being appended to the
blockchain are valid. There are mainly two rules that the shell uses
when receiving a new block:

* The shell does not accept a block whose level is below the current
  checkpoint. The checkpoint itself is updated based on information
  resulting from successful block applications by the protocol which
  depends on the protocol consensus algorithm. Previously accepted
  blocks with lower levels than the current checkpoint are considered
  finalized and immutable.

* The shell changes the head of the chain to this new block only if
  the block is valid, and it has a higher fitness than the current
  head; a block is valid only if all the operations included are also
  valid.

The support provided by the protocol for validating blocks can be
modulated by different validation modes. They allow using this same
interface for quite different use cases, as follows:

* being able to apply a block, typically used by the shell's validator
  component;

* being able to construct a block, typically used by the baker daemon
  to *bake* -- that is, to produce -- a new block;

* being able to partially construct a block, typically used by the
  prevalidator to determine valid operations in the mempool; and,

* being able to pre-apply a block, typically used in the validator to
  precheck a block, avoiding to further consider invalid blocks.


Blocks, Operations and their Validation
=======================================

A block consists of a header and operations. A block's header is
composed of two parts: the protocol-agnostic part and the protocol-
specific part. This separation enables the shell to interact with
different protocols. Each Tezos economic protocol can specify
different kinds of operations, which are described further in detail
in Blocks and Operations.

The semantics of, respectively, operations and blocks is indeed also
dependent on each economic protocol. The Validation and Application
entry explains the internals of *validation* -- that is, how to
determine whether operations and blocks can be safely included in the
Tezos blockchain -- and *application* -- that is, how the effects of
operations and blocks are taken into account -- for this economic
protocol.


Protocol constants
==================

Protocols are tuned by several *protocol constants*, such as the size
of a nonce, or the number of blocks per cycle. One can distinguish two
kinds of protocol constants:

* *fixed* protocol constants, such as the size of a nonce, are values
  wired in the code of a protocol, and can only be changed by protocol
  amendment (that is, by adopting a new protocol)

* *parametric* protocol constants, such as the number of blocks per
  cycle, are values maintained in a read-only data structure that can
  be instantiated differently, for the same protocol, from one network
  to another (for instance, test networks move faster).

* *derived* constants are not directly configurable, but they depend
  on parametric constants. For instance, the issuance modification
  delay is a derived constant which is always equal to the consensus
  rights delay.

The *list* of protocol constants can be found in the OCaml APIs:

* fixed protocol constants are defined in the module Constants_repr

* parametric constants are defined in the module
  Constants_parametric_repr

* derived constants are defined in the module Constants_repr.Derived

The *values* of protocol constants in any given protocol can be found
using specific RPC calls:

* one RPC for all constants (fixed and parametric and derived), as
  shown in this example

* one RPC for the parametric constants specifically

Further documentation of various protocol constants can be found in
the subsystems where they conceptually belong. See, for example:

* proof-of-stake parameters

* consensus-related parameters

* randomness generation parameters

* DAL parameters


See also
========

An in-depth description of the internals of developing a new Tezos
protocol can be found in the blog post: How to write a Tezos protocol.

Proof-of-stake
**************


Overview
========

The consensus algorithm in Tezos is based on the *proof-of-stake*
mechanism. Proof-of-stake means that participants in the consensus
algorithm are chosen in function of their stake (the amount of tokens
a participant has). The same mechanism is used in the Tezos
governance.

If one does not have enough stake to participate on its own or does
not want to set up the needed infrastructure, (s)he can use
delegation, possibly complemented with staking. Indeed, in Tezos, it
is the delegates that may participate in consensus. However, at each
level, not all delegates necessarily participate, and their
participation weight may differ. The selection of the delegates'
participation rights at a level is done by running a PRNG (pseudo-
random number generator). The PRNG's seeds are obtained from random
data that are regularly produced and stored on the blockchain. Thus,
the procedure is deterministic in that delegates' rights are uniquely
determined from the seed; and it is random, in that its seed (and
hence its results) cannot be predicted too much in advance.


Delegation and Staking
======================

A *delegate* is any user account registered as such. This is done by
*self-delegating*, that is, emitting a delegation operation (see
below) in which the specified delegate is the same as the operation
emitter (its signer). Note that "tz4" accounts cannot be registered as
delegates.

Any account (user account or smart contract) can specify a delegate
through a delegation operation.  Any non-delegate account can change
or revoke its delegate at any time, again through a delegation
operation. However, the change only becomes effective after
"CONSENSUS_RIGHTS_DELAY + 2" cycles.  The value
"CONSENSUS_RIGHTS_DELAY" is a protocol constant. A delegate cannot
stop self-delegating.

A delegate participates in consensus and in governance in proportion
to their *baking power* and *voting power* respectively.

* The voting power of a delegate is the total amount of tez owned by
  all the accounts that delegate to it (including the delegate
  itself), with no distinctions made between staked and non-staked
  tez.

* The baking power is similar, except that non-staked tez are weighted
  less than staked tez, and there are additional considerations such
  as overstaking and overdelegation. See the Baking Power page for
  more details.

Moreover, to participate in consensus and governance, the delegate
needs to be active and to meet minimal balance requirements.

Delegates and delegators may stake their tez. Staked tez are security
deposits that may be forfeited in case the baker does not follow (some
particular rules of) the protocol. Besides, as mentioned above, staked
tez are weighted higher than non-staked tez when computing the baking
power.


Consensus key
-------------

The key used by a delegate to sign blocks and consensus operations is
called the *consensus key*. By default, this is the delegate's private
key, called its *manager key*. However, a delegate may specify
another, dedicated key for this role.

Setting a new consensus key is accomplished via the
"Update_consensus_key" operation. There is delay of
"CONSENSUS_KEY_ACTIVATION_DELAY + 1" cycles before the new key
actually becomes the *active consensus key* that must be used to sign
blocks and consensus operations; until then, it is called a *pending
consensus key*. More precisely, the key becomes active after the cycle
containing the "Update_consensus_key" operation is over and then
another CONSENSUS_KEY_ACTIVATION_DELAY full cycles have passed: if the
update happens during cycle "n", then the key becomes active at the
beginning of cycle "n + CONSENSUS_KEY_ACTIVATION_DELAY + 1".

There may be multiple pending consensus keys, set to activate in
different future cycles, and each one will replace the previously
active consensus key in turn. If multiple "Update_consensus_key"
operations are performed within the same cycle "n", the last one takes
precedence for determining which key to activate at the start of cycle
"n + CONSENSUS_KEY_ACTIVATION_DELAY + 1".

Note that both the manager key and the consensus key give total
control over the delegate's spendable balance. See this page for
further important details, including client commands that are helpful
for handling consensus keys.


Active and passive delegates
----------------------------

A delegate can be marked as either active or passive. A passive
delegate cannot participate in the consensus algorithm.

A delegate is marked as active at its registration.

At the end of a cycle, a delegate gets deactivated if the chain has
not witnessed any consensus activity (baking, attesting) from it
during the past "TOLERATED_INACTIVITY_PERIOD" cycles, including the
currently ending cycle.

Note that there is an extra grace period of "CONSENSUS_RIGHTS_DELAY"
cycles when a delegate has just registered or has just been
reactivated. This is to account for the fact that it will not receive
consensus rights yet for the first "CONSENSUS_RIGHTS_DELAY" cycles, so
of course the chain cannot witness any activity from it during that
time.


Delegates' rights selection
===========================

Tezos being proof-of-stake, the delegates' rights are selected at
random based on their baking power. Let us detail the selection
mechanism used in Tezos.


Random seed
-----------

To each cycle is associated a random number called the seed. This seed
is used within its cycle to generate pseudo-random values in the
protocol, in particular for selecting delegates to participate in
consensus.

For more information on randomness generation, see randomness-
generation.


Slot selection
--------------

Delegates' rights to participate are determined using the alias
method, more precisely using Vose's algorithm (see also this more
pedagogic description; the algorithm is the last one listed there).
This algorithm samples from a discrete probability distribution, which
is given by the stakes of a specific cycle: the probability to sample
a particular delegate is its stake in the cycle over the total stake
in that cycle.

Concretely, the delegates' rights at a given level are expressed in
terms of the (quantity of) *slots* that the delegate owns at that
level. This quantity represents the delegate's weight in consensus. We
note that, in the long run (that is, on average over many levels), the
number of slots is proportional to its stake. The owner of a slot is
obtained by sampling using the algorithm mentioned above. More
precisely, given a level and a slot (which is just a non-negative
integer), the mentioned algorithm is invoked to assign a delegate to
the given slot. Its input is the probability distribution given by the
stakes retained for the cycle to which the level belongs. And whenever
the algorithm needs to draw a random value, this is obtained using a
simple procedure which has as its initial state: the level, the random
seed for the cycle to which the level belongs, and the slot.


Proof-of-stake parameters
=========================

+---------------------------------------------------------+---------------------------+
| Parameter name                                          | Parameter value           |
|=========================================================|===========================|
| "BLOCKS_PER_CYCLE"                                      | 10800 blocks              |
+---------------------------------------------------------+---------------------------+
| "CONSENSUS_RIGHTS_DELAY"                                | 2 cycles                  |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_STAKE"                                         | 6,000 ꜩ                   |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_FROZEN_STAKE"                                  | 600 ꜩ                     |
+---------------------------------------------------------+---------------------------+
| "TOLERATED_INACTIVITY_PERIOD"                           | 2 cycles                  |
+---------------------------------------------------------+---------------------------+


Further External Resources
==========================

The original design of the proof-of-stake mechanism in Tezos can be
found in the whitepaper.

The adaptive issuance feature documentation.

Other presentations of the Tezos' proof-of-stake mechanism can be
found in the Open Tezos entry.

Randomness generation
*********************


Overview
========

This document presents the randomness generation protocol, used to
select the delegates to participate in the consensus, as well as the
cryptographic primitives used.


Cryptographic primitives
========================

We introduce RANDAO and Verifiable Delay Functions, two cryptographic
protocols for generating random values.


RANDAO
------

RANDAO is a simple multi-party protocol used to generate a random seed
based on a ''commit and reveal'' scheme (similar in spirit with the
RANDAO protocol).

During the setup of the protocol, the cryptographic primitives and
security parameters are chosen. The members participating in the
protocol can also be chosen at this stage.

During the first phase (the "commitment" phase) each party commits to
a random value, called the nonce, and publishes the commitment.
Typically, a hash function is used for committing.

During the second phase, (the "nonce revelation" phase) each party
reveals their nonce. The revealed nonces are then verified with
respect to the commitments. Typically, this means checking that the
hash of each revealed nonce is equal to its corresponding commitment.
If the verification fails, the nonce is discarded, otherwise it is
accepted.

Once the revelation phase is finished, nonces are combined to generate
the seed. More precisely, the nonces are hashed together in the same
order as the commitment publication. In the case of a rolling RANDAO,
the previous seed may be used to initialize the hash.

We make the assumption that at least one participant is honest, that
is, it has indeed chosen a random value and this value was revealed.
This is a necessary condition for the seed to be random. The
randomness could however be biased as this protocol suffers from the
following low-impact weakness: if a malicious participant can make
sure she is the last revealer, then she can choose whether to reveal
its committed value, effectively choosing between two different
predetermined seeds.


Verifiable Delay Function
-------------------------

Verifiable Delay Functions, also called VDFs, are a recent
cryptographic primitive formalized in 2018. They can be seen as a
trapdoor-less timelock: the goal of a VDF is making sure a party
cannot compute a value before a specific time.

This new cryptographic building block is based on modular squaring in
a group of unknown order (e.g. class groups or MPC-generated RSA
groups) that is believed to be expensive and hard to parallelize.

More precisely, the goal of a VDF is for a user to compute a certain
value "h = g^2^T mod N ∈ G" and a proof of correctness "π_h" by
recursive modular squarings of "h". The variables "g", "h", "T" and
"N" are respectively the *challenge*, *solution* (or *output*), the
*difficulty parameter* and the -unknown- *group order*. The main
difference between VDF and timelocks is that the latter offers a
backdoor to efficiently generate the challenge from the solution.

To this day, two main schemes exist for generating VDF proofs:
Wesolowski and Pietrzak. The former presents shorter proofs and is
based on a stronger security assumption (adaptive root assumption)
while the latter is computationally cheaper and based on a weaker
security assumption (low order assumption).


Protocol
========


Randomness generation overview
------------------------------

The randomness generation uses both RANDAO and VDF, based on class
groups and using Wesolowski proofs. It can be summed up as follows. We
first use RANDAO to produce biasable entropy which is used as a VDF
challenge to generate an unbiasable seed (given the adversary cannot
compute the VDF solution before the reveal time ends). To ensure
liveness, we fallback to RANDAO entropy if no VDF output was published
and verified on-chain.

Concretely, the random seed for cycle "n" is a 256-bit long number
computed at the end of cycle "n-1-CONSENSUS_RIGHTS_DELAY". It is the
VDF output (or, in its absence, the RANDAO output) computed from
nonces to which delegates commit during cycle
"n-2-CONSENSUS_RIGHTS_DELAY".

Every "BLOCKS_PER_COMMITMENT" levels, the corresponding block contains
a nonce commitment. More precisely, a block contains a commitment if
and only if its cycle position modulo "BLOCKS_PER_COMMITMENT" is
"BLOCKS_PER_COMMITMENT - 1". The nonce is a 256-bit number generated
by the block proposer and its commitment is included in the block
header. The commitment is simply the hash of the nonce.

The committed nonce must be revealed by the original block proposer
during the nonce revelation phase, that is during the first
"NONCE_REVELATION_THRESHOLD" blocks, of cycle
"n-1-CONSENSUS_RIGHTS_DELAY" under penalty of forfeiting all of its
expected attesting rewards for that cycle. The associated security
deposit and baking rewards are not affected. The RANDAO output is then
computed and stored on-chain as the temporary seed for cycle "n". The
RANDAO output is the bitstring obtained by iterating through the
nonces revealed in cycle "n-1" as follows: initially it is the seed of
cycle "n-1"; at each iteration, the new bitstring is the hash of the
concatenation of the previous bitstring with the iterated revealed
nonce.

A *nonce revelation* is an operation and multiple nonce revelations
can thus be included in a block. A reward "seed_nonce_revelation_tip",
potentially adjusted by the adaptive issuance coefficient and weighted
by constant "SEED_NONCE_REVELATION_TIP_WEIGHT", is given for including
a revelation. Revelations are free operations which do not compete
with transactions for block space. Up to "MAX_ANON_OPS_PER_BLOCK"
revelations, wallet activations and denunciations can be contained in
any given block.

During the rest of the cycle, informally called the VDF revelation
period, any party can query the protocol for the *seed computation
status*, which can be one of the following:(1) the VDF revelation
period has not yet started, i.e. the nonce revelation phase is still
ongoing, (2) a VDF solution has already been successfully submitted,
and (3) no VDF solution has been submitted. In this latter case, the
status also provides the information needed to compute the VDF
solution:  hash seeds for computing the VDF discriminant (a prime
number defining the class group) and the VDF challenge; more precisely
the random seed of cycle "n-1"  for the VDF discriminant and the
current RANDAO output for the VDF challenge. Any party can compute a
VDF solution and publish it on-chain together with a proof of
correctness. If the verification of the solution and proof succeeds,
the seed for cycle "n" is then updated with the solution: its value is
set to be the hash of the RANDAO output and the VDF solution.

A *VDF revelation* is an operation. A reward
"seed_nonce_revelation_tip", potentially adjusted by the adaptive
issuance coefficient and weighted by constant
"SEED_NONCE_REVELATION_TIP_WEIGHT", is given for the first correct VDF
revelation, subsequent VDF revelation operations being discarded.


Randomness generation parameters
--------------------------------

+---------------------------------------------------------+---------------------------+
| Parameter name                                          | Parameter value           |
|=========================================================|===========================|
| "BLOCKS_PER_COMMITMENT"                                 | 240 blocks                |
+---------------------------------------------------------+---------------------------+
| "NONCE_REVELATION_THRESHOLD"                            | 960 blocks                |
+---------------------------------------------------------+---------------------------+
| "MAX_ANON_OPS_PER_BLOCK"                                | 132 revelations           |
+---------------------------------------------------------+---------------------------+
| "SEED_NONCE_REVELATION_TIP_WEIGHT"                      | 1                         |
+---------------------------------------------------------+---------------------------+
| "VDF_DIFFICULTY"                                        | 8,000,000,000             |
+---------------------------------------------------------+---------------------------+

The variables "BLOCKS_PER_CYCLE" and "CONSENSUS_RIGHTS_DELAY" are
already defined in the proof of stake page.

Baking Power
************

The proof-of-stake mechanism used for the consensus algorithm assigns
baking and attesting slots, called **baking rights**, to each delegate
a.k.a. baker. For this selection process, each baker is weighted
according to its **baking power** -- provided that it is active and
meets the minimal power and own staked requirements.

This page details how this baking power is determined from the staked
and non-staked funds owned by the baker itself and all its delegators.

Note that the amendment and voting process is based on each delegate's
voting power instead, which is computed in a similar but simpler way.

Note:

  In this page, the prefix ".../" in all RPC paths is standing for
  "/chain/<chain_id>/blocks/<block_id>/context/"Besides, many RPCs
  presented here used to be known under different names. They have
  been renamed in the Quebec protocol in order to disambiguate and
  normalize the baking power lexicon; the old names have been
  deprecated. See the changelog for more information.


Overview
========

At the end of cycle "n" (that is, the beginning of cycle "n + 1"), the
protocol randomly generates the baking rights for cycle "n + 1 +
CONSENSUS_RIGHTS_DELAY = n + 3", using the **current baking power** as
the weight for each delegate that meets the requirements.
("CONSENSUS_RIGHTS_DELAY = 2" is a protocol constant.)

The ".../delegates/<delegate_pkh>/baking_power" RPC can be used to
retrieve the current baking power of a delegate, that is, its baking
power as of the end of the requested block "<block_id>" (see the note
above on RPC paths). Therefore, the baking power used for the rights
of cycle "n + 3" is the one returned by this RPC called on the last
block of cycle "n".

The baking power of a delegate is defined as:

   baking_power = total_staked_after_limits + (total_delegated_after_limits / 3)

This page explains the relevant concepts and provides the detailed
computations of "total_staked_after_limits" and
"total_delegated_after_limits".


Delegate, delegators, stakers
=============================

A **delegate**, a.k.a. **baker**, is a user account that has
registered as a delegate by emitting a self-"delegation" manager
operation. The list of all registered delegates is queried with the
".../delegates" RPC.

A **delegator** for a given baker is an account that has registered
this baker as its delegate by emitting a "delegation" operation. This
includes the baker itself. A delegator may be a user account or a
smart contract. The list of delegators for a given delegate is queried
with the ".../delegates/<delegate_pkh>/delegators" RPC.

A **staker** is a delegator that has staked tez by emitting a stake
operation. This includes the delegate itself if it has staked funds.
Note that stakers are always user accounts, because smart contracts
cannot emit "stake" operations. The list of a delegate's stakers and
their respective staked balances (see below) are queried with the
".../delegates/<delegate_pkh>/stakers" RPC.

An **external delegator** (resp. **external staker**) is a delegator
(resp. staker) that is not the delegate itself.


Staked tez
==========

Delegates and delegators have the option to stake their tez. **Staked
tez** contribute to the baking power, but they also function as a
security deposit for baking, meaning that they may be slashed if the
delegate misbehaves. That's why they are also known as **frozen
deposits**.

The **staked balance** of an account is its amount of staked tez. It
can be queried with the ".../contracts/<contract_id>/staked_balance"
RPC (in mutez). Note that if an account does not have a delegate, then
it cannot have any staked tez so its staked balance is zero.

For a given delegate, we define the following:

* "own_staked" is the staked balance of the delegate itself. It can be
  queried with either RPC ".../contracts/<contract_id>/staked_balance"
  or ".../delegates/<delegate_pkh>/own_staked" (in mutez).

* "external_staked" is the sum of the staked balances of the
  delegate's external stakers. It is queried with the
  ".../delegates/<delegate_pkh>/external_staked" RPC (in mutez).

* "total_staked" is the sum of the staked balances of all stakers,
  including the delegate itself. It is queried with the
  ".../delegates/<delegate_pkh>/total_staked" RPC (in mutez).

All three values are of course related:

   total_staked = own_staked + external_staked


Delegated tez
=============

Non-staked tez owned by delegates and delegators are called
**delegated tez**. They also contribute to the delegate's baking
power, without being subject to slashing. However, delegated tez weigh
a third as much as staked tez for the purpose of computing the baking
power.


Delegated tez of an account
---------------------------

The **delegated balance** of an account is its amount of delegated
tez. It is the sum of the following balances:

* The **spendable balance** is the amount of tez that the account can
  freely access and spend at the current time. It can be queried with
  RPC ".../contracts/<contract_id>/spendable" (in mutez).

* The **unstaked balance** is the sum of the tez amounts contained in
  all of the account's **unstake requests** (both unfinalizable and
  finalizable unstake requests). These tez have been removed from the
  staked balance via an "unstake" operation, but have not been added
  back to the spendable balance yet; see Staked funds management.
  Unstake requests can be queried with RPC
  ".../contracts/<contract_id>/unstake_requests" (returns a detailed
  view with unfinalizable/finalizable status, delegate-at-creation-
  time, cycle, and amount in mutez).

* The **frozen bonds** are a deposit for rollup commitments. They can
  be queried with RPC ".../contracts/<contract_id>/frozen_bonds" (in
  mutez).

Together, the staked and delegated tez represent all the tez owned by
an account, called the **full balance**.

   delegated = spendable + unstaked + frozen_bonds

   full_balance = staked + delegated


Delegated tez to a baker
------------------------

Spendable tez and frozen bonds count as delegated to the account's
current delegate. However, the tez involved in an unstake request
count as delegated to the account's **delegate at the time of the
unstake request's creation** (which is the account's current delegate
in most cases, but might be a former delegate instead).

To sum up:

   delegated = (spendable + unstaked_from_current_delegate + frozen_bonds) + unstaked_from_former_delegates
   delegated =        delegated_to_current_delegate                        +  delegated_to_former_delegates

   delegated_to_current_delegate = delegated - delegated_to_former_delegates
   delegated_to_current_delegate = full_balance - staked - unstaked_from_former_delegates

For a given delegate, we define the following:

* "own_delegated" is the amount of delegated tez owned by the baker
  and counting as delegated to itself, that is, the
  "delegated_to_current_delegate" amount of the delegate's account. It
  corresponds to all non-staked tez owned by the baker (except for any
  tez involved in unstake requests created at a time when the baker
  was delegating to a different delegate, but it is rare for a
  delegate to still have such requests). It can be queried with RPC
  ".../delegates/<delegate_pkh>/own_delegated" (in mutez).

* "external_delegated" is the sum of tez that count as delegated to
  the baker but are not owned by the baker itself. In other words, it
  is the sum of "delegated_to_current_delegate" over all current
  external delegators, plus any tez involved in unstaked requests
  created by former external delegators when they were still
  delegating to the baker. It can be queried with RPC
  ".../delegates/<delegate_pkh>/external_delegated" (in mutez).

* "total_delegated" is the total amount that counts as delegated to
  this baker. It can be queried with RPC
  ".../delegates/<delegate_pkh>/total_delegated" (in mutez).

   total_delegated = own_delegated + external_delegated


Min-delegated-in-current-cycle
------------------------------

When computing baking rights for cycle "n + 3" at the end of cycle
"n", the "total_delegated" value used for each delegate is actually
the **minimum** of its "total_delegated" **over the whole cycle** "n",
called "min_delegated_in_current_cycle". The purpose of this mechanism
is to prevent any manipulation of baking rights through short-duration
transfers. (Note that such a mechanism is not needed for staked tez
because they are inherently frozen for at least four cycles, so short-
duration staking is already not possible.)

Since the Paris protocol, the considered minimum is the minimum at any
point during block applications, which can be reached in the middle of
executing a transaction.

For example, if a baker transfers tez to one of its delegators, this
is internally treated as first removing the transferred amount from
the total amount delegated to this baker, then adding it back. In
between executing both updates, the total delegated amount is lower so
it might be the new minimum over the whole cycle. In other words, the
transferred tez risk not counting towards the baking rights that will
be computed at the end of the cycle, even though they have been owned
by an account delegating to this baker during the whole cycle.

Besides, when the minimum is reached in the middle of a block's
operations, the context for this minimum is not directly accessible
via RPC. In that case, in order to retrieve this exact context, one
needs to replay the block's balance updates on their own.

In the Quebec protocol, to solve these problems, only the total
delegated amounts **at the end of blocks** count when determining this
minimum. This is known as the **per-block min-delegated feature**.
This solution no longer penalizes baking rights when funds are
transferred between two accounts delegated to the same baker.
Moreover, it lets users easily retrieve via RPC the exact context that
the minimum comes from, since it is guaranteed to correspond to the
end of a block.

The min-delegated-in-current-cycle can be retrieved with RPC "GET /ch
ains/<chain_id>/blocks/<block_id>/context/delegates/<delegate_pkh>/mi
n_delegated_in_current_cycle". This RPC returns the minimal value of
the baker’s "total_delegated" at the end of any block, from the first
block of the current cycle up to the current level, where "current
cycle" and "current level" are those of the "<block_id>" in the RPC
path; it also returns the earliest level at the end of which this
minimum has been reached in the current cycle. This means that calling
the RPC on the last block of cycle "n" returns the value actually used
during the generation of baking rights for cycle "n + 3".


Example
~~~~~~~

At level "150", the baker receives "50" tez from a contract that is
not a delegator for this baker. Then, at level "200", the baker
transfers "150" tez to one of its delegators. Finally, at level "205",
that delegator sends "70" tez to another contract that is not a
delegator for this baker.

The first transfer of "50" tez increases the baker’s "total_delegated"
by "50". Then, the transfer of "150" tez is internally implemented as
removing "150" tez from the total delegated of the sender’s delegate
-- which is the baker itself, then adding "150" tez to the
"total_delegated" of the destination’s delegate -- which is the same
baker in our case. Finally, the "70"-tez transfer just removes "70"
tez from the "total_delegated" of the same baker again.

Let's say that "blocks_per_cycle = 128", so the first level of the
current cycle is "129", and let's say that "total_delegated" was
"1000" at the beginning of the cycle.

+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
|                |                  | L129: first    | L150: add 50   | L200: remove   | L200: add 150  | L205: remove   |
|                |                  | level of cycle |                | 150            |                | 70             |
|================|==================|================|================|================|================|================|
|                | Baker's current  | 1000           | 1050           | 900            | 1050           | 980            |
|                | "total_delegate  |                |                |                |                |                |
|                | d" (tez)         |                |                |                |                |                |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
| Quebec RPC     | Returned min     | 1000           | 1000           |                | 1000           | 980            |
|                | (tez)            |                |                |                |                |                |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
| Quebec RPC     | Returned level   | 129            | 129            |                | 129            | 205            |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
| Paris RPC      | Returned min     | 1000           | 1000           |                | 900            | 900            |
|                | (tez)            |                |                |                |                |                |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
| Paris RPC      | Returned level   | None           | 150            |                | 200            | 200            |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+

Note there are empty cells in the table as RPCs cannot be called in
the middle of the block application. Also, the
"min_delegated_in_current_cycle" RPC returns the value in mutez, but
here we use tez for simplicity.

* In Quebec:

  * At levels "129" and "150", the earliest level at the end of which
    the "total_delegated" is equal to the minimum "1000" is the first
    level of the cycle, that is, level "129".

  * At level "200", the "900" value happens in the middle of the block
    application so it is not considered. The new end-of-block value
    "1050" is not lower than the old minimum of "1000", so
    "min_delegated_in_current_cycle" stays at (min: "1000", level:
    "129").

  * At level "205", the new end-of-block value "980" is lower than the
    old minimum of "1000", so "min_delegated_in_current_cycle" becomes
    (min: "980", level: "205").

* In Paris:

  * At level "129", the "total_delegated" has not changed since the
    start of the cycle. The Paris RPC returns level "None" in this
    case.

  * At level "150", the "total_delegated" has changed since the start
    of the cycle, but the minimum is actually the initial value it had
    at the start of the cycle. In this case, Paris RPC returns the
    earliest level at which the "total_delegated" has changed, that
    is, level "150".

  * At level "200", the "total_delegated" reaches a new minimum "900".
    Indeed, the Paris protocol does consider the values in the middle
    of the block application, so "min_delegated_in_current_cycle"
    becomes (min: "900", level: "200").

  * At level "205", the new value "980" is higher than the old minimum
    of "900", so "min_delegated_in_current_cycle" is still (min:
    "900", level: "200").


Overstaking
===========

The **limit_of_staking_over_baking** is a configurable delegate
parameter that limits how much staked tez the external stakers can
contribute to the baking power, relative to the baker's own staked
tez. It defaults to "0", meaning no staked contribution from external
stakers at all. It can be set to any non-negative value (with a one
millionth precision); however, the
"GLOBAL_LIMIT_OF_STAKING_OVER_BAKING" constant, set to "9", ensures
that external stakers may never contribute more than nine times as
much staked tez as the baker itself, regardless of the delegate's own
limit. If the amount of external staked tez exceeds this quota, the
baker is said to be **overstaked**, and we also call **overstaked**
the excess of external staked tez over the allowed maximum. Any
overstaked tez will count toward the baking power as delegated instead
of staked (provided that the baker is not overdelegated too), so they
will weigh a third as much.

   global_limit_of_staking_over_baking = 9
   actual_limit_of_staking_over_baking = min(limit_of_staking_over_baking, global_limit_of_staking_over_baking)
   max_allowed_external_staked = own_staked * actual_limit_of_staking_over_baking
   external_staked_after_limits = min(external_staked, max_allowed_external_staked)

   total_staked_after_limits = own_staked + external_staked_after_limits

   overstaked = external_staked - external_staked_after_limits

The purpose of this feature is to ensure that the baker's
"own_staked", that is, the part of the security deposit that belongs
to the baker itself, always represents a sizable portion of its baking
power. In other words, it guarantees that the baker always has its own
skin in the game. Besides, the global limit of "9" ensures that a
baker can never increase its own balance by denouncing its own double
baking or double attesting misbehavior; indeed, the reward that would
be given to the author of a denunciation is guaranteed to be lower
than the amount that would be slashed from the misbehaving baker's own
funds.

Note:

  The "limit_of_staking_over_baking" has an additional effect when set
  to "0": it prevents external delegators from using "stake"
  operations at all. This effect is completely removed when the limit
  is positive: external delegators can stake as much tez as they wish,
  even if this causes the baker to become overstaked or if the baker
  is already overstaked. If the limit is set to a positive value then
  back to "0", then external delegators are again prevented from
  staking new funds, but any previously staked tez remain staked
  (although they are now all considered overstaked, so they all count
  as delegated when computing baking the power).


Overdelegation
==============

The amount that counts as *delegated* cannot exceed "9" times the
baker's "own_staked". Any excess tez are called **overdelegated** and
do not contribute to the baking power at all. This mechanism also
contributes to ensuring that all baking rights are covered by
appropriate security deposits.

Recall that the delegated amount used for baking rights is
min_delegated_in_current_cycle, and any overstaked tez count as
delegated too. Therefore:

   total_delegated_after_limits = min(min_delegated_in_current_cycle + overstaked, own_staked * 9)

We finally have everything we need to compute the baking power as
defined above:

   baking_power = total_staked_after_limits + (total_delegated_after_limits / 3)


Minimal power and own staked requirements
=========================================

To receive baking rights, a delegate must meet the following
requirements:

* "baking_power >= MINIMAL_STAKE"

* "own_staked >= MINIMAL_FROZEN_STAKE"

* The delegate must be active

where "MINIMAL_STAKE = 6,000ꜩ" and "MINIMAL_FROZEN_STAKE = 600ꜩ" are
protocol constants.

If any of these conditions is not met at the end of cycle "n", the
delegate still has a *baking power* as computed above, but receives no
*baking rights* at all for cycle "n + 3".

The consensus algorithm
***********************

This document provides a high-level description of Tenderbake, the
Tezos proof-of-stake consensus algorithm.


History
=======

Before Tenderbake, there was Emmy*, a Nakamoto-style consensus
consisting of a series of improvements of the one in the Tezos
whitepaper.

Emmy*, like any Nakamoto-style consensus algorithm (such as Bitcoin or
Ouroboros), offers *probabilistic* finality: forks of arbitrary length
are possible but they collapse with a probability that increases
rapidly with fork length.

Tenderbake instead, like any classic BFT-style consensus algorithm
(such as PBFT or Tendermint), offers *deterministic* finality: a block
that has just been appended to the chain of some node is known to be
final once it has two additional blocks on top of it, regardless of
network latency.


Overview
========

The starting point for Tenderbake is Tendermint, the first classic-
style algorithm for blockchains.

Tenderbake adapts Tendermint to the Tezos blockchain, but the
adjustments required are substantive:

* Tenderbake is tailored to match the Tezos architecture by using only
  communication primitives and network assumptions which Tezos
  supports.

* Tenderbake makes weaker network assumptions than Tendermint, at the
  price of adding the extra assumption that participants have loosely
  synchronized clocks — which is fine, because Tezos already uses
  them.

The design of Tenderbake and its rationale are described at length in
the technical report and in a Nomadic Labs's blog post. Here we only
provide a user/developer perspective.

Tenderbake is executed for each new block level by a "committee" whose
members are called *validators*, which are delegates selected at
random based on their stake, in much the same way as endorsers were
selected in Emmy*. We let "CONSENSUS_COMMITTEE_SIZE" be the number of
validator slots per level. Furthermore, we use "CONSENSUS_THRESHOLD"
to denote two thirds of "CONSENSUS_COMMITTEE_SIZE".

For each level, Tenderbake proceeds in rounds. Each *round* represents
an attempt by the validators to agree on the content of the block for
the current level, that is, on the sequence of non-consensus
operations the block contains. We call this sequence the block's
*payload*.

Each round has an associated duration. Round durations are set to
increase so that for any possible message delay, there is a round that
is sufficiently long for all required messages to be exchanged. Round
durations depend on protocol parameters "MINIMAL_BLOCK_DELAY" and
"DELAY_INCREMENT_PER_ROUND". These parameters specify round durations
as follows:

   round\_duration(0) &= minimal\_block\_delay \\ round\_duration(r+1)
   &= round\_duration(r) + delay\_increment\_per\_round \\ & =
   minimal\_block\_delay + (r + 1) * delay\_increment\_per\_round

Round durations thus increase linearly with
"DELAY_INCREMENT_PER_ROUND".

Schematically, a round consists in the following steps:

* a validator designated for that round injects a *candidate block*
  (representing a proposal) and consensus operations (representing
  votes) into the node to which it is attached, which then

* diffuses those blocks and consensus operations to other nodes of the
  network, and thus

* communicates them to the validators attached to those nodes, to
  carry out voting on which block to accept.

Unlike Emmy*, Tenderbake has two types of votes: before attesting a
block "b", a validator preattests "b". Furthermore, to be able to
attest, a validator must have observed a preattestation *quorum*, that
is a set of preattestations from validators having at least
"CONSENSUS_THRESHOLD" validator slots. Similarly, to be able to
decide, a validator must have observed an attestation quorum, that is,
a set of attestations from validators having at least
"CONSENSUS_THRESHOLD" validator slots. The attestation quorum for a
block "b" is included in a block "b'" on top of "b", serving as a
certification that "b" has been agreed upon. We also say that block
"b'" confirms block "b".

The validator's whose turn is to inject a candidate block at a given
round is called the *proposer* at that round. Proposers in Tenderbake
are selected similarly to bakers in Emmy*: the proposer at round "r"
is the validator who has the validator slot "r". A proposer who has
observed a preattestation quorum for a candidate block in a previous
round, is required to propose a block with the same *payload* as the
initial block. We talk about a *re-proposal* in this case.


Transaction and block finality
==============================

A transaction is final as soon as the block including it has a
confirmation (that is, a block on top of it). Indeed, as hinted above,
a block contains the certification (that is, the attestation quorum)
for the previous payload. Thanks to the attestation quorum, Tenderbake
guarantees **transaction finality after 1 confirmation**.

It may be possible that different validators decide at different
rounds, though on the same payload. The blocks at these different
rounds differ precisely because they contain, in the header, as part
of the block fitness, the round at which they were proposed. Among
these "candidate" blocks, the block with the smallest round has the
highest fitness and so it will be the one decided. Consequently, to
agree on a block, that is, on both the payload and the header,
Tenderbake needs one more confirmation, and thus guarantees **block
finality after 2 confirmations**.


Time between blocks
===================

The time between blocks represents the difference between the
timestamps of the blocks. The timestamp of a block is given by the
beginning of the round at which the block has been agreed upon. Thus,
the time between blocks depends on the round at which decisions are
taken. For example, if the decision at the previous level was taken at
round 4 and at the current level at round 2, then the current block's
delay relative to its predecessor, is round\_duration(4) +
round\_duration(0) + round\_duration(1). The general case is as
follows, say that the decision at the previous level is taken at round
"m" and the decision at the current level is taken at round "n", then
the current block's delay relative to its predecessor is
round\_duration(m) + \sum_{i=0}^{n-1} round\_duration(i). We note
that, under normal network conditions, and with active and compliant
validators, decisions should be taken at round 0, meaning that the
time between blocks would be round\_duration(0) seconds i.e.,
parameter "MINIMAL_BLOCK_DELAY".


Validator selection
===================

Validator selection, that is, baking and attesting rights, are based
on the baking power of a delegate. The baking power is a function of
all tez owned by the delegate and its delegators, with staked tez
weighted more than non-staked tez, and there are additional
considerations such as overstaking and overdelegation; see the baking
power formula.

The baking rights are determined CONSENSUS_RIGHTS_DELAY in advance,
which is currently "2" cycles. More precisely, at the end of cycle "n"
and beginning of cycle "n+1", the baking rights for cycle
"n+1+CONSENSUS_RIGHTS_DELAY=n+3" are randomly generated based on the
current baking power of each delegate that meets the minimal power and
own staked requirements.


Economic Incentives
===================

As Emmy*, Tenderbake rewards participation in consensus and punishes
bad behavior. Notable changes however are as follows:

* Fees and baking rewards go to the payload producer, the one who
  selected the transactions to be included in the block (and was the
  first to propose a block with that payload). In case of re-proposal,
  the payload producer might be different from the block proposer, the
  baker who injects the block.

* Including extra attestations, that is, more than the minimal
  required to obtain a quorum, is rewarded with a bonus.

* Attesting rewards are shared equally among all validators.
  Participation above a minimal threshold per cycle is however
  required.

* Validators are rewarded instantaneously for baking blocks and
  including extra attestations, and not at the end of the cycle like
  in Emmy*.

* At the end of a cycle "c", the following actions happen:

  * the distribution of attesting rewards,

  * the selection of the consensus committee cycle "c +
    CONSENSUS_RIGHTS_DELAY", based on the current distribution of
    baking power.


Fees
----

The fees associated to the transactions included in a block go to the
payload producer. This is only natural given that this is the
validator that selects the transactions to be included; see an in-
depth blog post for further motivation.

The payload producer is usually the same delegate as the block
proposer (that is, the one that signs and injects the block): that's
always true for blocks at round 0; however, in case of re-proposals
this is not necessarily the case (see the algorithm description
above).

Fees are allocated to the payload producer's available balance
immediately, that is, they are already reflected in the blockchain
state obtained after applying the injected block.


Rewards
-------

There are three kinds of rewards: baking rewards, attesting rewards,
and a bonus for including extra attestations.

The baking rewards are treated in the same way as fees: they go to the
*payload* producer and are distributed immediately.

To encourage fairness and participation, the *block* proposer receives
a bonus for the extra attestations it includes in the block. The bonus
is proportional to the number of validator slots above the threshold
of "CONSENSUS_COMMITTEE_SIZE * 2 / 3" that the included attestations
represent. The bonus is also distributed immediately.

The attesting rewards are distributed at the end of the cycle.
Attesting rewards may be received even if not all of the validator's
attestations are included in the cycle's blocks and is proportional to
the validator's baking power (in other words, to its *expected* number
of validator slots, and not its actual number of slots). However, two
conditions must be met:

* the validator has revealed its nonces, and

* the validator has been present during the cycle.

Not giving rewards in case of missing revelations is not new as it is
adapted from Emmy*. The second condition is new. We say that a
delegate is *present* during a cycle if the attesting power (that is,
the number of validator slots at the corresponding level) of all the
delegate's attestations included during the cycle represents at least
"MINIMAL_PARTICIPATION_RATIO" of the delegate's expected number of
validator slots for the current cycle (which is "BLOCKS_PER_CYCLE *
CONSENSUS_COMMITTEE_SIZE * active_stake / total_active_stake").

The concrete values for rewards depend on the issuance which is
dynamically adjusted by Adaptive Issuance. For each block it issues an
amount "total_rewards" of rewarded tez, that varies with the total
amount of tez at stake on the chain. To obtain some concrete values,
we will use as an example the issuance before Adaptive Issuance, which
was "80" tez per minute. With "MINIMAL_BLOCK_DELAY = 8s", this
corresponds to a "total_rewards" per level of 10.67 tez. We define:

* "baking_reward_fixed_portion := baking_reward_ratio * total_rewards"

* "bonus := (1 - baking_reward_ratio) * bonus_ratio * total_rewards"
  is the max bonus

* "attesting_reward := (1 - baking_reward_ratio) * (1 - bonus_ratio) *
  total_rewards"

where:

* "baking_reward_ratio" to "1 / 4",

* "bonus_ratio" to "1 / 3".

Thus, we obtain "baking_reward_fixed_portion = 2.67" tez, (maximum)
"bonus = 2.67" tez, and "attesting_reward = 5.33" tez. The bonus per
additional attestation slot is in turn "bonus /
(CONSENSUS_COMMITTEE_SIZE / 3)" (because there are at most
"CONSENSUS_COMMITTEE_SIZE / 3" validator slots corresponding to the
additional attestations included in a block). The rewards per
attestation slot are "attesting_reward / CONSENSUS_COMMITTEE_SIZE".
Assuming "CONSENSUS_COMMITTEE_SIZE = 7000", we obtain a bonus per slot
of "2.67 / (7000 / 3) = 0.001143" tez and an attesting rewards per
slot of "5.33 / 7000 = 0.000761" tez.

Let's take an example. Say a block has round 1, is proposed by
delegate B, and contains the payload from round 0 produced by delegate
A. Also, B includes attestations with attesting power "5251". Then A
receives the fees and 10 tez (the "baking_reward_fixed_portion") as a
reward for producing the block's payload. Concerning the bonus, given
that "CONSENSUS_COMMITTEE_SIZE = 7000", the minimum required validator
slots is "4667", and there are "2333 = 7000 - 4667" additional
validator slots. Therefore B receives the bonus "(5251 - 4667) *
0.001143 = 0.667512" tez. (Note that B only included attestations
corresponding to "584 = 5251 - 4667" additional validator slots, about
a quarter of the maximum "2333" extra attestations it could have
theoretically included.) Finally, consider some delegate C, whose
baking power at some cycle is 1% of the total stake. Note that his
expected number of validator slots for that cycle is "1/100 *
BLOCKS_PER_CYCLE * CONSENSUS_COMMITTEE_SIZE = 1/100 * 10800 * 7000 =
756,000" slots. Assume also that the attesting power of C's
attestations included during that cycle has been "651,456" slots.
Given that this number is bigger than the minimum required ("756,000 *
2 / 3"), it receives an attesting reward of "756,000 * 0.000761 =
575.316" tez for that cycle.


Slashing
--------

Like in Emmy*, not revealing nonces and double signing are punishable.
If a validator does not reveal its nonces by the end of the cycle, it
does not receive its attesting rewards. If a validator double signs,
that is, it double bakes (which means signing different blocks at the
same level and same round) or it double (pre)attests (which means
voting on two different proposals at the same level and round), a part
of the frozen deposit is slashed. The slashed amount for double baking
is a fixed percentage of the frozen deposit
"PERCENTAGE_OF_FROZEN_DEPOSITS_SLASHED_PER_DOUBLE_BAKING". For double
(pre)attestations, the formula is more complex, as it depends on the
number of attestation slots that participated in the misbehavior; see
Adaptive Slashing for more details. The payload producer that includes
the misbehavior evidence is rewarded "1 /
(GLOBAL_LIMIT_OF_STAKING_OVER_BAKING + 2)" of the slashed amount; the
rest of the slashed amount is burned.

If a delegate's deposit is smaller than the slashed amount, the
deposit is simply emptied.

The evidence for double signing at a given level can be collected by
any accuser and included as a *denunciation* operation in a block in
the same cycle as the double signing or in the "DENUNCIATION_PERIOD"
next cycles.

As soon as a delegate is denounced for any double signing, it is
immediately forbidden from both baking and attesting for at least 2
cycles.

The actual slashing and denunciation rewarding happen at the end of
cycle "n + SLASHING_DELAY" for a misbehavior that happened in cycle
"n".

Note that selfish baking is not an issue in Tenderbake: say we are at
round "r" and the validator which is proposer at round "r+1" does not
(pre)attest at round "r" in the hope that the block at round "r" is
not agreed upon and its turn comes to propose at round "r+1". Under
the assumption that the correct validators have more than two thirds
of the total stake, these correct validators have sufficient power for
agreement to be reached, thus the lack of participation of a selfish
baker does not have an impact.


Fitness
=======

The fitness is given by the tuple "(version, level, locked_round, -
predecessor_round - 1, round)". The current version of the fitness is
2 (version 0 was used by Emmy, and version 1 by Emmy+ and Emmy*). The
fitness encapsulates more information than in Emmy* because Tenderbake
is more complex: recall that blocks at the last level only represent
candidate blocks. In Emmy*, only the level mattered. But in
Tenderbake, we need to, for instance, allow for new blocks at the same
level to be accepted by nodes. Therefore the fitness also includes the
block's round (as the fifth component). Furthermore, we also allow to
change the predecessor block when it has a smaller round. Therefore
the fitness also includes the opposite of predecessor block's round as
the forth component (the predecessor is taken for technical reasons).
Finally, to (partially) enforce the rule on re-proposals, the fitness
also includes, as the third component, the round at which a
preattestation quorum was observed by the baker, if any (this
component can therefore be empty). By the way, preattestations are
present in a block if and only if the locked round component is non-
empty and if so, the locked round has to match the round of the
included preattestations.

Next, we provide two examples of fitness values:
"02::00001000::::ffffffff::00000000" and
"02::00001000::00000000::fffffffe::00000001" (in the hexadecimal
format that one may observe in the node's logs). These two values have
the following components:

* the 1st component, "02", is the fitness version;

* the 2nd component, "00001000", is the block's level (level 4096);

* the 3rd component is the block's locked round: empty in the first
  case, 0 in the second;

* the 4th component is the round of the predecessor block, here 0 in
  the first case and 1 in the second case;

* the 5th component is the block's round: 0 in the first case, 1 in
  the second case.

We recall (see Shell header) that the fitness is, from the shell's
perspective, a sequence of sequences of unsigned bytes and comparison
is done first by the length of the sequence and then lexicographically
(both for the outer sequence, and for each of the inner sequences). So
the first fitness is smaller than the second one, because of the third
component, the empty bitstring being smaller than any other bitstring.


Consensus related protocol parameters
=====================================

+---------------------------------------------------------+---------------------------+
| Parameter name                                          | Parameter value           |
|=========================================================|===========================|
| "CONSENSUS_COMMITTEE_SIZE"                              | 7000                      |
+---------------------------------------------------------+---------------------------+
| "CONSENSUS_THRESHOLD"                                   | "ceil(2 *                 |
|                                                         | CONSENSUS_COMMITTEE_SIZE  |
|                                                         | / 3)" = 4667              |
+---------------------------------------------------------+---------------------------+
| "DENUNCIATION_PERIOD"                                   | 1 cycle                   |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_BLOCK_DELAY"                                   | 8s                        |
+---------------------------------------------------------+---------------------------+
| "BLOCKS_PER_CYCLE"                                      | 10800                     |
+---------------------------------------------------------+---------------------------+
| "DELAY_INCREMENT_PER_ROUND"                             | 4s                        |
+---------------------------------------------------------+---------------------------+
| "CONSENSUS_RIGHTS_DELAY"                                | 2 cycles                  |
+---------------------------------------------------------+---------------------------+
| "GLOBAL_LIMIT_OF_STAKING_OVER_BAKING"                   | 9                         |
+---------------------------------------------------------+---------------------------+
| "LIMIT_OF_DELEGATION_OVER_BAKING"                       | 9                         |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_STAKE"                                         | 6,000 ꜩ                   |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_FROZEN_STAKE"                                  | 600 ꜩ                     |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_PARTICIPATION_RATIO"                           | 2/3                       |
+---------------------------------------------------------+---------------------------+
| "PERCENTAGE_OF_FROZEN_DEPOSITS_SLASHED_PER_DOUBLE_BAKI  | 5%                        |
| NG"                                                     |                           |
+---------------------------------------------------------+---------------------------+
| "SLASHING_DELAY"                                        | 1 cycle                   |
+---------------------------------------------------------+---------------------------+
| "CONSENSUS_KEY_ACTIVATION_DELAY" [1]                    | 2 cycles                  |
+---------------------------------------------------------+---------------------------+
| "ISSUANCE_MODIFICATION_DELAY" [1]                       | 2 cycles                  |
+---------------------------------------------------------+---------------------------+
| "UNSTAKE_FINALIZATION_DELAY" [2]                        | 3 cycles                  |
+---------------------------------------------------------+---------------------------+

The above list of protocol parameters is a subset of the protocol
constants.

[1] These derived constants are automatically set to the same value as
    "CONSENSUS_RIGHTS_DELAY".

[2] This derived constant is automatically set to
    "CONSENSUS_RIGHTS_DELAY + SLASHING_DELAY".


Further External Resources
==========================

* Tenderbake report

* Tenderbake blog post.

* Tenderbake tzip.

The Amendment (and Voting) Process
**********************************

In the Tezos blockchain, the *economic protocol* can be amended.
Specifically, there is an on-chain mechanism to propose changes to the
economic protocol, to vote for-or-against these proposed changes, and,
depending on the result of the vote, to activate these changes or not.

Note that the proposal, voting and activation processes are part of
the economic protocol itself. Consequently the amendment rules
themselves are subject to amendments.

The rest of this page gives more details about the amendment and
voting process.


Periods
=======

The amendment process consists of five *periods*. Each period lasts
for "CYCLES_PER_VOTING_PERIOD" = 14 cycles (that is, around 14 days).
The periods (listed below) typically succeed one to another for a
total duration of approximately 2 months and a half, after which the
whole amendment process starts again.

The five periods are as follows:

* *Proposal period*: During this period, delegates can

  * submit *protocol amendment proposals* (or, simply, *proposals*)
    using the "Proposals" operation (see below);

  * upvote one or several proposals, using the same "Proposals"
    operation.

  Each delegate can submit a maximum of 20 proposals. Duplicates count
  towards this total.

  At the end of a **proposal period**, if participation reaches a
  proposal quorum, the proposal with most support is selected and we
  move to an **exploration period**. Note that support is measured in
  the voting power that delegates supporting the proposal have. E.g.,
  a proposal supported by a single delegate with 600,000 tz of voting
  power has more support than a proposal supported by two delegates
  with 100,000 tz each of voting power.

  If there are no proposals, or a tie between two or more proposals,
  or if participation did not reach the proposal quorum, the process
  moves back to a new **proposal period**.

* *Exploration period*: During this period delegates can cast one Yea,
  Nay, or Pass ballot on the selected proposal. They do so using the
  "Ballot" operation.

  If the voting participation reaches *quorum* and there is a *super-
  majority* of Yea, the process moves to the **cooldown period**. (See
  below for details on participation, quorum, and super-majority.)

  Otherwise the process moves back to the **proposal period**.

* *Cooldown period*: On-chain nothing specific happens during this
  period. Off-chain the delegates can read the proposal with more
  scrutiny, the community can discuss finer points of the proposal,
  the developers can perform additional tests, etc.

  At the end of this period, the process moves to the **promotion
  period**.

* *Promotion period*: During this period, delegates can cast a Yea,
  Nay, or Pass ballots using the "Ballot" operation.

  If the voting participation reaches *quorum* and there is a super-
  majority of Yea, the process moves to the **adoption period**.

  Otherwise the process moves back to the **proposal period**.

* *Adoption period*: On-chain nothing specific happens during this
  period except on the very last block (see below).

  Off-chain the developers release tools that include support for the
  soon-to-be activated protocol, other actors (bakers, indexers, etc.)
  update their infrastructure to support the newly released tools,
  smart-contract developers start working with soon-to-be-available
  features, etc.

  At the very end of the period, the proposal is *activated*. This
  means that the last block of the period is still interpreted by the
  current economic protocol, but the first block after the period is
  interpreted by the new economic protocol (the one that was voted
  in).

  And a new **proposal period** starts.

Warning:

  Thanks to an old spelling mistake, the most enthusiastic voters can
  cast "Yay" ballots. Enthusiasm does however not impact the result of
  the vote and the tools accepting "Yay" consider it as a deprecated
  synonym for "Yea".


Activation
==========

In the last block of the last cycle of an Adoption period (called
activation block), the current Protocol learns about the Protocol into
which the next block should be validated. Once the activation block is
applied, the Octez shell initializes the new protocol and performs the
protocol-stitching process. The first block of the cycle after the
Adoption period belongs to the new Protocol. It does not contain
attestations for the previous block.

After the activation step, the blocks added to the chain are
interpreted in the newly activated protocol. As a result gas costs may
differ, new operations may be available, contracts using new opcodes
may be injected, etc.

Because the amendment process is also part of the economic protocol,
the amendment process now unfolds according to the rules of the newly
activated protocol. As a result the periods may be lengthened or
shortened, a new period might be introduced, a different selection
mechanism may be used, the quorum requirement might differ, etc.


Voting Power
============

When supporting a proposal or casting a Yea, Nay, or Pass ballot, each
delegate has a **voting power** equal to the amount of tez owned by
all its delegators (including the delegate itself of course), no
matter whether they are staked or not.

More precisely, the voting power of a delegate during a voting period
is the sum of its total_staked and total_delegated amounts, measured
in *mutez*, and snapshotted at the beginning of the voting period.
(Note the differences from the validator selection for consensus,
which is based on the more complex baking power instead, and sampled
at the end of an earlier cycle.)


Super-majority and Quorum
=========================

As mentioned above, during either of the **exploration** or
**promotion** periods, delegates can cast ballots using the "Ballot"
operation (see below). In both cases, delegates can cast a single Yea,
Nay, or Pass ballot. A ballot has a weight equal to the delegate's
voting power as detailed above.

For either of these two periods, the process continues to the next
period if the *vote participation* reaches *quorum* and there is a
*super-majority* of Yea.

The *vote participation* is the ratio of all the cumulated stake of
cast ballots (including Pass ballots) to the total stake.

For the first vote, the *quorum* started at 80% of stake. The quorum
is adjusted after each vote as detailed below. This adjustment is
necessary to ensure that the amendment process can continue even if
some delegates stop participating. After each vote the new quorum is
updated based on the exponential moving average of the **vote
participation**:

   new_participation_ema = 0.8 * old_participation_ema + 0.2 * participation

with the following formula:

   new_quorum = 0.2 + (0.7 - 0.2) * old_participation_ema

This formula avoids establishing quorums close to 100% that would be
very difficult to attain, or, conversely, low quorums close to 0%
making little participation chronic, ensuring that the quorums are
lower- and upper-bounded by Quorum caps (0.2 and 0.7, respectively).

The *super-majority* is reached if the cumulated stake of Yea ballots
is greater than 8/10 of the cumulated stake of Yea and Nay ballots.

Note that Pass ballots do not count towards or against the super-
majority; they still counts towards participation and quorum.

More details can be found in the file
src/proto_022_PsRiotum/lib_protocol/amendment.ml.


The Hash and the Protocol
=========================

On the one hand, the voting part of the process revolves around the
**hash of a protocol**. Specifically, a delegate submits a hash of a
protocol, and all the delegates cast ballots on the proposed hash. The
*hash of a protocol* is the hash of the files that constitute the
source code of the protocol.

On the other hand, the **protocol activation** (at the end of the
**adoption period**) revolves around the compiled sources of the
protocol.

Basically, the voting process works on an identifier of the protocol
whilst the activation step works on the protocol itself. Consequently,
if a protocol hash is voted in and the protocol it identifies is
invalid, the activation step fails.


Checking a hash is of a valid protocol
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When a hash is proposed by a delegate, it is usually accompanied by
some blogposts and forum threads on community websites. These should
include directions for testing the proposed protocols. If you cannot
find such directions, do not hesitate to ask.


Localised failures
^^^^^^^^^^^^^^^^^^

It is possible that the activation step fails on a single node or a
few nodes of the network, but succeed on the others. In this case the
nodes with the failure are stuck, but the network as a whole
continues.

The most likely cause for this is nodes that have not been updated and
do not include a new protocol environment version.

If your node becomes stuck, you should start a fresh up-to-date node.

A protocol is *invalid* if its code cannot be compiled (e.g., if the
code is not valid source code), if its code uses functions not present
in the protocol environment, or if it downgrades the protocol
environment version.

If an invalid protocol is voted in, then the activation fails for all
the nodes, and then the chain becomes stuck. This is why it is
important to vote for hashes that designate valid protocols: ones with
sources that are available and that can be compiled.


Operations
==========

There are two operations used by the delegates: **proposals** and
**ballot**.

A *proposals* operation can only be injected during a proposal period.

   Proposals : {
     source: Signature.Public_key_hash.t ;
     period: Voting_period_repr.t ;
     proposals: Protocol_hash.t list ; }

The "source" is the public key hash of the delegate, "period" is the
unique identifier of each voting period and "proposals" is a non-empty
list of maximum 20 protocol hashes. The operation can be submitted
more than once but only as long as the cumulative length of the
proposals lists is less than 20. Duplicate proposals from the same
delegate are accounted for in the maximum number of proposals for that
delegate. However duplicates from the same delegate are not tallied at
the end of the proposal period.

For example, a delegate submits a *proposals* operation for protocols
A and B early in the proposal period, later a new protocol C is
revealed and the delegate submits another *proposals* operation for
protocols B and C. The list of submissions that will be tallied is
[A,B,C].

A *ballot* operation can only be submitted during periods where a vote
happens (e.g. exploration, promotion), and only once per period.

   Ballot : {
     source: Signature.Public_key_hash.t ;
     period: Voting_period_repr.t ;
     proposal: Protocol_hash.t ;
     ballot: Vote_repr.ballot ; }

The fields "source" and "period" are the same as above, while
"proposal" is the currently selected proposal and "ballot" is one of
"Yea", "Nay" or "Pass". The "Pass" vote allows a delegate to
contribute towards the quorum without contributing towards the super-
majority. This is important because, as detailed above, the quorum is
adaptive and that low participation would lower the quorum of the next
vote.

More details on the operations can be found in
src/proto_022_PsRiotum/lib_protocol/operation_repr.ml. The binary
format is described by "octez-client describe unsigned operation".


Client Commands
===============

The Octez client, "octez-client", provides commands for basic
exploration and interaction with the amendment and voting process.


Show
----

The client provides a command to show the status of a voting period.
It displays different information for different kind of periods, as in
the following samples:

   $ octez-client show voting period
   Current period: "proposal"
   Blocks remaining until end of period: 59
   Current proposals:
   PsNa6jTtsRfbGaNSoYXNTNM5A7c3Lji22Yf2ZhpFUjQFC17iZVp 2,400,000 ꜩ

   $ octez-client show voting period
   Current period: "exploration"
   Blocks remaining until end of period: 63
   Current proposal: PsNa6jTtsRfbGaNSoYXNTNM5A7c3Lji22Yf2ZhpFUjQFC17iZVp
   Ballots:
     Yay: 2,400,000 ꜩ
     Nay: 0 ꜩ
     Pass: 0 ꜩ
   Current participation 20.00%, necessary quorum 80.00%
   Current in favor 2,400,000 ꜩ, needed supermajority 1,920,000 ꜩ

   $ octez-client show voting period
   Current period: "cooldown"
   Blocks remaining until end of period: 64
   Current proposal: PsNa6jTtsRfbGaNSoYXNTNM5A7c3Lji22Yf2ZhpFUjQFC17iZVp

The proposal has a total stake of 2,400,000 ꜩ, which may come from a
single ballot from a delegate having a staking balance of 2,400,000 ꜩ
or it may come from multiple ballots from delegates with a combined
stake of 2,400,000 ꜩ.


Submit proposals
----------------

During a proposal period, a list of proposals can be submitted with:

   octez-client submit proposals for <delegate> <proposal1> <proposal2> ...

Remember that each delegate can submit a maximum of 20 protocol hashes
and that duplicates count towards this total. Moreover each proposal
is accepted only if it meets one of the following two conditions:

* the protocol hash was already proposed on the network. In this case
  we can submit an additional proposal that "upvotes" an existing one
  and our voting power are added to the ones already supporting the
  proposal.

* the protocol is known by the node. In particular the first proposer
  of a protocol should be able to successfully inject the protocol in
  its node which performs some checks, compiles and loads the
  protocol.

These are protection measures that the Octez client takes to prevent
the accidental injection of invalid protocols. As mentioned above, it
is still important to check the validity of the protocols that you
vote for as they may have been injected via different means.


Submit ballots
--------------

During either of the **exploration** or **promotion** periods, ballots
can be submitted once with:

   octez-client submit ballot for <delegate> <proposal> <yea|nay|pass>


Further External Resources
==========================

Further details and explanations on the voting procedure can be found
at:

* Governance on-chain on Open Tezos

* Tezos Governance on Tezos Agora.

For more details on the client commands refer to the manual at Client
manual.

For vote related RPCs check the Rio RPCs - Reference under the prefix
"votes/".

For Ledger support refer to Obsidian Systems' documentation.

Accounts and addresses
**********************

The Tezos ledger currently supports two types of accounts that can
hold tokens (and be the destinations of transactions), identified by
distinct addresses:

* A user account (or implicit account) is a non-programmable account,
  whose tokens are spendable and delegatable by the owner of a private
  key. Its address is the hash of the public key, prefixed by "tz1",
  "tz2", "tz3" or "tz4".

* A smart contract is a programmable account, associated with some
  Michelson code and a piece of data storage. The address of a smart
  contract is a unique hash that depends on the operation that led to
  its creation, called origination, prefixed by "KT1". This is why
  smart contracts are also called originated accounts.

Finally, addresses prefixed with "sr1" identify Smart Rollups.


User accounts
=============

From the economic protocol's point of view, user accounts are
considered as a particular case of smart contracts that always succeed
in receiving tokens or tickets, and do nothing else. Transferring
toward a user account succeeds even if the address of the account has
never appeared on the chain before. This is why user accounts are also
called implicit accounts.

Transactions that are signed by the private key corresponding to the
public key hash, *i.e.* address of the account can spend its tokens.
Each prefix for addresses denotes a different cryptographic signing
scheme. They are briefly described below from a user point of view.

The sizes of public keys, secret keys and signatures may differ
between the different schemes but addresses are always 20 bytes long.


"tz1": Ed25519
--------------

Addresses that start with the "tz1" prefix are hashes of Ed25519
public keys and signatures must be produced by using the EdDSA
signature scheme with the Curve25519 curve. This is the default scheme
of Octez when, *e.g.*, generating key pairs. It is also the
recommended cryptographic scheme to use because it offers better
security guarantees than EcDSA and has good performance on most
hardware. It may not be available in all wallets or on all dedicated
chips which is why Tezos supports multiple schemes.


"tz2": Secp256k1
----------------

Addresses that start with the "tz2" prefix are hashes of Secp256k1
public keys and signatures must be produced by using the EcDSA
signature scheme with the Secp256k1 curve. Secp256k1 is notably the
cryptographic scheme used by Bitcoin and Ethereum. This means that
private keys and addresses used on Bitcoin can also be used on Tezos.


"tz3": P-256
------------

Addresses that start with the "tz3" prefix are hashes of P-256 public
keys and signatures must be produced by using the EcDSA signature
scheme with the P-256 curve, also known as Secp256r1. This is one of
the curves for EcDSA recommended by NIST. It is also often the only
cryptographic scheme supported by HSMs (Hardware Security Modules) of
cloud providers.


"tz4": BLS
----------

Addresses that start with the "tz4" prefix are hashes of BLS public
keys and signatures must be produced by using the BLS signature scheme
with the BLS12-381 curve. One particularity of BLS signatures is that
they are aggregatable. This means that multiple signatures can be
aggregated into one, and later verified as having been produced for
the correct expected public keys. This allows for numerous
applications like mutli-signatures schemes, multi-party key exchanges,
signatures compaction, etc. BLS is notably used by Zcash and Ethereum
2.0.


Smart contracts
===============

A transaction to a smart contract address can provide data and
optionally some tokens, and triggers the execution of the code, which
may read and update the storage. The transaction can succeed or fail,
according to the transaction semantics.

Michelson: the language of Smart Contracts in Tezos
***************************************************

This specification gives a detailed formal semantics of the Michelson
language and a short explanation of how smart contracts are executed
and interact in the blockchain.

The language is stack-based, with high level data types and
primitives, and strict static type checking. Its design cherry-picks
traits from several language families. Vigilant readers will notice
direct references to Forth, Scheme, ML and Cat.

A Michelson program is a series of instructions that are run in
sequence: each instruction receives as input the stack resulting from
the previous instruction, and rewrites it for the next one. The stack
contains both immediate values and heap allocated structures. All
values are immutable and garbage collected.

The types of the input and output stack are fixed and monomorphic, and
the program is typechecked before being introduced into the system. No
smart contract execution can fail because an instruction has been
executed on a stack of unexpected length or contents.

This specification gives the complete instruction set, type system and
semantics of the language. It is meant as a precise reference manual,
not an easy introduction. Even though, some examples are provided at
the end of the document and can be read first or at the same time as
the specification. The document also starts with a less formal
explanation of the context: how Michelson code interacts with the
blockchain.


Semantics of smart contracts and transactions
=============================================

The Tezos ledger currently has two types of accounts that can hold
tokens. Accounts can be used in transactions as senders, to send
tokens, or as destinations, to receive tokens.

* User account: non-programmable account whose address is the public
  key hash, prefixed by "tz" and one digit. User accounts are
  sometimes called implicit accounts. A transaction to such an address
  cannot provide data, except tickets.

* Smart contract: programmable account associated to some Michelson
  code, whose address is a unique hash, prefixed by "KT1". A
  transaction to such an address can provide data, and can fail for
  reasons detailed below. Smart contracts are sometimes called
  originated accounts.

Finally, addresses prefixed with "sr1" identify Smart Rollups, which
cannot hold tokens but can be the destination of transactions.

See Accounts and addresses for more details.


Smart contract call semantics
-----------------------------

The only way to call a smart contract is to perform a transaction
operation whose destination is the address of the smart contract.

Alongside their tokens, smart contracts keep a piece of storage. Both
are ruled by a specific logic specified by a Michelson program. A
transaction to a smart contract will provide an input value and in
option some tokens, and in return, the smart contract can modify its
storage and transfer its tokens.

The Michelson program receives as input a stack containing a single
pair whose first element is an input value and second element the
content of the storage space. It must return a stack containing a
single pair whose first element is the list of operations that it
wants to emit, and second element is the new contents of the storage
space. Alternatively, a Michelson program can fail, explicitly using a
specific opcode, or because something went wrong that could not be
caught by the type system (e.g. gas exhaustion).

A bit of polymorphism can be used at contract level, with a
lightweight system of named entrypoints: instead of an input value,
the contract can be called with an entrypoint name and an argument,
and these two components are transformed automatically in a simple and
deterministic way to an input value. This feature is available both
for users and from Michelson code. See the dedicated section.


Transaction semantics
---------------------

On one hand, a smart call may result from an external operation of
kind Transaction. External operations are blockchain operations
included in a block, signed by a user account.

On the other hand, a smart contract call may be emitted as an internal
operation of kind Transaction, either when executing another smart
contract call or when triggering the execution of an outbox message
from a smart rollup.

This section explains the whole semantics of an external transaction,
including the execution of the smart contract called directly by it
and the others smart contracts called indirectly via emitted internal
operations.

The subset of blockchain operations that can be emitted by Michelson
programs as internal operations are of the following kinds:

* Transaction transferring:

  * tokens and optionally tickets to a user account, or

  * tokens and parameters to a smart contract (or, optionally, to a
    specified entrypoint of a smart contract), or

  * parameters to a smart rollup.

* Origination creating a new smart contract from its Michelson source
  code, an initial amount of tokens transferred from the source, and
  an initial storage content.

* Delegation assigning the tokens of the sender account to the stake
  of a user account (without transferring any tokens).

* Contract event delivering live information from a smart contract to
  external applications.

Internal operations are not included in any block, and are not signed.

Internal operations are run in an atomic sequence with the external
operation who generated them, right after it, in depth-first order.

Note that manager operations batches contain a sequence of external
operations signed as a whole by a source user account, which are
executed atomically. For example, in case of a batch of two external
operations, execution proceeds as follows:

   +------+----------------+-------+----------------+
   | op 1 | internal ops 1 |  op 2 | internal ops 2 |
   +------+----------------+-------+----------------+

Smart contracts called by internal operations can in turn also emit
internal operations. The interpreter uses a stack of internal
operations to perform them in depth-first order, as in the following
more detailed example, corresponding to the two external operations
above.

   +-----------+---------------+--------------------------+
   | executing | emissions     | resulting stack          |
   +-----------+---------------+--------------------------+
   | op 1      | 1a, 1b, 1c    | 1a, 1b, 1c               |
   | op 1a     | 1ai, 1aj      | 1ai, 1aj, 1b, 1c         |
   | op 1ai    |               | 1aj, 1b, 1c              |
   | op 1aj    |               | 1b, 1c                   |
   | op 1b     | 1bi           | 1bi, 1c                  |
   | op 1bi    |               | 1c                       |
   | op 1c     |               |                          |
   | op 2      | 2a, 2b        | 2a, 2b                   |
   | op 2a     | 2ai           | 2ai, 2b                  |
   | op 2ai    | 2ai1          | 2ai1, 2b                 |
   | op 2ai1   |               | 2b                       |
   | op 2b     | 2bi           | 2bi                      |
   | op 2bi    | 2bi1          | 2bi1                     |
   | op 2bi1   | 2bi2          | 2bi2                     |
   | op 2bi2   |               |                          |
   +-----------+---------------+--------------------------+


Failures
--------

All transactions can fail for a few reasons, mostly:

* Not enough tokens in the source to spend the specified amount.

* The script took too many execution steps.

* The script failed programmatically using the "FAILWITH" instruction.

External transactions can also fail for these additional reasons:

* The signature of the external operations was wrong.

* The code or initial storage in an origination did not typecheck.

* The parameter in a transfer did not typecheck.

* The destination did not exist.

* The specified entrypoint did not exist.

All these errors cannot happen in internal transactions, as the type
system catches them at operation creation time. In particular,
Michelson has two types to talk about other addresses: "address" and
"contract t". The "address" type merely gives the guarantee that the
value has the form of a Tezos address. The "contract t" type, on the
other hand, guarantees that the address is indeed a valid destination
of transfers whose parameter type is "t". To make a transaction from
Michelson, a value of type "contract t" must be provided, and the type
system checks that the argument to the transaction is indeed of type
"t". Hence, all transactions made from Michelson are well formed by
construction.

In any case, when a failure happens, either total success or total
failure is guaranteed. If a transaction (internal or external) fails,
then the whole sequence fails and all the effects up to the failure
are reverted. These transactions can still be included in blocks, and
the transaction fees are given to the user account who baked the
block.


Language semantics
==================

This specification explains in a symbolic way the computation
performed by the Michelson interpreter on a given program and initial
stack to produce the corresponding resulting stack. The Michelson
interpreter is a pure function: it only builds a result stack from the
elements of an initial one, without affecting its environment. This
semantics is then naturally given in what is called a big step form: a
symbolic definition of a recursive reference interpreter. This
definition takes the form of a list of rules that cover all the
possible inputs of the interpreter (program and stack), and describe
the computation of the corresponding resulting stacks.


Rules form and selection
------------------------

The rules have the main following form.

   > (syntax pattern) / (initial stack pattern)  =>  (result stack pattern)
       iff (conditions)
       where (recursions)
       and (more recursions)

The left hand side of the "=>" sign is used for selecting the rule.
Given a program and an initial stack, one (and only one) rule can be
selected using the following process. First, the toplevel structure of
the program must match the syntax pattern. This is quite simple since
there are only a few non-trivial patterns to deal with instruction
sequences, and the rest is made of trivial patterns that match one
specific instruction. Then, the initial stack must match the initial
stack pattern. Finally, some rules add extra conditions over the
values in the stack that follow the "iff" keyword. Sometimes, several
rules may apply in a given context. In this case, the one that appears
first in this specification is to be selected. If no rule applies, the
result is equivalent to the one for the explicit "FAILWITH"
instruction. This case does not happen on well-typed programs, as
explained in the next section.

The right hand side describes the result of the interpreter if the
rule applies. It consists in a stack pattern, whose parts are either
constants, or elements of the context (program and initial stack) that
have been named on the left hand side of the "=>" sign.


Recursive rules (big step form)
-------------------------------

Sometimes, the result of interpreting a program is derived from the
result of interpreting another one (as in conditionals or function
calls). In these cases, the rule contains a clause of the following
form.

   where (intermediate program) / (intermediate stack)  =>  (partial result)

This means that this rule applies in case interpreting the
intermediate state on the left gives the pattern on the right.

The left hand sign of the "=>" sign is constructed from elements of
the initial state or other partial results, and the right hand side
identify parts that can be used to build the result stack of the rule.

If the partial result pattern does not actually match the result of
the interpretation, then the result of the whole rule is equivalent to
the one for the explicit "FAILWITH" instruction. Again, this case does
not happen on well-typed programs, as explained in the next section.


Format of patterns
------------------

Code patterns are of one of the following syntactical forms.

* "INSTR" (an uppercase identifier) is a simple instruction (e.g.
  "DROP").

* "INSTR (arg) ..." is a compound instruction, whose arguments can be
  code, data or type patterns (e.g. "PUSH nat 3").

* "{ (instr) ; ... }" is a possibly empty sequence of instructions,
  (e.g. "IF { SWAP ; DROP } { DROP }"), nested sequences can drop the
  braces.

* "name" is a pattern that matches any program and names a part of the
  matched program that can be used to build the result.

* "_" is a pattern that matches any instruction.

Stack patterns are of one of the following syntactical forms.

* "[FAILED]" is the special failed state.

* "[]" is the empty stack.

* "(top) : (rest)" is a stack whose top element is matched by the data
  pattern "(top)" on the left, and whose remaining elements are
  matched by the stack pattern "(rest)" on the right (e.g. "x : y :
  rest").

* "name" is a pattern that matches any stack and names it in order to
  use it to build the result.

* "_" is a pattern that matches any stack.

Data patterns are of one of the following syntactical forms.

* integer/natural number literals, (e.g. "3").

* string literals, (e.g. ""contents"").

* raw byte sequence literals (e.g. "0xABCDEF42").

* "Tag" (capitalized) is a symbolic constant, (e.g. "Unit", "True",
  "False").

* "(Tag (arg) ...)" tagged constructed data, (e.g. "(Pair 3 4)").

* a code pattern for first class code values.

* "name" to name a value in order to use it to build the result.

* "_" to match any value.

The domain of instruction names, symbolic constants and data
constructors is fixed by this specification. Michelson does not let
the programmer introduce its own types.

Be aware that the syntax used in the specification may differ from the
concrete syntax. In particular some instructions are annotated with
types that are not present in the concrete language because they are
synthesized by the typechecker.


Shortcuts
---------

Sometimes, it is easier to think (and shorter to write) in terms of
program rewriting than in terms of big step semantics. When it is the
case, and when both are equivalents, we write rules of the form:

   p / S  =>  S''
   where   p' / S'  =>  S''

using the following shortcut:

   p / S  =>  p' / S'

The concrete language also has some syntax sugar to group some common
sequences of operations as one. This is described in this
specification using a simple regular expression style recursive
instruction rewriting.


Introduction to the type system and notations
=============================================

This specification describes a type system for Michelson. To make
things clear, in particular to readers that are not accustomed to
reading formal programming language specifications, it does not give a
typechecking or inference algorithm. It only gives an intentional
definition of what we consider to be well-typed programs. For each
syntactical form, it describes the stacks that are considered well-
typed inputs, and the resulting outputs.

The type system is sound, meaning that if a program can be given a
type, then if run on a well-typed input stack, the interpreter will
never apply an interpretation rule on a stack of unexpected length or
contents. Also, it will never reach a state where it cannot select an
appropriate rule to continue the execution. Well-typed programs do not
block, and do not go wrong.


Type notations
--------------

The specification introduces notations for the types of values, terms
and stacks. Apart from a subset of value types that appear in the form
of type annotations in some places throughout the language, it is
important to understand that this type language only exists in the
specification.

A stack type can be written:

* "[]" for the empty stack.

* "(top) : (rest)" for the stack whose first value has type "(top)"
  and queue has stack type "(rest)".

Instructions, programs and primitives of the language are also typed,
their types are written:

   (type of stack before) -> (type of stack after)

The types of values in the stack are written:

* "identifier" for a primitive data-type (e.g. "bool").

* "identifier (arg)" for a parametric data-type with one parameter
  type "(arg)" (e.g. "list nat").

* "identifier (arg) ..." for a parametric data-type with several
  parameters (e.g. "map string int").

* "[ (type of stack before) -> (type of stack after) ]" for a code
  quotation, (e.g. "[ int : int : [] -> int : [] ]").

* "lambda (arg) (ret)" is a shortcut for "[ (arg) : [] -> (ret) : []
  ]".


Meta type variables
-------------------

The typing rules introduce meta type variables. To be clear, this has
nothing to do with polymorphism, which Michelson does not have. These
variables only live at the specification level, and are used to
express the consistency between the parts of the program. For
instance, the typing rule for the "IF" construct introduces meta
variables to express that both branches must have the same type.

Here are the notations for meta type variables:

* "'a" for a type variable.

* "'A" for a stack type variable.

* "_" for an anonymous type or stack type variable.


Typing rules
------------

The system is syntax directed, meaning that it defines a single typing
rule for each syntax construct. A typing rule restricts the type of
input stacks that are authorized for this syntax construct, links the
output type to the input type, and links both of them to the
subexpressions when needed, using meta type variables.

Typing rules are of the form:

   (syntax pattern)
   :: (type of stack before) -> (type of stack after) [rule-name]
      iff (premises)

Where premises are typing requirements over subprograms or values in
the stack, both of the form "(x) :: (type)", meaning that value "(x)"
must have type "(type)".

A program is shown well-typed if one can find an instance of a rule
that applies to the toplevel program expression, with all meta type
variables replaced by non variable type expressions, and of which all
type requirements in the premises can be proven well-typed in the same
manner. For the reader unfamiliar with formal type systems, this is
called building a typing derivation.

Here is an example typing derivation on a small program that computes
"(x+5)*10" for a given input "x", obtained by instantiating the typing
rules for instructions "PUSH", "ADD" and for the sequence, as found in
the next sections. When instantiating, we replace the "iff" with "by".

   { PUSH nat 5 ; ADD ; PUSH nat 10 ; MUL }
   :: [ nat : [] -> nat : [] ]
      by { PUSH nat 5 ; ADD }
         :: [ nat : [] -> nat : [] ]
            by PUSH nat 5
               :: [ nat : [] -> nat : nat : [] ]
                  by 5 :: nat
           and ADD
               :: [ nat : nat : [] -> nat : [] ]
     and { PUSH nat 10 ; MUL }
         :: [ nat : [] -> nat : [] ]
            by PUSH nat 10
               :: [ nat : [] -> nat : nat : [] ]
                  by 10 :: nat
           and MUL
               :: [ nat : nat : [] -> nat : [] ]

Producing such a typing derivation can be done in a number of manners,
such as unification or abstract interpretation. In the implementation
of Michelson, this is done by performing a recursive symbolic
evaluation of the program on an abstract stack representing the input
type provided by the programmer, and checking that the resulting
symbolic stack is consistent with the expected result, also provided
by the programmer.


Type normalization
------------------

For convenience, Michelson developers can use some compact type
notations for types that do not exist in the language implementation.
These are just shorthands that are expanded into existing types during
the so-called "type normalization".

Currently, the only such notation concerns N-ary pairs (also called
tuples), which are expanded into "right combs", that is, nested binary
pairs with right associativity. See type pair for details.

The node RPC "/helpers/script/normalize_type" is available to
normalize a given Michelson type (see RPCs - OpenAPI reference, within
the protocol-dependent RPCs). This RPC is intended for tool developers
wanting to support the type shorthands in their tools without
reimplementing their normalization. However, one side effect of this
RPC is the stripping of annotations. As a consequence, a tool needing
to preserve annotations on shorthand data types should implement its
own type normalization instead of relying on this RPC.


Side note
---------

As with most type systems, it is incomplete. There are programs that
cannot be given a type in this type system, yet that would not go
wrong if executed. This is a necessary compromise to make the type
system usable. Also, it is important to remember that the
implementation of Michelson does not accept as many programs as the
type system describes as well-typed. This is because the
implementation uses a simple single pass typechecking algorithm, and
does not handle any form of polymorphism.


Types and instructions
======================

The complete sets of Michelson types and instructions are detailed in
the interactive Michelson reference page.

* Specifically, it contains synthesis tables for types and for
  instructions.

* Instructions are also organized by categories.

* Each instruction is precisely defined using typing and semantic
  inference rules.


Removed instructions and types
==============================

Protocol Babylon deprecated the following instructions. Because no
smart contract used these on Mainnet before they got deprecated, they
have been removed. The Michelson type-checker will reject any contract
using them.

* "CREATE_CONTRACT { parameter 'p ; storage 'g ; code ... }": Forge a
  new contract from a literal.

     Γ ⊢ CREATE_CONTRACT { parameter 'p ; storage 'g ; code ... }
     :: key_hash : option key_hash : bool : bool : mutez : 'g : 'S
     ⇒ operation : address : 'S

  There is a new version of this instruction, see its documentation.

* "CREATE_ACCOUNT": Forge an account creation operation.

     Γ ⊢ CREATE_ACCOUNT :: key_hash : option key_hash : bool : mutez : 'S
     ⇒ operation : address : 'S

  Takes as argument the manager, optional delegate, the delegatable
  flag and finally the initial amount taken from the currently
  executed contract. This instruction originates a contract with two
  entrypoints; "%default" of type "unit" that does nothing and "%do"
  of type "lambda unit (list operation)" that executes and returns the
  parameter if the sender is the contract's manager.

* "STEPS_TO_QUOTA": Push the remaining steps before the contract
  execution must terminate.

     Γ ⊢ STEPS_TO_QUOTA :: 'S ⇒ nat : 'S

Protocol Mumbai deprecated the following type. Because no smart
contract used it on Mainnet before it got deprecated, it has been
removed. The Michelson type-checker will reject any contract using it.

* "tx_rollup_l2_address": An address used to identify an account in a
  transaction rollup ledger. It is the hash of a BLS public key, used
  to authenticate layer-2 operations to transfer tickets from this
  account.


Macros
======

In addition to the instructions listed in the interactive Michelson
reference manual, several extensions have been added to the language's
concrete syntax. If you are interacting with the node via RPC,
bypassing the client, which expands away these macros, you will need
to desugar them yourself.

These macros are designed to be unambiguous and reversible, meaning
that errors are reported in terms of desugared syntax. Below you'll
see these macros defined in terms of other syntactic forms. That is
how these macros are seen by the node.


Compare
-------

Syntactic sugar exists for merging "COMPARE" and comparison
combinators, and also for branching.

* "CMP{EQ|NEQ|LT|GT|LE|GE}"

   > CMP(\op) / S  =>  COMPARE ; (\op) / S

* "IF{EQ|NEQ|LT|GT|LE|GE} bt bf"

   > IF(\op) bt bf / S  =>  (\op) ; IF bt bf / S

* "IFCMP{EQ|NEQ|LT|GT|LE|GE} bt bf"

   > IFCMP(\op) / S  =>  COMPARE ; (\op) ; IF bt bf / S


Fail
----

The "FAIL" macros is equivalent to "UNIT; FAILWITH" and is callable in
any context since it does not use its input stack.

* "FAIL"

   > FAIL / S  =>  UNIT; FAILWITH / S


Assertion macros
----------------

All assertion operations are syntactic sugar for conditionals with a
"FAIL" instruction in the appropriate branch. When possible, use them
to increase clarity about illegal states.

* "ASSERT"

   > ASSERT  =>  IF {} {FAIL}

* "ASSERT_{EQ|NEQ|LT|LE|GT|GE}"

   > ASSERT_(\op)  =>  IF(\op) {} {FAIL}

* "ASSERT_CMP{EQ|NEQ|LT|LE|GT|GE}"

   > ASSERT_CMP(\op)  =>  IFCMP(\op) {} {FAIL}

* "ASSERT_NONE"

   > ASSERT_NONE  =>  IF_NONE {} {FAIL}

* "ASSERT_SOME"

   > ASSERT_SOME @x =>  IF_NONE {FAIL} {RENAME @x}

* "ASSERT_LEFT"

   > ASSERT_LEFT @x =>  IF_LEFT {RENAME @x} {FAIL}

* "ASSERT_RIGHT"

   > ASSERT_RIGHT @x =>  IF_LEFT {FAIL} {RENAME @x}


Syntactic Conveniences
----------------------

These macros are simply more convenient syntax for various common
operations.

* "P(\left=A|P(\left)(\right))(\right=I|P(\left)(\right))R": A
  syntactic sugar for building nested pairs. In the case of right
  combs, "PAIR n" is more efficient.

   > PA(\right)R / S => DIP ((\right)R) ; PAIR / S
   > P(\left)IR / S => (\left)R ; PAIR / S
   > P(\left)(\right)R =>  (\left)R ; DIP ((\right)R) ; PAIR / S

A good way to quickly figure which macro to use is to mentally parse
the macro as "P" for pair constructor, "A" for left leaf and "I" for
right leaf. The macro takes as many elements on the stack as there are
leaves and constructs a nested pair with the shape given by its name.

Take the macro "PAPPAIIR" for instance:

   P A  P P A  I    I R
   ( l, ( ( l, r ), r ))

A typing rule can be inferred:

   PAPPAIIR
   :: 'a : 'b : 'c : 'd : 'S  ->  (pair 'a (pair (pair 'b 'c) 'd))

* "UNP(\left=A|P(\left)(\right))(\right=I|P(\left)(\right))R": A
  syntactic sugar for destructing nested pairs. These macros follow
  the same convention as the previous one.

   > UNPA(\right)R / S => UNPAIR ; DIP (UN(\right)R) / S
   > UNP(\left)IR / S => UNPAIR ; UN(\left)R / S
   > UNP(\left)(\right)R => UNPAIR ; DIP (UN(\right)R) ; UN(\left)R / S

* "C[AD]+R": A syntactic sugar for accessing fields in nested pairs.
  In the case of right combs, "CAR k" and "CDR k" are more efficient.

   > CA(\rest=[AD]+)R / S  =>  CAR ; C(\rest)R / S
   > CD(\rest=[AD]+)R / S  =>  CDR ; C(\rest)R / S

* "CAR k": Access the "k" -th part of a right comb of size "n > k +
  1". "CAR 0" is equivalent to "CAR" and in general "CAR k" is
  equivalent to "k" times the "CDR" instruction followed by once the
  "CAR" instruction. Note that this instruction cannot access the last
  element of a right comb; "CDR k" should be used for that.

   > CAR n / S  =>  GET (2n+1) / S

* "CDR k": Access the rightmost element of a right comb of size "k".
  "CDR 0" is a no-op, "CDR 1" is equivalent to "CDR" and in general
  "CDR k" is equivalent to "k" times the "CDR" instruction. Note that
  on a right comb of size "n > k >= 2", "CDR k" will return the right
  comb composed of the same elements but the "k" leftmost ones.

   > CDR n / S  =>  GET (2n) / S

* "IF_SOME bt bf": Inspect an optional value.

   > IF_SOME bt bf / S  =>  IF_NONE bf bt / S

* "IF_RIGHT bt bf": Inspect a value of a union.

   > IF_RIGHT bt bf / S  =>  IF_LEFT bf bt / S

* "SET_CAR": Set the left field of a pair. This is equivalent to
  "SWAP; UPDATE 1".

   > SET_CAR  =>  CDR ; SWAP ; PAIR

* "SET_CDR": Set the right field of a pair. This is equivalent to
  "SWAP; UPDATE 2".

   > SET_CDR  =>  CAR ; PAIR

* "SET_C[AD]+R": A syntactic sugar for setting fields in nested pairs.
  In the case of right combs, "UPDATE n" is more efficient.

   > SET_CA(\rest=[AD]+)R / S   =>
       { DUP ; DIP { CAR ; SET_C(\rest)R } ; CDR ; SWAP ; PAIR } / S
   > SET_CD(\rest=[AD]+)R / S   =>
       { DUP ; DIP { CDR ; SET_C(\rest)R } ; CAR ; PAIR } / S

* "MAP_CAR" code: Transform the left field of a pair.

   > MAP_CAR code  =>  DUP ; CDR ; DIP { CAR ; code } ; SWAP ; PAIR

* "MAP_CDR" code: Transform the right field of a pair.

   > MAP_CDR code  =>  DUP ; CDR ; code ; SWAP ; CAR ; PAIR

* "MAP_C[AD]+R" code: A syntactic sugar for transforming fields in
  nested pairs.

   > MAP_CA(\rest=[AD]+)R code / S   =>
       { DUP ; DIP { CAR ; MAP_C(\rest)R code } ; CDR ; SWAP ; PAIR } / S
   > MAP_CD(\rest=[AD]+)R code / S   =>
       { DUP ; DIP { CDR ; MAP_C(\rest)R code } ; CAR ; PAIR } / S


Concrete syntax
===============

The concrete language is very close to the formal notation of the
specification. Its structure is extremely simple: an expression in the
language can only be one of the five following constructs.

1. An integer in decimal notation.

2. A character string.

3. A byte sequence in hexadecimal notation prefixed by "0x".

4. The application of a primitive to a sequence of expressions.

5. A sequence of expressions.

This simple five cases notation is called Micheline.

In the Tezos protocol, the primitive "constant" with a single
character string applied has special meaning. See Global Constants.


Constants
---------

There are three kinds of constants:

1. Integers or naturals in decimal notation.

2. Strings, with some usual escape sequences: "\n", "\\", "\"".
   Unescaped line-breaks (both "\n" and "\r") cannot appear in a
   Michelson string. Moreover, the current version of Michelson
   restricts strings to be the printable subset of 7-bit ASCII, namely
   characters with codes from within "[32, 126]" range, plus the
   escaped characters mentioned above.

3. Byte sequences in hexadecimal notation, prefixed with "0x".


Differences with the formal notation
------------------------------------

The concrete syntax follows the same lexical conventions as the
specification: instructions are represented by uppercase identifiers,
type constructors by lowercase identifiers, and constant constructors
are capitalized.

All domain specific constants are Micheline constants with specific
formats. Some have two variants accepted by the data type checker: a
readable one in a string, and an optimized one using a more compact
encoding.

* "mutez" amounts are written as naturals.

* "timestamp"s are written either using "RFC3339" notation in a string
  (readable), or as the number of seconds since Epoch (when positive)
  or before Epoch (when negative) (optimized).

* "contract"s, "address"es, "key"s and "signature"s are written as
  strings, in their usual Base58 encoded versions (readable), or as
  their raw bytes (optimized).

* "bls12_381_g1"s and "bls12_381_g2"s are written as their raw bytes,
  using a big-endian point encoding, as specified here.

* "bls12_381_fr"s are written as their raw bytes, using a little-
  endian encoding.

The optimized versions should not reach the RPCs, the protocol code
will convert to optimized by itself when forging operations, storing
to the database, and before hashing to get a canonical representation
of a datum for a given type.

To prevent errors, control flow primitives that take instructions as
parameters require sequences in the concrete syntax.

   IF { instr1_true ; instr2_true ; ... }
      { instr1_false ; instr2_false ; ... }


Main program structure
----------------------

The toplevel of a smart contract file must be an un-delimited sequence
of three primitive applications (in no particular order) that provide
its "code", "parameter" and "storage" fields.

See the next section for a concrete example.


Annotations
===========

The annotation mechanism of Michelson provides ways to better track
data on the stack and to give additional type constraints. Except for
a single exception specified just after, annotations are only here to
add constraints, *i.e.* they cannot turn an otherwise rejected program
into an accepted one. The notable exception to this rule is for
entrypoints: the semantics of the "CONTRACT" and "SELF" instructions
vary depending on their constructor annotations, and some contract
origination may fail due to invalid entrypoint constructor
annotations.

Stack visualization tools like the Michelson's Emacs mode print
annotations associated with each type in the program, as propagated by
the typechecker as well as variable annotations on the types of
elements in the stack. This is useful as a debugging aid.

We distinguish three kinds of annotations:

* type annotations, written ":type_annot",

* variable annotations, written "@var_annot",

* and field or constructors annotations, written "%field_annot".

Note that all annotations are stripped during type normalization.


Type annotations
----------------

Each type can be annotated with at most one type annotation. They are
used to give names to types. For types to be equal, their unnamed
version must be equal and their names must be the same or at least one
type must be unnamed.

For instance, the following Michelson program which put its integer
parameter in the storage is not well typed:

   parameter (int :p) ;
   storage (int :s) ;
   code { UNPAIR ; SWAP ; DROP ; NIL operation ; PAIR }

Whereas this one is:

   parameter (int :p) ;
   storage int ;
   code { UNPAIR ; SWAP ; DROP ; NIL operation ; PAIR }

Inner components of composed typed can also be named.

   (pair :point (int :x_pos) (int :y_pos))

Push-like instructions, that act as constructors, can also be given a
type annotation. The stack type will then have on top a type with a
corresponding name.

   UNIT :t
   :: 'A -> (unit :t) : 'A

   PAIR :t
   :: 'a : 'b : 'S -> (pair :t 'a 'b) : 'S

   SOME :t
   :: 'a : 'S -> (option :t 'a) : 'S

   NONE :t 'a
   :: 'S -> (option :t 'a) : 'S

   LEFT :t 'b
   :: 'a : 'S -> (or :t 'a 'b) : 'S

   RIGHT :t 'a
   :: 'b : 'S -> (or :t 'a 'b) : 'S

   NIL :t 'a
   :: 'S -> (list :t 'a) : 'S

   EMPTY_SET :t 'elt
   :: 'S -> (set :t 'elt) : 'S

   EMPTY_MAP :t 'key 'val
   :: 'S -> (map :t 'key 'val) : 'S

   EMPTY_BIG_MAP :t 'key 'val
   :: 'S -> (big_map :t 'key 'val) : 'S

A no-op instruction "CAST" ensures the top of the stack has the
specified type, and change its type if it is compatible. In
particular, this allows to change or remove type names explicitly.

   CAST 'b
   :: 'a : 'S   ->   'b : 'S
      iff  'a = 'b

   > CAST t / a : S  =>  a : S


Variable annotations
--------------------

Variable annotations can only be used on instructions that produce
elements on the stack. An instruction that produces "n" elements on
the stack can be given at most "n" variable annotations.

The stack type contains both the types of each element in the stack,
as well as an optional variable annotation for each element. In this
sub-section we note:

* "[]" for the empty stack,

* "@annot (top) : (rest)" for the stack whose first value has type
  "(top)" and is annotated with variable annotation "@annot" and whose
  queue has stack type "(rest)".

The instructions which do not accept any variable annotations are:

   DROP
   SWAP
   DIG
   DUG
   IF_NONE
   IF_LEFT
   IF_CONS
   ITER
   IF
   LOOP
   LOOP_LEFT
   DIP
   FAILWITH

The instructions which accept at most one variable annotation are:

   DUP
   PUSH
   UNIT
   SOME
   NONE
   PAIR
   CAR
   CDR
   LEFT
   RIGHT
   NIL
   CONS
   SIZE
   MAP
   MEM
   EMPTY_SET
   EMPTY_MAP
   EMPTY_BIG_MAP
   UPDATE
   GET
   LAMBDA
   LAMBDA_REC
   EXEC
   ADD
   SUB
   CONCAT
   MUL
   OR
   AND
   XOR
   NOT
   ABS
   ISNAT
   INT
   NEG
   EDIV
   LSL
   LSR
   COMPARE
   EQ
   NEQ
   LT
   GT
   LE
   GE
   ADDRESS
   CONTRACT
   SET_DELEGATE
   IMPLICIT_ACCOUNT
   NOW
   LEVEL
   AMOUNT
   BALANCE
   HASH_KEY
   CHECK_SIGNATURE
   BLAKE2B
   SOURCE
   SENDER
   SELF
   SELF_ADDRESS
   CAST
   RENAME
   CHAIN_ID
   NAT
   BYTES

The instructions which accept at most two variable annotations are:

   UNPAIR
   CREATE_CONTRACT

Annotations on instructions that produce multiple elements on the
stack will be used in order, where the first variable annotation is
given to the top-most element on the resulting stack. Instructions
that produce "n" elements on the stack but are given less than "n"
variable annotations will see only their top-most stack type elements
annotated.

   UNPAIR @first @second
   :: pair 'a 'b : 'S
      ->  @first 'a : @second 'b : 'S

   UNPAIR @first
   :: pair 'a 'b : 'S
      ->  @first 'a : 'b : 'S

A no-op instruction "RENAME" allows to rename variables in the stack
or to erase variable annotations in the stack.

   RENAME @new
   :: @old 'a ; 'S -> @new 'a : 'S

   RENAME
   :: @old 'a ; 'S -> 'a : 'S


Field and constructor annotations
---------------------------------

Components of pair types, option types and or types can be annotated
with a field or constructor annotation. This feature is useful to
encode records fields and constructors of sum types.

   (pair :point
         (int %x)
         (int %y))

The previous Michelson type can be used as visual aid to represent the
record type (given in OCaml-like syntax):

   type point = { x : int ; y : int }

Similarly,

   (or :t
       (int %A)
       (or
          (bool %B)
          (pair %C
                (nat %n1)
                (nat %n2))))

can be used to represent the algebraic data type (in OCaml-like
syntax):

   type t =
     | A of int
     | B of bool
     | C of { n1 : nat ; n2 : nat }

Field annotations are part of the type (at the same level as type name
annotations), and so types with differing field names (if present) are
not considered equal.

Instructions that construct elements of composed types can also be
annotated with one or multiple field annotations (in addition to type
and variable annotations).

   PAIR %fst %snd
   :: 'a : 'b : 'S -> (pair ('a %fst) ('b %snd)) : 'S

   LEFT %left %right 'b
   :: 'a : 'S -> (or ('a %left) ('b %right)) : 'S

   RIGHT %left %right 'a
   :: 'b : 'S -> (or ('a %left) ('b %right)) : 'S

To improve readability and robustness, instructions "CAR" and "CDR"
accept one field annotation. For the contract to type check, the name
of the accessed field in the destructed pair must match the one given
here.

   CAR %fst
   :: (pair ('a %fst) 'b) : S -> 'a : 'S

   CDR %snd
   :: (pair 'a ('b %snd)) : S -> 'b : 'S


Syntax
------

Primitive applications can receive one or many annotations.

An annotation is a sequence of characters that matches the regular
expression "@%|@%%|%@|[@:%][_0-9a-zA-Z][_0-9a-zA-Z\.%@]*". Note
however that "@%", "@%%" and "%@" are special annotations and are not
allowed everywhere.

Annotations come after the primitive name and before its potential
arguments.

   (prim @v :t %x arg1 arg2 ...)

Ordering between different kinds of annotations is not significant,
but ordering among annotations of the same kind is. Annotations of the
same kind must be grouped together.

For instance these two annotated instructions are equivalent:

   PAIR :t @my_pair %x %y

   PAIR %x %y :t @my_pair

An annotation can be empty, in this case it will mean *no annotation*
and can be used as a wildcard. For instance, it is useful to annotate
only the right field of a pair instruction "PAIR % %right" or to
ignore field access constraints, *e.g.* in the macro "UNPPAIPAIR %x1 %
%x3 %x4".


Annotations and macros
----------------------

Macros also support annotations, which are propagated on their
expanded forms. As with instructions, macros that produce "n" values
on the stack accept "n" variable annotations.

   DUU+P @annot
   > DUU(\rest=U*)P @annot / S  =>  DIP (DU(\rest)P @annot) ; SWAP / S

   C[AD]+R @annot %field_name
   > CA(\rest=[AD]+)R @annot %field_name / S  =>  CAR ; C(\rest)R @annot %field_name / S
   > CD(\rest=[AD]+)R @annot %field_name / S  =>  CDR ; C(\rest)R @annot %field_name / S

   CMP{EQ|NEQ|LT|GT|LE|GE} @annot
   > CMP(\op) @annot / S  =>  COMPARE ; (\op) @annot / S

The variable annotation on "SET_C[AD]+R" and "MAP_C[AD]+R" annotates
the resulting toplevel pair while its field annotation is used to
check that the modified field is the expected one.

   SET_C[AD]+R @var %field
   > SET_CAR @var %field =>  CDR %field ; SWAP ; PAIR @var
   > SET_CDR @var %field =>  CAR %field ; PAIR @var
   > SET_CA(\rest=[AD]+)R @var %field / S   =>
     { DUP ; DIP { CAR ; SET_C(\rest)R %field } ; CDR ; SWAP ; PAIR @var } / S
   > SET_CD(\rest=[AD]+)R  @var %field/ S   =>
     { DUP ; DIP { CDR ; SET_C(\rest)R %field } ; CAR ; PAIR @var } / S

   MAP_C[AD]+R @var %field code
   > MAP_CAR code  =>  DUP ; CDR ; DIP { CAR %field ; code } ; SWAP ; PAIR @var
   > MAP_CDR code  =>  DUP ; CDR %field ; code ; SWAP ; CAR ; PAIR @var
   > MAP_CA(\rest=[AD]+)R @var %field code / S   =>
     { DUP ; DIP { CAR ; MAP_C(\rest)R %field code } ; CDR ; SWAP ; PAIR @var} / S
   > MAP_CD(\rest=[AD]+)R @var %field code / S   =>
    { DUP ; DIP { CDR ; MAP_C(\rest)R %field code } ; CAR ; PAIR @var} / S

Macros for nested "PAIR" accept multiple annotations. Field
annotations for "PAIR" give names to leaves of the constructed nested
pair, in order.  This next snippet gives examples instead of generic
rewrite rules for readability purposes.

   PAPPAIIR @p %x1 %x2 %x3 %x4
   :: 'a : 'b : 'c : 'd : 'S
      -> @p (pair ('a %x1) (pair (pair ('b %x) ('c %x3)) ('d %x4))) : 'S

   PAPAIR @p %x1 %x2 %x3
   :: 'a : 'b : 'c : 'S  ->  @p (pair ('a %x1) (pair ('b %x) ('c %x3))) : 'S

Annotations for nested "UNPAIR" are deprecated.


Automatic variable and field annotations inferring
--------------------------------------------------

When no annotation is provided by the Michelson programmer, the
typechecker infers some annotations in specific cases. This greatly
helps users track information in the stack for bare contracts.

For unannotated accesses with "CAR" and "CDR" to fields that are named
will be appended (with an additional "." character) to the pair
variable annotation.

   CDAR
   :: @p (pair ('a %foo) (pair %bar ('b %x) ('c %y))) : 'S ->  @p.bar.x 'b : 'S

If fields are not named but the pair is still named in the stack then
".car" or ".cdr" will be appended.

   CDAR
   :: @p (pair 'a (pair 'b 'c)) : 'S ->  @p.cdr.car 'b : 'S

If the original pair is not named in the stack, but a field annotation
is present in the pair type the accessed value will be annotated with
a variable annotation corresponding to the field annotation alone.

   CDAR
   :: (pair ('a %foo) (pair %bar ('b %x) ('c %y))) : 'S ->  @bar.x 'b : 'S

A similar mechanism is used for context dependent instructions:

   ADDRESS  :: @c contract _ : 'S   ->   @c.address address : 'S

   CONTRACT 'p  :: @a address : 'S   ->   @a.contract contract 'p : 'S

   BALANCE :: 'S   ->   @balance mutez : 'S

   SOURCE  :: 'S   ->   @source address : 'S

   SENDER  :: 'S   ->   @sender address : 'S

   SELF  :: 'S   ->   @self contract 'p : 'S

   SELF_ADDRESS  :: 'S   ->   @self address : 'S

   AMOUNT  :: 'S   ->   @amount mutez : 'S

   NOW  :: 'S   ->   @now timestamp : 'S

   LEVEL :: 'S  ->   @level nat : 'S

Inside nested code blocks, bound items on the stack will be given a
default variable name annotation depending on the instruction and
stack type (which can be changed). For instance the annotated typing
rule for "ITER" on lists is:

   ITER body
   :: @l (list 'e) : 'A  ->  'A
      iff body :: [ @l.elt e' : 'A -> 'A ]


Special annotations
-------------------

The special variable annotations "@%" and "@%%" can be used on
instructions "CAR", "CDR", and "UNPAIR". It means to use the accessed
field name (if any) as a name for the value on the stack. The
following typing rule demonstrates their use for instruction "CAR".

   CAR @%
   :: @p (pair ('a %fst) ('b %snd)) : 'S   ->   @fst 'a : 'S

   CAR @%%
   :: @p (pair ('a %fst) ('b %snd)) : 'S   ->   @p.fst 'a : 'S

The special field annotation "%@" can be used on instructions "PAIR",
"LEFT" and "RIGHT". It means to use the variable name annotation in
the stack as a field name for the constructed element. Two examples
with "PAIR" follows, notice the special treatment of annotations with
".".

   PAIR %@ %@
   :: @x 'a : @y 'b : 'S   ->   (pair ('a %x) ('b %y)) : 'S

   PAIR %@ %@
   :: @p.x 'a : @p.y 'b : 'S   ->  @p (pair ('a %x) ('b %y)) : 'S
   :: @p.x 'a : @q.y 'b : 'S   ->  (pair ('a %x) ('b %y)) : 'S


Entrypoints
===========

The specification up to this point has been mostly ignoring existence
of entrypoints: a mechanism of contract level polymorphism. This
mechanism is optional, non intrusive, and transparent to smart
contracts that don't use them. This section is to be read as a patch
over the rest of the specification, introducing rules that apply only
in presence of contracts that make use of entrypoints.


Defining and calling entrypoints
--------------------------------

Entrypoints piggyback on the constructor annotations. A contract with
entrypoints is basically a contract that takes a disjunctive type (a
nesting of "or" types) as the root of its input parameter, decorated
with constructor annotations. An extra check is performed on these
constructor annotations: a contract cannot define two entrypoints with
the same name.

An external transaction can include an entrypoint name alongside the
parameter value. In that case, if there is a constructor annotation
with this name at any position in the nesting of "or" types, the value
is automatically wrapped into the according constructors. If the
transaction specifies an entrypoint, but there is no such constructor
annotation, the transaction fails.

For instance, suppose the following input type.

"parameter (or (or (nat %A) (bool %B)) (or %maybe_C (unit %Z) (string
%C)))"

The input values will be wrapped as in the following examples.

   +------------+-----------+---------------------------------+
   | entrypoint | input     | wrapped input                   |
   +------------+-----------+---------------------------------+
   | %A         | 3         | Left (Left 3)                   |
   | %B         | False     | Left (Right False)              |
   | %C         | "bob"     | Right (Right "bob")             |
   | %Z         | Unit      | Right (Left Unit)               |
   | %maybe_C   | Right "x" | Right (Right "x")               |
   | %maybe_C   | Left Unit | Right (Left Unit)               |
   +------------+-----------+---------------------------------+
   | not given  | value     | value (untouched)               |
   | %BAD       | _         | failure, contract not called    |
   +------------+-----------+---------------------------------+


The "default" entrypoint
------------------------

A special semantics is assigned to the "default" entrypoint. If the
contract does not explicitly declare a "default" entrypoint, then it
is automatically assigned to the root of the parameter type.
Conversely, if the contract is called without specifying an
entrypoint, then it is assumed to be called with the "default"
entrypoint. This behaviour makes the entrypoint system completely
transparent to contracts that do not use it.

This is the case for the previous example, for instance. If a value is
passed to such a contract specifying entrypoint "default", then the
value is fed to the contract untouched, exactly as if no entrypoint
was given.

A non enforced convention is to make the entrypoint "default" of type
unit, and to implement the crediting operation (just receive the
transferred tokens).

A consequence of this semantics is that if the contract uses the
entrypoint system and defines a "default" entrypoint somewhere else
than at the root of the parameter type, then it must provide an
entrypoint for all the paths in the toplevel disjunction. Otherwise,
some parts of the contracts would be dead code.

Another consequence of setting the entrypoint somewhere else than at
the root is that it makes it impossible to send the raw values of the
full parameter type to a contract. A trivial solution for that is to
name the root of the type. The conventional name for that is "root".

Let us recapitulate this by tweaking the names of the previous
example.

"parameter (or %root (or (nat %A) (bool %B)) (or (unit %default)
string))"

The input values will be wrapped as in the following examples.

   +------------+---------------------+-----------------------+
   | entrypoint | input               | wrapped input         |
   +------------+---------------------+-----------------------+
   | %A         | 3                   | Left (Left 3)         |
   | %B         | False               | Left (Right False)    |
   | %default   | Unit                | Right (Left Unit)     |
   | %root      | Right (Right "bob") | Right (Right "bob")   |
   +------------+---------------------+-----------------------+
   | not given  | Unit                | Right (Left Unit)     |
   | %BAD       | _                   | failure, contract not |
   +------------+---------------------+-----------------------+


Calling entrypoints from Michelson
----------------------------------

Michelson code can also produce transactions to a specific entrypoint.

For this, both types "address" and "contract" have the ability to
denote not just an address, but a pair of an address and an
entrypoint. The concrete notation is ""address%entrypoint"". Note that
""address"" is strictly equivalent to ""address%default"", and for
clarity, the second variant is forbidden in the concrete syntax.

When the "TRANSFER_TOKENS" instruction is called, it places the
entrypoint provided in the contract handle in the transaction.

The "CONTRACT t" instruction has a variant "CONTRACT %entrypoint t",
that works as follows. Note that "CONTRACT t" is strictly equivalent
to "CONTRACT %default t", and for clarity, the second variant is
forbidden in the concrete syntax.

   +---------------+---------------------+------------------------------------------+
   | input address | instruction         | output contract                          |
   +---------------+---------------------+------------------------------------------+
   | "addr"        | CONTRACT t          | (Some "addr") if contract exists, has a  |
   |               |                     | default entrypoint of type t, or has no  |
   |               |                     | default entrypoint and parameter type t  |
   +---------------+---------------------+------------------------------------------+
   | "addr%name"   | CONTRACT t          | (Some "addr%name") if addr exists and    |
   +---------------+---------------------+ has an entrypoint %name of type t        |
   | "addr"        | CONTRACT %name t    |                                          |
   +---------------+---------------------+------------------------------------------+
   | "addr%_"      | CONTRACT %_ t       | None                                     |
   +---------------+---------------------+------------------------------------------+

Similarly, the "SELF" instruction has a variant "SELF %entrypoint",
that is only well-typed if the current contract has an entrypoint
named "%entrypoint".

* "SELF %entrypoint"

   :: 'S   ->   contract 'p : 'S
      where   contract 'p is the type of the entrypoint %entrypoint of the current contract

User accounts are considered to have a single "default" entrypoint of
type "Unit".


JSON syntax
===========

Micheline expressions are encoded in JSON like this:

* An integer "N" is an object with a single field ""int"" whose value
  is the decimal representation as a string.

  "{ "int": "N" }"

* A string ""contents"" is an object with a single field ""string""
  whose value is the decimal representation as a string.

  "{ "string": "contents" }"

* A sequence is a JSON array.

  "[ expr, ... ]"

* A primitive application is an object with two fields ""prim"" for
  the primitive name and ""args"" for the arguments (that must contain
  an array). A third optional field ""annots"" contains a list of
  annotations, including their leading "@", "%" or ":" sign.

     "{ "prim": "pair", "args": [ { "prim": "nat", "args": [] }, {
     "prim": "nat", "args": [] } ], "annots": [":t"] }"

As in the concrete syntax, all domain specific constants are encoded
as strings.


Development tools
=================

To ease the development of Michelson scripts, some tools are provided
to Michelson developers.


Emacs mode
----------

Emacs can be used as a practical environment for writing, editing and
debugging Michelson programs. Install it and follow the configuration
instructions in the Michelson Emacs README here.


Interactive toplevel
--------------------

An interactive Michelson toplevel (also known as a REPL) built on the
Mockup mode mode of Octez client is available in
"scripts/michelson_repl.sh", the typical usage is:

   $ octez-client --mode mockup --base-dir /tmp/mockup create mockup
   $ rlwrap scripts/michelson_repl.sh
   > UNIT
     { Stack_elt unit Unit }
   > UNIT
     { Stack_elt unit Unit ; Stack_elt unit Unit }
   > COMPARE
     { Stack_elt int 0 }


Examples
========

Contracts in the system are stored as a piece of code and a global
data storage. The type of the global data of the storage is fixed for
each contract at origination time. This is ensured statically by
checking on origination that the code preserves the type of the global
data. For this, the code of the contract is checked to be of  type
"lambda (pair 'arg 'global) -> (pair (list operation) 'global)" where
"'global" is the type of the original global store given on
origination. The contract also takes a parameter and returns a list of
internal operations, hence the complete calling convention above. The
internal operations are queued for execution when the contract
returns.


Empty contract
--------------

The simplest contract is the contract for which the "parameter" and
"storage" are all of type "unit". This contract is as follows:

   code { CDR ;           # keep the storage
          NIL operation ; # return no internal operation
          PAIR };         # respect the calling convention
   storage unit;
   parameter unit;


Example contract with entrypoints
---------------------------------

The following contract maintains a number in its storage. It has two
entrypoints "add" and "sub" to modify it, and the default entrypoint,
of type "unit" will reset it to "0".

   { parameter (or (or (nat %add) (nat %sub)) (unit %default)) ;
     storage int ;
     code { AMOUNT ; PUSH mutez 0 ; ASSERT_CMPEQ ; UNPAIR ;
            IF_LEFT
              { IF_LEFT { ADD } { SWAP ; SUB } }
              { DROP ; DROP ; PUSH int 0 } ;
            NIL operation ; PAIR } }


Example contract with recursive lambda
--------------------------------------

The following contract computes the factorial of the given parameter
using a recursive function and then saves the result in the storage.

In Michelson regular functions start with a stack containing a single
value, the function argument. If the function is of type "lambda int
int", when calling the function the stack will have just an "int".
Recursive functions start with two values, the argument and the
function itself. Therefore, if the recursive function is of type
"lambda int int" then, when it is being called, the stack will have an
"int" at the top and a "lambda int int" at the bottom.

In this recursive factorial we can see the first branch of the "IF",
this is the base case. The second one performs the recursive call. To
do that, we need to access the function. This is what the "DUP 3"
instruction does. Then we decrement the argument and finally make the
recursive call with "EXEC".

   { parameter int;
     storage int;
     code { CAR ;
            LAMBDA_REC  int int
                        { DUP;
                          EQ;
                          IF { PUSH int 1 }
                             { DUP;
                               DUP 3;
                               PUSH int 1;
                               DUP 4;
                               SUB;
                               EXEC;
                               MUL};
                          DIP { DROP 2 }};
            SWAP;
            EXEC;
            NIL operation;
            PAIR}}


Multisig contract
-----------------

The multisig is a typical access control contract. The ownership of
the multisig contract is shared between "N" participants represented
by their public keys in the contract's storage. Any action on the
multisig contract needs to be signed by "K" participants where the
threshold "K" is also stored in the storage.

To avoid replay of the signatures sent to the contract, the signed
data include not only a description of the action to perform but also
the address of the multisig contract and a counter that gets
incremented at each successful call to the contract.

The multisig commands of Octez command line client use this smart
contract. Moreover, functional correctness of this contract has been
verified using the Coq proof assistant.

   parameter (pair
                (pair :payload
                   (nat %counter) # counter, used to prevent replay attacks
                   (or :action    # payload to sign, represents the requested action
                      (pair :transfer    # transfer tokens
                         (mutez %amount) # amount to transfer
                         (contract %dest unit)) # destination to transfer to
                      (or
                         (option %delegate key_hash) # change the delegate to this address
                         (pair %change_keys          # change the keys controlling the multisig
                            (nat %threshold)         # new threshold
                            (list %keys key)))))     # new list of keys
                (list %sigs (option signature)));    # signatures

   storage (pair (nat %stored_counter) (pair (nat %threshold) (list %keys key))) ;

   code
     {
       UNPAIR ; SWAP ; DUP ; DIP { SWAP } ;
       DIP
         {
           UNPAIR ;
           # pair the payload with the current contract address, to ensure signatures
           # can't be replayed across different contracts if a key is reused.
           DUP ; SELF ; ADDRESS ; CHAIN_ID ; PAIR ; PAIR ;
           PACK ; # form the binary payload that we expect to be signed
           DIP { UNPAIR @counter ; DIP { SWAP } } ; SWAP
         } ;

       # Check that the counters match
       UNPAIR @stored_counter; DIP { SWAP };
       ASSERT_CMPEQ ;

       # Compute the number of valid signatures
       DIP { SWAP } ; UNPAIR @threshold @keys;
       DIP
         {
           # Running count of valid signatures
           PUSH @valid nat 0; SWAP ;
           ITER
             {
               DIP { SWAP } ; SWAP ;
               IF_CONS
                 {
                   IF_SOME
                     { SWAP ;
                       DIP
                         {
                           SWAP ; DIIP { DIP { DUP } ; SWAP } ;
                           # Checks signatures, fails if invalid
                           CHECK_SIGNATURE ; ASSERT ;
                           PUSH nat 1 ; ADD @valid } }
                     { SWAP ; DROP }
                 }
                 {
                   # There were fewer signatures in the list
                   # than keys. Not all signatures must be present, but
                   # they should be marked as absent using the option type.
                   FAIL
                 } ;
               SWAP
             }
         } ;
       # Assert that the threshold is less than or equal to the
       # number of valid signatures.
       ASSERT_CMPLE ;
       DROP ; DROP ;

       # Increment counter and place in storage
       DIP { UNPAIR ; PUSH nat 1 ; ADD @new_counter ; PAIR} ;

       # We have now handled the signature verification part,
       # produce the operation requested by the signers.
       NIL operation ; SWAP ;
       IF_LEFT
         { # Transfer tokens
           UNPAIR ; UNIT ; TRANSFER_TOKENS ; CONS }
         { IF_LEFT {
                     # Change delegate
                     SET_DELEGATE ; CONS }
                   {
                     # Change set of signatures
                     DIP { SWAP ; CAR } ; SWAP ; PAIR ; SWAP }} ;
       PAIR }


Views
-----

Here is an example using views, consisting of two contracts. The first
contract defines two views at toplevel that are named "add_v" and
"mul_v".

   { parameter nat;
     storage nat;
     code { CAR; NIL operation ; PAIR };
     view "add_v" nat nat { UNPAIR; ADD };
     view "mul_v" nat nat { UNPAIR; MUL };
   }

The second contract calls the "add_v" view of the above contract and
obtains a result immediately.

   { parameter (pair nat address) ;
     storage nat ;
     code { CAR ; UNPAIR; VIEW "add_v" nat ;
            IF_SOME { } { FAIL }; NIL operation; PAIR }; }


Full grammar
============

   <data> ::=
     | <int constant>
     | <string constant>
     | <byte sequence constant>
     | Unit
     | True
     | False
     | Pair <data> <data> ...
     | Left <data>
     | Right <data>
     | Some <data>
     | None
     | Lambda_rec <instruction>
     | { <data> ; ... }
     | { Elt <data> <data> ; ... }
     | instruction
   <natural number constant> ::=
     | [0-9]+
   <int constant> ::=
     | <natural number constant>
     | -<natural number constant>
   <string constant> ::=
     | "<string content>*"
   <string content> ::=
     | \"
     | \r
     | \n
     | \t
     | \b
     | \\
     | [^"\]
   <byte sequence constant> ::=
     | 0x[0-9a-fA-F]+
   <instruction> ::=
     | { <instruction> ... }
     | DROP
     | DROP <natural number constant>
     | DUP
     | DUP <natural number constant>
     | SWAP
     | DIG <natural number constant>
     | DUG <natural number constant>
     | PUSH <type> <data>
     | SOME
     | NONE <type>
     | UNIT
     | NEVER
     | IF_NONE { <instruction> ... } { <instruction> ... }
     | PAIR
     | PAIR <natural number constant>
     | CAR
     | CDR
     | UNPAIR
     | UNPAIR <natural number constant>
     | LEFT <type>
     | RIGHT <type>
     | IF_LEFT { <instruction> ... } { <instruction> ... }
     | NIL <type>
     | CONS
     | IF_CONS { <instruction> ... } { <instruction> ... }
     | SIZE
     | EMPTY_SET <comparable type>
     | EMPTY_MAP <comparable type> <type>
     | EMPTY_BIG_MAP <comparable type> <type>
     | MAP { <instruction> ... }
     | ITER { <instruction> ... }
     | MEM
     | GET
     | GET <natural number constant>
     | UPDATE
     | UPDATE <natural number constant>
     | IF { <instruction> ... } { <instruction> ... }
     | LOOP { <instruction> ... }
     | LOOP_LEFT { <instruction> ... }
     | LAMBDA <type> <type> { <instruction> ... }
     | LAMBDA_REC <type> <type> { <instruction> ... }
     | EXEC
     | APPLY
     | DIP { <instruction> ... }
     | DIP <natural number constant> { <instruction> ... }
     | FAILWITH
     | CAST
     | RENAME
     | CONCAT
     | SLICE
     | PACK
     | UNPACK <type>
     | ADD
     | SUB
     | MUL
     | EDIV
     | ABS
     | ISNAT
     | INT
     | NEG
     | LSL
     | LSR
     | OR
     | AND
     | XOR
     | NOT
     | COMPARE
     | EQ
     | NEQ
     | LT
     | GT
     | LE
     | GE
     | SELF
     | SELF_ADDRESS
     | CONTRACT <type>
     | TRANSFER_TOKENS
     | SET_DELEGATE
     | CREATE_CONTRACT { <instruction> ... }
     | IMPLICIT_ACCOUNT
     | VOTING_POWER
     | NOW
     | LEVEL
     | AMOUNT
     | BALANCE
     | CHECK_SIGNATURE
     | BLAKE2B
     | KECCAK
     | SHA3
     | SHA256
     | SHA512
     | HASH_KEY
     | SOURCE
     | SENDER
     | ADDRESS
     | CHAIN_ID
     | TOTAL_VOTING_POWER
     | PAIRING_CHECK
     | SAPLING_EMPTY_STATE <natural number constant>
     | SAPLING_VERIFY_UPDATE
     | TICKET
     | READ_TICKET
     | SPLIT_TICKET
     | JOIN_TICKETS
     | OPEN_CHEST
     | BYTES
     | NAT
   <type> ::=
     | <comparable type>
     | option <type>
     | list <type>
     | set <comparable type>
     | operation
     | contract <type>
     | ticket <comparable type>
     | pair <type> <type> ...
     | or <type> <type>
     | lambda <type> <type>
     | map <comparable type> <type>
     | big_map <comparable type> <type>
     | bls12_381_g1
     | bls12_381_g2
     | bls12_381_fr
     | sapling_transaction <natural number constant>
     | sapling_state <natural number constant>
     | chest
     | chest_key
   <comparable type> ::=
     | unit
     | never
     | bool
     | int
     | nat
     | string
     | chain_id
     | bytes
     | mutez
     | key_hash
     | key
     | signature
     | timestamp
     | address
     | option <comparable type>
     | or <comparable type> <comparable type>
     | pair <comparable type> <comparable type> ...


Reference implementation
========================

The language is implemented in OCaml as follows:

* The lower internal representation is written as a GADT whose type
  parameters encode exactly the typing rules given in this
  specification. In other words, if a program written in this
  representation is accepted by OCaml's typechecker, it is guaranteed
  type-safe. This is of course also valid for programs not handwritten
  but generated by OCaml code, so we are sure that any manipulated
  code is type-safe.

  In the end, what remains to be checked is the encoding of the typing
  rules as OCaml types, which boils down to half a line of code for
  each instruction. Everything else is left to the venerable and well
  trusted OCaml.

* The interpreter is basically the direct transcription of the
  rewriting rules presented above. It takes an instruction, a stack
  and transforms it. OCaml's typechecker ensures that the
  transformation respects the pre and post stack types declared by the
  GADT case for each instruction.

  The only things that remain to be reviewed are value dependent
  choices, such as we did not swap true and false when interpreting
  the IF instruction.

* The input, untyped internal representation is an OCaml ADT with only
  5 grammar constructions: "String", "Int", "Bytes", "Seq" and "Prim".
  It is the target language for the parser, since not all parsable
  programs are well typed, and thus could simply not be constructed
  using the GADT.

* The typechecker is a simple function that recognizes the abstract
  grammar described in section X by pattern matching, producing the
  well-typed, corresponding GADT expressions. It is mostly a checker,
  not a full inferrer, and thus takes some annotations (basically the
  input and output of the program, of lambdas and of uninitialized
  maps and sets). It works by performing a symbolic evaluation of the
  program, transforming a symbolic stack. It only needs one pass over
  the whole program.

  Here again, OCaml does most of the checking, the structure of the
  function is very simple, what we have to check is that we transform
  a "Prim ("If", ...)" into an "If", a "Prim ("Dup", ...)" into a
  "Dup", etc.


TZT, a Syntax extension for writing unit tests
==============================================

This section describes the TZT format, an extension of the Michelson
language allowing to run Michelson unit tests at a finer level than a
full smart contract script. This extension adds syntax to specify an
instruction (or sequence of instructions) to test, a concrete input
stack and the expected output stack.

These unit tests can be useful for both smart contract developers who
need to independently test various parts of the smart contracts they
develop and to the developers of new implementations of the Michelson
interpreter who need to check that their new implementations behave as
the reference implementation by passing a conformance test suite.

Similarly to Michelson scripts, the concrete syntax of TZT unit tests
is Micheline.

TZT unit test files usually have the extension ".tzt". A unit test
file describes a single unit test. It consists of a Micheline sequence
of primitive applications (see Micheline), in no particular order.
This is similar to Michelson scripts but the set of primitives allowed
at the toplevel differ; in Michelson scripts, the allowed toplevel
primitives are "parameter" (mandatory), "storage" (mandatory), "code"
(mandatory), and "view" (optional and repeated). For TZT unit tests,
the toplevel primitives which can be used are:

* "input",

* "code",

* "output",

* "now",

* "sender",

* "source",

* "chain_id",

* "self",

* "parameter",

* "amount",

* "balance",

* "other_contracts", and

* "big_maps".


Mandatory primitives
--------------------

Each of the mandatory primitives "input", "code", and "output" must
occur exactly once in a unit test file in no particular order.

The "input" primitive is used to declare the input stack (see the
syntax of concrete stacks).

The "code" primitive is used to declare the instruction or sequence of
instructions to execute.

The "output" primitive is used to declare if the execution is expected
to succeed or fail and what result is expected from the execution. For
executions expected to succeed, the argument of the "output" primitive
is simply the expected output stack (see the syntax of errors). For
executions expected to fail, the argument is the expected error. In
both cases, the wildcard pattern can be used to omit part of the
expected output.

The simplest test which can be written asserts that executing no
instruction on the empty stack successfully returns the empty stack:

   input {};
   code {};
   output {}

Here is a slightly more involved test which demonstrates the effect of
the SWAP instruction:

   input
     {
       Stack_elt nat 8 ;
       Stack_elt bool False
     };
   code SWAP;
   output
     {
       Stack_elt bool False ;
       Stack_elt nat 8
     }

It is possible to test the effect of several instructions by wrapping
them in a sequence:

   input
     {
       Stack_elt nat 8 ;
       Stack_elt bool False
     };
   code { SWAP ; SWAP };
   output
     {
       Stack_elt nat 8 ;
       Stack_elt bool False
     }

Here is an example showing how to test the "FAILWITH" instruction:

   input {Stack_elt nat 2};
   code FAILWITH;
   output (Failed 2)


Optional primitives
-------------------

Optional primitives are used to set the execution context for the
test. Each of the optional primitives can be used at most once, in no
particular order.

* "amount" (optional, defaults to 0): the amount, expressed in mutez,
  that should be pushed by the AMOUNT instruction

* "balance" (optional, defaults to 0): the balance, expressed in
  mutez, that should be pushed by the BALANCE instruction

* "now" (optional, defaults to ""1970-01-01T00:00:00Z""): the
  timestamp that should be pushed by the NOW instruction

* "sender" (optional, defaults to
  ""tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx""): the sender address that
  should be pushed by the SENDER instruction

* "source" (optional, defaults to
  ""tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx""): the source address that
  should be pushed by the SOURCE instruction

* "chain_id" (optional, defaults to ""NetXdQprcVkpaWU""): the chain
  identifier that should be pushed by the CHAIN_ID instruction

* "self" (optional, defaults to
  ""KT1BEqzn5Wx8uJrZNvuS9DVHmLvG9td3fDLi""): the address that should
  be pushed by the SELF and SELF_ADDRESS instructions

* "parameter" (optional, defaults to "unit"): the type of the
  parameter of the contract pushed by the SELF instruction

* "other_contracts" (optional, defaults to "{}"): mapping between the
  contract addresses that are assumed to exist and their parameter
  types (see the syntax of other contracts specifications)

* "big_maps" (optional, defaults to "{}"): mapping between integers
  representing "big_map" indices and descriptions of big maps (see the
  syntax of extra big maps specifications)

The following test example asserts that the default value for the NOW
instruction is the unix epoch:

   input {};
   code NOW;
   output { Stack_elt timestamp "1970-01-01T00:00:00Z" }

The following example shows how to use the "now" toplevel primitive to
make the NOW instruction return a chosen timestamp:

   input {};
   now "2020-01-08T07:13:51Z";
   code NOW;
   output { Stack_elt timestamp "2020-01-08T07:13:51Z" }


Syntax of concrete stacks
-------------------------

A concrete stack is written as a Micheline sequence whose elements are
of the form "Stack_elt <ty> <x>" where "<x>" is a Michelson value and
"<ty>" is its type. For example, "{ Stack_elt bool True ; Stack_elt
nat 42 }" is a concrete stack of length 2 whose top element is the
boolean "True" and the bottom element is the natural number "42".


Omitting parts of the output
----------------------------

Any part of the "output" specification can be replaced with the
wildcard pattern "_".

For example, let's consider the following test of the "PAIR"
instruction:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {Stack_elt (pair bool string) (Pair True "foo")}

Omitting the "True" argument to the "Pair" primitive can be done as
follows:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {Stack_elt (pair bool string) (Pair _ "foo")}

Omitting the "Pair" primitive:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {Stack_elt (pair bool string) (_ True "foo")}

Omitting the "pair bool string" type:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {Stack_elt _ (Pair True "foo")}

Omitting the resulting stack element:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {_}

Omitting all of the output:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output _

The difference between the last two examples is that "output {_}"
means that the instruction is expected to successfully return a stack
of length 1 while "output _" means that nothing in particular is
expected from the execution of the instruction, not even being
successful.

The wildcard pattern is typically used to omit unspecified aspects of
the Michelson language when writing portable tests; in particular the
cryptographic nonces in values of type "operation" (see the syntax of
concrete operations) or implementation-specific parts of error outputs
(see the syntax of errors).


Output normalization
--------------------

The input and output stacks can use the readable and optimized formats
for Michelson values and even mix the formats; for a test to pass, the
expected output does not need to syntactically match the result of the
execution but only to match up to conversion between optimized and
readable formats; the TZT test runner is responsible for normalizing
the actual output and the expected one to common format. This means in
particular that conversion between readable and optimized formats can
be tested by using "{}" as the "code" instruction sequence to test;
for example these two tests pass:

   input {Stack_elt address 0x0000e7670f32038107a59a2b9cfefae36ea21f5aa63c};
   code {};
   output {Stack_elt address "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"}

   input {Stack_elt address "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"};
   code {};
   output {Stack_elt address 0x0000e7670f32038107a59a2b9cfefae36ea21f5aa63c}

This normalization feature is however incompatible with using the
wildcard pattern in the output; when using wildcards the output must
be formatted using the readable format so the following test does not
pass:

   input {Stack_elt address "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"};
   code {};
   output {Stack_elt _ 0x0000e7670f32038107a59a2b9cfefae36ea21f5aa63c}

but the following test does pass:

   input {Stack_elt address 0x0000e7670f32038107a59a2b9cfefae36ea21f5aa63c};
   code {};
   output {Stack_elt _ "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"}


Syntax of errors
----------------

To test that the execution of an instruction fails, the following
syntaxes can be used instead of the output stack as the argument of
the "output" toplevel primitive to specify which error the instruction
is expected to raise:

* "(StaticError <error description>)": an error occurred before the
  instruction was executed; the error description format is
  unspecified so consider using a wildcard such as "(StaticError _)"
  to write portable tests;

* "(Failed <value>)": the execution reached a "FAILWITH" instruction
  and the topmost element of the stack at this point was "<value>";

* "MutezUnderflow": a mutez subtraction resulted in a negative value.
  This should only happen in the case of the deprecated "mutez" case
  of the "SUB" instruction;

* "Overflow": an overflow was detected. This can happen when an
  addition or multiplication on type "mutez" produces a result which
  is too large to be represented as a value of type "mutez", or when
  the number of bits to shift using the "LSL" or "LSR" instruction is
  too large.

The following example shows how to test a runtime failure; it asserts
that the FAILWITH instruction produces a runtime error containing the
top of the stack.

   input { Stack_elt nat 4 ; Stack_elt bytes 0x };
   code FAILWITH;
   output (Failed 4)

The following example shows how to test type checking failure; it
asserts that the DUP instruction cannot be used on an empty stack.

   input {};
   code DUP;
   output (StaticError _)

The following example shows another kind of static failure: a string
cannot be passed as argument to the DUP instruction.

   input { Stack_elt nat 8 };
   code { DUP "foo" };
   output (StaticError _)


Syntax of concrete operations
-----------------------------

The operation type has no concrete syntax in Michelson. In order to
specify the result of the operation forging instructions
TRANSFER_TOKENS, CREATE_CONTRACT, and SET_DELEGATE , the following
data constructors are added:

* "Transfer_tokens",

* "Create_contract", and

* "Set_delegate".

They take as arguments the inputs of the corresponding operation
forging instructions plus a cryptographic nonce represented as a byte
sequence. The result of "TRANSFER_TOKENS", "CREATE_CONTRACT", and
"SET_DELEGATE" have respectively the following shapes:

* "Transfer_tokens <argument> <amount in mutez> <address of
  destination> <nonce>",

* "Create_contract { <script> } <optional delegate> <initial balance
  in mutez> <initial storage> <nonce>", and

* "Set_delegate <optional delegate> <nonce>".

The computation of the cryptographic nonce is not specified. To write
portable tests, the nonces appearing in output stack expectations
should be replaced by a wildcard pattern.

Here is an example unit test for the "SET_DELEGATE" instruction used
to set the delegate of the current contract to the account at address
"tz1NwQ6hkenkn6aYYio8VnJvjtb4K1pfeU1Z":

   input { Stack_elt (option key_hash) (Some "tz1NwQ6hkenkn6aYYio8VnJvjtb4K1pfeU1Z") } ;
   code SET_DELEGATE ;
   output { Stack_elt operation (Set_delegate (Some "tz1NwQ6hkenkn6aYYio8VnJvjtb4K1pfeU1Z") _) }


Syntax of other contracts specifications
----------------------------------------

The behaviour of the CONTRACT instruction depends on whether or not
its input is the address of a smart contract accepting the expected
type as parameter. To test it, the "other_contract" toplevel primitive
can be used to specify which contracts are assumed to be originated
and which type they accept as parameter.

The mapping given to the "other_contract" toplevel primitive is a
Micheline sequence whose elements have the form "Contract "KT1..."
<ty>" where ""KT1..."" is a valid smart contract address and "<ty>" is
the type of its parameter. Each address should appear at most once and
the order is irrelevant.


Syntax of extra big maps specifications
---------------------------------------

The behaviour of the instructions operating on type big_map depend on
the contents of big maps stored in the context. To test them, the
"big_maps" toplevel primitive can be used to specify the types and
contents of the big maps which are assumed to be present.

The mapping given to the "big_maps" toplevel primitive is a Micheline
sequence whose elements have the form "Big_map <i> <kty> <vty> { Elt
<k1> <v1>; Elt <k2> <v2>; ...}" where "<i>" is an integer (the
identifier of the big map), "<kty>" is the comparable type of keys,
"<vty>" is the type of values, each "<ki>" is of type "<kty>" and each
"<vi>" is of type "<vty>". Each identifier should appear at most once
and the order in which big maps are specified is irrelevant but each
"{ Elt <k1> <v1>; Elt <k2> <v2>; ...}" description of big map contents
should be given in increasing order of keys.

The following example tests the GET instruction in the big_map case:

   big_maps { Big_map 4 string nat { Elt "bar" 42 } };
   input { Stack_elt (big_map string nat) 4 };
   code { PUSH string "foo"; GET };
   output { Stack_elt (option nat) None }

On-chain Views
**************

Views are a mechanism that allows smart contracts to call smart
contracts, and that:

* are read-only: they may depend on the storage of the contract
  declaring the view but cannot modify it nor emit operations (but
  they can call other views),

* take arguments as input in addition to the contract storage,

* return results as output,

* are synchronous: the result is immediately available on the stack of
  the caller contract.


Homonyms
========

Beware that the term *view* has also been used for other kinds of
mechanisms to access some data from the blockchain.

* Off-chain views cannot be called from smart contracts but only by
  off-chain tools such as wallets.

* Legacy on-chain views are a conventional way of making some contract
  data available. Their purpose is subsumed by that of the on-chain
  views defined here, but they are still referenced in FA1.2 and FA2
  standards which predate the introduction of native on-chain views in
  the Hangzhou amendment.

In the following, the term "view" denotes the modern notion of on-
chain views, described in this page.


Execution flow
==============

The execution of a view is included in the operation of the caller’s
contract, but accesses the storage of the declarer’s contract, in
read-only mode. Thus, in terms of execution, views are more like
lambda functions rather than contract entrypoints, Here is an example:

   code {
     ...;
     TRANSFER_TOKENS;
     ...;
     VIEW "view_ex" unit;
     ...;
   }

This contract calls a contract "TRANSFER_TOKENS", and, later on, a
view called "view_ex". No matter if the callee "view_ex" is defined in
the same contract with this caller contract or not, this view will be
executed immediately in the current operation, while the operations
emitted by "TRANSFER_TOKENS" will be executed later on. As a result,
although it may seem that "view_ex" receives the storage modified by
"TRANSFER_TOKENS", this is not the case. In other words, the storage
of the view is the same as when the execution of the contract calling
the view started. In particular, in case of re-entrance, i.e., if a
contract "A" calls a contract "B" that calls a view on "A", the
storage of the view will be the same as when "B" started, not when "A"
started.


Declaration
===========

Views are **declared** at the toplevel of the script of the contract
on which they operate, alongside the contract parameter type, storage
type, and code. To declare a view, the "view" keyword is used; its
syntax is "view name 'arg 'return { instr; ... }" where:

* "name" is a string of at most 31 characters matching the regular
  expression "[a-zA-Z0-9_.%@]*"; it is used to identify the view,
  hence it must be different from the names of the other views
  declared in the same script;

* "'arg" is the type of the argument of the view;

* "'return" is the type of the result returned by the view;

* "{ instr; ... }" is a sequence of instructions of type "lambda (pair
  'arg 'storage_ty) 'return" where "'storage_ty" is the type of the
  storage of the current contract. Certain specific instructions have
  different semantics in views: "BALANCE" represents the current
  amount of mutez held by the contract where the view is declared;
  "SENDER" represents the contract which is calling the view;
  "SELF_ADDRESS" represents the contract declaring the view; "AMOUNT"
  is always 0 mutez.


Forbidden types
===============

Note that in both view input (type "'arg") and view output (type
"'return"), the following types are forbidden: "ticket", "operation",
"big_map" and "sapling_state".

Time-lock
*********

Time-lock is a cryptographic primitive that can be used as part of a
commit & reveal scheme, to provide a guarantee that the information
associated to the commit phase is eventually revealed.

For more background information on time-locks and their uses, see
Timelocks in the Tezos documentation.


Cryptographic design
====================

The time-lock features are supported in Octez by the
Tezos_crypto.Timelock library .

Users first generate a RSA modulus and a symmetric encryption key.
They use authenticated encryption to encrypt a packed Michelson value
(an array of bytes computed with "PACK") and encrypt that encryption
key using a time-lock puzzle. They then combine the RSA modulus, the
time-locked symmetric key, the constant "T" and the encrypted value as
a single value as well (called "chest" in our library).

A proof of decryption can be the symmetric key itself. However, a
malicious user could propose an authenticated ciphertext that does not
yield a valid value even when decrypted with the symmetric key that
was indeed time locked. To avoid this threat, an opening (called
"chest_key" in our library) includes the symmetric key and a proof
that the symmetric key proposed is indeed the one hidden in the time-
lock puzzle. In this way one can differentiate whether the chest or
the chest_key was proposed by a malicious user.


Opcode and types
================

To expose the features of this library, the Michelson language
provides the following types:

* "chest", which represents time-locked arbitrary bytes with the
  necessary public parameters to open it.

* "chest_key", which represents the decryption key, alongside with a
  proof that the key is correct.

and the following opcode:

"open_chest ::  chest_key → chest → time → bytes option"

"open_chest" takes a "chest" and a "chest_key", and produces either
the underlying plaintext or indicates that the "chest_key" is
malicious. If the "chest" is not well-formed (which is possible since
we use authenticated encryption), the empty Byte is returned.


Implementation of the time-lock puzzle
======================================

The implementation of the time-lock puzzle and proof scheme is located
in src/lib_crypto/timelock.ml.

To facilitate the use of time-locks,  commands have also been
implemented in Octez client to generate a "chest" and "chest_key" as
well as to open and verify them. An additional command "precompute"
was implemented to fasten the time-lock "chest" generation.

For more information on the client commands, see cli-commands.


Example
=======

The built-in coin flip contract  gives an example of using time-lock.
Beware this contract is for educational purpose only and is not
secure.

Sapling support
***************

This page first give some details on Sapling and its underlying
concepts, and then describes the support for Sapling in Octez and in
the Michelson language (part of the Tezos protocol). For support in
high-level smart contract languages, see the Sapling documentation in
OpenTezos.


Sapling
=======

Sapling is a protocol enabling privacy-preserving transactions of
fungible tokens in a decentralised environment. It was designed and
implemented by the Electric Coin Company as the last iteration over a
series of previous protocols and academic works starting with the
Zerocoin seminal paper.


Keys
----

Sapling offers a rich set of keys, each allowing different operations.
A *spending key* allows to spend tokens so if it is lost or
compromised the tokens could remain locked or be stolen. From a
spending key it is possible to derive a corresponding *viewing key*
which allows to view all incoming and outgoing transactions. The
viewing key allows the owner of the tokens to check their balance and
transaction history so if compromised there is a complete loss of
privacy. On the other hand a viewing key can willingly be shared with
a third party, for example with an auditor for regulatory compliance
purposes.

A viewing key can also derive several diversified addresses. An
address can be used to receive funds, much like the address of a user
account.

Additionally *proving keys* can be used to allow the creation of
proofs, thus revealing private information, without being able to
spend funds. They are useful for example in case the spending key is
stored in a hardware wallet but we'd like to use our laptop to craft
the transaction and produce the zero-knowledge proofs, which are
computationally too intensive for an embedded device.

More details can be found in the specification document.


Shielded transactions
---------------------

Transactions use Bitcoin's UTXO model with the important difference
that each input and output, instead of containing an amount and an
address, are just cryptographic *commitments*. In order to avoid
double spends, it's important to be able to check that a commitment
has not already been spent. In Bitcoin we just need to check if an
output is also later used as an input to verify if it's already spent.
In Sapling however we can't know because inputs are not linked to
outputs. For this reason for each input of a transaction, the owner
must also publish a *nullifier*, which invalidates it. The nullifier
can only be produced by the owner of a commitment and it's
deterministic so that everybody can check that it hasn't been already
published. Note however that it is not possible to infer which
commitment has been nullified. Transactions of this form are privacy
preserving and are referred to as *shielded*, because they reveal
neither the amount, the sender nor the receiver.

The existing set of transactions is referred to as the *shielded
pool*. Unlike Bitcoin, where everybody can compute the set of unspent
outputs of every user, in Sapling only the owner of a viewing key can
find their outputs and verify that they are not already spent. For
this reason, to an external observer, the shielded pool is always
increasing in size and the more transactions are added the harder it
is to pinpoint the commitments belonging to a user.

When we spend a commitment there is some additional information that
we need to transmit to the recipient in order for them to spend the
corresponding output. This data is encrypted under a symmetric key
resulting from a Diffie-Hellman key exchange using the recipient
address and an ephemeral key. In principle this *ciphertext* can be
transmitted off-chain as it's not needed to verify the integrity of
the pool. For convenience, in Tezos, it is stored together with the
commitment and the nullifier on chain.

For reasons of efficiency the commitments are stored in an incremental
Merkle tree which allows for compact proofs of membership. The root of
the tree is all that is needed to refer to a certain state of the
shielded pool.

In order to ensure the correctness of a transaction, given that there
is information that we wish to remain secret, the spender must also
generate proofs that various good properties are true. Thanks to the
use of SNARKs these proofs are very succinct in size, fast to verify
and they don't reveal any private information.

This model of transaction adapts elegantly to the case when we need to
mint or burn tokens, which is needed to shield or unshield from a
transparent token. It suffices to add more values in the outputs than
in the inputs to mint and to have more in inputs than outputs to burn.


Preventing Malleability
-----------------------

A Sapling transaction contains a "bound_data" field with arbitrary
bytes that gets signed by the user's Sapling key. This field can be
used to bind the transaction to another application's logic. For
example during an unshield operation it is important to include in the
"bound_data" the Tezos account that will receive the unshielded
tokens. Without any "bound_data", a Sapling unshield operation
authorizes any party that submits it to claim the unshielded tokens.
An adversary can intercept the Tezos operation containing the unshield
and resubmit it using its own Tezos account. This malleability attack
is prevented by including the recipient Tezos account in the
"bound_data", which is signed by the owner of the Sapling keys, and
that can be used by the Smart Contract to transfer the tokens.

**All Smart Contracts managing a Sapling shielded pool must include a
`bound_data` field in their unshield operations.**

Note that the "bound_data" field is not encrypted and could leak
information about a transaction, therefore for a transfer (e.g. an
operation with balance 0) it should be left empty. This is not a
concern for shielding and unshielding which are already linking a
shielded and a transparent address.


Privacy guarantees
------------------

We explained that the shielded pool contains one commitment for each
input (spent or not), and one nullifier for each spent input. These
cryptographic commitments hide the amount and the owner of the tokens
they represent. Additionally commitments are unlinkable meaning that
we can not deduce which input is spent to create an output.

It should be noted that the number of inputs and outputs of a
transaction is public, which could help link a class of transactions.
This problem can be mitigated by adding any number of dummy inputs or
outputs at the cost of wasting some space.

The shielded pool communicates with the public ledger by minting and
burning shielded tokens in exchange for public coins. Therefore going
in and out of the shielded pool is public: we know which address
shielded or unshielded and how much. We can among other things infer
the total number of shielded coins.

Timing and network information can also help to deduce some private
information. For example by observing the gossip network we might
learn the IP address of somebody that is submitting a shielded
transaction. This can be mitigated by using TOR.


Good practices
--------------

When blending in a group of people, one should always pay attention to
the size and the variety of the group.

We recommend two good practices. First, do not originate a second
contract if another one has the same functionalities, it will split
the anonymity set.

Second, remember that shielding and unshielding are public operations.
A typical anti-pattern is to shield from tz1-alice 15.3 tez, and then
unshield 15.3 tez to tz1-bob. It's fairly clear from timing and
amounts that Alice transferred 15.3 tez to Bob. To decorrelate the two
transfers it is important to change the amounts, let some time pass
between the two and perform the transactions when there is traffic in
the pool. Similar problems exist in Zcash and they are illustrated in
this introductory blog post.

There are a number of more sophisticated techniques to deanonymise
users using timing of operations, network monitoring, side-channels on
clients and analysis of number of inputs/outputs just to mention a few
(A fistful of Bitcoins is a good first read). We advice users to be
familiar with the use of the TOR network and to use clients developed
specifically to protect their privacy.


Sapling integration
===================

The reference implementation of Sapling, librustzcash, has been
integrated in the Octez codebase, and Sapling features were made
available for production use with the Edo protocol amendment in early
2021.

Librustzcash and the Octez integration implement the protocol
described in this specification, version 2020.1.0.


Michelson support
-----------------

Sapling support uses two new Michelson types "sapling_state" and
"sapling_transaction", and two instructions called
"SAPLING_VERIFY_UPDATE" and "SAPLING_EMPTY_STATE" (see the Michelson
reference for more details).

"SAPLING_EMPTY_STATE" pushes an empty "sapling_state" on the stack.

"SAPLING_VERIFY_UPDATE" takes a transaction and a state and returns an
option type which is "Some(bound_data, balance and updated state)" if
the transaction is correct, "None" otherwise. A transaction has a list
of inputs, outputs, a balance, the root of the Merkle tree containing
its inputs, some bound data and a signature.

The verification part checks:

* the zero-knowledge proofs of all inputs and outputs of the
  transaction, which guarantee several properties of correctness

* a (randomised) signature associated with each input (which
  guarantees that the owner forged the transaction), and the signature
  that binds the whole transaction together and guarantees the
  correctness of the balance

* that the root of the Merkle tree appears in one of the past states
  and that the nullifiers are not already present (i.e. no double
  spending is happening).

All the signatures are over the hash of the data that we wish to sign
and the hash function used is Blake2-b, prefixed with the anti-replay
string. The anti-replay string is the the concatenation of the chain
id and the smart contract address. The same string has to be used by
the client for signing.

If one of the checks fails, the instruction returns None. Otherwise
the function adds to the new state the nullifiers given with each
input and adds the outputs to the Merkle tree, which will produce a
new root. It should be noted that it is possible to generate
transactions referring to an old root, as long as the inputs used were
present in the Merkle tree with that root and were not spent after. In
particular, the protocol keeps 120 previous roots and guarantees that
roots are updated only once per block. Considering 2 blocks per minute
and that each block contains at least one call to the same contract, a
client has 1 hour to have its transaction accepted before it is
considered invalid.

The nullifiers are stored in a set. The ciphertexts and other relevant
information linked to the commitment of the Merkle tree are stored in
a map indexed by the position of the commitment in the Merkle tree.

Lastly the instruction pushes on the stack an option with the bound
data, the balance, and the updated state.

A smart contract typically shields or unshields tokens if the balance
is positive or negative, and simply updates the state if the balance
is zero. Additionally in case of an unshield, it must use the bound
data to authorize the transfer of unshielded tokens. For example it
could convert the bound data to a public-key hash and use it as
recipient address of a Tezos transfer.


Example contracts
-----------------


Shielded tez
~~~~~~~~~~~~

An example contract implementing a shielded pool of tokens with a 1 to
1 conversion rate to mutez is available in the tests of the protocol
at src/proto_022_PsRiotum/lib_protocol/test/integration/michelson/con
tracts/sapling_contract.tz.


Simple Voting Contract
~~~~~~~~~~~~~~~~~~~~~~

One might envision using Sapling to implement private voting (i.e., a
secret ballot). It is possible to adapt shielded transactions to
express and quantify preferences. **However, this is not what Sapling
is designed for and it does not provide the same properties as an
actual private voting protocol.**

A naive idea is the following.

Suppose we want a set of users to express a preference for option A or
B. We can generate two Sapling keys with two addresses that are
published and represent the two options. The contract lets each user
create a token which represents one vote that can then be transferred
to address A or B. Using the published viewing keys everyone can check
the outcome of the vote.

However note that:

* A transaction can be replayed and we can see the balance of A or B
  going up.

* This system does not offer ballot privacy.

Therefore, this idea may be usable in some practical situations, but
**we recommend in general not to use it for any important vote.**

Note that using a random elliptic curve element as incoming viewing
key (IVK) allows to generate a dummy address that cannot be spent.
This eases the counting of the votes. To ensure that the IVK does not
correspond to a normal address with spending key, one can use the
Fiat-Shamir heuristic.


Fees issue
----------

The Sapling integration in Tezos exhibits a privacy issue that Z-cash
doesn't have. When interacting with a shielded pool one interacts with
a smart contract via a normal transaction and therefore have to pay
fees from a user account. One could guess that private transactions
whose fees are paid by the same user account are from the same user.
This can be mitigated by making a service that act as a proxy by
forwarding the user transactions and paying its fees. The user would
then include in the transaction a shielded output for the service that
covers the fees plus a small bonus to pay the service. This output can
be open by the service before sending the transaction to check that
there is enough money to cover its fees.

As usually done for mitigating other Z-cash privacy issues, users
interacting with the proxy should use TOR or mitigate network analysis
as they wish.


RPCs
----

There are two Sapling RPCs under the prefix "context/sapling".
"get_size" returns a pair with the size of the set of commitments and
the size of the set of nullifiers. "get_diff" takes two optional
starting offsets "cm_from" and "nf_from" and returns the sapling state
that was added from the offsets to the current size. In particular it
returns three lists:

* commitments,

* ciphertexts from position "cm_from" up to the last one added, and

* nullifiers from "nf_from" to the last one added.

Additionally it returns the last computed root of the Merkle tree so
that a client updating its tree using the diff can verify the
correctness of the result.


Client support
--------------


Wallet
~~~~~~

"octez-client" supports Sapling keys and can send shielded
transactions to smart contracts.

The client supports two ways to generate a new Sapling spending key.
It can be generated from a mnemonic using BIP39, so that it can be
recovered in case of loss using the mnemonic. Alternatively it is
possible to derive new keys from existing ones using ZIP32, a Sapling
variant of BIP32 for hierarchical deterministic wallets. As usual, in
this case it is important to note the derivation path of the key to be
able to recover it in case of loss. At the moment there is no hardware
wallet support, keys are stored in "~/.tezos-client/sapling_keys" by
default encrypted with a password. **Users should take care to backup
this file.**

The client can also derive addresses from viewing keys. By default
addresses are generated using an increasing counter called the address
index. Not all indexes correspond to valid addresses for each key so
it is normal to see an increasing counter that occasionally skips a
few positions.

The client binds each newly generated key to a specific smart contract
address.


Operations
~~~~~~~~~~

The client also facilitates the creation of shielded transactions and
their transfer as arguments of smart contracts. For now there is
seamless integration to send transactions to the reference shielded-
tez contract and there are plans to support a larger class of
contracts.

For the shielded-tez smart contract, the client supports shielding,
unshielding and shielded transactions. In the case of shielded
transactions there are two commands, one to forge a transaction and
save it to file and one to submit it to the smart contract. The idea
is that a user should not use their own transparent tz{1,2,3} address
to submit a shielded address but rather have a third party inject it.


Code base
---------

The current code-base is organized in three main components. There is
a core library called "lib_sapling" which binds "librustzcash", adds
all the data structures necessary to run the Sapling protocol and
includes a simple client and baker. Under the protocol directory there
is a "lib_client_sapling" library which implements a full client
capable of handling Sapling keys and forging transactions. Lastly in
the protocol there is a efficient implementation of the Sapling
storage, in the spirit of "big_map"s, and the integration of
"SAPLING_VERIFY_UPDATE" in the Michelson interpreter.


Protocol
~~~~~~~~

In order to make the Sapling library available to the protocol is has
been exposed through the environment that sandboxes the protocol. The
changes to src/lib_protocol_environment were delivered as part of
version V1 of the environment.

There are two main forms of support in the economic protocol: the
storage for Sapling and the addition of "SAPLING_VERIFY_UPDATE" to the
Michelson interpreter.

Given that the storage of a Sapling contract can be substantially
large, it is important to provide an efficient implementation.
Similarly to what is done for big_maps, the storage of Sapling can't
be entirely deserialized and modified in memory but only a diff of the
changes is kept by the interpreter and applied at the end of each
smart contract call.

The Michelson language offers two special-purpose types:
"sapling_state" and "sapling_transaction", and one instruction:
"SAPLING_VERIFY_UPDATE".


Client
~~~~~~

Under "lib_client_sapling" there is the client integration with the
support for Sapling keys and forging of transactions. The main
difference from the "lib_client" library is the need for the Sapling
client to keep an additional state, for each contract. Because Sapling
uses a UTXO model it is necessary for a client to compute the set of
unspent outputs in order to forge new transactions. Computing this set
requires scanning all the state of a contract which can be expensive.
For this reason the client keeps a local state of the unspent outputs
after the last synchronization and updates it before performing any
Sapling command. The update is done using the RPCs to recover the new
updates since the last known position.

The state of all Sapling contracts is stored in "~/.tezos-
client/sapling_states". This file can be regenerated from the chain in
case of loss. However disclosure of this file will reveal the balance
and the unspent outputs of all viewing keys.


Memo
~~~~

Sapling offers the possibility to add an arbitrary memo to any created
output. The memo is encrypted and available to anyone owning the
outgoing viewing key or the spending key. For privacy reasons the size
of the memo is fixed per contract and it is chosen at origination
time. A transaction containing an output with a different memo-size
will be rejected. Our client adds a default zero-filled message of the
right length. If a message is provided with the "--message" option,
the client will pad it or truncate it if necessary. A warning message
is printed only if the user's message is truncated.


Sandbox tutorial
----------------

Let us show how to test the system end-to-end using the Sandboxed
mode. After having set up the sandbox and originated the contract, a
good way to get familiar with the system is to generate keys and then
perform the full cycle of shielding, shielded transfer and
unshielding.

   # set up the sandbox
   ./src/bin_node/octez-sandboxed-node.sh 1 --connections 0 &
   eval `./src/bin_client/octez-init-sandboxed-client.sh 1`
   octez-activate-alpha

   # originate the contract with its initial empty sapling storage,
   # bake a block to include it.
   # { } represents an empty Sapling state.
   octez-client originate contract shielded-tez transferring 0 from bootstrap1 \
   running src/proto_022_PsRiotum/lib_protocol/test/integration/michelson/contracts/sapling_contract.tz \
   --init '{ }' --burn-cap 3 &
   octez-client bake for bootstrap1

   # if necessary, you can get information from the octez-client manual
   octez-client sapling man

   # generate two shielded keys for Alice and Bob and use them for the shielded-tez contract
   # the memo size has to be indicated
   octez-client sapling gen key alice
   octez-client sapling use key alice for contract shielded-tez --memo-size 8
   octez-client sapling gen key bob
   octez-client sapling use key bob for contract shielded-tez --memo-size 8

   # generate an address for Alice to receive shielded tokens.
   octez-client sapling gen address alice
   zet1AliceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # Alice's address


   # shield 10 tez from bootstrap1 to alice
   octez-client sapling shield 10 from bootstrap1 to zet1AliceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX using shielded-tez --burn-cap 2 &
   octez-client bake for bootstrap1
   octez-client sapling get balance for alice in contract shielded-tez

   # generate an address for Bob to receive shielded tokens.
   octez-client sapling gen address bob
   zet1BobXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # Bob's address

   # forge a shielded transaction from alice to bob that is saved to a file
   octez-client sapling forge transaction 10 from alice to zet1BobXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX using shielded-tez

   # submit the shielded transaction from any transparent account
   octez-client sapling submit sapling_transaction from bootstrap2 using shielded-tez --burn-cap 1 &
   octez-client bake for bootstrap1
   octez-client sapling get balance for bob in contract shielded-tez

   # unshield from bob to any transparent account
   octez-client sapling unshield 10 from bob to bootstrap1 using shielded-tez --burn-cap 1
   ctrl+z # to put the process in background
   octez-client bake for bootstrap1
   fg # to put resume the transfer

Liquidity Baking
****************

Liquidity baking incentivizes large amounts of decentralized liquidity
provision between tez and tzBTC by minting a small amount of tez every
block and depositing it inside of a constant product market making
smart-contract.


Contracts
=========

During activation of Granada protocol, a constant product market
making (CPMM) Michelson contract has been deployed on the chain with
address "KT1TxqZ8QtKvLu3V3JH7Gx58n7Co8pgtpQU5" as well as an
associated liquidity token contract (LQT) with address
"KT1AafHA1C1vk959wvHWBispY9Y2f3fxBUUo".

Warning:

  While the CPMM and LQT contract originations provide an
  "Origination_result", the LQT contract contains two big maps not
  included in a "lazy_storage_diff" field. Indexers and other tooling
  may need manual updates to include these.

The CPMM maintains a balance of "a" tez and "b" tzBTC, where tzBTC is
the FA1.2 token  found at address
"KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn". The smart contract accepts
deposits of "da" tez and returns "db" tzBTC (or vice versa) where the
invariant "(a + da * (1 - f - n)) * (b - db) = a b" is preserved, and
"f" and "n" are a fee and burn, set at 0.1% each. Calculations are
done with precision of 1000, rounding down on division.

To implement this contract, we use a fork of the open source code base
used by version two of the "Dexter" project. The implementation of
this contract has been formally verified against its functional
specification. The contract code is modified in the following way:

1. The fee is set to 0.1% only (the fee in Dexter v2 is set to 0.3%).
   Rationale: given the subsidy it is not necessary to charge a large
   fee and better to improve liquidity.

2. An additional 0.1% of every trade is burned by being transferred to
   the null user account. Rationale: this mechanism offsets inflation
   from the subsidy. The inflation is exactly balanced at a daily
   trade volume of 7.2 million tez.

3. The ability to set a delegate has been removed. Rationale: the
   subsidy means there is no need for a baker for that contract and
   having one would create an imbalance.

4. The ability to set a manager has been removed. Rationale: the only
   privilege of the Dexter manager is to set Dexter's delegate so this
   role is now unnecessary.

The LIGO and Michelson code for these contracts, as well as detailed
documentation, can be found on the liquidity baking branch of the
Dexter 2 repository.


Subsidy
=======

At every block in the chain, a small amount of tez is minted and
credited to the CPMM contract, and the CPMM's "%default" entrypoint is
called to update the "xtz_pool" balance in its storage. The amount
that is minted and sent to the CPMM contract for each block is a
protocol constant ("LIQUIDITY_BAKING_SUBSIDY"), defining a fixed
target rate of 5 tez/minute. One can easily compute the value of the
per block subsidy by taking into account the minimal block time, under
the assumption that all blocks are produced at round 0.

So the credits to the CPMM contract can be accounted for by indexers,
they are included in block metadata as a balance update with a new
constructor for "update_origin", "Subsidy".


Toggle vote
===========

The subsidy can be paused by a mechanism called the Liquidity Baking
Toggle Vote. At every block, the baker producing the block includes a
flag that requests ending the subsidy or on the contrary continuing or
restarting it. The context maintains an exponential moving average of
that flag. The baker has three options for this flag: "Off" to request
ending the subsidy, "On" to request continuing or restarting the
subsidy, and "Pass" to abstain.

"e[n+1] = e[n]" if the flag is set to "Pass". "e[n+1] = (1999 * e[n]
// 2000) + 1_000_000" if the flag is set to "Off". "e[n+1] = (1999 *
e[n] // 2000)" if the flag is set to "On". When computing "e[n+1]",
the division is rounded toward "1_000_000_000".

If at any block "e[n] >= LIQUIDITY_BAKING_EMA_THRESHOLD" (currently
"1_000_000_000") then it means that an exponential moving average with
a window size on the order of two thousand non-abstaining blocks has
had roughly at least a half of the blocks demanding the end of the
subsidy. If that is the case, the subsidy is halted but can be
reactivated if for some later block "e[n] <
LIQUIDITY_BAKING_EMA_THRESHOLD".

For indicative purposes, if among the non-abstaining blocks a fraction
"f" of blocks use it to request ending the subsidy, the threshold is
reached after roughly "2*(log(1-1/(2f)) / log(0.999))" non-abstaining
blocks, about 1386 blocks if everyone signals, 1963 blocks if 80% do,
3583 blocks if 60% do etc. Recall for comparison that assuming six
blocks per minute there are 8640 blocks per day.

When producing blocks using Octez baking daemon "octez-baker", there
are two command-line options affecting toggle vote. The "--liquidity-
baking-toggle-vote <on|off|pass>" option sets a static value to be
used in each block. Note that this option must be placed **after**
"run" on the command-line. Moreover, the path of a JSON file can be
given to the "--votefile <path>" option e.g. "octez-baker-<protocol
codename> run with local node ~/.tezos-node alice --liquidity-baking-
toggle-vote on --votefile "per_block_votes.json"", or placed in a
default location: "per_block_votes.json" in the current working
directory **or** in the client data directory (e.g. "~/.tezos-
client/per_block_votes.json"); the former location takes precedence.
The content of the JSON file will be repeatedly submitted on each
baked block, where "per_block_votes.json" contains just
"{"liquidity_baking_toggle_vote": "pass"}" (to abstain),
"{"liquidity_baking_toggle_vote": "off"}" (to request ending the
subsidy), or "{"liquidity_baking_toggle_vote": "on"}" (to request
continuing the subsidy). When the "--votefile" option is present it
takes precedence over "--liquidity-baking-toggle-vote". If the JSON
file is deleted or becomes malformed while the baker is running, the
last valid value read is used. If neither a valid vote file is
provided nor a CLI value given, the baker will fail on the first block
after it was started. See also the baker man page.

Global Constants
****************

The size limit for Michelson contracts is quite small, limited to 60
kilobytes as of Granada protocol. Global constants are a feature added
in Hangzhou protocol that enables the re-use of user-defined Micheline
chunks in Michelson scripts, allowing for larger and more complex
contracts on the chain. It works in the following way:

* Fragments of Michelson code (written in the Micheline format) are
  registered on the chain via a new operation
  "register_global_constant". An example expression might be the
  integer "999" or the lambda expression "{ PUSH int 999; ADD }"

* Included in the receipt of the operation is a hash of the expression
  registered. For example the hash "999" is
  "expruQN5r2umbZVHy6WynYM8f71F8zS4AERz9bugF8UkPBEqrHLuU8".

* Constants can be referenced inside a Michelson script with the new
  primitive "constant". For example, we could write a lambda
  equivalent to the one above like so: "{ PUSH int (constant
  "expruQN5r2umbZVHy6WynYM8f71F8zS4AERz9bugF8UkPBEqrHLuU8"); ADD }"


Global Constant Registration
============================

The new "register_global_constant" operation includes an object with a
single key ""value"", the value of which is the Micheline expression
to be registered.

You can submit this operation conveniently through a new octez-client
command. For example, the command:

   octez-client register global constant "999" from bootstrap1 --burn-cap 0.017

would result in the output:

   Node is bootstrapped.
   Estimated gas: 1440 units (will add 100 for safety)
   Estimated storage: 68 bytes added (will add 20 for safety)
   Operation successfully injected in the node.
   Operation hash is 'onsFknW5iWa6eiTYqAghY4peQZ7JYQUJg5fR8MwAQkMKjXfNqGf'
   NOT waiting for the operation to be included.
   Use command
     octez-client wait for onsFknW5iWa6eiTYqAghY4peQZ7JYQUJg5fR8MwAQkMKjXfNqGf to be included --confirmations 5 --branch BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU
   and/or an external block explorer to make sure that it has been included.
   This sequence of operations was run:
     Manager signed operations:
       From: tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx
       Fee to the baker: ꜩ0.000385
       Expected counter: 1
       Gas limit: 1540
       Storage limit: 88 bytes
       Balance updates:
         tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx ................ -ꜩ0.000385
         fees(the baker who will include this operation,0) ... +ꜩ0.000385
       Register Global:
         Value: 999
         This global constant registration was successfully applied
         Balance updates:
           tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx ... -ꜩ0.017
         Consumed gas: 1440
         Storage size: 68 bytes
         Global address: expruQN5r2umbZVHy6WynYM8f71F8zS4AERz9bugF8UkPBEqrHLuU8

As you can see, the address of the constant is returned in the
operation receipt in the field "Global address". This address is the
Base58-encode Blake2b hash of the binary serialization of the
registered Micheline expression. This means constants are content-
addressable - given a particular Micheline expression, you can always
calculate its on-chain address and check if it’s registered.

A few points about registering global constants:

* Global constants may contain references to other constants; however,
  any referenced constants must already be registered on the chain. As
  a corollary, you cannot have cyclic references.

* Global constants are not type-checked before registration - any
  valid Micheline expression may be registered. That said, attempting
  to originate a contract that uses a constant in an ill-typed way
  will fail.

* The total depth of the expression registered as a constant (after
  expanding all constant references) may not exceed 10,000.

* The total number of nodes in the Micheline expression being
  registered (after expanding all constant references) may not exceed
  the "max_micheline_node_count" protocol constant. As of Hangzhou
  this is 50,000.

* The total number of bytes in the Micheline expression being
  registered (after expanding all constant references) may not exceed
  the "max_micheline_bytes_limit" protocol constant. As of Hangzhou
  this is 50,000.


Originating a Contract that uses Global Constants
=================================================

A global constant can be referenced in Michelson scripts via the
primitive "constant", which accepts a single string argument, being
the hash of the expression to be referenced at runtime. This primitive
can be used to replace any Micheline node in the bodies of the
"parameter", "storage", "code", or "view" fields of a Michelson
script. For example, we replace every instance of the type "lambda
unit unit" and value 999 with their respective hashes:

   parameter (constant "exprtYirrFwYKm6yKLzJNtYRbq49zedYq16BonRvMzHiwSbUekB9YL");
   storage (big_map int (constant "exprtYirrFwYKm6yKLzJNtYRbq49zedYq16BonRvMzHiwSbUekB9YL"));
   code {
     PUSH int (constant "expruQN5r2umbZVHy6WynYM8f71F8zS4AERz9bugF8UkPBEqrHLuU8");
     # <rest of code>
   };

The full expansion of this contract would be:

   parameter (lambda unit unit);
   storage (big_map int (lambda unit unit));
   code {
     PUSH int 999;
     # <rest of code>
   };

During origination, all constants are expanded recursively. The
operation will fail if the resulting contract is ill-typed. Global
constant expansion consumes gas; thus, the operation may also fail due
to gas exhaustion.


Global Constants at Runtime
===========================

Contracts that use global constants are semantically equivalent to the
contract with all constants expanded.

Note that using the UNPACK operation to deserialize a lambda which
contains a reference to a global constant is not supported. Similarly,
originating a contract which contains a reference to a global constant
using the CREATE_CONTRACT instruction will also fail.

Michelson Anti-Patterns
***********************

Even though Michelson is designed to make it easy to write secure
contracts and difficult to write vulnerable ones, it is still possible
to write buggy contracts that leak data and funds. This is a list of
mistakes that you can make when writing or interacting with contracts
on the Tezos blockchain and alternative ways to write code that avoid
these problems.

This list is not exhaustive and will never be. The following resources
partially complement it:

* https://opentezos.com/smart-contracts/avoiding-flaws/

* https://ligolang.org/docs/tutorials/security/

* https://github.com/InferenceAG/TezosSmartContractDetails

* https://medium.com/protofire-blog/recommendations-to-enhance-
  security-of-tezos-smart-contracts-d14c0e53a6d3


Storing unbounded data
======================

The gas costs for serializing and deserializing a contract storage are
proportional to its size. Contracts allowing arbitrary users to add
data, or allowing authenticated users to add data of unbounded size,
are vulnerable to malicious users increasing the storage size to make
legitimate interactions with the contract consume a lot of gas or even
deadlocking the contract.


Possible issues:
----------------

* Malicious users may increase the storage size by adding large chunks
  of data.

* Even if each account can only store a bounded amount of data, a
  malicious user may create many accounts to bypass the limit.


Alternatives/Solutions:
-----------------------

* Store unbounded data offchain. When some data is not required for
  the execution of a smart contract, the contract does not need to
  store it. Typically, in most cases there is no need to store the
  full contents of an NFT in a smart contract, but rather some
  metadata or pointer. Even when some data is genuinely required, it
  can often be replaced in storage by its hash and revealed by the
  user when it is required for the contract execution.

* Store user data in big maps. Since big map are lazy data structures,
  instead of deserializing the full content of a stored big map before
  the execution, deserialization happens during the execution and only
  for the accessed keys. The size of the part of a big map which is
  not accessed during the execution has no impact on the gas costs of
  the execution (with two exceptions: transferring or deleting a
  complete big map containing tickets leads to an update of the ticket
  table which is linear in the number of keys). By storing user data
  under a big map whose keys are linked to the user addresses having
  added each key, it is possible to guarantee that the gas costs for
  legitimate users is not impacted by the interactions of malicious
  users.


Refunding to a list of contracts
================================

One common pattern in contracts is to refund a group of people’s funds
at once. This is problematic if you accepted arbitrary contracts as a
malicious user can do cause various issues for you.


Possible issues:
----------------

* One contract swallows all the gas through a series of callbacks

* One contract writes transactions until the block is full

* Reentrancy bugs. Michelson intentionally makes these difficult to
  write, but it is still possible if you try.

* A contract calls the `FAIL` instruction, stopping all computation.


Alternatives/Solutions:
-----------------------

* Create a default account from people’s keys. Default accounts cannot
  execute code, avoiding the bugs above. Have people submit keys
  rather than contracts.

* Have people pull their funds individually. Each user can break their
  own withdrawal only. **This does not protect against reentrancy
  bugs.**


Avoid batch operations when users can increase the size of the batch
====================================================================

Contracts that rely on linear or super-linear operations are
vulnerable to malicious users supplying values until the contract
cannot finish without running into fuel limits. This can deadlock your
contract.


Possible issues:
----------------

* Malicious users can force your contract into a pathological worst
  case, stopping it from finishing with available gas. Note that in
  the absence of hard gas limits, this can still be disabling as node
  operators may not want to run contracts that take more than a
  certain amount of gas.

* You may hit the slow case of an amortized algorithm or data
  structure at an inopportune time, using up all of your contract’s
  available gas.


Alternatives/Solutions:
-----------------------

* Avoid data structures and algorithms that rely on amortized
  operations, especially when users may add data.

* Restrict the amount of data your contract can store to a level that
  will not overwhelm the available gas.

* Write your contract so that it may pause and resume batch
  operations. This would complicate these sequences and require
  constant checking of available gas, but it prevents various attacks.

*Do not assume an attack will be prohibitively expensive*
Cryptocurrencies have extreme price fluctuations frequently and an
extremely motivated attacker may decide that an enormous expense is
justified. Remember, an attack that disables a contract is not just
targeted at the authors, but also the users of that contract.


Signatures alone do not prevent replay attacks
==============================================

If your contract uses signatures to authenticate messages, beware of
replay attacks. If a user ever signs a piece of data, you *must* make
sure that that piece of data is never again a valid message to the
contract. If you do not do this, anyone else can call your contract
with the same input and piggyback on the earlier approval.


Possible issues:
----------------

* A previously approved action can be replayed.


Alternatives/Solutions
----------------------

* Use an internal counter to make the data you ask users to sign
  unique. This counter should be per key so that users can find out
  what they need to approve. This should be paired with a signed hash
  of your contract to prevent cross-contract replays.

* Use the "SENDER" instruction to verify that the expected sender is
  the source of the message.


Do not assume users will use a unique key for every smart contract
------------------------------------------------------------------

Users should always use a different key for every contract with which
they interact. If this is not the case, a message the user signed for
another contract can be sent to your contract. An internal counter
alone does not protect against this attack. It *must* be paired with a
hash of your contract. You must verify the source of the message.


Storing/transferring private data
=================================

Once data is published to anyone, including broadcasting a
transaction, that data is public. Never transmit secret information
via any part of the blockchain ecosystem. As soon as you have
broadcast a transaction including that piece of information, anyone
can see it. Furthermore, malicious nodes in the system can manipulate
unsigned transactions by delaying, modifying, or reordering them.


Possible Issues
---------------

* If data is not signed, it can be modified

* Transactions can be delayed

* Secret information will become public


Alternatives/Solutions
----------------------

* Do not store private information on the blockchain or broadcast it
  in transactions.

* Sign all transactions that contain information that, if manipulated,
  could be abused.

* Use counters to enforce transaction orders.

This will at least create a logical clock on messages sent to your
contract.


Not setting all state before a transfer
=======================================

Reentrancy is a potential issue on the blockchain. When a contract
makes a transfer to another contract, that contract can execute its
own code, and can make arbitrary further transfers, including back to
the original contract. If state has not been updated before the
transfer is made, a contract can call back in and execute actions
based on old state.


Possible Issues
---------------

* Multiple withdrawals/actions

* Generating illegal state if state is updated twice later


Alternatives/Solutions
----------------------

* Forbid reentrancy by means of a flag in your storage, unless you
  have a good reason to allow users to reenter your contract, this is
  likely the best option.

* Only make transfers to trusted contracts or default accounts.
  Default accounts cannot execute code, so it is always safe to
  transfer to them. Before trusting a contract, make sure that its
  behavior cannot be modified and that you have an extremely high
  degree of confidence in it.

Token transfers and balance updates
***********************************

This page describes the reporting of tokens transfers in block
metadata, as a sequence of balance updates. They serve among others to
the Octez client to derive and print the "Balance updates" sections of
the receipts reported when operations are included in blocks.


Overview
========

Minting, transferring or burning tokens is handled by the Token
module. The module provides functions ("Token.transfer" and
"Token.transfer_n") to transfer tokens from one, respectively from
more accounts, to another account. Balance updates found in block
metadata are generated by these functions as a trace of the token
movements having taken place.

Balance updates indicate that either: tokens have been minted and
deposited into an account, transferred from or to another account, or
taken from an account and burned. In the Json format, a balance update
consists of three parts:

* an account identification indicated by a combination of fields such
  as: "kind", "category", "contract", ...

* the amount transferred (in mutez) indicated by the field "change". A
  positive amount indicates that the account has been credited, and a
  negative amount indicates that the account has been debited.

* the cause of the update given by the field "origin" which may have
  the following values:

  * ""block"" means that the balance update originates from the
    application of a block

  * ""migration"" means that the balance update originates from
    migration

  * ""subsidy"" means that the balance update originates from
    subsidies for liquidity baking

  * ""delayed_operation"" means that the balance update originates
    from the delayed application of the operation whose hash is given
    in the additional field ""delayed_operation_hash"".

A transfer of tokens is represented by a continuous and ordered
sequence of (balance) updates. That sequence starts with a series of
debits, and ends with a credit matching those debits. In block
metadata, the field ""balance updates"" contains one or more
transfers, represented by their sequences of updates, concatenated in
a flat list. Consider for example the following list:

   [ {"kind": "...", ..., "change": "-100", "origin": "block"},
     {"kind": "...", ..., "change": "100", "origin": "block"},
     {"kind": "...", ..., "change": "-125", "origin": "block"},
     {"kind": "...", ..., "change": "-75", "origin": "block"},
     {"kind": "...", ..., "change": "200", "origin": "block"} ]

This list reports that two transfers have occurred: a transfer of
"100" mutez from one account to another, and a transfer of "200" mutez
from two accounts to a third. There is one exception to this though:
migration balance updates are a bit compressed to save space, so the
list of balance updates in the corresponding metadata does not
necessarily allow to correlate debits to a credit by analyzing the
sequence. However, for those balance updates, the "delegate" and
"contract" fields can be used to establish correlations between debits
and credits.


Source, container, and sink accounts
====================================

There are three kinds of accounts: source accounts, container
accounts, and sink accounts. Tokens can be transferred from source or
container accounts, to container or sink accounts. All balance updates
contain the field "kind" which allows to determine the kind of account
it refers to. The possible values of this field are described in the
following sections (they are not simply ""source"", ""container"", and
""sink""). Depending on the kind of account, more fields such as the
"category" field may be used to identify accounts more specifically.


Source accounts
---------------

Source accounts are debited whenever new tokens are minted. A balance
update refers to a source account if and only if the field "kind" has
the value ""minted"". The value of the additional field "category"
designates one of the following fictitious accounts, each containing a
virtually unlimited number of tokens:

* ""nonce revelation rewards"" is the source of tokens minted to
  reward delegates for revealing their nonces

* ""double signing evidence rewards"" is the source of tokens minted
  to reward delegates for injecting a double signing evidence

* ""attesting rewards"" is the source of tokens minted to reward
  delegates for attesting blocks

* ""baking rewards"" is the source of tokens minted to reward
  delegates for creating blocks

* ""baking bonuses"" is the source of tokens minted to reward
  delegates for validating blocks and including extra attestations

* ""DAL attesting rewards"" is the source of tokens minted to reward
  delegates for attesting DAL slots

* ""subsidy"" is the source of tokens minted to subsidize the
  liquidity baking CPMM contract

* ""invoice"" is the source of tokens minted to compensate some users
  who have contributed to the betterment of the chain

* ""commitment"" is the source of tokens minted to match commitments
  made by some users to supply funds for the chain

* ""Sc_rollup_refutation_rewards"" is the source of tokens minted to
  reward an account for winning a smart-contract rollup refutation
  game

* ""bootstrap"" is analogous to ""commitment"" but is for internal use
  or testing. It will not be used during normal operation on mainnet,
  but may be used on test networks or in sandboxed mode

* ""minted"" is only for internal use and may be used to mint tokens
  for testing. It will not be used during normal operation on mainnet,
  but may appear on test networks or in sandboxed mode.


Container accounts
------------------

Container accounts are regular (user and smart contract) accounts, or
convenience accounts that hold tokens temporarily (e.g. when parts of
a delegate's funds are frozen). The field "kind" allows to identify
the type of container account, it can have one of the following
values:

* ""contract"" represents a user account or smart contract, and comes
  with the additional field (also called) "contract" whose value is
  the public key hash of the user account or the contract hash of the
  smart contract.

* ""freezer"" represents frozen accounts, and comes with the
  additional field "category" that can have one of the following
  values:

  * ""deposits"" represents the accounts of frozen deposits. Accounts
    in this category are further identified by the additional field
    ""staker"" whose value is either

    * a ""contract"" field, that contains the public key hash of the
      staker owning the funds and a ""delegate"" (public key hash)
      which gets staking power from the deposit,

    * just a ""delegate"" to designate collectively the deposits of
      all stakers delegating to the provided user account.

    * a ""baker_own_stake"" field to designate the delegate's own
      deposits received from its own stake rewards.

    * a ""baker_edge"" field to designate the delegate's own deposits
      received from the edge on its staker's rewards.

  * ""unstaked_deposits"" represents the accounts of unstaked frozen
    tokens. Accounts in this category are further identified by the
    following additional fields:

    * the field ""staker""  whose value is either

      * a ""contract"" field, that contains the public key hash of the
        staker owning the funds and a ""delegate"" (public key hash)
        which gets staking power from the deposit,

      * just a ""delegate"" to designate collectively the deposits of
        all stakers and the delegate itself.

    * the field ""cycle"" contains either the cycle at which the funds
      have been unstaked or the last unslashable cycle
      ("UNSTAKE_FINALIZATION_DELAY + SLASHING_DELAY + 1" before
      current cycle) if it is greater than the unstaking cycle.

  * ""bonds"" represents the accounts of frozen bonds. Bonds are like
    deposits. However, they can be associated to user accounts or
    smart contracts, unlike deposits that only apply to user accounts
    that are also delegates. Accounts in this category are further
    identified by the following additional fields:

    * the field "contract" contains the public key hash of the user
      account, or the contract hash of the smart contract

    * the field "bond_id" contains the identifier of the bond (e.g. a
      rollup hash if the bond is associated to a transaction or a
      smart contract rollup).

* ""accumulator"" represents accounts used to store tokens for some
  short period of time. This type of account is further identified by
  the additional field "category" whose (only possible) value ""block
  fees"" designates the container account used to collect manager
  operation fees while block's operations are being applied. Other
  categories may be added in the future.

* ""commitment"" represents the accounts of commitments awaiting
  activation. This type of account is further identified by the
  additional field "committer" whose value is the encrypted public key
  hash of the user who has committed to provide funds.

* ""staking"" represents abstractions used for accounting staking by
  delegators, and comes with the additional field "category" that can
  have one of the following values:

  * ""delegator numerator"" abstracts the delegator's stake, and comes
    with the additional field ""delegator"" whose value is the public
    key hash of the delegator.

  * ""delegate denominator"" abstracts the total stake of delegate's
    delegators, and comes with the additional field ""delegate"" whose
    value is the public key hash of the delegate.


Sink accounts
-------------

Sink accounts are credited whenever tokens are burned. A balance
update refers to a sink account if and only if the field "kind" has
the value ""burned"". The value of the additional field "category"
allows to identify more specifically a fictitious account able to
receive a virtually unlimited number of tokens. The field "category"
of a sink account may have one of the following values:

* ""storage fees"" is the destination of storage fees burned for
  consuming storage space on the chain

* ""punishments"" is the destination of tokens burned as punishment
  for a delegate that has double baked or double attested

* ""lost attesting rewards"" is the destination of rewards that were
  not distributed to a delegate. This category comes with the
  following additional fields:

  * the field "delegate" contains the public key hash of the delegate

  * the field "participation" has the value ""true"" if participation
    was not sufficient and has the value ""false"" otherwise

  * the field "revelation" has the value ""true"" if the delegate has
    not revealed his nonce and has the value ""false"" otherwise.

* ""lost DAL attesting rewards"" is the destination of DAL rewards
  that were not distributed to a delegate. This category comes with
  the additional field "delegate" containing the public key hash of
  the delegate.

* ""Sc_rollup_refutation_punishments"" is the destination of tokens
  burned as punishment for submitting bad commitments that have been
  refuted

* ""burned"" is only for internal use and testing. It will not appear
  on mainnet, but may appear on test networks or in sandboxed mode.


Token transfers and metadata
============================

Balance updates in block metadata give a complete account of all token
transfers that have occurred when a block is applied. A few cases of
token transfers and the associated metadata are illustrated below. All
other cases of token transfers in the protocol follow the same
pattern. The only differences are the accounts involved.


Origination and transaction
---------------------------

When an origination or transaction operation is applied, tokens are
transferred from one contract to another. Depending on whether or not
storage space has been allocated on the chain by the application of
the operation, storage fees may also be burned. For example, a
transaction of "100" mutez from address "tz1a..." to address "KT1b..."
that allocates storage space for a cost of "10" mutez produces the
following list of balance updates:

   [ {"kind": "contract", "contract": "tz1a...", "change": "-100", "origin": "block"},
    {"kind": "contract", "contract": "KT1b...", "change": "100", "origin": "block"}
    {"kind": "contract", "contract": "tz1a...", "change": "-10", "origin": "block"}
    {"kind": "burned", "category": "storage fees", "change": "10", "origin": "block"} ]


Baking fees, rewards and bonuses
--------------------------------

When a contract pays the baking fees associated to an operation it has
emitted, those fees are temporarily collected (during the processing
of the block) into the container account ""block fees"". For example,
when a manager operation is applied, the account of the payer contract
is debited with the amount of fees and the ""block fees"" account is
credited with the same amount. Hence, for "100" mutez in fees, the
following balance updates are generated :

   [ {"kind": "contract", "contract": "tz1x...", "change": "-100", ...},
     {"kind": "accumulator", "category": "block fees", "change": "100", ...} ]

When all operations of a block have been applied baking fees rewards
and bonuses are distributed. The total amount of fees collected and
the baking rewards are transferred from the container account ""block
fees"" and the source account ""baking rewards"", respectively, to the
contract of the payload producer that selected the transactions to be
included in the block. So, for a total amount of "1000" mutez in fees
collected and an amount of "500" mutez in baking rewards, assuming
that the staking parameter of the delegate are such that 50 mutez are
frozen -- with 5 mutez being the delegates edge, 10 mutez being the
delegates share, and 35 mutez going to the stakers -- and 450 are
spendable, the following balance updates are generated:

   [ {"kind": "accumulator", "category": "block fees", "change": "-1000", ...},
     {"kind": "contract", "contract": "tz1a...", "change": "1000", ...}
     {"kind": "minted", "category": "baking rewards", "change": "-5", ...},
     {"kind": "freezer", "category": "deposits", "staker": { "baker_edge": "tz1a..."}, "change": "5", ...},
     {"kind": "minted", "category": "baking rewards", "change": "-10", ...},
     {"kind": "freezer", "category": "deposits", "staker": { "baker_own_stake": "tz1a..."}, "change": "10", ...},
     {"kind": "minted", "category": "baking rewards", "change": "-35", ...},
     {"kind": "freezer", "category": "deposits", "staker": { "delegate": "tz1a..."}, "change": "35", ...},
     {"kind": "minted", "category": "baking rewards", "change": "-450", ...},
     {"kind": "contract", "contract": "tz1a...", "change": "450", ...} ]

The baking bonus go to the block proposer that signed and injected the
block. Hence the amount of the bonus is transferred from the source
account ""baking bonuses"" to the contract of the block producer
and/or to its frozen balance. For example, the balance updates
generated for an amount of "100" mutez in baking bonus with 90% sent
to spendable balance and 10% to bakers frozen deposit (case with no
stakers and mainnet ratios) are:

   [ {"kind": "minted", "category": "baking bonus", "change": "-90", ...},
     {"kind": "contract", "contract": "tz1b...", "change": "90", ...},
     {"kind": "minted", "category": "baking bonus", "change": "-10", ...},
     {"kind": "freezer", "category": "deposits", "staker": { "baker_own_stake": "tz1b..."}, "change": "10", ...}]


Attesting, double signing evidence, and nonce revelation rewards
----------------------------------------------------------------

Attesting rewards are reflected in balance updates as a transfer of
tokens from the ""attesting rewards"" source account to the account of
the delegate that receives the reward. Hence, for a reward of "100"
mutez,  the following two balance updates are generated:

   [ {"kind": "minted", "category": "attesting rewards", "change": "-100", ...},
     {"kind": "contract", "contract": "tz1...", "change": "100", ...} ]

When attesting rewards are not distributed to the delegate due to
insufficient participation or for not revealing nonces, they are
transferred instead to the sink account identified by the quadruple
"("lost attesting rewards", delegate, participation, revelation)". For
example, for an amount of "100" mutez in rewards not distributed due
to insufficient participation, the following balance updates are
generated:

   [ {"kind": "minted", "category": "attesting rewards", "change": "-100", ...},
     {"kind": "burned",
      "category": "lost attesting rewards",
      "delegate": "tz1...",
      "participation": "true",
      "revelation": "false",
      "change": "100", ...} ]

DAL attesting rewards are analogous to attesting rewards, except that
the source account used is "DAL attesting rewards".

Depending on the staking parameters set by the delegate, some portion
of the (DAL) attesting rewards will go to the freezer container, as
for baking rewards and bonuses.

Double signing evidence rewards and nonce revelation rewards are
analogous to attesting rewards, except that the source accounts used
are ""double signing evidence rewards"" and ""nonce revelation
rewards"".

Mempool
*******

The economic protocol provides a protocol-side mempool module data
structure intended for use by the shell prevalidator (see The
Prevalidator) and by "octez-baker" to incrementally accumulate
operations that can be safely used to bake a new block.

It ensures that :

* Every operation contained are valid;

* every operation are co-valid: they can be safely included in a block
  in any arbitrary order, meaning operations commute;

* the merging of two mempools also maintains the aforementioned
  properties.

The protocol leverages the partial construction mode to incrementally
validate new operations while maintaining the aforementioned
invariants.

During validation, operations are never actually applied, as it is
unnecessary for asserting their validity.


Merging Mempools
================

The protocol-side mempool data structure is designed to allow the
merging of two mempools. This is useful in practice when injecting an
external mempool into the current one. The design also considers the
future possibility of parallelizing operations validation and then
merging mempools in a divide-and-conquer style, an approach that could
be implemented later to accelerate validation.


Conflict
========

Of course, two mempools containing conflicting operations cannot be
merged while maintaining the desired invariants. Similarly, if an
injected operation conflicts with existing ones, the properties would
not hold. To solve conflicts, the structure requires a
"conflict_handler", a function called upon encountering conflicts to
determine which operation remains and which is discarded.

Smart Optimistic Rollups
************************

A **rollup** is a processing unit that receives, retrieves and
interprets input messages to update its local state and to produce
output messages targetting the Tezos blockchain. In this
documentation, we will generally refer to the rollup under
consideration as the Layer 2 on top of the Tezos blockchain,
considered as the Layer 1.


Overview
========

Smart rollups are a permissionless scaling solution for the Tezos
blockchain.  Indeed, anyone can originate and operate one or more
rollups, allowing to increase the throughput of the Tezos blockchain,
(almost) arbitrarily.

The purpose of this documentation is to provide an overview of the
terminology and basic principles of smart rollups. In the Smart rollup
node, we provide a complete tour of smart rollups related workflows
and reference documentation for the development of a WASM kernel.

The integration of these rollups in the Tezos protocol is
*optimistic*: this means that when a participant publishes a claim
about the state of the rollup, this claim is *a priori* trusted.
However, a refutation mechanism allows anyone to economically punish
someone who has published an invalid claim. Therefore, thanks to the
refutation mechanism, a single honest participant is enough to
guarantee that the input messages are correctly interpreted.


Rollup kernel
=============

In the Tezos protocol, the subsystem of smart rollups is generic with
respect to the syntax and the semantics of the input messages. More
precisely, the originator of a smart rollup provides a program named a
**kernel** (in one of the languages supported by Tezos) responsible
for interpreting input messages. During the refutation mechanism, the
execution of this kernel is handled by a **Proof-generating Virtual
Machine (PVM)** for this language, provided by the Tezos protocol,
which allows to prove that the result of applying an input message to
the rollup context is correct. The rest of the time, any VM
implementation of the chosen language can be used to run the smart
rollup kernel, provided that it is compliant with the PVM.

The smart rollup infrastructure currently supports the WebAssembly
language. A WASM rollup runs a kernel expressed in WASM. The role of
the kernel is to process input messages, to update a state, and to
output messages targeting the Layer 1 following a user-defined logic.
Anyone can develop a kernel or reuse existing kernels. A typical use
case of WASM rollups is to deploy a kernel that implements the
Ethereum Virtual Machine (EVM) and to get as a result an EVM-
compatible Layer 2 running on top of the Tezos blockchain. WASM
rollups are not limited to this use case though: they are fully
programmable, hence their names, smart optimistic rollups, as they are
very close to smart contracts in terms of expressiveness.


User roles
==========

Just like smart contracts, smart rollups are decentralized software
components. However, contrary to smart contracts that are processed by
the network validators automatically, a smart rollup requires a
dedicated *rollup node* to function.

Any user can originate, operate, and interact with a rollup. For the
sake of clarity, we will distinguish three kinds of users in this
documentation: operators, kernel developers, and end-users. An
operator deploys the rollup node to make the rollup progress. A kernel
developer writes a kernel to be executed within a rollup. An end-user
interacts with the rollup through Layer 1 operations or Layer 2 input
messages.


Address
=======

When a Smart Rollup is originated on the Layer 1, a unique address is
generated to uniquely identify it. A smart rollup address starts with
the prefix "sr1" (see also Accounts and addresses).


Inputs
======

There are two channels of communication to interact with smart
rollups:

1. a global **rollups inbox** allows the Layer 1 to transmit
   information to all the rollups.

2. a **reveal data channel** allows each rollup to retrieve data
   coming from data sources external to the Layer 1. Rollups request
   data through that channel to the runner of that rollup kernel (i.e.
   the smart rollup node).


Rollups inbox
-------------

A single global inbox serves all rollups and contains two kinds of
messages: *external* messages are pushed through a Layer 1 manager
operation while *internal* messages are pushed by Layer 1 smart
contracts or by the protocol itself. All messages (external and
internal) pushed to the inbox also contain the Layer 1 level of their
insertion and a counter. The counter is the index of the message and
it is reset at each Layer 1 level.


External messages
~~~~~~~~~~~~~~~~~

Anyone can push a message to the rollups inbox. This message is a mere
sequence of bytes following no particular underlying format. The
interpretation of this sequence of bytes is the responsibility of each
kernel.

There are two ways for end-users to push an external message to the
rollups inbox: first, they can inject the dedicated Layer 1 operation
using the Octez client (see command "send smart rollup message
<messages> from <src>"); second, they can use the batcher of a smart
rollup node. More details can be found in the Sending an external
inbox message.


Internal messages
~~~~~~~~~~~~~~~~~

Contrary to external messages, which are submitted by the end users,
internal messages are constructed by the Layer 1.

At the beginning of every Tezos block, the Layer 1 pushes two internal
messages: “Start of level”, and “Info per level”. “Start of level”
does not have any payload associated to it, while “Info per level”
provides to the kernel the timestamp and block hash of the predecessor
of the current Tezos block. If the Tezos block is the first block of a
protocol, then the Layer 1 pushes another message “Protocol migration”
just after the “Info per level” that provides the new protocol version
(i.e. "<proto-name>_<NNN>").

A rollup is identified by an address and has an associated Michelson
type (defined at origination time). Any Layer 1 smart contract can
perform a transfer to this address with a payload of this type. This
transfer is realized as an internal message pushed to the rollups
inbox.

Finally, after the application of the operations of the Tezos block,
the Layer 1 pushes one final internal message “End of level”.
Similarly to “Start of level“, this internal message does not come
with any payload.


Reveal data channel
-------------------

The reveal data channel is a communication interface that allows the
rollup to request data from sources that are external to the inbox and
can be unknown to the Layer 1. The rollup node has the responsibility
to answer the rollup requests.

A rollup can do the following requests through the reveal data
channel:

1. **preimage requests**: The rollup can request arbitrary data of at
   most 4kBytes, provided that it knows its (blake2b) hash. The
   request is fulfilled by the rollup node, see Populating the reveal
   channel.

2. **metadata requests** The rollup can request information from the
   protocol, namely the address and the origination level of the
   rollup itself. The rollup node retrieves this information through
   RPCs to answer the rollup.

Information passing through the reveal data channel does not have to
be considered by the Layer 1: for this reason, the volume of
information is not limited by the bandwidth of the Layer 1. Thus, the
reveal data channel can be used to upload large volumes of data to the
rollup.


Origination
===========

A smart rollup is characterized by: - the kind of Proof-generating
Virtual Machine (PVM), - the kernel written in a language that the PVM
can interpret, - the Michelson type of the entrypoint used by Layer 1
smart contracts to send internal messages to it, and - an optional
list of addresses used as a white-list of allowed committers (see
Private rollups).

All these characteristics are provided when originating a new smart
rollup.


Processing
==========

Each time a Tezos block is finalized, a rollup reacts to three kinds
of events: the beginning of the block, the input messages possibly
contained in that block, and the end of the block. A **rollup node**
implements this reactive process: it downloads the Tezos block and
interprets it according to the semantics of the PVM. This
interpretation can require updating a state, downloading data from
other sources, or performing some cryptographic verifications. The
state of the rollup contains an **outbox**, which is a sequence of
latent calls to Layer 1 contracts.

The behavior of the rollup node is deterministic and fully specified
by a reference implementation of the PVM embedded in the protocol.
Notice that the PVM implementation is meant for verification, not
performance: for this reason, a rollup node does not normally run a
PVM to process inputs but a **fast execution engine** (e.g., based on
the Wasmer runtime for the WASM PVM in the case of the rollup node
distributed with Octez). This fast execution engine implements the
exact same semantics as the PVM. The PVM is only ever used by the
rollup node when it needs to produce a proof during the last step of
the refutation mechanism.


Commitments
===========

Starting from the rollup origination level, levels are partitioned
into **commitment periods** of a number of consecutive blocks
corresponding to about 15 minutes (currently 112 blocks).

A **commitment** claims that the interpretation of all inbox messages
published during a given commitment period, and applied on the state
of a parent commitment, led to a given new state by performing a given
number of execution steps of the PVM. Execution steps are called
**ticks** in Smart Rollups terminology.

A commitment must be published on the Layer 1 any time after each
commitment period, to have the rollup progress. A new commitment
period starts right after the previous commitment period, no matter if
commitments were published or not for the previous commitment
period(s). For example, if an operator rollup node stops running for
one day long, when it comes back, it will be able to resume publishing
commitments for the passed periods, in chronological order. Indeed, a
commitment is always based on a parent commitment (except for the
genesis commitment that is automatically published at origination
time), so publishing a commitment fails if the parent commitment has
not yet been published.

Since the PVM is deterministic and the inputs are completely
determined by the Layer 1 rollups inbox and the reveal channel, there
is only one honest commitment. In other words, if two distinct
commitments are published for the same commitment period, one of them
must be wrong.

An operator publishing a commitment is called a **committer**. Notice
that, in order to publish a commitment, the operator must freeze a
deposit of 10,000 tez, called **frozen bonds**. For this reason, the
committer is sometimes called a (smart rollup) *staker*. However, in
order to avoid confusion with the staker role in Tezos Layer 1's
Proof-of-Stake mechanism, we prefer to use the term "committer"
throughout this documentation.

Several committers can publish (and thus stake on) the same
commitment. When a committer *C* publishes a new commitment based on a
commitment that *C* has published, *C* does not have to provide a new
deposit: the deposit also applies to this new commitment.

There is no need to synchronize between operators: if two honest
operators publish the same commitment for a given commitment period,
the commitment will be published with two stakes on it.

A commitment is optimistically trusted but it can be refuted until it
is said to be **cemented** (i.e., final, unchangeable). Indeed, right
after a commitment is published, a two-week refutation period starts.
During the refutation period, anyone noticing that a commitment for a
given commitment period is invalid can post a concurrent commitment
for the same commitment period to force the removal of the invalid
commitment. If no one posts such a concurrent commitment during the
refutation period, the commitment can be cemented with a dedicated
operation injected in Layer 1, and the outbox messages can be executed
by the Layer 1 by an explicit Layer 1 operation (see Smart rollup
node), typically to transfer assets from the rollup to the Layer 1.

The outbox messages can follow three different formats. Firstly, the
Layer 1 operations contained in the outbox messages can be left
untyped, meaning only the Micheline expression is provided by the
kernel. Before executing the transaction, the Layer 1 typechecks said
expression against the expected type of the targeted entrypoint. Since
Nairobi, it is also possible for the kernel to provide its expected
type of the targeted entrypoint. This additional safety mechanism is
to avoid type confusion: namely, a kernel transferring a tuple that
the Layer 1 interprets as a ticket. Lastly, the outbox message can
contain a white-list update. This message can only be executed for a
rollup that is private since its origination (see Private rollups).


Bonds
=====

The committer's bond of 10,000 tez is frozen automatically when the
committer makes their first commitment. The bond remains frozen until
the committer is no longer staking on any uncemented commitment.

Because nodes have the length of the refutation period to challenge
another node's commitment, the bond stays locked until the end of the
refutation period for the last commitment that the node posted.
Recovering the bond safely takes a few steps; in general, node
operators follow these steps:

1. Switch the rollup node to "bailout" mode, which defends existing
   commitments but does not post new commitments.

2. Wait until the last commitment is cemented. If operators shut down
   the node before the last commitment is cemented, they risk losing
   their bond if another node challenges their commitments.

3. Recover the bond by running the "recover bond" operation, which
   unlocks their tez. Nodes running in "bailout" mode attempt to run
   this command automatically when the last commitment is cemented.
   For more information, see Deploying a rollup node.


Refutation
==========

Because of concurrent commitments, a rollup is generally related to a
**commitment tree** where branches correspond to different claims
about the rollup state.

By construction, only one view of the rollup state is valid (as the
PVM is deterministic). When two concurrent branches exist in the
commitment tree, the cementation process is stopped at the first fork
in the tree. To unfreeze the cementation process, a **refutation
game** must be started between *two concurrent committers* of these
branches. Refutation games are automatically played by rollup nodes to
defend their stakes: honest participants are guaranteed to win these
games. Therefore, an honest participant should not have to worry about
refutation games. Finally, a running refutation game does not prevent
new commitments to be published on top of the disputed commitments.

A refutation game is decomposed into two main steps: a dissection
mechanism and a final conflict resolution phase. During the first
phase, the two committers exchange hashes about intermediate states of
the rollups in a way that allows them to converge to the very first
tick on which they disagree. The exact number of hashes exchanged at a
given step is PVM-dependent. During the final phase, the committers
must provide a proof that they correctly interpreted this conflicting
tick.

The Layer 1 PVM then determines whether these proofs are valid. There
are only two possible outcomes: either one of the committers, that we
dub *C* in the sequel, has provided a valid proof, then *C* wins the
game, and is rewarded with half of the opponent's deposit (the other
half being burnt); or, both committers have provided an invalid proof
and they both lose their deposit. In the end, at most one stake will
be kept in the commitment tree. When a commitment has no more stake on
it (because all committers have lost the related refutation games), it
is removed from the tree. An honest player *H* must therefore play as
many refutation games as there are stakes on the commitments in
conflict with *H*'s own commitment.

Finally, notice that each player is subject to a timer similar to a
chess clock, allowing each player to play only up to one week: after
this time is elapsed, a player can be dismissed by any Layer 1 user
playing a timeout operation. Thus, the refutation game played by the
two players can last at most 2 weeks.

There is no timeout for starting a refutation game after having
published a concurrent commitment. However, assuming the existence of
an honest participant *H*, then *H* will start the refutation game
with all concurrent committers to avoid the rollup getting stuck.


Private rollups
===============

A **private** Smart Rollup guarantees that private data cannot be
leaked by any means, whereas in a public rollup, one can force a
rollup to leak part of the data by starting a refutation game. This is
achieved by restricting the set of allowed committers using a
*whitelist*. With that restriction, only addresses on the whitelist
can publish commitments and therefore participate in a refutation
game.

The whitelist is optionally defined at origination. The rollup is
considered public if no white-list is defined, private otherwise. The
whitelist can be updated with a specific outbox message. This message
contains an optional list, the new list completely replaces the stored
whitelist in layer 1. If the message contains no list, then the rollup
becomes public. In turn, it is forbidden to make a public rollup
private by sending an outbox message with a non-empty whitelist.

It is the responsibility of the kernel to maintain the white-list by
submitting outbox messages. Kernels must therefore implement their own
access control list logic to add and remove addresses.

Also, it is important to remember that because of the refutation
logic, an outbox message can only be executed when the associated
commitment has been cemented (see Smart rollup node).

Protocol Plugins
****************

This document describes the protocol plugins implemented in Octez.

Protocol-specific shell plugins, simply called protocol plugins,
implement extra APIs needed by the shell in order to interact with the
economic protocol, beyond the one provided by the protocol
environment. This code is not strictly speaking part of the protocol
code base, so this is not subject to on-chain governance (see voting
procedure), but it is still protocol-dependent, which means that it
may vary with different protocols. For instance, the plugin code for
protocol Alpha is located in file
src/proto_022_PsRiotum/lib_plugin/plugin.ml. Thus, a specific version
is included in the Octez node for each protocol version (recall that a
new release of Octez is usually delivered for each new protocol
proposal, see Release System)

In contrast to the Octez code, plugins are protocol-specific and as
such, know how to read the content of an operation.

So what kind of features may a protocol plugin provide? For instance,
protocol plugins do not define the context, or restrict the validity
of operations. In turn protocol plugins may, for example:

* perform protocol-dependent computations that require data not
  available in the amendable part of the protocol like accessing the
  current time to reason on timestamps (see Consensus filter);

* preserve the opacity/abstraction barrier of the protocol's internal
  data by performing computations on internal data without revealing
  it: e.g., there are some RPCs that can introspect the protocol-
  dependent content for certain operations;

* implement some common operations that are customized for each
  protocol (e.g., Prevalidator filters).


Prevalidator filters
====================

**Prevalidator filters**, which are detailed in the rest of this page,
enable the node to discard some operations (that cannot be included in
the next block) faster and restrict the operations it propagates to
the network. Filters are implemented as a node plugin and a specific
filter is delivered with each protocol version. When the chain
switches to a new protocol, the node installs its corresponding
filters, *in lieu of* the filters of the previous protocol. Notice
that prevalidator filters are not mandatory, their absence does not
break the Tezos blockchain protocol.

The interface of the prevalidator plugin is described at the mempool
plugin API . The following filtering strategies are implemented in the
pre_filter.


Fees filter
-----------

A very affordable technique for attackers is based on flooding the
network with, valid but useless, zero-fees operations. This is why the
prevalidator filter currently restricts operations based on their
associated fees, to reject "too cheap" or "zero-fees" operations. This
can be configured via the "minimal_fees",
"minimal_nanotez_per_gas_unit" and "minimal_nanotez_per_byte" (see
filter RPCs) parameters of the filter configuration of your node.


Consensus filter
----------------

To prevent the network from being flooded by valid (pre)attestations
regarding fanciful future levels and rounds, the plugin provides a
filter that narrows down the number of consensus operations declared
as valid based on a timestamp heuristic.

This filter will classify a consensus operation as "Branch_refused" if
the operation concerns a level and round combination that is far-
fetched in the future in regard to the latest proposal predecessor and
the current timestamp. It can be configured via the "clock_drift" (see
filter RPCs) parameter of the filter configuration of your node.


Operations prioritization and ordering
--------------------------------------

In addition to quick filtering of undesired operations, the
"prefilter" provides a priority for each successfully filtered
operation. Concretely, the priority is either "High", "Medium" or
"Low" in the current implementation, depending on the validation pass.
Some extra information (like the fees, or the gas/fees ratio of
manager operations) are also provided along the priorities to enable
fine-grained operations ordering. This extra information is similar to
the one used by the baker's operations selection mechanism, that
decides which operations will be included in the next block.


Filters RPCs
============

Filters are tunable by several parameters, whose values can be
retrieved and changed by users via the following RPC calls,
respectively:

* "rpc get /chains/<chain>/mempool/filter"

* "rpc post /chains/<chain>/mempool/filter"

The following parameters can be thus inspected and modified:

* "minimal_fees": type "int", default "100"

* "minimal_nanotez_per_gas_unit": type "int", default "100"

* "minimal_nanotez_per_byte": type "int", default "1000"

* "allow_script_failure": type "bool", default "true"

* "clock_drift" : type "Period.t option", default "None"

* "max_operations" : type "int", default "10_000"

* "max_total_bytes" : type "int", default "10_000_000"

* "replace_by_fee_factor" : type "rational", default "21/20" (ie.
  "1.05%")

For example, each command below modifies the provided parameter and
resets all the others to their default values:

   octez-client rpc post /chains/main/mempool/filter with '{ "minimal_fees": "42" }'
   octez-client rpc post /chains/main/mempool/filter with '{ "replace_by_fee_factor": [ "23", "20" ] }'
   octez-client rpc post /chains/main/mempool/filter with '{ "max_operations": 7500 }'


Changing filters default configuration
--------------------------------------

Changing filters configuration may not have the expected outcome.
Because the configuration is only changed on your node, and not on all
the gossip network nodes, this will only impact how operations of your
nodes are propagated. For example, assuming there are three nodes "A",
"B" and "C" such that "A" is connected to "B" and "B" is connected to
"C". Assume that "A" modifies its filter configuration so that
"minimal_fees" is now "0". Then "A" may propagate an operation with
"0" fee to "B". However, because "B" has the default filter
configuration, this operation will not be propagated to "C" (so "C"
may never see it).

Contract events
***************

Contract events provide a way for smart contracts to deliver live
information to external applications. This mechanism allows off-chain
applications to react to Tezos contracts execution.

Note:

  Contract events should not be confused with log events, which are
  generated by the various Octez tools to include information about
  their execution in their own logs.

This page explains how events can be generated using specific support
in the Michelson language. For using events in higher-level languages
for smart contracts, consult their own documentation.


Emitting events
===============

Contracts can create event emission operations by invoking the
Michelson instruction "EMIT". "EMIT %tag ty" pops an item of type "ty"
off the stack and pushes an "operation" onto the stack. The type
ascription "ty" is special since annotations in "ty" are preserved and
eventually transmitted in the event so that indexers have a choice to
present the event body with appropriate labels. Both the "ty" type
ascription and "%tag" are optional. If "ty" is absent, this type is
inferred from the type of the top element of the stack. If "%tag" is
absent, no tag is attached to the event.

To actually send out the events, most importantly, the produced
"operation"s must be included in the list of operations returned by
the contract, along with "TRANSFER_TOKEN" operations for example.

Upon successful contract execution, the events appearing in the
resultant list of operations will be attached, in the order of
appearance, to the transaction receipt. There, the events are made
ready for consumption by services observing the chain.


Example
-------

Suppose a contract wants to emit events with the following type:

   or (nat %int) (string %str)

Then, this contract may generate an event emission operation with the
following instructions:

   PUSH string "right";
   RIGHT nat;
   EMIT %notify_client (or (nat %int) (string %str));

Note that the "EMIT" instruction will emit an event with a tag
"notify_client". The additional tags "int" and "str", allow indexers
to recognise the two variants and present the values with the
appropriate variant labels.


Retrieving events
=================

Events successfully emitted can be read off from transaction results.

This is typically achieved by making JSON RPCs to the block service.
It will return a list of operations, each including the event entries
with the information above.

Here is a sample result from a call, corresponding to the example
above.

   {
     "protocol": "ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK",
     "hash": "opNX59asPwNZGu2kFiHFVJzn7QwtyaExoLtxdZSm3Q3o4jDdSmo",
     // ... fields elided for brevity
     "contents": [
       {
         "kind": "transaction",
         // ... fields elided for brevity
         "metadata": {
           // ... fields elided for brevity
           "internal_operation_results": [
             {
               "kind": "event",
               // ... fields elided for brevity
               "source": "KT1M1ynE3YXkM7qLZoMppq6szMbBvxX9yQVL",
               "nonce": 0,
               "type": {
                  "prim": "or",
                  "args": [
                    {
                      "prim": "nat",
                      "annots": [
                        "%int"
                      ]
                    },
                    {
                      "prim": "string",
                      "annots": [
                        "%str"
                      ]
                    }
                  ]
               },
               "tag": "notify_client",
               "payload": {
                 "prim": "Right",
                 "args": [
                   {
                     "string": "right"
                   }
                 ]
               },
               "result": {
                 "status": "applied",
                 "consumed_milligas": "1000000"
               }
             }
           ]
         }
       }
     ]
   }

Note that the "operation" produced by "EMIT" does not constitute a
call to any other contract.

Blocks and Operations
*********************

The content of a Tezos block is made up of a block header and a
payload consisting of a list of operations.

This page first describes the protocol-specific part of the block
header, and then explains what operations are. For the protocol-
independent part of the block header, see Shell header.


Protocol-specific block header
==============================

Recall that, for the shell to interact with the economic protocol, two
notions are defined abstractly at the level of the shell and made
concrete at the level of the consensus protocol. Namely, these two
notions are the protocol-specific header and the fitness.

As in Emmy*, the protocol-specific header contains the fields:

* "signature": a digital signature of the shell and protocol headers
  (excluding the signature itself)

* "seed_nonce_hash": a commitment to a random number, used to generate
  entropy on the chain

* "proof_of_work_nonce": a nonce used to pass a low-difficulty proof-
  of-work for the block, as a spam prevention measure

* "liquidity_baking_toggle_vote": a vote to continue the Liquidity
  Baking Subsidy, stop it, or abstain.

There are two additional fields: "payload_hash" and "payload_round"
which are needed for establishing if a block is final.


Operations
==========

Operations implement and reify different functionalities provided by a
Tezos economic protocol: from reaching consensus on the state of the
Tezos blockchain, to performing smart contract calls and transactions.
Each Tezos economic protocol can specify different kinds of
operations.

This page only describes the operations supported by the economic
protocol that implement *enabled* features -- that is, those available
to end-users on Tezos Mainnet. The complete list of operations,
including those corresponding to features in development or available
only on test networks, is given in the OCaml Documentation .


Validation Passes
=================

The different kinds of operations are grouped into classes. Each class
has an associated index, a natural number, also known as a validation
pass. There are currently four classes of operations: consensus,
voting, anonymous, and manager operations. This order also specifies
the validation and application priority of each of these classes.
Consensus operations are considered the highest priority ones, and
manager operations the lowest.

Each kind of operation belongs to exactly one validation pass, except
for the Failing_noop operation which belongs to no validation pass and
therefore cannot be applied.

In the sequel, we describe the different classes of operations, and
the different kinds of operations belonging to each class.


Consensus Operations
====================

Consensus operations are administrative operations that are necessary
to implement the consensus algorithm. There are two kinds of consensus
operations, each belonging to the different voting phases required to
agree on the next block.

* A "Preattestation" operation implements a first vote for a candidate
  block with the aim of building a preattestation quorum.

* An "Attestation" operation implements a vote for a candidate block
  for which a preattestation quorum certificate (PQC) has been
  observed.


Voting Operations
=================

Voting operations are operations related to the on-chain Tezos
Amendment process. In this economic protocol, there are two voting
operations:

* The "Proposal" operation enables delegates to submit (also known as
  to "inject") protocol amendment proposals, or to up-vote previously
  submitted proposals, during the Proposal period.

* The "Ballot" operation enables delegates to participate in the
  Exploration and Promotion periods. Delegates use this operation to
  vote for ("Yea"), against ("Nay"), or to side with the majority
  ("Pass"), when examining a protocol amendment proposal.

Further details on each operation's implementation and semantics are
provided in the dedicated entry for on-chain governance.


Anonymous Operations
====================

This class groups all operations that do not require a signature from
a Tezos account (with an exception, detailed below). They implement
different functionalities of the protocol, and their common
characteristic is that they allow the account originating these
operations to remain anonymous in order to avoid censorship.

Two operations in this class implement functionality pertaining to the
protocol's random seeds generation mechanism:

* The "Seed_nonce_revelation" operation allows a baker to anonymously
  reveal the nonce seed for the commitment it had included in a
  previously baked block (in the previous cycle).

* The "Vdf_revelation" operation allows the submission of a solution
  to, and a proof of correctness of, the VDF challenge corresponding
  to the VDF revelation period of the randomness generation protocol.

Further details on the latter operation's implementation and semantics
are provided in the random seed generation protocol.

Three operations in this class are used to punish participants which
engage in Byzantine behaviour -- notably delegates which "double sign"
blocks, or emit conflicting consensus operations:

* The "Double_preattestation_evidence" operation allows for accusing a
  delegate of having *double-preattested* -- i.e., of having
  preattested two different block candidates, at the same level and at
  the same round. The bulk of the evidence, the two arguments
  provided, consists of the two offending preattestations.

* Similarly, the "Double_attestation_evidence" operation allows for
  accusing a delegate of having *double-attested* -- i.e., of having
  attested two different block candidates at the same level and the
  same round -- by providing the two offending attestations.

* The "Double_baking_evidence" allows for accusing a delegate of
  having "double-baked" a block -- i.e., of having signed two
  different blocks at the same level and at same round. The bulk of
  the evidence consists of the block headers of each of the two
  offending blocks.

See here for further detail on the semantics of evidence-providing
operations.

The "Activation" operation allows users which participated in the
Tezos fundraiser to make their accounts operational.

Finally, the "Drain_delegate" operation allows an active consensus-key
account, i.e., an account to which a baker delegated its consensus-
signing responsibility, to **empty** its delegate account. This
operation is used as a deterrent to ensure that a delegate secures its
consensus key as much as its manager (or main) key.


Manager Operations
==================

Manager operations enable end-users to interact with the Tezos
blockchain -- e.g., transferring funds or calling smart contracts. A
manager operation is issued by a single *manager* account which signs
the operation and pays the fees to the baker for its inclusion in a
block. Indeed, manager operations are the only fee-paying and gas-
consuming operations.

* The "Reveal" operation reveals the public key of the sending
  manager. Knowing this public key is indeed necessary to check the
  signature of future operations signed by this manager.

* The "Transaction" operation allows users to transfer tez between
  accounts, to invoke a smart contract, or to invoke pseudo-operations
  on user accounts.

* The "Delegation" operation allows users to designate a delegate (a
  *baker*) for delegating or staking their coins, or to register
  themselves as delegates.

* The "Update_consensus_key" operation allows users to register a
  consensus key, which is a dedicated key for signing blocks and
  consensus-related operations.

* The "Origination" operation is used to originate, that is to deploy,
  smart contracts in the Tezos blockchain.

* The "Set_deposits_limit" operation enables delegates to adjust the
  amount of stake a delegate has locked in bonds.

* Support for registering global constants is implemented with the
  "Register_global_constant" operation.

* The "Increase_paid_storage" operation allows a sender to increase
  the paid storage of some previously deployed contract.

* The "Event" operation enables sending event-like information to
  external applications from Tezos smart contracts -- see Contract
  Events for further detail.

Moreover, all operations necessary to implement Tezos' *enshrined*
Layer 2 solutions into the economic protocol are also manager
operations.

In particular, smart rollups maintenance is handled with dedicated
manager operations.

* The "Smart_rollup_originate" operation is used to originate, that
  is, to deploy smart rollups in the Tezos blockchain.

* The "Smart_rollup_add_messages" operation is used to add messages to
  the inbox shared by all the smart rollups originated in the Tezos
  blockchain. These messages are interpreted by the smart rollups
  according to their specific semantics.

* The "Smart_rollup_publish" operation is used to regularly declare
  what is the new state of a given smart rollup in a so-called
  “commitment”. To publish commitments, a user account has to own at
  least ꜩ 10,000, which are frozen as long as at least one of their
  commitments is disputable.

* The "Smart_rollup_cement" operation is used to cement a commitment,
  if the following requirements are met: it has been published for
  long enough, and there is no concurrent commitment for the same
  state update. Once a commitment is cemented, it cannot be disputed
  anymore.

* The "Smart_rollup_recover_bond" operation is used by a user account
  to unfreeze their ꜩ 10,000. This operation only succeeds if and only
  if all the commitments published by the user account have been
  cemented.

* The "Smart_rollup_refute" operation is used to start or pursue a
  dispute. A dispute is resolved on the Tezos blockchain through a so-
  called refutation game, where two players seek to prove the
  correctness of their respective commitment. The game consists in a
  dissection phase, where the two players narrow down their
  disagreement to a single execution step, and a resolution, where the
  players provide a proof sustaining their claims. The looser of a
  dispute looses their frozen bond: half of it is burned, and the
  winner receives the other half in compensation.

* The "Smart_rollup_timeout" operation is used to put an end to a
  dispute if one of the two players takes too much time to send their
  next move (with a "Smart_rollup_refute" operation). It is not
  necessary to be one of the players to send this operation.

* The "Smart_rollup_execute_outbox_message" operation is used to enact
  a transaction from a smart rollup to a smart contract, as authorized
  by a cemented commitment. The targeted smart contract can determine
  if it is called by a smart rollup using the "SENDER" Michelson
  instruction.


Manager Operation Batches
-------------------------

Manager operations can be grouped, forming a so-called **batch**.
Batches enable the inclusion of several manager operations from the
same manager in a single block.

Batches satisfy the following properties:

* All operations in a batch are issued by the same manager, which
  provides a single signature for the entire batch.

* A batch is applied atomically: all its operations are executed
  sequentially, without interleaving other operations. Either all the
  operations in the batch succeed, or none is applied.


Failing_noop operation
======================

The "Failing_noop" operation is not executable in the protocol:

* it can only be validated in mempool mode, by the prevalidator
  component;

* consequently, this operation cannot be applied, and in fact will
  never be included into a block.

Rather, the "Failing_noop" operation allows to sign an arbitrary
string, without introducing an operation that could be misinterpreted
in the protocol.

The Octez client provides commands to sign and verify the signature of
input messages by a given key. These commands create a "failing_noop"
operation from the message that is being signed or checked.

   octez-client sign message "hello world" for <account>

   octez-client check that message "hello world" was signed by <account> to
   produce <signature>

Validation and Application
**************************

The economic protocol is responsible for providing the rules that
govern the Tezos network, and for enforcing that these rules implement
a correct blockchain protocol. However, it does so in coordination
with a Tezos shell, who ultimately implements these rules on its
behalf. To this end, a Tezos economic protocol must provide the shell
with an interface enabling roughly the following functionalities:

* deciding whether an operation is *valid*, and thus can be safely
  included into a new block in the blockchain;

* deciding whether a new block is *valid*, and thus can be safely
  appended to the blockchain;

* including an operation in a block, and executing it on the
  blockchain state, effectively modifying the ledger state; and,

* appending a new block to a Tezos blockchain, and computing the
  updated ledger state.

From a higher-level, *abstract* perspective, the validation system in
the Tezos protocol implements this business logic in a functional,
state-passing machine where:

* Its state is given by the context, the internal representation of
  the state of the Tezos ledger at a given blockchain level. For
  instance, the context contains the information of all activated
  accounts and contracts, and their balances. More generally, the
  context must provide enough information to determine the validity of
  operations and blocks.

* The *apply_operation* method determines whether an operation is safe
  to be executed in a given context. If so, it proceeds to execute it
  and outputs the context resulting from the applied operation's
  effects.

* The *apply* method takes as input a block and a context, determines
  whether the block can be safely and correctly appended to the
  current head of the chain. If so, it appends the block to the chain
  and outputs the resulting context, reflecting the state of the
  ledger after the block has successfully been applied. *apply* relies
  on (i.e., it should call) *apply_operation* to validate and apply
  each operation in the block, and compute intermediate states.

However, the *concrete* API exported from the Tezos economic protocol
does not implement this business logic *monolithically*, as described
above, but it rather presents a more fine-grained API. The rationale
is to provide specialized variations of the core *validation* and
*application* functionality, dubbed Validation modes. For example,
these modes enable the protocol to distinguish operations "in the
mempool", whose validation is triggered by the prevalidator, from
operations included in newly received blocks, whose validation is
triggered by the block validator, in order to localize validation
rules as needed. The resulting concrete API is specified by the
Protocol  module in the protocol environment "V14", and it is
implemented by this protocol in the Main module.

The rest of this document is organized as follows: we first describe
the different validation modes implemented by this Tezos economic
protocol, and then we delve deeper into the particulars of validation
and application for blocks and the operations supported.


Validation modes
================

The Tezos protocol provides different validation modes, intended to be
used by the Tezos *shell* and *baker* software implementations when
needing to apply (or to assert the validity of) blocks and operations
under different, or specialized, circumstances -- for example, in
order to *bake* a block. For each of these validation modes, the API
specified by the protocol environment offers an entry point so that
protocol-agnostic components, the Tezos shell for instance, are able
to use these different modes.


Full Application
----------------

The "Full application" mode is intended to be used to *fully* validate
and apply blocks. In particular, this mode is used to validate and
apply a **known** block, with a known operation trace. A Tezos shell
implementation should use the full application mode to decide whether
an incoming block can be safely included in the blockchain. That is,
all validity checks are enabled: the block's signature is correct, and
**all** operations included in the block are valid; the correct amount
of consensus operations have been included in order to satisfy the
consensus' threshold, etc.


Full Construction
-----------------

The "Full construction" mode is intended to be used when a *delegate*
is trying to bake a block and therefore needs to filter the validity
of the desired operation trace, to include only valid operations. This
mode is mostly similar to the "Full application" mode except that
*some* global block validity checks are disabled, and consensus
operations are validated with slightly different preconditions. For
instance, since a delegate cannot sign a block while it is being
built, the signature check is disabled, and it will be left to the
baker to correctly sign the resulting block after its construction is
finalized.

In Octez, this mode is mainly used by the baker daemon.


Partial Construction
--------------------

The "Partial construction" mode, also known as "Mempool mode" is used
by the prevalidator component of an Octez node to validate incoming
operations -- that is, those not-yet included into blocks. This mode's
business-logic is very close to the "Full construction" mode, and the
differences boil down to the intended usage. The partial construction
mode does not try to fully bake a block, but rather to inform the
Octez prevalidator on the potential validity of operations (and
whether they can safely included into a block), so that the latter can
**classify** incoming operations, and further decide how to process
them accordingly.

The protocol provides the shell with the following classification of
an operation, consisting of one valid kind -- "Applied" --, and four
error kinds defined by the protocol environment:

* "Applied": the operation is valid and can be included in a potential
  block in the current context.

* "Temporary": the operation is invalid in the current context, but it
  could *later* become valid -- in the context associated to a
  successor block of the current head. For instance, a manager
  operation whose counter value is greater than the one expected (a
  *"counter-in-the-future"* error), or the manager's balance is
  insufficient to pay the operation's fees, etc.

* "Branch": the operation is invalid in the current context and in any
  possible context from its future successors, but it might still be
  valid in an alternative branch. For example: a manager operation
  with a smaller counter than the one expected (a *"counter-in-the-
  past"* error), an unexpected attestation for the current level, etc.

* "Permanent": the operation is invalid in the current context, and
  there isn't any plausible context where it might be or become valid.
  For example, an operation carrying an invalid signature.

* "Outdated": the operation is *too old* to be included in a block.
  Furthermore, there might be still some value in the information
  provided by an "Outdated" operation. An example is the case of an
  attestation which was received *too late*, but that could still be
  used to form a consensus quorum.


Partial Application
-------------------

The "Partial application" mode is no longer used, but it subsists in
the code. Its aim is to provide Tezos shell implementations with a
light-weight (read "fast") block application mechanism, which can
determine whether a block has a *chance* of being valid or not, in a
situation when the provided context is *not a recent one*. That is,
when the block candidate succeeds neither the head of the chain, nor a
close ancestor.

This validation mode could be used when the node receives a
significantly large branch -- for instance, while bootstrapping. To
check whether this branch is plausibly valid or potentially malicious
spam, the shell retrieves the context from the most recent common
ancestor between its current head and the announced branch, and
proceeds to "partially apply" each block of this branch using the
common ancestor's context.

Indeed, by relying on the ancestor context, this mode can *only*
assert the validity of consensus-related preconditions (attesting
power, block fitness, etc.), as future consensus slots are known in
advance -- how much in advance being specified by the
"<CONSENSUS_RIGHTS_DELAY>" protocol constant. Thus, the "Partial
application" mode provides an over-approximation of the branch's
validity, and as a result intermediate results are not committed on
disk in order to prevent potential attacks.


Block Validation
================

The validity of a block depends on a set of precondition checks
implemented in different steps, which happen at different stages of
the application (and the construction) of a block.

The first step in the process is to decide whether a candidate block
is *well-formed*, that is, that it has the expected "shape" of a valid
block under the current Tezos economic protocol. Given a block
candidate, the block validation process will then verify that the
candidate block declares consistent level, round, and timestamp
values; that it carries a valid signature, etc. At this step, the
block validation process will also initialize the data-structures
required for subsequent steps.

The second step iterates over the block's operations and proceeds to
apply them sequentially. When at least one operation is found to be
invalid, under the conditions described in Operation Validation and
Application further below, the whole block is considered as invalid.

The last step in the block validation process, known as "block
finalization", aims to verify that the collected consensus operations
constitute a sufficiently large quorum. That is, it will verify that
the total attesting power present in the block is greater than the
"CONSENSUS_THRESHOLD" constant.

This sequence of three steps also yields a new context -- the
resulting state of the Tezos ledger after the application of the
candidate block. The shell may decide to commit this context to disk.

The Tezos economic protocol also offers a cheap (read "faster")
alternative to determine an over-approximation of the validity of a
block (see Partial Application above). This feature allows the shell
to propagate blocks faster without needing to fully validate them,
speeding-up block propagation over the network. Of course, as this is
an over-approximation, this feature cannot be considered to provide a
safe guarantee that a block will be valid: in particular, it does not
validate all kinds of operations.


Operation Validation and Application
====================================

In the Tezos economic protocol, we dissociate the notion of *validity*
from the notion of *applicability* for operations. A valid operation
is an operation that can be included safely in a block without
affecting the block's validity. Applying an operation, on the other
hand, actually performs the operation's side-effects which can be:
registering a new delegate, executing a smart contract, voting for a
new protocol amendment proposal, etc.

Note that an operation may fail during the application phase, even
though it has been checked as valid. For example, a smart contract
call that exceeds its gas limit can be included in a block even if an
error is raised at run-time. The application (that is, the operation's
side-effects) will not take effect, but fees will nonetheless be taken
from the account submitting the smart contract call.

In the sequel we refine the validity conditions and describe the
application process for each of the different validation passes.


Validity of operations
----------------------

For consensus, voting, and anonymous validation passes, validation is
sufficient to ensure successful application. For manager operations,
validation only ensures that the manager’s balance is sufficient to
cover the fees, in addition to verifying that the operation is well-
formed. It does not guarantee that the operation’s execution will be
error-free. If an operation does result in an error, all side effects
will be canceled, but the fees will be forfeited.


Validity of Manager Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In this sub-section, we explain the conditions for manager operations
(and batches of managers operations) to be considered valid and hence
suitable for inclusion in a block.


Validity of Individual Manager Operations
"""""""""""""""""""""""""""""""""""""""""

Manager operations are a class of operations, issued by a single
*manager* account which signs the operation and pays their fees. The
different manager operation kinds share several common fields:

* "source": the public key's hash of the *source* account of the
  manager operation -- that is, the *manager*.

* "fee": the amount of tez paid to the baker which decides to include
  this operation;

* "counter": the manager account's counter, incremented each time this
  account executes a manager operation, to prevent replay-attacks.

* "gas_limit": the maximum amount of gas that this operation may
  consume before failing.

* "storage_limit": the maximum amount of storage that this operation
  may require before failing.

* "operation": the actual operation(s) -- e.g., transfers, smart-
  contract calls, originations, delegations, etc.

* "signature": the manager's signature used to prove its identity.

A manager operation is **valid** if and only if all of the following
conditions hold:

* The operation source's public key has been previously *revealed*,
  unless the operation is itself a "Reveal" operation.

* The operation's signature is correct with regard to the manager
  account's public key.

* The provided "counter" value is the expected one for the manager.

* Depending on the operation's kind, the "gas_limit" is high enough to
  cover the minimal cost of parsing the operation and further minimal
  treatment.

* The manager account is solvent to pay the announced fees.


Validity of Manager Operation Batches
"""""""""""""""""""""""""""""""""""""

A batch of manager operations includes one or more manager operations
for sequential and atomic execution. The atomicity property imposes
that the validity of a batch should entail the validity of each
individual operation in the batch, as defined above. However, it also
entails some additional *global* constraints on manager batches.

For each of the operations present in a batch, the validation process
must check that the individual constraints above are satisfied, *with
the exception of* the signature constraint. Given that the signature
concerns the whole batch, as all operations in the batch are signed by
the same manager, it suffices to verify the signature only once.

The **global batch validity** constraint for this economic protocol is
defined as the conjunction of the following conditions:

* When a "Reveal" operation is present, **it must only occur once**,
  and **it must be placed at the head** of the batch -- that is, the
  "Reveal" operation must be the first operation in the batch.

* Every operation in the batch should declare the same "source".

* Each of the individual operation counters must be incremented
  correctly and sequentially.

* The sum of each individual operation's declared fees must be lower
  than the account's balance. That is, the manager account must be
  solvent to pay the announced fees for all the operations in the
  batch.


Co-valid operations
"""""""""""""""""""

However, two valid manager operations (or batches) do not always form
a valid block together. For example, operations "op1" and "op2" might
both be valid in the context "C", but after applying "op1" to "C",
"op2" might no longer be valid in the resulting context because the
manager's balance may no longer cover the fees.

In order to enforce that a set of valid managers operations forms a
valib block, the protocol over-approximate this property by requiring
the property of *co-validity*:

Two manager operations are *compatible* if they correspond to distinct
managers.

Two manager operations are *co-valid* in a context if they are
compatible and valid in this context.

Considering two co-valid operations in a context, the application of
one of them preserves the solvability of the other.

This property extends to a set of operations if every pair of distinct
operations is co-valid. In this case, the operations could be included
in the next block in any order, modulo block limits (eg. maximum gas,
block size limit, etc).


Application of Manager Operations
---------------------------------

Once the validity of a manager operation (or, a batch of manager
operations) is established, the protocol proceeds to apply the
operation. This first step in this application phase is to transfer
the operation's fees to the baker that included this operation. Then,
the actual application depends on the operation kind. For instance,
this could be a smart contract execution, enacting a delegation, or
multiple actions executed as a batch. The application of a batch of
manager operations consists of the sequential application of each
operation in the batch, following their inclusion order -- the head of
the batch being the first manager operation being applied.

The application of each individual manager operation may either
succeed -- and therefore be reported as "Applied" --, or indeed fail
with an error. In both cases, the fees are taken and the counter for
the operation's manager is incremented.

When a manager operation fails, every side-effect which was previously
performed is backtracked. Moreover, the (rest of the) batch has to be
aborted.  Thus, depending on the position of the manager operation in
a batch, its failure has to be propagated accordingly:

* If there were other successfully applied operations in the batch
  prior to the offending one, the effect of each of them has to be
  reverted, and each of them will be reported as "Backtracked".

* If there were other operations pending application after the
  offending one, their application is aborted, and they are reported
  as "Skipped".

For example, let's consider a simple batch of three manager operations
"[op1, op2, op3]", if "op1" is successful but "op2" fails, the ticket
result for the application of the manager operation batch will report:

* "op1" -- "Backtracked", "op1" was applied successfully, but after
  "op2" failed, the operation was canceled;

* "op2" -- "Failed", the application of this particular operation
  failed;

* "op3" -- "Skipped", this operation was never executed because "op2"
  had previously failed.

Tickets
*******

Tickets are a special class of assets on Tezos that contracts can
construct and store, and that can then be transferred between
contracts, user accounts, or rollups. They are useful to model
ownership of some digital artefact, existing in a limited number of
copies (or instances); this can further serve as a basis for
restricting access to the artefact and permissions to use it. In this
document, we will explain the semantics and lifetime of tickets,
including creation, transfer, and destruction.


Semantics
=========

A ticket represents a digital artefact existing in a limited number of
copies (or instances). As such, a ticket has the following four
attributes:

1. "ticketer" is the creator of the ticket, which is always a Tezos
   smart contract.

2. "contents" is the Michelson expression attached to the ticket.

3. "content_ty" is the Michelson type of the "contents" data. It can
   be any comparable type.

4. "amount" is the amount of the ticket. It is a strictly positive
   whole number.

Furthermore, a ticket cannot be duplicated with the DUP or the DUP n
instructions.


Operations on tickets
=====================


Creating
--------

Tickets can be constructed by smart contracts. To construct tickets,
smart contracts use the "TICKET" instruction with some ticket amount,
ticket content type, and value as inputs. Smart contracts then may
store them in contract storage or transfer them along to either other
contracts, rollups, or user accounts.

Tickets cannot be constructed with instructions for duplication, such
as "DUP". Therefore, the issuance of tickets is completely within the
control of the ticketer contract by invoking "TICKET" instructions.


Splitting
---------

A ticket can be split into two tickets with the same "contents" and
"ticketer" values, as long as the sum of the "amount" of the two
tickets is equal to that of the original. This operation allows
tickets to be spent across several transactions by breaking tickets
into smaller tickets. Smart contracts may split tickets by invoking
the SPLIT_TICKET instruction. The ticket splitting is done
automatically when a user account transfers part of a ticket.


Joining
-------

Tickets issued by the same "ticketer" contract with the same
"contents" data of the same "contents_ty" type are considered of the
same kind. Therefore, two such tickets can be **joined** into one
ticket and the output "amount" will be the sum of those of the two
input tickets. Smart contracts can join tickets via the JOIN_TICKETS
instruction. Tickets of the same kind are automatically joined when
they belong to the same user account.


Transferring
------------

Once a ticket has been constructed by a smart contract, it may be
transferred to other contracts as follows:

* *Smart contract to user account*: Smart contracts can transfer a
  ticket to user accounts via TRANSFER_TOKENS. To do so, the contract
  needs to cast the address of the target user account to type
  "contract (ticket cty)" where "cty" is the type of the content of
  the ticket to be sent. This can be done using "CONTRACT (ticket
  cty)". The rest is the same as making a contract call. The following
  Michelson snippet is an example sending a ticket of amount "10" with
  a "string" content ""some ticket"" to a user account address made
  available at the top of the stack.

   # Stack: address :: S
   CONTRACT (ticket string) ;
   # Stack: option (contract (ticket string)) :: S
   ASSERT_SOME ;
   # Stack: contract (ticket string) :: S
   PUSH mutez 0 ;
   # Stack: mutez :: contract (ticket string) :: S
   PUSH nat 10 ;
   # Stack: nat :: mutez :: contract (ticket string) :: S
   PUSH string "some ticket" ;
   # Stack: string :: nat :: mutez :: contract (ticket string) :: S
   TICKET ;
   # Stack: option (ticket string) :: mutez :: contract (ticket string) :: S
   ASSERT_SOME ;
   # Stack: ticket string :: mutez :: contract (ticket string) :: S
   TRANSFER_TOKENS ;
   # Stack: operation :: S

* *Between smart contracts*: Contracts can send tickets to other
  contracts via regular contract calls using the instruction
  "TRANFSER_TOKENS", as long as target contracts accept tickets of
  matching content type in their "parameter"s.

* *Between user accounts*: User accounts can transfer existing tickets
  they own to other user accounts with "Transfer_ticket" operation
  from their wallets. For instance, "octez-client" can be invoked in
  the following way by a user account holder "alice" to transfer a
  ticket of amount "10", type "string", content ""some ticket"" and
  ticketer "ticketer" to another user account held by "bob".

   octez-client transfer 10 tickets from alice to bob with entrypoint default and type string and content '"some ticket"' and ticketer 'ticketer'

If the amount of the ticket ""some ticket"" owned by "alice" was
greater than 10, this transfer would cause an automatic ticket split,
after which "alice" would continue to own the remaining amount.

Note that the above command uses the expression "transfer 10 tickets"
of the given kind, instead of "transfer a ticket of amount 10".
Indeed, by virtue of automatic joining of tickets of the same kind
belonging to the same user account, this formulation does not
introduce any ambiguity: the user account cannot hold several distinct
tickets of this kind. Of course, these expressions would not be
interchangeable for tickets belonging to a smart contract.

* *User account to smart contract*: Using the same "Transfer_ticket"
  operation, user accounts can also send their tickets to smart
  contracts. In this case, the entrypoint, as defined by the
  specification of the target contract, must accept this kind of
  tickets. Here is an example using "octez-client" to transfer a
  ticket of amount "10", type "string", content ""some ticket"" and
  ticketer "ticketer" owned by "alice" to a contract "receiver"
  accepting tickets at the entrypoint "save".

   octez-client transfer 10 tickets from alice to receiver with entrypoint save and type string and content '"some ticket"' and ticketer 'ticketer'

In this case, too, automatic ticket splitting may happen, under the
same circumstances.

Note:

  This page does not cover transfers of tickets to/from rollups. For
  that, refer to the documentation pages of particular rollups (e.g.
  Smart Optimistic Rollups).


Destroying
----------

Only smart contracts can destroy tickets, by simply dropping them.
That is, by not storing them anymore in the contract storage and not
sending them to other contracts. User accounts, on the other hand,
cannot destroy any ticket in their possession.

For instance, a user account "A" may receive a "string" ticket of
amount "2" with content "Lorem ipsum" created by a smart contract "B".
"A" may send amount "1" of it to a smart contract "C". This
automatically splits the ticket into two tickets of amount "1". Now
"C" may destroy this ticket by dropping it and "A" will still hold a
ticket of amount "1" with a "string" content "Lorem ipsum" created by
"B". It will remain in the possession of "A" until "A" sends it to
another user account or smart contract.

\newcommand\F[2]{\mathit{#1}\left(#2\right)}
\newcommand\IL[1]{\normalsize{#1}} \newcommand\exp[1]{\F{exp}{#1}}


Adaptive Issuance
*****************

Adaptive Issuance is a novel mechanism regulating tez issuance in
Tezos.

The Tezos economic protocol issues new tez via:

* Participation rewards: incentives given to delegates for
  participation in consensus and random seed generation.

* The Liquidity Baking (LB) subsidy.

* Protocol "invoices": lump sums of tez issued and allocated during
  protocol migration.

Participation rewards and the LB subsidy are regularly issued by the
protocol, whereas the value and recipients of invoices are defined
discretionarily by the developers of a protocol proposal. The values
for participation rewards and the LB subsidy, if any, are currently
defined by the Tezos protocol using fixed constants.

Adaptive Issuance lets the amount of participation rewards depend on
the global **staked ratio** – that is, the ratio of staked tez to the
total supply. This lets issuance roughly match the *actual* security
budget the chain requires, the amount needed to encourage participants
to stake and produce blocks, but *no more*.

At the end of each blockchain cycle, the regular issuance is adjusted,
to nudge the staked ratio towards a protocol-defined target (set at
50% starting in the Paris protocol). Participation rewards are
recomputed to match that budget. When the staked ratio decreases and
diverges from the target, emission rates increase, incentivizing
participants to stake funds to re-approach the target. Conversely,
incentives decrease as the ratio increases beyond the target.


Adaptive issuance rate
======================

The adaptive issuance rate determines, at the end of cycle \IL{c}, the
issuance for cycle \IL{c + 3}. The adaptive issuance rate is the sum
of a static rate and a dynamic rate. This value is kept within a
minimal and a maximal value, to ensure nominal emissions remain within
reasonable bounds.


Staked ratio
------------

The **staked ratio** is the ratio of staked tez to the total supply.
It is computed at the end of a given "cycle":

   def staked_ratio(cycle):
     return total_frozen_stake(cycle + 1 + issuance_modification_delay) / total_supply(cycle)

Where:

* "issuance_modification_delay" is a derived protocol constant which
  is set to the same value as "CONSENSUS_RIGHTS_DELAY" (see Validator
  selection), that is, 2 cycles.

* "total_supply(cycle)" returns the total supply of tez at the end of
  the given "cycle".

* "total_frozen_stake(cycle)" returns the total frozen stake at the
  given "cycle".


Static rate
-----------

The **static rate** is a mechanism which approximates a Dutch auction
to compute a nominal issuance rate as a function of the staked ratio
for a given cycle. Its value decreases as the staked ratio increases.
The static rate is defined as follows:

   def static_rate(cycle):
     return 1 / 1600 * (1 / (staked_ratio(cycle) ** 2))

The choice of a scaling factor ensures that the curve takes reasonable
values for plausible staked ratios. Moreover, since Adaptive Issuance
is activated with a dynamic rate of 0, and at current staked ratio
(that is, ~7.5% of the total supply), this factor allows for a smooth
transition from the issuance rate (~4.6%) from the Oxford protocol
(before the activation of Adaptive Issuance).


Dynamic rate
------------

The **dynamic reward rate** adjusts itself over time based on the
distance between the staked ratio and the 50% (±2%) target ratio,
increasing when < 48% and decreasing when > 52%. The dynamic rate is
defined as follows:

   def dynamic_rate(cycle):
     seconds_per_cycle = blocks_per_cycle * minimal_block_delay
     days_per_cycle = seconds_per_cycle / 86400
     previous_dynamic = dynamic_rate(cycle - 1)
     staked_ratio = staked_ratio(cycle)
     if staked_ratio < 0.48:
       delta_d = (0.48 - staked_ratio) * growth_rate * days_per_cycle
     elif staked_ratio > 0.52:
       delta_d = (0.52 - staked_ratio) * growth_rate * days_per_cycle
     else:
       delta_d = 0
     return previous_dynamic + delta_d

Where:

* "blocks_per_cycle" denotes the number of blocks in a Tezos cycle.

* "minimal_block_delay" denotes the minimal duration of a block in
  seconds.

* "days_per_cycle" denotes the minimal duration in days of a Tezos
  cycle, assuming all blocks in the cycle are produced at the minimal
  allowed time – that is, every 8 seconds in Quebec.

* "growth_rate" controls the speed at which the dynamic rate adjusts.
  The value is set so that a one percentage point deviation of the
  staked ratio changes the dynamic rate by 0.01 percentage points per
  day.

In a nutshell, "dynamic_rate(c)" increases and decreases by an amount
proportional to the distance between the target rate and the interval
"[48%; 52%]". Note that to ensure that the issuance rate is kept
within the minimum and maximum bounds, the dynamic rate might be
adjusted accordingly. More precisely, if the issuance rate would
surpass the maximum issuance allowed for a given cycle, then
"dynamic_rate(c)" would be reduced to keep the issuance rate within
the bounds (this part of the formula has been omitted from the above
pseudocode for brevity).


Minimum and maximum rates
-------------------------

Starting in Paris, the minimum and maximum issuance rates will evolve
slowly over a set period of time, so that the range of possible
issuance rate values widens progressively.

The following figure describes the progressive maximum and minimum
values of Adaptive Issuance.

   [image]Figure 1. A gradual widening of the range ensures a smooth
   transition to Adaptive Issuance.

The schedule consists of three periods:

* an **initial** period, set to 1 month, where the minimum and maximum
  issuance rates are close to the issuance rate from the Oxford
  protocol (before the activation of Adaptive Issuance) and stay
  constant,

* a **transition** period, set to 5 months, where they evolve
  linearly, with a decreasing minimum, and an increasing maximum, and

* a **final** period where the minimum and maximum have reached their
  final values.

Formally, the functions for the minimum and maximum values are
piecewise linear functions of time, and can be generally defined as
follows:

   def compute_extremum(cycle, initial_value, final_value):
     trans = transition_period + 1
     initial_limit = ai_activation_cycle + initial_period
     trans_limit = initial_limit + trans
     if cycle <= initial_limit:
         return initial_value
     elif cycle >= trans_limit:
         return final_value
     else:
         t = cycle - initial_limit
         res = (t * (final_value - initial_value) / trans) + initial_value
         return res

Where:

* "ai_activation_cycle" is the first cycle with Adaptive Issuance
  active, which was cycle 748 on mainnet.

* "initial_period" is a predefined period of time, set to 1 month in
  Paris.

* "transition_period" is a predefined period of time, set to 5 months
  in Paris.

The issuance minimum rate for Adaptive Issuance curve is then defined
as follows.

   def minimum_rate(cycle):
     return compute_extremum(cycle, issuance_initial_min, issuance_global_min)

Where:

* "issuance_initial_min" (4.5%) is the initial minimum value. The
  issuance rate is kept above this bound for the initial period.

* "issuance_global_min" (0.25%) is the final value for the lower
  bound, reached at the end of the transition period.

The issuance maximum rate for Adaptive Issuance curve is then defined
as follows.

   def maximum_rate(cycle):
     return compute_extremum(cycle, issuance_initial_max, issuance_global_max)

Where:

* "issuance_initial_max" (5.5%) controls the initial maximum value.
  The issuance rate is kept below this bound for the initial period.

* "issuance_global_max" (10%) is the final value for the upper bound,
  reached at the end of the transition period.


Adaptive Maximum
----------------

In addition to the issuance bounds described above, the Quebec
protocol introduces another upper bound on the total issuance rate.
This new bound, called **adaptive maximum**, prevents the issuance
rate from taking undesirably high values when the staked ratio of the
chain is already close to the 50% target.

The adaptive maximum is defined as a function of the staked ratio that
decreases smoothly as the staked ratio grows closer to the 50% target,
down to a 1% maximal issuance rate when the staked ratio is at 50% or
above. See the blue curve on Figure 2, as well as a few sample values
in Figure 3. The exact formula is provided further below.

Thanks to the adaptive maximum, the total issuance rate (static rate +
dynamic rate) cannot be too far above the static rate (green dashed
curve). More precisely, the total issuance rate is forced to stay
between the static rate (green) and the adaptive maximum (blue). In
other words, the dynamic rate is effectively bounded to stay below the
red dotted curve, which plots the adaptive maximum minus the static
rate. Note that this no longer holds if the adaptive maximum gets
overridden by the minimum issuance, as discussed below.

   [image]Figure 2. Adaptive maximum compared to the static rate, in
   the range from 5% to 50% staked ratio.


Figure 3. Adaptive maximum: a few data points.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Staked ratio   | 5% and below   | 10%            | 20%            | 30%            | 40%            | 50% and above  |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Adaptive       | 10%            | 9.2%           | 5.6%           | 3%             | 1.5%           | 1%             |
| maximum        |                |                |                |                |                |                |
| issuance       |                |                |                |                |                |                |
| (approx.)      |                |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+

The function that defines the adaptive maximum is:

   def adaptive_maximum(r): # r is the staked ratio of the chain
     if r >= 0.5:
       return 0.01
     elif r <= 0.05:
       return 0.1
     y = (1 + 9 * ((50 - 100 * r) / 42 ) ** 2 ) / 100
     if y > 0.1:
       return 0.1
     elif y < 0.01:
       return 0.01
     else:
       return y

Note:

  Until the final value of the minimum rate is reached, it is
  possible, with a high enough staked ratio, for the corresponding
  adaptive maximum to be smaller than the minimum rate. If this
  happens, then the minimum rate takes priority, that is, the total
  issuance rate (static + dynamic) is set to the minimum rate exactly.
  This way, the total issuance rate is as close to the adaptive
  maximum as can be while fully satisfying the minimum rate.


Issuance rate
-------------

Finally, as mentioned before, the nominal adaptive issuance rate [1]
for a cycle "c + issuance_modification_delay + 1" is defined as the
sum of the static rate and the dynamic rate computed for the cycle
"c", bounded within the minimum and maximum rates, along with the
adaptive maximum, computed for the cycle "c + 1".

   def issuance_rate(cycle):
     adjusted_cycle = cycle - issuance_modification_delay
     static_rate = static_rate(adjusted_cycle - 1)
     dynamic_rate = dynamic_rate(adjusted_cycle - 1)
     minimum_rate = minimum_rate(adjusted_cycle)
     maximum_rate = min(maximum_rate(adjusted_cycle), adaptive_maximum(stake_ratio(adjusted_cycle)))
     total_rate = static_rate + dynamic_rate
     return max( min(total_rate, maximum_rate), minimum_rate )


Adaptive rewards
================

Before Adaptive Issuance activation, participation rewards were fixed
values defined by protocol constants. With the new mechanism, the
adaptive issuance rate provides instead a budget for the whole cycle,
which gets allocated equally to each block of the cycle and
distributed between the various rewards, in proportion to their
relative weights.


Reward weights
--------------

The weights for participation rewards are defined as:

* Attestation rewards: 10,240.

* Fixed baking reward: 5,120.

* Bonus baking reward: 5,120.

* DAL rewards: 2,275.

* Nonce revelation tip: 1.

* VDF tip: 1.

The total sum "sum_rewards_weight" of all weights is 22,757.

   sum_rewards_weight = (
     attestation_rewards +
     fixed_baking_rewards +
     bonus_baking_rewards +
     dal_rewards +
     nonce_revelation_tip +
     vdf_tip)

The coefficient to apply for reward computation is defined as follows.

   def reward_coeff(cycle):
     rate = issuance_rate(cycle)
     total_supply = total_supply(cycle - issuance_modification_delay - 1)
     return (rate / 525600) * total_supply / base_total_issued_per_minute

Where:

* "base_total_issued_per_minute" (80007812) is the expected amount of
  mutez issued per minute.

* 525600 is the number of minutes per year.

The issuance per block is then distributed amongst the different
rewards in proportion to their weight.

   def tez_from_weights(weight):
     num = weight * minimal_block_delay
     den = sum_rewards_weight * 60
     return base_total_issued_per_minute * num / den

   def reward_from_constants(cycle, weight):
     return tez_from_weights(weight) * reward_coeff(cycle)

**Consensus rewards.** Since the adoption of Tenderbake, Tezos
protocols before Paris have rewarded delegates for their participation
in consensus with the following rewards per block:

* A fixed **baking** reward, given to the delegate which produced the
  *payload* of the block (i.e. choosing transactions, and other non-
  consensus operations).

* A variable, baking **bonus** reward given to the delegate which
  produced the block included in the chain. This bonus is given for
  including attestations, if their combined attesting power exceeds
  the minimal threshold (two thirds of total slots).

* A *collective* **attestation** reward, for attesting block
  proposals, distributed at the end of the cycle to the delegates
  selected in the consensus committees for that cycle, proportionnaly
  to their expected participation.

We refer to the consensus page for further insight on the pre-
requisites and distribution of these rewards. Here, we derive the new
functions which compute their values per block.

   def baking_reward_fixed_portion(cycle):
     return reward_from_constants(cycle, fixed_baking_rewards)

   def baking_reward_bonus_per_slot(cycle):
     bonus_committee_size = consensus_committee_size - consensus_threshold
     return reward_from_constants(cycle, bonus_baking_rewards) / bonus_committee_size

   def attestation_reward_per_slot(cycle):
     return reward_from_constants(cycle, attestation_rewards) / consensus_committee_size

Where:

* "consensus_committee_size" (7000) is the number of attestation slots
  available in a block.

* "consensus_threshold" (4667) is the required number of attestations
  for a baker to propose a block.

**Nonce and VDF revelation tips.** The rewards allocated to delegates
for contributing to random seed generation (that is, for revealing
nonce seeds and posting VDF proofs) are not paid each block, but
rather every 192 blocks.

   def seed_nonce_revelation_tip(cycle):
     return reward_from_constants(cycle, nonce_revelation_tip * blocks_per_commitment)

   def vdf_revelation_tip(cycle):
     return reward_from_constants(cycle, vdf_tip * blocks_per_commitment)

Where:

* "blocks_per_commitment" (192) is the interval in blocks between each
  revelation, both VDF and nonce.

The RPC endpoint, "/issuance/expected_issuance" reports the
precomputed values of all participation rewards for the provided block
and the next "issuance_modification_delay" cycles.

[1] Note that if the nominal annual issuance rate is r, the annualized
    rate is close to \IL{\exp{r} - 1} as it is compounded at every
    cycle.

Note:

  For operational details about the staking mechanism and its
  configuration, see a staking mechanism tutorial.


Staking mechanism
*****************

Staking is an evolution of the existing Tezos Liquid Proof-of-Stake
mechanism. It introduces a new role for network participants, called
**staker**, complementary to the existing delegate (also known as
*baker*) and *delegator* roles. A staker must also be a *delegator* –
that is, they must first choose a delegate.

When stakers **stake** funds towards a delegate’s **staking**
**balance**, the associated **baking** and **voting powers** accrue to
that delegate. Similarly to how delegated funds work, staked funds
remain within the staker’s account at all times.

Staked and delegated funds **have different weights** in the
computation of delegates’ baking and voting powers: staked funds (both
external stakes by stakers and the delegate’s own) count **three
times** as much as delegated funds.

Unlike delegated funds, staked funds are considered to contribute to
the security deposit associated with their chosen delegate. Thus, they
are subject to slashing if the delegate misbehaves by double-signing
block proposals or consensus operations, and are subject to the same
withdrawal delays – colloquially, they are "frozen".

Stakers are slashed proportionally to their contribution to the
delegate’s staking balance. To simplify slashing, double-baking
penalties are now proportional to staked funds: instead of the
previous fixed sum of 640 tez they are now set to 5% of the delegate’s
stake. Moreover, denunciation rewards (both for double-baking and
double-attestations) are reduced from one half to one eleventh of the
slashed funds. The chosen value prevents adversarial delegates from
abusing the slashing mechanism for profit at the expense of their
stakers.

Participation rewards are automatically shared between delegates and
their stakers. Staker's rewards are proportional to their
participation in the delegate's total staked at the time the rewards
are given. This means that the staker gets rewards for staked tez as
soon as they are staked, and stops receiving rewards as soon as the
tez are unstaked, disregarding the fact that baking rights for the
delegate are computed with some delays. *Delegates* configure their
staking policy by setting staking parameters which regulate whether
they accept stakers (the default being to reject them), and if so, up
to which fraction of their total staking balance. They can also
configure which proportion of the staking rewards from other stakers
is set to accrue to their own staked balance instead, thereby
collecting an *edge* from the rewards attributable to their stakers.

Freezing and unfreezing of staked funds is controlled directly by
delegates and stakers. This entails that staked funds are frozen until
manually unfrozen by stakers. This is a two step process which spans
for at least "UNSTAKE_FINALIZATION_DELAY" cycles (cf. Staked funds
management).

A user interface is provided for delegates and stakers to interact
with the mechanism. It is based on four *pseudo-operations*: "stake",
"unstake", "finalize_unstake", and "set_delegate_parameters". Pseudo-
operations are self-transfers: a transfer operation where the
destination matches the source – each involving a special entry-point
of the same name introduced for user accounts. This approach was
chosen to minimize the work required by wallets, custodians,
exchanges, and other parties to support the functionality.

**NB** In the current implementation, only *user accounts* can become
stakers. In other words, smart contracts cannot stake funds (they can
of course still delegate them).


Staking policy configuration
============================

*Delegates* can configure their staking policy by setting the
following parameters:

* "edge_of_baking_over_staking": a ratio between 0 and 1, whose
  default value is 1. This parameter determines the fraction of the
  rewards that accrue to the delegate's frozen deposit – the remainder
  is shared among its stakers.

* "limit_of_staking_over_baking": a non-negative number, denoting the
  maximum portion of external stake by stakers over the delegate’s own
  staked funds. It defaults to 0 – which entails that delegates do not
  accept external stakes by default. It is moreover capped by a global
  constant, set to 9 starting in the Quebec protocol, which ensures
  the baker controls a significant part of the stake.

Delegates can modify these staking parameters at all times, using the
"set_delegate_parameters" pseudo-operation: that is, by transferring 0
tez to their own "set_delegate_parameters" entry-point. The chosen
values for both parameters need to be supplied. The new parameters are
then applied "DELEGATE_PARAMETERS_ACTIVATION_DELAY" (currently 5)
cycles later.

   octez-client transfer 0 from <delegate> to <delegate> --entrypoint set_delegate_parameters --arg "Pair <limit as int value in millionth> (Pair <edge as int value in billionth> Unit)"

or more conveniently:

   octez-client set delegate parameters for <delegate> --limit-of-staking-over-baking <value> --edge-of-baking-over-staking <value>

Once you have set your parameters, if you want to update only one
parameter you can use:

   octez-client update delegate parameters for <delegate> [--limit-of-staking-over-baking <value>] [--edge-of-baking-over-staking <value>]

The command will fill the missing parameter with the value of the last
pending parameters (or the active parameters if no set/update are
pending).

Please ensure to not send two updates in a row without waiting for the
first update operation to be included (i.e. do not use --wait none) or
you might end up with inconsistent parameters (The modification of one
parameter could be reverted by the subsequent modification of another
parameter).

**On overstaking and overdelegation.** Note that if a delegate’s
"limit_of_staking_over_baking" is exceeded (that is, the delegate is
*overstaked*), the exceeding stake is automatically considered as
*delegation* for the delegate’s baking and voting power calculation,
but it does remain slashable. The new mechanism does not alter
*overdelegation* (delegated funds beyond 9 times the delegate’s own
stake) nor its consequence on voting and baking powers. That is,
overdelegated funds are not counted towards a delegate baking power,
but they do increase their voting power.


Staked funds management
=======================

Stakers (and delegates) can use the "stake", "unstake", and
"finalize_unstake" pseudo-operations to control their stakes. Figure 1
illustrates their effect on a staker’s funds. Note that while these
pseudo-operations change the *state* of the involved funds, they
remain otherwise within the staker’s account at all times.

Note:

  Due to rounding that occurs at various stages of fund management,
  stakers' staked balances may deviate by a few mutez (millionth of
  tez) with respect to transferred amounts.

   [image]Figure 1: staked funds management using pseudo-operations.

To *stake* funds, a delegator uses the "stake" pseudo-operation,
transferring the chosen amount of **spendable** tez to their own
"stake" entry-point. The **staked** tez will then be frozen and
contribute to their chosen delegate’s staking balance. Note that the
"stake" pseudo-operation will fail if the sender account is not
*delegated*.

   octez-client transfer <amount> from <staker> to <staker> --entrypoint stake

or more conveniently:

   octez-client stake <amount> for <staker>

The staked amount and remaining spendable funds can then be checked
respectively with:

   octez-client get staked balance for <staker>
   octez-client get balance for <staker>

To *unstake* funds, a staker first submits an unstake request with the
"unstake" pseudo-operation. This is implemented by transferring the
chosen amount in tez to their "unstake" entry-point:

   octez-client transfer <amount> from <staker> to <staker> --entrypoint unstake

or more conveniently:

   octez-client unstake <amount|"everything"> for <staker>

The requested amount will be **unstaked** but will remain **frozen**,
a.k.a. **unfinalizable**.

After "UNSTAKE_FINALIZATION_DELAY + 1" cycles (more precisely, after
the cycle in which the unstake was requested has ended and then
another UNSTAKE_FINALIZATION_DELAY full cycles have passed), unstaked
frozen tokens are no longer considered at stake nor slashable. They
are said then to be both **unstaked** and **finalizable**.

A staker can retrieve all unstaked and finalizable tokens at any time,
making them spendable again. This is done using the "finalize_unstake"
entrypoint -– that is, by transferring 0 tez to their
"finalize_unstake" entry-point:

   octez-client transfer 0 from <staker> to <staker> --entrypoint finalize_unstake

or more conveniently:

   octez-client finalize unstake for <staker>

In some circumstances, unstake and finalize can be done implicitly:
any call to "stake" or "unstake" will implicitly finalize all
currently finalizable pending unstake requests. Also, as we will see
next, change of delegate triggers an unstake operation.


Change of delegate
==================

When a staker changes its delegate, the operation will trigger an
implicit unstake request for the full frozen deposit of the staker.

As long as the unstake request is not finalized, the frozen tokens
will continue to be delegated to the old delegate, however the
spending balance of the account is accounted in the new delegate's
stake. It will not be possible to stake with the new delegate as long
as there are unfinalizable unstake request for token staked with the
old delegate.

Adaptive Slashing
*****************


Overview
========

In Oxford2, each instance of a baker's misconduct results in the
slashing of a fixed percentage of their staked funds. However, this
approach does not distinguish between innocent mistakes and deliberate
attacks. The rule conservatively punishes every misconduct with strong
penalties. The rationale for this model is its simplicity and the
observation that the cost of avoiding these mistakes is low anyway.
Nonetheless, it overlooks the impact of double signing of attestations
on the entire block committee, treating all cases uniformly and
ignoring collusion among bakers.

To better reflect this distinction, protocols starting with Paris
adjusts the amount of slashing based on the fraction of double-
attestation in a single block. A low fraction of misconduct incurs
moderate penalties, while a high fraction of misconduct is deemed to
be critical and faces more serious repercussions.

This document presents the definition of the adaptive slashing
function implementing this idea, as well as the new forbidden period.


Adaptive Slashing Function
==========================


Informal presentation
---------------------

For a given block, we call “fraction of double-attestations” the ratio
between the total weight of attestations for which a valid
denunciation has been included and the total weight of possible
attestations in a block.

The shape of the slashing function for double attestations is the
following:

   [image]

+--------------------------------------------+------------------------------------------+
| f: fraction of double attestations         | S: slashing ratio                        |
|============================================|==========================================|
| 1.00%                                      | 0.09%                                    |
+--------------------------------------------+------------------------------------------+
| 5.00%                                      | 2.25%                                    |
+--------------------------------------------+------------------------------------------+
| 10.00%                                     | 9.00%                                    |
+--------------------------------------------+------------------------------------------+
| 20.00%                                     | 36.00%                                   |
+--------------------------------------------+------------------------------------------+
| 23.57%                                     | 50.00%                                   |
+--------------------------------------------+------------------------------------------+
| 33.34%                                     | 100.00%                                  |
+--------------------------------------------+------------------------------------------+
| 100.00%                                    |                                          |
+--------------------------------------------+------------------------------------------+

Instead of using a constant function as in Oxford2, we use a convex
function that saturates at 100% when a critical fraction of doubled
attestations are issued. Accidental double attestations are unlikely
to cause a large amount of slashing to be applied, but concerted
attacks result in severe penalties.

**Remark 1.** Even though the baker does not attest to a weight
exactly equivalent to its staked funds at each block, the slashing
function is applied to bakers’ total staked funds for simplicity.

**Remark 2.** As in Oxford2, slashing happens at the end of each
cycle. Since the denunciation period for a block ranges over its cycle
and the next one, a baker can be punished for misbehaving during the
previous cycle, not only the one that has just ended.

**Remark 3.** If a baker wants to get back their at-stake funds, it
takes more than 2 cycles to complete the unstaking process. This
ensures that the baker can't decrease their at-stake funds after being
denunciated and before facing penalties.


A formal definition of slashing function for double-attestations
----------------------------------------------------------------

* \mathcal{W} denotes the maximal possible *weight* of attestations in
  a block, that is, the fixed number of available slots in any block.
  It is also known as CONSENSUS_COMMITTEE_SIZE.

* f(B) is the *fraction of double attestations* for block B, that is,
  the ratio of the total weight of double attestations in B, over
  \mathcal{W}.

* T is the *threshold* for the fraction of double attestations to be
  considered critical. A typical value for T is {1 \over 3}
  \mathcal{W}, which is the difference between \mathcal{W} and the
  CONSENSUS_THRESHOLD which is set to {2 \over 3} \mathcal{W}.

We define S(B) the percentage of slashed funds for all misbehaving
bakers at the block B as follows:

S(B) = \text{min} (100\%, {1 \over T^2} \cdot f(B)^2 \cdot 100\%)

Then, the percentage of slashed funds S(b,C) for a baker b at the end
of the cycle C is defined as follows:

S(b, C) = \text{min} (100\%, \sum_{(b, B) \in C} S(B))

where (b, B) \in C means that:

* the double attestation by baker b of block B has been denounced
  before the end of cycle C, and

* C is the last cycle of the denunciation period for B.


New definition for the forbidden period
=======================================

Given that slashing occurs with a delay, immediate action at
denunciation time is necessary upon clear evidence of a baker's
misbehavior to prevent further misconduct, or to protect the baker
against their own faulty setup. Any double-signing denunciation
immediately triggers the beginning of a **forbidden period** that
lasts at least 2 cycles, to make sure the slashing occurs before
accepting new attestations or blocks from the baker.

Note that it is still possible for one baker to commit multiple double
signings, but only if they all happen before any corresponding
denunciation gets included in a block.

This forbidding is lifted as soon as both following conditions are
met:

* all pending slashings for the delegate have occurred, and

* the current total frozen stake for the delegate (sum of the staking
  balances of the delegate itself and its stakers) is at least as high
  as the active stake that was used "CONSENSUS_RIGHTS_DELAY" cycles
  ago to compute the consensus rights for the next cycle.

The second condition may be fulfilled when the delegate and/or stakers
stake additional funds so that the total frozen stake grows back to
its pre-slashing value, thus matching the rights computed before the
slashing. Or it may be fulfilled "CONSENSUS_RIGHTS_DELAY" cycles after
the slashing, when the rights for the next cycle are finally based on
the post-slashing stake.

DAL integration
***************

The integration for the DAL within the economic protocol relies on
three operations:

1. "DAL_publish_commitment": a manager operation, allowing anyone to
   publish a DAL commitment

2. "attestation": the existing consensus operation, allowing bakers to
   attach a DAL payload attesting the data seen on the DAL P2P network

3. "DAL_entrapment_evidence": an anonymous operation to denounce a
   baker that has attested a trap shard

and on an incentives scheme for the DAL.


DAL publish commitment
======================

"DAL_publish_commitment" is a manager operation that can be issued by
a user wishing to publish data onto the DAL. The payload of this
operation consists in the following fields:

* Slot index: Identifies the specific slot for which the data is being
  published. It is an integer between "0" and "NUMBER_OF_SLOTS - 1".

* Commitment: The KZG commitment over the data.

* Commitment proof: A proof that the commitment commits over data that
  does not exceed the size "SLOT_SIZE".

Users can create and manage these commitments and proofs through the
DAL node using these RPCs:

* To create a commitment: "POST /commitment"

* To retrieve a commitment’s proof: "GET
  /commitments/<commitment>/proof"


Concurrent operations
---------------------

If a block contains a valid "DAL_publish_commitment" operation, any
subsequent operations of the same kind within the same block (and the
same slot index) will be recognized as valid but will fail during
execution. They still incur transaction fees. More details about this
can be found in the validator documentation.


Economics
---------

Currently, the fees are estimated based on the execution cost of this
operation alone. There are no additional charges related to the
bandwidth required for bakers to download data from the DAL for this
commitment. However, this might be subject to changes in the future.


DAL attestation payloads
========================

The attestation operation includes an optional field "dal_content".
This field allows attesters participating to the DAL to announce
whether they were able to successfully download the shards assigned to
them. Concretely, this field is a bitset reflecting the status of each
slot. The size of the bitset corresponds to the total number of slots.
A value of 1 indicates successful retrieval of all assigned shards by
the baker for that slot, while 0 indicates an unsuccessful attempt.
The least significant bit corresponds to the smallest slot index.


Attestation timing
------------------

When a commitment is published at a certain level, say level "n", the
corresponding DAL payloads are expected to be included in the
attestations contained in the block at level "n + ATTESTATION_LAG".


Block metadata
--------------

In the block’s metadata, there is a specific field for the DAL. This
field reflects the availability of slots based on the DAL payloads
received. It is a bitfield with one bit per slot (its format is the
same as the attestation payload of the "attestation" operation). The
bit is set to 1 if the slot is declared available. The smallest slot
index corresponds to the least significant bit. To consider a slot as
available, there must be a minimum number of shards, as defined by the
"AVAILABILITY_THRESHOLD" parameter, marked as available by the
attesters for that slot (e.g. if the number of shards is 2048 and the
availability threshold is 50%, then 1024 shards are required).

Therefore, for data committed (published) at level "n", the slot's
availability is determined by the metadata of the block at level "n +
ATTESTATION_LAG". Consequently, a smart rollup can only utilize this
data from level "n + ATTESTATION_LAG + 1" onward.


DAL incentives scheme
=====================


Overview
--------

Bakers must meet a 64% minimal participation threshold in a cycle to
earn a fixed percentage of the total participation rewards allocated
for them. As part of participation rewards, the DAL rewards are
subject to the adjustments done by Adaptive Issuance.

To ensure DAL attestations match the actual availability of data
shards, there are special shards, known as *traps*, which are designed
to test whether bakers have genuinely downloaded and processed their
assigned shards. Bakers must correctly identify these traps to avoid
losing the DAL rewards allocated to them.


Minimal participation
---------------------

Any baker that can take part in consensus is eligible for rewards.

Bakers meeting the minimum participation ratio
"MINIMAL_PARTICIPATION_RATIO" over a cycle, set to 64%, receive
rewards for that cycle.

The participation ratio of the baker is the proportion of slots the
baker attested over the total number of slots deemed available by the
protocol in the cycle.


DAL participation rewards
-------------------------

A fixed percentage, defined by a protocol parameter called
"REWARDS_RATIO", set to 10%, of the total participation rewards is
allocated to the DAL.

The DAL rewards per level are implicitly given by their weight,
"DAL_REWARDS_WEIGHT", as for the other types of participation rewards.
The value of "DAL_REWARDS_WEIGHT" is such that it represents
"REWARDS_RATIO" of all reward weights.

The rewards are distributed at the end of a cycle, and are computed in
the same manner as for the other participation rewards. For instance,
the stakers' share of these reward is proportional to the weight of
their stake in relation to their baker's baking power.


Trap mechanism
--------------

A deterministic function "trap(pkh, shard)" returning a boolean flag
indicates whether a shard is a trap for a specific baker identified by
its public key hash ("pkh").

The protocol parameter "TRAPS_FRACTION" controls the fraction of
shards marked as traps.

Bakers detect traps by retrieving shard content via their DAL node and
applying the trap function. A trap invalidates the corresponding
attestation: the baker should not attest a slot if one of the slot’s
shards assigned to him is a trap.

The "DAL_entrapment_evidence" accusation operation can be used to
accuse a baker of wrongly attesting a slot due to an undetected trap.
This accusation operation includes the attestation operation
containing the wrongly attested slot, the slot index, and the
undetected shard.

As for double-signing accusations, any baker can include a DAL
accusation in its block. Accusations can be included during a period
of "DENUNCIATION_PERIOD" cycles after the misbehavior event, which is
that of the corresponding attestation operation.


Penalties
---------

A baker that is correctly accused, through an accusation operation
included in a block, loses their DAL rewards for the cycle containing
the block.


Smart Rollups integration
=========================

The DAL is integrated with smart rollups so that kernels can request
pages from the DAL via the reveal data channel. A smart rollup can
fetch any page from the DAL node if the commitment respects some
conditions:

* The commitment should have been published after the rollup
  origination (this constraint will be leveraged so that the kernel
  can request any commitment in the past)

* The commitment should not have been published in a level in the
  future after the level of the next commitment of the state (at most
  30 levels in the future). This constraint might be leveraged thanks
  to the internal messages "Start of Level" and "End of Level"
  (abbreviated as SOL and EOL), which allow the kernel to track the L1
  level, so that the kernel cannot request a page for a commitment
  published after the current L1 level.

If the kernel requests a page that does not satisfy the mentioned
conditions, then the rollup node must answer with an empty page
indicating there is no data. Similarly, if the kernel requests for a
commitment that was not attested, the rollup node must also provide an
empty page. As a consequence, if a kernel requests a page for a
commitment for which its availability is still unknown, the rollup
node cannot answer and will have to wait for the availability status
to be confirmed by the L1.

Moreover, the rollup kernel has access to the protocol constants so
that the same kernel code can be used on different test networks.


DAL-related protocol constants
==============================

This section describes the protocol constants specific to the DAL as
well as their default values on mainnet (see Protocol constants on how
to find the values for tests networks):

* "FEATURE_ENABLE" (true): whether the DAL is available

* "INCENTIVES_ENABLE" (true): whether baker incentives are available

* "NUMBER_OF_SLOTS" (32): how many slots are available per block

* "ATTESTATION_LAG" (8 level): the timeframe for bakers to download
  shards between the published level of a commitment and the time they
  must attest the availability of those shards

* "ATTESTATION_THRESHOLD" (66%): the minimum percentage of shards
  attested for a given slot to declare the slot available

* "PAGE_SIZE" (3967B, ~4KiB): the size of a page (see DAL slots)

* "SLOT_SIZE" (126944B, ~128KiB): the size of a slot (see DAL slots)

* "REDUNDANCY_FACTOR" (8): the erasure-code factor (see DAL slots)

* "NUMBER_OF_SHARDS" (512): the number of shards per slot (see DAL
  slots)

* "MINIMAL_PARTICIPATION_RATIO" (64%): the minimum percentage of slots
  attested by a baker during a cycle (among all slots deemed
  available) that entitles them to rewards

* "REWARDS_RATIO" (10%): the ratio of the DAL rewards over the total
  participation rewards

* "DAL_REWARDS_WEIGHT" (2275): the weight of the DAL rewards (relative
  to other participation rewards)

* "TRAPS_FRACTION" (0.0005): the fraction of shards that are traps

Glossary
********

This glossary is divided in two sections:

* The first section contains Tezos generic terms (i.e., which are not
  related to a particular protocol), such as Tezos or block.

* The second section contains terms defined by the economic protocol,
  such as Tez/XTZ, account, smart contract, or delegate. The list of
  terms in this section and/or their definitions may be different for
  other protocol versions.


Tezos terms
===========

   The Tezos blockchain is a linked list of blocks (or actually, a
   tree when several competing branches exist). Blocks conceptually
   contain a header and a list of operations, which are specific to
   the economic protocol.

   The header itself decomposes into a shell header (common to all
   Tezos economic protocols), and a protocol-specific header. The
   shell header contains protocol-agnostic data such as the
   predecessor's block hash and the block's timestamp.

   The state of the blockchain. The context is defined by the economic
   protocol and typically includes information such as “this account
   is credited with this many tez” and “this is the code for that
   smart contract.”

   The context is modified by operations. For example, an operation
   can transfer tez from one account to another, which modifies the
   part of the context that tracks account credit.

   The economic protocol is the set of rules defining valid operations
   and blocks, how the network agrees on the next block to build (the
   consensus algorithm), and how operations update the blockchain
   state, also called context.

   In Tezos, the economic protocol can be upgraded without
   interruption or forking of the blockchain. This is because the
   procedure for an upgrade is also defined within the economic
   protocol, which can thus update itself.

 (a.k.a. score, weight)
   For each block, the consensus algotrithm can compute a score called
   fitness which determines the quality of the chain leading to that
   block. The shell changes the head of the chain to the valid block
   that has the highest fitness.

   See level.

 (a.k.a. block height)
   The position of a block in the blockchain, that is, the number of
   blocks since the genesis block, where the genesis block is at level
   0.

   A pool (set) of operations maintained by a node and not yet
   included in a block.

   A (block or operation) metadata is a piece of data computed as a
   result of the application of the block or operation on an
   associated context. The metadata consists of many pieces of
   information such as the operation receipts, rewards updates, voting
   period, etc.

   A block's metadata is the collections of operations metadata for
   all the operations included in the block (if the validation was
   successful).

   For a detailed metadata content check the Rio RPCs - Reference
   under the prefix "../<block_id>/metadata".

   A peer in the P2P network. It maintains a local state and
   propagates blocks and operations.

   An operation transforms the context; this is what makes the state
   of the chain change. Operations are grouped into blocks; thus, the
   chain progresses in batches. For the different kinds of operations
   defined by the protocol, see operation kinds.

   See fitness.

   The shell is a software component of the node. It is parameterized
   by a specific economic protocol. It serves as the bridge between
   the P2P layer (handling communication between nodes) and the
   economic protocol layer (handling the context, operation
   application, scoring, etc.).

   Used either as a noun or an adjective to designate:

   * An open-source project and software (as in, "contributing to the
     Tezos protocol")

   * A peer-to-peer network of nodes maintaining a blockchain (as in
     "a Tezos node")

   * The specific Tezos chain with the most economic relevance (as in
     "the Tezos chain"). In particular, the chain whose millionth
     block had hash
     BKtC4QCWoF73kxLj773vFpQuuwrnye6PS7T1aM3XEPvFXiQbNu7

   See fitness.


Protocol terms
==============

   When a delegate attempts double signing (or when it tries to abuse
   the network in another similar way), another delegate can make an
   accusation, by providing evidence of the offense. The delegate
   injecting the accusation in a newly baked block is called the
   accuser.

   The accuser is awarded some funds from the security deposit of the
   accused.

   When using Octez, accusation operations are emitted by the accuser
   daemon. Note that this daemon is not associated to a delegate:
   accusation operations are anonymous, and any delegate can include
   them in a block.

   An account is an address managed by the protocol. In the context,
   each account is associated with a balance (an amount of tez
   available).

   An account can be either a user account or a smart contract.

   When a block is created and propagated on the network, delegates
   that have attesting rights for the matching block level and round
   can emit an attestation operation. Attestation operations are
   included in the next block.

   See baking rights.

   When a delegate creates a new block, it is called the baker of this
   block. Baking rights are distributed to different delegates based
   on their baking power. Only a delegate with baking rights is
   allowed to bake. The baker selects transactions from the mempool to
   be included in the block it bakes.

   When using Octez, baking and other consensus actions are handled by
   the baker daemon, on behalf of one or more delegate accounts. By
   extension, a baker designates the owner of such a delegate account,
   typically running the baker daemon on its behalf.

   The act of creating a new block by a baker.

   The amount of tokens that determines a delegate's weight in the
   selection of its baking and attesting rights. A delegate's baking
   power is computed from the delegate's own tokens and the sums of
   tokens delegated and staked to it (where staked tokens weigh more
   than tokens just delegated). See Baking Power for details.

   Baking/attesting a block can only be done by a delegate who holds
   the baking/attesting right for that block level and round. At the
   start of a cycle, baking and attesting rights are computed for all
   the block levels and rounds in the cycle, based on the baking power
   of each delegate.

   For each block level and round, there is exactly one account that
   is allowed to bake, but several accounts are allowed to attest.

   To ensure responsible use of the storage space on the public
   blockchain, there are some costs charged to users for consuming
   storage. These costs are burnt (i.e., the amount of tez is
   destroyed). For example, a per-byte storage cost is burnt for
   increasing the storage space of a smart contract; a fixed amount is
   burnt for allocating a new contract (which consumes space by
   storing its address on the blockchain).

   See also fee.

   Protocols are parameterized by several parameters called protocol
   constants, which may vary from one protocol to another or from one
   network to another.

   A cycle is a sequence of consecutive blocks of fixed length (given
   by a protocol constant). E.g., cycle 12 started at block level
   49152 and ended at block level 53248.

   Cycles are used as a unit of “time” in the block chain. For
   example, the different phases in the amendment voting procedures
   are defined based on numbers of cycles.

   The length of a cycle is a (parametric) protocol constant, and thus
   might change across different Tezos protocols.

   A user account that can participate in consensus and in governance.
   Actual participation is under further provisions, like having a
   minimal stake. A user account becomes a delegate by registering as
   such. Through delegation, other accounts can delegate their rights
   to a delegate account.

   Note that "tz4" accounts cannot be delegates.

   An operation in which an account designates a delegate. The
   delegating account's balance increases the delegate's baking power
   and consequently its baking rights and attesting rights; it also
   increases its voting power. However, the delegate does not control
   the funds of the delegating account, e.g., it can not spend them.

   The situation when a baker signs two different blocks at the same
   level and same round, is called double baking. Double baking is
   detrimental to the network and might be indicative of an attempt to
   double spend. The same goes for signing two different attestations
   at the same level and the same round. As such, double signing
   (i.e., double baking or double attesting) is punished by the
   network: an accuser can provide proof of the double signing to be
   awarded part of the double signer's deposit -- see Slashing.

   The "Failing_noop" operation implements a *No-op*, which always
   fails at application time, and should never appear in applied
   blocks. This operation allows end-users to sign arbitrary messages
   which have no computational semantics.

   To ensure responsible use of computation resources of other nodes,
   and also to encourage active participation in the consensus
   protocol, users pay fees to bakers for including their operations
   in blocks. For example, fees are paid to a baker for operations
   such as a transaction or a revelation of a public key.

   Currently, only manager operations require collecting fees from its
   sender account.

   See also burn.

   A measure of the number of elementary steps performed during the
   execution of a smart contract. Gas is used to measure how much
   computing power is used to execute a smart contract.

   See user account.

   The primary blockchain i.e. the Tezos chain. Within any blockchain
   ecosystem, Layer 1 (L1) refers to the main chain to which side
   chains, rollups, or other protocols connect and settle to. The
   Layer 1 chain is deemed to be most secure, since it has the most
   value (or stake) tied to it, and be most decentralized and
   censorship resistant. However, transaction space is limited leading
   to low throughput and possibly high transaction costs. See Layer 2.

   Layer 2 (L2) includes sidechains, rollups, payment channels, etc.
   that batch their transactions and write to the Layer 1 chain. By
   processing transactions on layer 2 networks, greater scalability in
   speed and throughput can be achieved by the ecosystem overall,
   since the number of transactions the Layer 1 can process directly
   is limited. By cementing transactions from a L2 to L1, the security
   of the L1 chain backs those operations. Currently, Layer 2
   solutions on Tezos are built as smart rollups.

   The built-in language used by a smart contract.

   An amount of tez (e.g., 6000ꜩ) serving as a minimal amount for a
   delegate to have baking rights and voting rights in a cycle.

   The main kinds of operations in the protocol are transactions (to
   transfer funds or to execute smart contracts), accusations,
   activations, delegations, attestations, and originations. For the
   full list of operations, see Blocks and Operations.

   See smart contract.

   A manager operation whose purpose is to create -- that is, to
   deploy -- a smart contract on the Tezos blockchain.

   A PVM (Proof-generating Virtual Machine) is a reference
   implementation for a device on top of which a smart rollup can be
   executed. This reference implementation is part of the economic
   protocol and is the unique source of truth regarding the semantics
   of rollups. The PVM is able to produce proofs enforcing this truth.
   This ability is used during the final step of a refutation game.

   A process by which the economic protocol solves a conflict between
   two rollup committers. Note that the refutation mechanism used in
   Tezos smart rollups corresponds to the notion of fraud proofs used
   in other blockchain/Layer 2 ecosystems.

   When the first rollup commitment for a rollup commitment period is
   published, a refutation period of two weeks starts to allow this
   commitment to be challenged.

   deprecated; see minimal stake.

   A claim that the interpretation of all rollup inbox messages
   published during a given period, and applied on the state of a
   parent rollup commitment, led to a given new state by performing a
   given number of execution steps of the PVM.

   A period of roughly 15 minutes during which all rollup inbox
   messages must be processed by the rollup node state to compute a
   rollup commitment. A commitment must be published for each
   commitment period.

   A user account that has published and made a deposit on a rollup
   commitment.

   A sequence of messages from the Layer 1 to all the smart rollups.
   The contents of the inbox are determined by the consensus of the
   economic protocol.

   A daemon required for deploying and operating smart rollups. The
   rollup node is responsible for making the rollup progress by
   publishing rollup commitments and by playing refutation games.

   A sequence of messages from a smart rollup to the Layer 1. Messages
   are smart contract calls, potentially containing tickets. These
   calls can be triggered only when the related rollup commitment is
   cemented (hence, at least two weeks after the actual execution of
   the operation).

   An attempt to reach consensus on a block at a given level. A round
   is represented by an index, starting with 0. Each round corresponds
   to a time span. A baker with baking rights at a given round is only
   allowed to bake during the round's corresponding time span. Baking
   outside of one's designated round results in an invalid block.

   Account which is associated to a Michelson script. They are created
   with an explicit origination operation and are therefore sometimes
   called originated accounts. The address of a smart contract always
   starts with the letters "KT1".

   Smart rollups constitute a Layer 2 solution that can be used to
   deploy either a general-purpose polyvalent Layer 2 blockchain
   (e.g., an EVM-compatible one), or an application-specific DApp. See
   Smart Optimistic Rollups.

   A user account that made a security deposit. The user account must
   have set a delegate. The security deposit accrues to the stake of
   the user account's delegate and is subject to slashing in case the
   delegate misbehaves -- see Slashing.

   A unit of the cryptocurrency native to a Tezos chain, such as in "I
   sent you 2 tez." Tez is invariable. It is not capitalized except at
   the beginning of a sentence or when you would otherwise capitalize
   a noun. See also XTZ.

   An operation to transfer tez between two accounts, or to run the
   code of a smart contract.

   An account that is linked to a public key. Contrary to a smart
   contract, a user account cannot include a script and it cannot
   reject incoming transactions. User accounts are sometimes called
   "implicit accounts".

   If *registered*, a user account can act as a delegate.

   The address of a user account always starts with the letters "tz"
   followed by "1", "2", "3", or "4" (depending on the signature
   scheme) and finally the hash of the public key. See Accounts and
   addresses for a more detailed explanation on addresses.

   An index (a natural number) associated with a particular kind of
   operations, allowing to group them into classes. Validation passes
   enable prioritizing the validation and application of certain
   classes of operations.

   Any of the "proposal", "exploration", "cooldown", "promotion" or
   "adoption" stages in the voting procedure when amending the
   economic protocol.

   The amount of tokens that determines a delegate's weight in the
   voting process. A delegate's voting power is computed from the
   delegate's own tokens and the sum of tokens delegated to it. See
   Voting Power for details.

   The list calculated at the beginning of each voting period that
   contains the staking balance (in number of mutez) of each delegate
   that owns more than the minimal stake at that moment. For each
   delegate, the voting listings reflect the weight of the vote
   emitted by the delegate when amending the economic protocol.

   XTZ, tez, or ꜩ ("\ua729", "Latin small letter tz") is the native
   currency of Tezos.

   "XTZ" is an ISO-4217-compatible code for representing tez on the
   most economically relevant Tezos chain. Unless there is a very
   specific reason to use an ISO code for it, the term tez is
   preferred. Situations where the ISO code might be useful typically
   involve accounting systems, exchange rates with other currencies,
   and anything that might need some sort of standardized code.

The economic protocol
*********************

The economic protocol provides the rules for checking the validity of
the blocks and operations, and for updating the blockchain state
accordingly, by applying new valid blocks and operations on the
current blockchain state. These rules can be changed through voting.
Thus, the economic protocol represents the amendable part of Tezos.

The following pages provide the full documentation for this protocol.
The differences with respect to other protocols are summarized in the
protocol release page Protocol Alpha.

* Economic protocol features

  * Overview of the economic protocol

  * Proof-of-stake

  * Randomness generation

  * Baking Power

  * The consensus algorithm

  * The Amendment (and Voting) Process

  * Accounts and addresses

  * Native Multisig accounts

  * Michelson: the language of Smart Contracts in Tezos

  * On-chain Views

  * Time-lock

  * Sapling support

  * Liquidity Baking

  * Global Constants

  * Michelson Anti-Patterns

  * Token transfers and balance updates

  * Mempool

  * Smart Optimistic Rollups

  * Protocol Plugins

  * Contract events

  * Blocks and Operations

  * Validation and Application

  * Tickets

  * Adaptive Issuance

  * Staking mechanism

  * Adaptive Slashing

  * DAL integration

The following pages provide a reference for this protocol:

* Glossary

  * Tezos terms

  * Protocol terms

* Command Line Interface

  * Client manual

  * Baker manual

  * Accuser manual

  * Protocol-independent tools

* Seoul RPCs - Reference

  * Index

  * Full description

Economic protocol features
**************************

This page groups the documentation helping developers and users in
understanding the basic concepts of the economic protocol (Proof Of
Stake, consensus, voting, etc), its features (Michelson, Sapling,
etc), and some details about its implementation.

* Overview of the economic protocol

  * The role of the economic protocol

  * Shell-protocol interaction

  * Blocks, Operations and their Validation

  * Protocol constants

  * See also

* Proof-of-stake

  * Overview

  * Delegation and Staking

  * Delegates' rights selection

  * Proof-of-stake parameters

  * Further External Resources

* Randomness generation

  * Overview

  * Cryptographic primitives

  * Protocol

* Baking Power

  * Overview

  * Delegate, delegators, stakers

  * Staked tez

  * Delegated tez

  * Overstaking

  * Overdelegation

  * Minimal power and own staked requirements

* The consensus algorithm

  * History

  * Overview

  * Transaction and block finality

  * Time between blocks

  * Validator selection

  * Economic Incentives

  * Fitness

  * Consensus related protocol parameters

  * Further External Resources

* The Amendment (and Voting) Process

  * Periods

  * Activation

  * Voting Power

  * Super-majority and Quorum

  * The Hash and the Protocol

  * Operations

  * Client Commands

  * Further External Resources

* Accounts and addresses

  * User accounts

  * Smart contracts

* Native Multisig accounts

  * Enshrined multisig signing scenarios

  * Octez CLI commands and RPC endpoints

* Michelson: the language of Smart Contracts in Tezos

  * Semantics of smart contracts and transactions

  * Language semantics

  * Introduction to the type system and notations

  * Types and instructions

  * Removed instructions and types

  * Macros

  * Concrete syntax

  * Annotations

  * Entrypoints

  * JSON syntax

  * Development tools

  * Examples

  * Full grammar

  * Reference implementation

  * TZT, a Syntax extension for writing unit tests

* On-chain Views

  * Homonyms

  * Execution flow

  * Declaration

  * Forbidden types

* Time-lock

  * Cryptographic design

  * Opcode and types

  * Implementation of the time-lock puzzle

  * Example

* Sapling support

  * Sapling

  * Sapling integration

* Liquidity Baking

  * Contracts

  * Subsidy

  * Toggle vote

* Global Constants

  * Global Constant Registration

  * Originating a Contract that uses Global Constants

  * Global Constants at Runtime

* Michelson Anti-Patterns

  * Storing unbounded data

  * Refunding to a list of contracts

  * Avoid batch operations when users can increase the size of the
    batch

  * Signatures alone do not prevent replay attacks

  * Storing/transferring private data

  * Not setting all state before a transfer

* Token transfers and balance updates

  * Overview

  * Source, container, and sink accounts

  * Token transfers and metadata

* Mempool

  * Merging Mempools

  * Conflict

* Smart Optimistic Rollups

  * Overview

  * Rollup kernel

  * User roles

  * Address

  * Inputs

  * Origination

  * Processing

  * Commitments

  * Bonds

  * Refutation

  * Private rollups

* Protocol Plugins

  * Prevalidator filters

  * Filters RPCs

* Contract events

  * Emitting events

  * Retrieving events

* Blocks and Operations

  * Protocol-specific block header

  * Operations

  * Validation Passes

  * Consensus Operations

  * Voting Operations

  * Anonymous Operations

  * Manager Operations

  * Failing_noop operation

* Validation and Application

  * Validation modes

  * Block Validation

  * Operation Validation and Application

* Tickets

  * Semantics

  * Operations on tickets

* Adaptive Issuance

  * Adaptive issuance rate

  * Adaptive rewards

* Staking mechanism

  * Staking policy configuration

  * Staked funds management

  * Change of delegate

* Adaptive Slashing

  * Overview

  * Adaptive Slashing Function

  * New definition for the forbidden period

* DAL integration

  * DAL publish commitment

  * DAL attestation payloads

  * DAL incentives scheme

  * Smart Rollups integration

  * DAL-related protocol constants

Overview of the economic protocol
*********************************

Tezos is a self-amending blockchain, in that a large part of Tezos can
be changed through a so-called amendment procedure. To this end, as
mentioned in the big picture, a Tezos node consists of two components:

* the shell, which comprises the network and storage layer, and embeds

* the economic protocol component, which is the part that can be
  changed through amendment.


The role of the economic protocol
=================================

At a very high level, a protocol must:

* implement protocol-specific types, such as the type of operations or
  protocol-specific block header data (in addition to the shell
  generic header),

* define under which conditions a block is a valid extension of the
  current blockchain, and define an ordering on blocks to arbitrate
  between concurrent extensions.

Validity conditions are implemented in the "apply" function which is
called whenever the node processes a block---see the dedicated
protocol validation and operation entry for further detail into the
validation and application process for blocks and their operations.


Shell-protocol interaction
==========================

In the Tezos architecture, the economic protocol and the shell
interact in order to ensure that the blocks being appended to the
blockchain are valid. There are mainly two rules that the shell uses
when receiving a new block:

* The shell does not accept a block whose level is below the current
  checkpoint. The checkpoint itself is updated based on information
  resulting from successful block applications by the protocol which
  depends on the protocol consensus algorithm. Previously accepted
  blocks with lower levels than the current checkpoint are considered
  finalized and immutable.

* The shell changes the head of the chain to this new block only if
  the block is valid, and it has a higher fitness than the current
  head; a block is valid only if all the operations included are also
  valid.

The support provided by the protocol for validating blocks can be
modulated by different validation modes. They allow using this same
interface for quite different use cases, as follows:

* being able to apply a block, typically used by the shell's validator
  component;

* being able to construct a block, typically used by the baker daemon
  to *bake* -- that is, to produce -- a new block;

* being able to partially construct a block, typically used by the
  prevalidator to determine valid operations in the mempool; and,

* being able to pre-apply a block, typically used in the validator to
  precheck a block, avoiding to further consider invalid blocks.


Blocks, Operations and their Validation
=======================================

A block consists of a header and operations. A block's header is
composed of two parts: the protocol-agnostic part and the protocol-
specific part. This separation enables the shell to interact with
different protocols. Each Tezos economic protocol can specify
different kinds of operations, which are described further in detail
in Blocks and Operations.

The semantics of, respectively, operations and blocks is indeed also
dependent on each economic protocol. The Validation and Application
entry explains the internals of *validation* -- that is, how to
determine whether operations and blocks can be safely included in the
Tezos blockchain -- and *application* -- that is, how the effects of
operations and blocks are taken into account -- for this economic
protocol.


Protocol constants
==================

Protocols are tuned by several *protocol constants*, such as the size
of a nonce, or the number of blocks per cycle. One can distinguish two
kinds of protocol constants:

* *fixed* protocol constants, such as the size of a nonce, are values
  wired in the code of a protocol, and can only be changed by protocol
  amendment (that is, by adopting a new protocol)

* *parametric* protocol constants, such as the number of blocks per
  cycle, are values maintained in a read-only data structure that can
  be instantiated differently, for the same protocol, from one network
  to another (for instance, test networks move faster).

* *derived* constants are not directly configurable, but they depend
  on parametric constants. For instance, the issuance modification
  delay is a derived constant which is always equal to the consensus
  rights delay.

The *list* of protocol constants can be found in the OCaml APIs:

* fixed protocol constants are defined in the module Constants_repr

* parametric constants are defined in the module
  Constants_parametric_repr

* derived constants are defined in the module Constants_repr.Derived

The *values* of protocol constants in any given protocol can be found
using specific RPC calls:

* one RPC for all constants (fixed and parametric and derived), as
  shown in this example

* one RPC for the parametric constants specifically

Further documentation of various protocol constants can be found in
the subsystems where they conceptually belong. See, for example:

* proof-of-stake parameters

* consensus-related parameters

* randomness generation parameters

* DAL parameters


See also
========

An in-depth description of the internals of developing a new Tezos
protocol can be found in the blog post: How to write a Tezos protocol.

Proof-of-stake
**************


Overview
========

The consensus algorithm in Tezos is based on the *proof-of-stake*
mechanism. Proof-of-stake means that participants in the consensus
algorithm are chosen in function of their stake (the amount of tokens
a participant has). The same mechanism is used in the Tezos
governance.

If one does not have enough stake to participate on its own or does
not want to set up the needed infrastructure, (s)he can use
delegation, possibly complemented with staking. Indeed, in Tezos, it
is the delegates that may participate in consensus. However, at each
level, not all delegates necessarily participate, and their
participation weight may differ. The selection of the delegates'
participation rights at a level is done by running a PRNG (pseudo-
random number generator). The PRNG's seeds are obtained from random
data that are regularly produced and stored on the blockchain. Thus,
the procedure is deterministic in that delegates' rights are uniquely
determined from the seed; and it is random, in that its seed (and
hence its results) cannot be predicted too much in advance.


Delegation and Staking
======================

A *delegate* is any user account registered as such. This is done by
*self-delegating*, that is, emitting a delegation operation (see
below) in which the specified delegate is the same as the operation
emitter (its signer).

Any account (user account or smart contract) can specify a delegate
through a delegation operation.  Any non-delegate account can change
or revoke its delegate at any time, again through a delegation
operation. However, the change only becomes effective after
"CONSENSUS_RIGHTS_DELAY + 2" cycles.  The value
"CONSENSUS_RIGHTS_DELAY" is a protocol constant. A delegate cannot
stop self-delegating.

A delegate participates in consensus and in governance in proportion
to their *baking power* and *voting power* respectively.

* The voting power of a delegate is the total amount of tez owned by
  all the accounts that delegate to it (including the delegate
  itself), with no distinctions made between staked and non-staked
  tez.

* The baking power is similar, except that non-staked tez are weighted
  less than staked tez, and there are additional considerations such
  as overstaking and overdelegation. See the Baking Power page for
  more details.

Moreover, to participate in consensus and governance, the delegate
needs to be active and to meet minimal balance requirements.

Delegates and delegators may stake their tez. Staked tez are security
deposits that may be forfeited in case the baker does not follow (some
particular rules of) the protocol. Besides, as mentioned above, staked
tez are weighted higher than non-staked tez when computing the baking
power.


Consensus key
-------------

The key used by a delegate to sign blocks and consensus operations is
called the *consensus key*. By default, this is the delegate's private
key, called its *manager key*. However, a delegate may specify
another, dedicated key for this role.

Setting a new consensus key is accomplished via the
"Update_consensus_key" operation. There is delay of
"CONSENSUS_KEY_ACTIVATION_DELAY + 1" cycles before the new key
actually becomes the *active consensus key* that must be used to sign
blocks and consensus operations; until then, it is called a *pending
consensus key*. More precisely, the key becomes active after the cycle
containing the "Update_consensus_key" operation is over and then
another CONSENSUS_KEY_ACTIVATION_DELAY full cycles have passed: if the
update happens during cycle "n", then the key becomes active at the
beginning of cycle "n + CONSENSUS_KEY_ACTIVATION_DELAY + 1".

There may be multiple pending consensus keys, set to activate in
different future cycles, and each one will replace the previously
active consensus key in turn. If multiple "Update_consensus_key"
operations are performed within the same cycle "n", the last one takes
precedence for determining which key to activate at the start of cycle
"n + CONSENSUS_KEY_ACTIVATION_DELAY + 1".

Note that both the manager key and the consensus key give total
control over the delegate's spendable balance. See this page for
further important details, including client commands that are helpful
for handling consensus keys.


Active and passive delegates
----------------------------

A delegate can be marked as either active or passive. A passive
delegate cannot participate in the consensus algorithm.

A delegate is marked as active at its registration.

At the end of a cycle, a delegate gets deactivated if the chain has
not witnessed any consensus activity (baking, attesting) from it
during the past "TOLERATED_INACTIVITY_PERIOD" cycles, including the
currently ending cycle.

Note that there is an extra grace period of "CONSENSUS_RIGHTS_DELAY"
cycles when a delegate has just registered or has just been
reactivated. This is to account for the fact that it will not receive
consensus rights yet for the first "CONSENSUS_RIGHTS_DELAY" cycles, so
of course the chain cannot witness any activity from it during that
time.


Delegates' rights selection
===========================

Tezos being proof-of-stake, the delegates' rights are selected at
random based on their baking power. Let us detail the selection
mechanism used in Tezos.


Random seed
-----------

To each cycle is associated a random number called the seed. This seed
is used within its cycle to generate pseudo-random values in the
protocol, in particular for selecting delegates to participate in
consensus.

For more information on randomness generation, see randomness-
generation.


Slot selection
--------------

Delegates' rights to participate are determined using the alias
method, more precisely using Vose's algorithm (see also this more
pedagogic description; the algorithm is the last one listed there).
This algorithm samples from a discrete probability distribution, which
is given by the stakes of a specific cycle: the probability to sample
a particular delegate is its stake in the cycle over the total stake
in that cycle.

Concretely, the delegates' rights at a given level are expressed in
terms of the (quantity of) *slots* that the delegate owns at that
level. This quantity represents the delegate's weight in consensus. We
note that, in the long run (that is, on average over many levels), the
number of slots is proportional to its stake. The owner of a slot is
obtained by sampling using the algorithm mentioned above. More
precisely, given a level and a slot (which is just a non-negative
integer), the mentioned algorithm is invoked to assign a delegate to
the given slot. Its input is the probability distribution given by the
stakes retained for the cycle to which the level belongs. And whenever
the algorithm needs to draw a random value, this is obtained using a
simple procedure which has as its initial state: the level, the random
seed for the cycle to which the level belongs, and the slot.


Proof-of-stake parameters
=========================

+---------------------------------------------------------+---------------------------+
| Parameter name                                          | Parameter value           |
|=========================================================|===========================|
| "BLOCKS_PER_CYCLE"                                      | 10800 blocks              |
+---------------------------------------------------------+---------------------------+
| "CONSENSUS_RIGHTS_DELAY"                                | 2 cycles                  |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_STAKE"                                         | 6,000 ꜩ                   |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_FROZEN_STAKE"                                  | 600 ꜩ                     |
+---------------------------------------------------------+---------------------------+
| "TOLERATED_INACTIVITY_PERIOD"                           | 2 cycles                  |
+---------------------------------------------------------+---------------------------+


Further External Resources
==========================

The original design of the proof-of-stake mechanism in Tezos can be
found in the whitepaper.

The adaptive issuance feature documentation.

Other presentations of the Tezos' proof-of-stake mechanism can be
found in the Open Tezos entry.

Randomness generation
*********************


Overview
========

This document presents the randomness generation protocol, used to
select the delegates to participate in the consensus, as well as the
cryptographic primitives used.


Cryptographic primitives
========================

We introduce RANDAO and Verifiable Delay Functions, two cryptographic
protocols for generating random values.


RANDAO
------

RANDAO is a simple multi-party protocol used to generate a random seed
based on a ''commit and reveal'' scheme (similar in spirit with the
RANDAO protocol).

During the setup of the protocol, the cryptographic primitives and
security parameters are chosen. The members participating in the
protocol can also be chosen at this stage.

During the first phase (the "commitment" phase) each party commits to
a random value, called the nonce, and publishes the commitment.
Typically, a hash function is used for committing.

During the second phase, (the "nonce revelation" phase) each party
reveals their nonce. The revealed nonces are then verified with
respect to the commitments. Typically, this means checking that the
hash of each revealed nonce is equal to its corresponding commitment.
If the verification fails, the nonce is discarded, otherwise it is
accepted.

Once the revelation phase is finished, nonces are combined to generate
the seed. More precisely, the nonces are hashed together in the same
order as the commitment publication. In the case of a rolling RANDAO,
the previous seed may be used to initialize the hash.

We make the assumption that at least one participant is honest, that
is, it has indeed chosen a random value and this value was revealed.
This is a necessary condition for the seed to be random. The
randomness could however be biased as this protocol suffers from the
following low-impact weakness: if a malicious participant can make
sure she is the last revealer, then she can choose whether to reveal
its committed value, effectively choosing between two different
predetermined seeds.


Verifiable Delay Function
-------------------------

Verifiable Delay Functions, also called VDFs, are a recent
cryptographic primitive formalized in 2018. They can be seen as a
trapdoor-less timelock: the goal of a VDF is making sure a party
cannot compute a value before a specific time.

This new cryptographic building block is based on modular squaring in
a group of unknown order (e.g. class groups or MPC-generated RSA
groups) that is believed to be expensive and hard to parallelize.

More precisely, the goal of a VDF is for a user to compute a certain
value "h = g^2^T mod N ∈ G" and a proof of correctness "π_h" by
recursive modular squarings of "h". The variables "g", "h", "T" and
"N" are respectively the *challenge*, *solution* (or *output*), the
*difficulty parameter* and the -unknown- *group order*. The main
difference between VDF and timelocks is that the latter offers a
backdoor to efficiently generate the challenge from the solution.

To this day, two main schemes exist for generating VDF proofs:
Wesolowski and Pietrzak. The former presents shorter proofs and is
based on a stronger security assumption (adaptive root assumption)
while the latter is computationally cheaper and based on a weaker
security assumption (low order assumption).


Protocol
========


Randomness generation overview
------------------------------

The randomness generation uses both RANDAO and VDF, based on class
groups and using Wesolowski proofs. It can be summed up as follows. We
first use RANDAO to produce biasable entropy which is used as a VDF
challenge to generate an unbiasable seed (given the adversary cannot
compute the VDF solution before the reveal time ends). To ensure
liveness, we fallback to RANDAO entropy if no VDF output was published
and verified on-chain.

Concretely, the random seed for cycle "n" is a 256-bit long number
computed at the end of cycle "n-1-CONSENSUS_RIGHTS_DELAY". It is the
VDF output (or, in its absence, the RANDAO output) computed from
nonces to which delegates commit during cycle
"n-2-CONSENSUS_RIGHTS_DELAY".

Every "BLOCKS_PER_COMMITMENT" levels, the corresponding block contains
a nonce commitment. More precisely, a block contains a commitment if
and only if its cycle position modulo "BLOCKS_PER_COMMITMENT" is
"BLOCKS_PER_COMMITMENT - 1". The nonce is a 256-bit number generated
by the block proposer and its commitment is included in the block
header. The commitment is simply the hash of the nonce.

The committed nonce must be revealed by the original block proposer
during the nonce revelation phase, that is during the first
"NONCE_REVELATION_THRESHOLD" blocks, of cycle
"n-1-CONSENSUS_RIGHTS_DELAY" under penalty of forfeiting all of its
expected attesting rewards for that cycle. The associated security
deposit and baking rewards are not affected. The RANDAO output is then
computed and stored on-chain as the temporary seed for cycle "n". The
RANDAO output is the bitstring obtained by iterating through the
nonces revealed in cycle "n-1" as follows: initially it is the seed of
cycle "n-1"; at each iteration, the new bitstring is the hash of the
concatenation of the previous bitstring with the iterated revealed
nonce.

A *nonce revelation* is an operation and multiple nonce revelations
can thus be included in a block. A reward "seed_nonce_revelation_tip",
potentially adjusted by the adaptive issuance coefficient and weighted
by constant "SEED_NONCE_REVELATION_TIP_WEIGHT", is given for including
a revelation. Revelations are free operations which do not compete
with transactions for block space. Up to "MAX_ANON_OPS_PER_BLOCK"
revelations, wallet activations and denunciations can be contained in
any given block.

During the rest of the cycle, informally called the VDF revelation
period, any party can query the protocol for the *seed computation
status*, which can be one of the following:(1) the VDF revelation
period has not yet started, i.e. the nonce revelation phase is still
ongoing, (2) a VDF solution has already been successfully submitted,
and (3) no VDF solution has been submitted. In this latter case, the
status also provides the information needed to compute the VDF
solution:  hash seeds for computing the VDF discriminant (a prime
number defining the class group) and the VDF challenge; more precisely
the random seed of cycle "n-1"  for the VDF discriminant and the
current RANDAO output for the VDF challenge. Any party can compute a
VDF solution and publish it on-chain together with a proof of
correctness. If the verification of the solution and proof succeeds,
the seed for cycle "n" is then updated with the solution: its value is
set to be the hash of the RANDAO output and the VDF solution.

A *VDF revelation* is an operation. A reward
"seed_nonce_revelation_tip" potentially adjusted by the adaptive
issuance coefficient and weighted by constant
"SEED_NONCE_REVELATION_TIP_WEIGHT", is given for the first correct VDF
revelation, subsequent VDF revelation operations being discarded.


Randomness generation parameters
--------------------------------

+---------------------------------------------------------+---------------------------+
| Parameter name                                          | Parameter value           |
|=========================================================|===========================|
| "BLOCKS_PER_COMMITMENT"                                 | 84 blocks                 |
+---------------------------------------------------------+---------------------------+
| "NONCE_REVELATION_THRESHOLD"                            | 300 blocks                |
+---------------------------------------------------------+---------------------------+
| "MAX_ANON_OPS_PER_BLOCK"                                | 132 revelations           |
+---------------------------------------------------------+---------------------------+
| "SEED_NONCE_REVELATION_TIP_WEIGHT"                      | 1                         |
+---------------------------------------------------------+---------------------------+
| "VDF_DIFFICULTY"                                        | 2,400,000,000             |
+---------------------------------------------------------+---------------------------+

The variables "BLOCKS_PER_CYCLE" and "CONSENSUS_RIGHTS_DELAY" are
already defined in the proof of stake page.

Baking Power
************

The proof-of-stake mechanism used for the consensus algorithm assigns
baking and attesting slots, called **baking rights**, to each delegate
a.k.a. baker. For this selection process, each baker is weighted
according to its **baking power** -- provided that it is active and
meets the minimal power and own staked requirements.

This page details how this baking power is determined from the staked
and non-staked funds owned by the baker itself and all its delegators.

Note that the amendment and voting process is based on each delegate's
voting power instead, which is computed in a similar but simpler way.

Note:

  In this page, the prefix ".../" in all RPC paths is standing for
  "/chain/<chain_id>/blocks/<block_id>/context/"Besides, many RPCs
  presented here used to be known under different names. They have
  been renamed in the Quebec protocol in order to disambiguate and
  normalize the baking power lexicon; the old names have been
  deprecated. See the changelog for more information.


Overview
========

At the end of cycle "n" (that is, the beginning of cycle "n + 1"), the
protocol randomly generates the baking rights for cycle "n + 1 +
CONSENSUS_RIGHTS_DELAY = n + 3", using the **current baking power** as
the weight for each delegate that meets the requirements.
("CONSENSUS_RIGHTS_DELAY = 2" is a protocol constant.)

The ".../delegates/<delegate_pkh>/baking_power" RPC can be used to
retrieve the current baking power of a delegate, that is, its baking
power as of the end of the requested block "<block_id>" (see the note
above on RPC paths). Therefore, the baking power used for the rights
of cycle "n + 3" is the one returned by this RPC called on the last
block of cycle "n".

The baking power of a delegate is defined as:

   baking_power = total_staked_after_limits + (total_delegated_after_limits / 3)

This page explains the relevant concepts and provides the detailed
computations of "total_staked_after_limits" and
"total_delegated_after_limits".


Delegate, delegators, stakers
=============================

A **delegate**, a.k.a. **baker**, is a user account that has
registered as a delegate by emitting a self-"delegation" manager
operation. The list of all registered delegates is queried with the
".../delegates" RPC.

A **delegator** for a given baker is an account that has registered
this baker as its delegate by emitting a "delegation" operation. This
includes the baker itself. A delegator may be a user account or a
smart contract. The list of delegators for a given delegate is queried
with the ".../delegates/<delegate_pkh>/delegators" RPC.

A **staker** is a delegator that has staked tez by emitting a stake
operation. This includes the delegate itself if it has staked funds.
Note that stakers are always user accounts, because smart contracts
cannot emit "stake" operations. The list of a delegate's stakers and
their respective staked balances (see below) are queried with the
".../delegates/<delegate_pkh>/stakers" RPC.

An **external delegator** (resp. **external staker**) is a delegator
(resp. staker) that is not the delegate itself.


Staked tez
==========

Delegates and delegators have the option to stake their tez. **Staked
tez** contribute to the baking power, but they also function as a
security deposit for baking, meaning that they may be slashed if the
delegate misbehaves. That's why they are also known as **frozen
deposits**.

The **staked balance** of an account is its amount of staked tez. It
can be queried with the ".../contracts/<contract_id>/staked_balance"
RPC (in mutez). Note that if an account does not have a delegate, then
it cannot have any staked tez so its staked balance is zero.

For a given delegate, we define the following:

* "own_staked" is the staked balance of the delegate itself. It can be
  queried with either RPC ".../contracts/<contract_id>/staked_balance"
  or ".../delegates/<delegate_pkh>/own_staked" (in mutez).

* "external_staked" is the sum of the staked balances of the
  delegate's external stakers. It is queried with the
  ".../delegates/<delegate_pkh>/external_staked" RPC (in mutez).

* "total_staked" is the sum of the staked balances of all stakers,
  including the delegate itself. It is queried with the
  ".../delegates/<delegate_pkh>/total_staked" RPC (in mutez).

All three values are of course related:

   total_staked = own_staked + external_staked


Delegated tez
=============

Non-staked tez owned by delegates and delegators are called
**delegated tez**. They also contribute to the delegate's baking
power, without being subject to slashing. However, delegated tez weigh
a third as much as staked tez for the purpose of computing the baking
power.


Delegated tez of an account
---------------------------

The **delegated balance** of an account is its amount of delegated
tez. It is the sum of the following balances:

* The **spendable balance** is the amount of tez that the account can
  freely access and spend at the current time. It can be queried with
  RPC ".../contracts/<contract_id>/spendable" (in mutez).

* The **unstaked balance** is the sum of the tez amounts contained in
  all of the account's **unstake requests** (both unfinalizable and
  finalizable unstake requests). These tez have been removed from the
  staked balance via an "unstake" operation, but have not been added
  back to the spendable balance yet; see Staked funds management.
  Unstake requests can be queried with RPC
  ".../contracts/<contract_id>/unstake_requests" (returns a detailed
  view with unfinalizable/finalizable status, delegate-at-creation-
  time, cycle, and amount in mutez).

* The **frozen bonds** are a deposit for rollup commitments. They can
  be queried with RPC ".../contracts/<contract_id>/frozen_bonds" (in
  mutez).

Together, the staked and delegated tez represent all the tez owned by
an account, called the **full balance**.

   delegated = spendable + unstaked + frozen_bonds

   full_balance = staked + delegated


Delegated tez to a baker
------------------------

Spendable tez and frozen bonds count as delegated to the account's
current delegate. However, the tez involved in an unstake request
count as delegated to the account's **delegate at the time of the
unstake request's creation** (which is the account's current delegate
in most cases, but might be a former delegate instead).

To sum up:

   delegated = (spendable + unstaked_from_current_delegate + frozen_bonds) + unstaked_from_former_delegates
   delegated =        delegated_to_current_delegate                        +  delegated_to_former_delegates

   delegated_to_current_delegate = delegated - delegated_to_former_delegates
   delegated_to_current_delegate = full_balance - staked - unstaked_from_former_delegates

For a given delegate, we define the following:

* "own_delegated" is the amount of delegated tez owned by the baker
  and counting as delegated to itself, that is, the
  "delegated_to_current_delegate" amount of the delegate's account. It
  corresponds to all non-staked tez owned by the baker (except for any
  tez involved in unstake requests created at a time when the baker
  was delegating to a different delegate, but it is rare for a
  delegate to still have such requests). It can be queried with RPC
  ".../delegates/<delegate_pkh>/own_delegated" (in mutez).

* "external_delegated" is the sum of tez that count as delegated to
  the baker but are not owned by the baker itself. In other words, it
  is the sum of "delegated_to_current_delegate" over all current
  external delegators, plus any tez involved in unstaked requests
  created by former external delegators when they were still
  delegating to the baker. It can be queried with RPC
  ".../delegates/<delegate_pkh>/external_delegated" (in mutez).

* "total_delegated" is the total amount that counts as delegated to
  this baker. It can be queried with RPC
  ".../delegates/<delegate_pkh>/total_delegated" (in mutez).

   total_delegated = own_delegated + external_delegated


Min-delegated-in-current-cycle
------------------------------

When computing baking rights for cycle "n + 3" at the end of cycle
"n", the "total_delegated" value used for each delegate is actually
the **minimum** of its "total_delegated" **over the whole cycle** "n",
called "min_delegated_in_current_cycle". The purpose of this mechanism
is to prevent any manipulation of baking rights through short-duration
transfers. (Note that such a mechanism is not needed for staked tez
because they are inherently frozen for at least four cycles, so short-
duration staking is already not possible.)

Since the Paris protocol, the considered minimum is the minimum at any
point during block applications, which can be reached in the middle of
executing a transaction.

For example, if a baker transfers tez to one of its delegators, this
is internally treated as first removing the transferred amount from
the total amount delegated to this baker, then adding it back. In
between executing both updates, the total delegated amount is lower so
it might be the new minimum over the whole cycle. In other words, the
transferred tez risk not counting towards the baking rights that will
be computed at the end of the cycle, even though they have been owned
by an account delegating to this baker during the whole cycle.

Besides, when the minimum is reached in the middle of a block's
operations, the context for this minimum is not directly accessible
via RPC. In that case, in order to retrieve this exact context, one
needs to replay the block's balance updates on their own.

In the Quebec protocol, to solve these problems, only the total
delegated amounts **at the end of blocks** count when determining this
minimum. This is known as the **per-block min-delegated feature**.
This solution no longer penalizes baking rights when funds are
transferred between two accounts delegated to the same baker.
Moreover, it lets users easily retrieve via RPC the exact context that
the minimum comes from, since it is guaranteed to correspond to the
end of a block.

The min-delegated-in-current-cycle can be retrieved with RPC "GET /ch
ains/<chain_id>/blocks/<block_id>/context/delegates/<delegate_pkh>/mi
n_delegated_in_current_cycle". This RPC returns the minimal value of
the baker’s "total_delegated" at the end of any block, from the first
block of the current cycle up to the current level, where "current
cycle" and "current level" are those of the "<block_id>" in the RPC
path; it also returns the earliest level at the end of which this
minimum has been reached in the current cycle. This means that calling
the RPC on the last block of cycle "n" returns the value actually used
during the generation of baking rights for cycle "n + 3".


Example
~~~~~~~

At level "150", the baker receives "50" tez from a contract that is
not a delegator for this baker. Then, at level "200", the baker
transfers "150" tez to one of its delegators. Finally, at level "205",
that delegator sends "70" tez to another contract that is not a
delegator for this baker.

The first transfer of "50" tez increases the baker’s "total_delegated"
by "50". Then, the transfer of "150" tez is internally implemented as
removing "150" tez from the total delegated of the sender’s delegate
-- which is the baker itself, then adding "150" tez to the
"total_delegated" of the destination’s delegate -- which is the same
baker in our case. Finally, the "70"-tez transfer just removes "70"
tez from the "total_delegated" of the same baker again.

Let's say that "blocks_per_cycle = 128", so the first level of the
current cycle is "129", and let's say that "total_delegated" was
"1000" at the beginning of the cycle.

+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
|                |                  | L129: first    | L150: add 50   | L200: remove   | L200: add 150  | L205: remove   |
|                |                  | level of cycle |                | 150            |                | 70             |
|================|==================|================|================|================|================|================|
|                | Baker's current  | 1000           | 1050           | 900            | 1050           | 980            |
|                | "total_delegate  |                |                |                |                |                |
|                | d" (tez)         |                |                |                |                |                |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
| Quebec RPC     | Returned min     | 1000           | 1000           |                | 1000           | 980            |
|                | (tez)            |                |                |                |                |                |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
| Quebec RPC     | Returned level   | 129            | 129            |                | 129            | 205            |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
| Paris RPC      | Returned min     | 1000           | 1000           |                | 900            | 900            |
|                | (tez)            |                |                |                |                |                |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+
| Paris RPC      | Returned level   | None           | 150            |                | 200            | 200            |
+----------------+------------------+----------------+----------------+----------------+----------------+----------------+

Note there are empty cells in the table as RPCs cannot be called in
the middle of the block application. Also, the
"min_delegated_in_current_cycle" RPC returns the value in mutez, but
here we use tez for simplicity.

* In Quebec:

  * At levels "129" and "150", the earliest level at the end of which
    the "total_delegated" is equal to the minimum "1000" is the first
    level of the cycle, that is, level "129".

  * At level "200", the "900" value happens in the middle of the block
    application so it is not considered. The new end-of-block value
    "1050" is not lower than the old minimum of "1000", so
    "min_delegated_in_current_cycle" stays at (min: "1000", level:
    "129").

  * At level "205", the new end-of-block value "980" is lower than the
    old minimum of "1000", so "min_delegated_in_current_cycle" becomes
    (min: "980", level: "205").

* In Paris:

  * At level "129", the "total_delegated" has not changed since the
    start of the cycle. The Paris RPC returns level "None" in this
    case.

  * At level "150", the "total_delegated" has changed since the start
    of the cycle, but the minimum is actually the initial value it had
    at the start of the cycle. In this case, Paris RPC returns the
    earliest level at which the "total_delegated" has changed, that
    is, level "150".

  * At level "200", the "total_delegated" reaches a new minimum "900".
    Indeed, the Paris protocol does consider the values in the middle
    of the block application, so "min_delegated_in_current_cycle"
    becomes (min: "900", level: "200").

  * At level "205", the new value "980" is higher than the old minimum
    of "900", so "min_delegated_in_current_cycle" is still (min:
    "900", level: "200").


Overstaking
===========

The **limit_of_staking_over_baking** is a configurable delegate
parameter that limits how much staked tez the external stakers can
contribute to the baking power, relative to the baker's own staked
tez. It defaults to "0", meaning no staked contribution from external
stakers at all. It can be set to any non-negative value (with a one
millionth precision); however, the
"GLOBAL_LIMIT_OF_STAKING_OVER_BAKING" constant, set to "9", ensures
that external stakers may never contribute more than nine times as
much staked tez as the baker itself, regardless of the delegate's own
limit. If the amount of external staked tez exceeds this quota, the
baker is said to be **overstaked**, and we also call **overstaked**
the excess of external staked tez over the allowed maximum. Any
overstaked tez will count toward the baking power as delegated instead
of staked (provided that the baker is not overdelegated too), so they
will weigh a third as much.

   global_limit_of_staking_over_baking = 9
   actual_limit_of_staking_over_baking = min(limit_of_staking_over_baking, global_limit_of_staking_over_baking)
   max_allowed_external_staked = own_staked * actual_limit_of_staking_over_baking
   external_staked_after_limits = min(external_staked, max_allowed_external_staked)

   total_staked_after_limits = own_staked + external_staked_after_limits

   overstaked = external_staked - external_staked_after_limits

The purpose of this feature is to ensure that the baker's
"own_staked", that is, the part of the security deposit that belongs
to the baker itself, always represents a sizable portion of its baking
power. In other words, it guarantees that the baker always has its own
skin in the game. Besides, the global limit of "9" ensures that a
baker can never increase its own balance by denouncing its own double
baking or double attesting misbehavior; indeed, the reward that would
be given to the author of a denunciation is guaranteed to be lower
than the amount that would be slashed from the misbehaving baker's own
funds.

Note:

  The "limit_of_staking_over_baking" has an additional effect when set
  to "0": it prevents external delegators from using "stake"
  operations at all. This effect is completely removed when the limit
  is positive: external delegators can stake as much tez as they wish,
  even if this causes the baker to become overstaked or if the baker
  is already overstaked. If the limit is set to a positive value then
  back to "0", then external delegators are again prevented from
  staking new funds, but any previously staked tez remain staked
  (although they are now all considered overstaked, so they all count
  as delegated when computing baking the power).


Overdelegation
==============

The amount that counts as *delegated* cannot exceed "9" times the
baker's "own_staked". Any excess tez are called **overdelegated** and
do not contribute to the baking power at all. This mechanism also
contributes to ensuring that all baking rights are covered by
appropriate security deposits.

Recall that the delegated amount used for baking rights is
min_delegated_in_current_cycle, and any overstaked tez count as
delegated too. Therefore:

   total_delegated_after_limits = min(min_delegated_in_current_cycle + overstaked, own_staked * 9)

We finally have everything we need to compute the baking power as
defined above:

   baking_power = total_staked_after_limits + (total_delegated_after_limits / 3)


Minimal power and own staked requirements
=========================================

To receive baking rights, a delegate must meet the following
requirements:

* "baking_power >= MINIMAL_STAKE"

* "own_staked >= MINIMAL_FROZEN_STAKE"

* The delegate must be active

where "MINIMAL_STAKE = 6,000ꜩ" and "MINIMAL_FROZEN_STAKE = 600ꜩ" are
protocol constants.

If any of these conditions is not met at the end of cycle "n", the
delegate still has a *baking power* as computed above, but receives no
*baking rights* at all for cycle "n + 3".

The consensus algorithm
***********************

This document provides a high-level description of Tenderbake, the
Tezos proof-of-stake consensus algorithm.


History
=======

Before Tenderbake, there was Emmy*, a Nakamoto-style consensus
consisting of a series of improvements of the one in the Tezos
whitepaper.

Emmy*, like any Nakamoto-style consensus algorithm (such as Bitcoin or
Ouroboros), offers *probabilistic* finality: forks of arbitrary length
are possible but they collapse with a probability that increases
rapidly with fork length.

Tenderbake instead, like any classic BFT-style consensus algorithm
(such as PBFT or Tendermint), offers *deterministic* finality: a block
that has just been appended to the chain of some node is known to be
final once it has two additional blocks on top of it, regardless of
network latency.


Overview
========

The starting point for Tenderbake is Tendermint, the first classic-
style algorithm for blockchains.

Tenderbake adapts Tendermint to the Tezos blockchain, but the
adjustments required are substantive:

* Tenderbake is tailored to match the Tezos architecture by using only
  communication primitives and network assumptions which Tezos
  supports.

* Tenderbake makes weaker network assumptions than Tendermint, at the
  price of adding the extra assumption that participants have loosely
  synchronized clocks — which is fine, because Tezos already uses
  them.

The design of Tenderbake and its rationale are described at length in
the technical report and in a Nomadic Labs's blog post. Here we only
provide a user/developer perspective.

Tenderbake is executed for each new block level by a "committee" whose
members are called *validators*, which are delegates selected at
random based on their stake, in much the same way as endorsers were
selected in Emmy*. We let "CONSENSUS_COMMITTEE_SIZE" be the number of
validator slots per level. Furthermore, we use "CONSENSUS_THRESHOLD"
to denote two thirds of "CONSENSUS_COMMITTEE_SIZE".

For each level, Tenderbake proceeds in rounds. Each *round* represents
an attempt by the validators to agree on the content of the block for
the current level, that is, on the sequence of non-consensus
operations the block contains. We call this sequence the block's
*payload*.

Each round has an associated duration. Round durations are set to
increase so that for any possible message delay, there is a round that
is sufficiently long for all required messages to be exchanged. Round
durations depend on protocol parameters "MINIMAL_BLOCK_DELAY" and
"DELAY_INCREMENT_PER_ROUND". These parameters specify round durations
as follows:

   round\_duration(0) &= minimal\_block\_delay \\ round\_duration(r+1)
   &= round\_duration(r) + delay\_increment\_per\_round \\ & =
   minimal\_block\_delay + (r + 1) * delay\_increment\_per\_round

Round durations thus increase linearly with
"DELAY_INCREMENT_PER_ROUND".

Schematically, a round consists in the following steps:

* a validator designated for that round injects a *candidate block*
  (representing a proposal) and consensus operations (representing
  votes) into the node to which it is attached, which then

* diffuses those blocks and consensus operations to other nodes of the
  network, and thus

* communicates them to the validators attached to those nodes, to
  carry out voting on which block to accept.

Unlike Emmy*, Tenderbake has two types of votes: before attesting a
block "b", a validator preattests "b". Furthermore, to be able to
attest, a validator must have observed a preattestation *quorum*, that
is a set of preattestations from validators having at least
"CONSENSUS_THRESHOLD" validator slots. Similarly, to be able to
decide, a validator must have observed an attestation quorum, that is,
a set of attestations from validators having at least
"CONSENSUS_THRESHOLD" validator slots. The attestation quorum for a
block "b" is included in a block "b'" on top of "b", serving as a
certification that "b" has been agreed upon. We also say that block
"b'" confirms block "b".

The validator's whose turn is to inject a candidate block at a given
round is called the *proposer* at that round. Proposers in Tenderbake
are selected similarly to bakers in Emmy*: the proposer at round "r"
is the validator who has the validator slot "r". A proposer who has
observed a preattestation quorum for a candidate block in a previous
round, is required to propose a block with the same *payload* as the
initial block. We talk about a *re-proposal* in this case.


Transaction and block finality
==============================

A transaction is final as soon as the block including it has a
confirmation (that is, a block on top of it). Indeed, as hinted above,
a block contains the certification (that is, the attestation quorum)
for the previous payload. Thanks to the attestation quorum, Tenderbake
guarantees **transaction finality after 1 confirmation**.

It may be possible that different validators decide at different
rounds, though on the same payload. The blocks at these different
rounds differ precisely because they contain, in the header, as part
of the block fitness, the round at which they were proposed. Among
these "candidate" blocks, the block with the smallest round has the
highest fitness and so it will be the one decided. Consequently, to
agree on a block, that is, on both the payload and the header,
Tenderbake needs one more confirmation, and thus guarantees **block
finality after 2 confirmations**.


Time between blocks
===================

The time between blocks represents the difference between the
timestamps of the blocks. The timestamp of a block is given by the
beginning of the round at which the block has been agreed upon. Thus,
the time between blocks depends on the round at which decisions are
taken. For example, if the decision at the previous level was taken at
round 4 and at the current level at round 2, then the current block's
delay relative to its predecessor, is round\_duration(4) +
round\_duration(0) + round\_duration(1). The general case is as
follows, say that the decision at the previous level is taken at round
"m" and the decision at the current level is taken at round "n", then
the current block's delay relative to its predecessor is
round\_duration(m) + \sum_{i=0}^{n-1} round\_duration(i). We note
that, under normal network conditions, and with active and compliant
validators, decisions should be taken at round 0, meaning that the
time between blocks would be round\_duration(0) seconds i.e.,
parameter "MINIMAL_BLOCK_DELAY".


Validator selection
===================

Validator selection, that is, baking and attesting rights, are based
on the baking power of a delegate. The baking power is a function of
all tez owned by the delegate and its delegators, with staked tez
weighted more than non-staked tez, and there are additional
considerations such as overstaking and overdelegation; see the baking
power formula.

The baking rights are determined CONSENSUS_RIGHTS_DELAY in advance,
which is currently "2" cycles. More precisely, at the end of cycle "n"
and beginning of cycle "n+1", the baking rights for cycle
"n+1+CONSENSUS_RIGHTS_DELAY=n+3" are randomly generated based on the
current baking power of each delegate that meets the minimal power and
own staked requirements.


Economic Incentives
===================

As Emmy*, Tenderbake rewards participation in consensus and punishes
bad behavior. Notable changes however are as follows:

* Fees and baking rewards go to the payload producer, the one who
  selected the transactions to be included in the block (and was the
  first to propose a block with that payload). In case of re-proposal,
  the payload producer might be different from the block proposer, the
  baker who injects the block.

* Including extra attestations, that is, more than the minimal
  required to obtain a quorum, is rewarded with a bonus.

* Attesting rewards are shared equally among all validators.
  Participation above a minimal threshold per cycle is however
  required.

* Validators are rewarded instantaneously for baking blocks and
  including extra attestations, and not at the end of the cycle like
  in Emmy*.

* At the end of a cycle "c", the following actions happen:

  * the distribution of attesting rewards,

  * the selection of the consensus committee cycle "c +
    CONSENSUS_RIGHTS_DELAY", based on the current distribution of
    baking power.


Fees
----

The fees associated to the transactions included in a block go to the
payload producer. This is only natural given that this is the
validator that selects the transactions to be included; see an in-
depth blog post for further motivation.

The payload producer is usually the same delegate as the block
proposer (that is, the one that signs and injects the block): that's
always true for blocks at round 0; however, in case of re-proposals
this is not necessarily the case (see the algorithm description
above).

Fees are allocated to the payload producer's available balance
immediately, that is, they are already reflected in the blockchain
state obtained after applying the injected block.


Rewards
-------

There are three kinds of rewards: baking rewards, attesting rewards,
and a bonus for including extra attestations.

The baking rewards are treated in the same way as fees: they go to the
*payload* producer and are distributed immediately.

To encourage fairness and participation, the *block* proposer receives
a bonus for the extra attestations it includes in the block. The bonus
is proportional to the number of validator slots above the threshold
of "CONSENSUS_COMMITTEE_SIZE * 2 / 3" that the included attestations
represent. The bonus is also distributed immediately.

The attesting rewards are distributed at the end of the cycle.
Attesting rewards may be received even if not all of the validator's
attestations are included in the cycle's blocks and is proportional to
the validator's baking power (in other words, to its *expected* number
of validator slots, and not its actual number of slots). However, two
conditions must be met:

* the validator has revealed its nonces, and

* the validator has been present during the cycle.

Not giving rewards in case of missing revelations is not new as it is
adapted from Emmy*. The second condition is new. We say that a
delegate is *present* during a cycle if the attesting power (that is,
the number of validator slots at the corresponding level) of all the
delegate's attestations included during the cycle represents at least
"MINIMAL_PARTICIPATION_RATIO" of the delegate's expected number of
validator slots for the current cycle (which is "BLOCKS_PER_CYCLE *
CONSENSUS_COMMITTEE_SIZE * active_stake / total_active_stake").

The concrete values for rewards depend on the issuance which is
dynamically adjusted by Adaptive Issuance. For each block it issues an
amount "total_rewards" of rewarded tez, that varies with the total
amount of tez at stake on the chain. To obtain some concrete values,
we will use as an example the issuance before Adaptive Issuance, which
was "80" tez per minute. With "MINIMAL_BLOCK_DELAY = 8s", this
corresponds to a "total_rewards" per level of 10.67 tez. We define:

* "baking_reward_fixed_portion := baking_reward_ratio * total_rewards"

* "bonus := (1 - baking_reward_ratio) * bonus_ratio * total_rewards"
  is the max bonus

* "attesting_reward := (1 - baking_reward_ratio) * (1 - bonus_ratio) *
  total_rewards"

where:

* "baking_reward_ratio" to "1 / 4",

* "bonus_ratio" to "1 / 3".

Thus, we obtain "baking_reward_fixed_portion = 2.67" tez, (maximum)
"bonus = 2.67" tez, and "attesting_reward = 5.33" tez. The bonus per
additional attestation slot is in turn "bonus /
(CONSENSUS_COMMITTEE_SIZE / 3)" (because there are at most
"CONSENSUS_COMMITTEE_SIZE / 3" validator slots corresponding to the
additional attestations included in a block). The rewards per
attestation slot are "attesting_reward / CONSENSUS_COMMITTEE_SIZE".
Assuming "CONSENSUS_COMMITTEE_SIZE = 7000", we obtain a bonus per slot
of "2.67 / (7000 / 3) = 0.001143" tez and an attesting rewards per
slot of "5.33 / 7000 = 0.000761" tez.

Let's take an example. Say a block has round 1, is proposed by
delegate B, and contains the payload from round 0 produced by delegate
A. Also, B includes attestations with attesting power "5251". Then A
receives the fees and 10 tez (the "baking_reward_fixed_portion") as a
reward for producing the block's payload. Concerning the bonus, given
that "CONSENSUS_COMMITTEE_SIZE = 7000", the minimum required validator
slots is "4667", and there are "2333 = 7000 - 4667" additional
validator slots. Therefore B receives the bonus "(5251 - 4667) *
0.001143 = 0.667512" tez. (Note that B only included attestations
corresponding to "584 = 5251 - 4667" additional validator slots, about
a quarter of the maximum "2333" extra attestations it could have
theoretically included.) Finally, consider some delegate C, whose
baking power at some cycle is 1% of the total stake. Note that his
expected number of validator slots for that cycle is "1/100 *
BLOCKS_PER_CYCLE * CONSENSUS_COMMITTEE_SIZE = 1/100 * 10800 * 7000 =
756,000" slots. Assume also that the attesting power of C's
attestations included during that cycle has been "651,456" slots.
Given that this number is bigger than the minimum required ("756,000 *
2 / 3"), it receives an attesting reward of "756,000 * 0.000761 =
575.316" tez for that cycle.


Slashing
--------

Like in Emmy*, not revealing nonces and double signing are punishable.
If a validator does not reveal its nonces by the end of the cycle, it
does not receive its attesting rewards. If a validator double signs,
that is, it double bakes (which means signing different blocks at the
same level and same round) or it double (pre)attests (which means
voting on two different proposals at the same level and round), a part
of the frozen deposit is slashed. The slashed amount for double baking
is a fixed percentage of the frozen deposit
"PERCENTAGE_OF_FROZEN_DEPOSITS_SLASHED_PER_DOUBLE_BAKING". For double
(pre)attestations, the formula is more complex, as it depends on the
number of attestation slots that participated in the misbehavior; see
Adaptive Slashing for more details. The payload producer that includes
the misbehavior evidence is rewarded "1 /
(GLOBAL_LIMIT_OF_STAKING_OVER_BAKING + 2)" of the slashed amount; the
rest of the slashed amount is burned.

If a delegate's deposit is smaller than the slashed amount, the
deposit is simply emptied.

The evidence for double signing at a given level can be collected by
any accuser and included as a *denunciation* operation in a block in
the same cycle as the double signing or in the "DENUNCIATION_PERIOD"
next cycles.

As soon as a delegate is denounced for any double signing, it is
immediately forbidden from both baking and attesting for at least 2
cycles.

The actual slashing and denunciation rewarding happen at the end of
cycle "n + SLASHING_DELAY" for a misbehavior that happened in cycle
"n".

Note that selfish baking is not an issue in Tenderbake: say we are at
round "r" and the validator which is proposer at round "r+1" does not
(pre)attest at round "r" in the hope that the block at round "r" is
not agreed upon and its turn comes to propose at round "r+1". Under
the assumption that the correct validators have more than two thirds
of the total stake, these correct validators have sufficient power for
agreement to be reached, thus the lack of participation of a selfish
baker does not have an impact.


Fitness
=======

The fitness is given by the tuple "(version, level, locked_round, -
predecessor_round - 1, round)". The current version of the fitness is
2 (version 0 was used by Emmy, and version 1 by Emmy+ and Emmy*). The
fitness encapsulates more information than in Emmy* because Tenderbake
is more complex: recall that blocks at the last level only represent
candidate blocks. In Emmy*, only the level mattered. But in
Tenderbake, we need to, for instance, allow for new blocks at the same
level to be accepted by nodes. Therefore the fitness also includes the
block's round (as the fifth component). Furthermore, we also allow to
change the predecessor block when it has a smaller round. Therefore
the fitness also includes the opposite of predecessor block's round as
the forth component (the predecessor is taken for technical reasons).
Finally, to (partially) enforce the rule on re-proposals, the fitness
also includes, as the third component, the round at which a
preattestation quorum was observed by the baker, if any (this
component can therefore be empty). By the way, preattestations are
present in a block if and only if the locked round component is non-
empty and if so, the locked round has to match the round of the
included preattestations.

Next, we provide two examples of fitness values:
"02::00001000::::ffffffff::00000000" and
"02::00001000::00000000::fffffffe::00000001" (in the hexadecimal
format that one may observe in the node's logs). These two values have
the following components:

* the 1st component, "02", is the fitness version;

* the 2nd component, "00001000", is the block's level (level 4096);

* the 3rd component is the block's locked round: empty in the first
  case, 0 in the second;

* the 4th component is the round of the predecessor block, here 0 in
  the first case and 1 in the second case;

* the 5th component is the block's round: 0 in the first case, 1 in
  the second case.

We recall (see Shell header) that the fitness is, from the shell's
perspective, a sequence of sequences of unsigned bytes and comparison
is done first by the length of the sequence and then lexicographically
(both for the outer sequence, and for each of the inner sequences). So
the first fitness is smaller than the second one, because of the third
component, the empty bitstring being smaller than any other bitstring.


Consensus related protocol parameters
=====================================

+---------------------------------------------------------+---------------------------+
| Parameter name                                          | Parameter value           |
|=========================================================|===========================|
| "CONSENSUS_COMMITTEE_SIZE"                              | 7000                      |
+---------------------------------------------------------+---------------------------+
| "CONSENSUS_THRESHOLD"                                   | "ceil(2 *                 |
|                                                         | CONSENSUS_COMMITTEE_SIZE  |
|                                                         | / 3)" = 4667              |
+---------------------------------------------------------+---------------------------+
| "DENUNCIATION_PERIOD"                                   | 1 cycle                   |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_BLOCK_DELAY"                                   | 8s                        |
+---------------------------------------------------------+---------------------------+
| "BLOCKS_PER_CYCLE"                                      | 10800                     |
+---------------------------------------------------------+---------------------------+
| "DELAY_INCREMENT_PER_ROUND"                             | 4s                        |
+---------------------------------------------------------+---------------------------+
| "CONSENSUS_RIGHTS_DELAY"                                | 2 cycles                  |
+---------------------------------------------------------+---------------------------+
| "GLOBAL_LIMIT_OF_STAKING_OVER_BAKING"                   | 9                         |
+---------------------------------------------------------+---------------------------+
| "LIMIT_OF_DELEGATION_OVER_BAKING"                       | 9                         |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_STAKE"                                         | 6,000 ꜩ                   |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_FROZEN_STAKE"                                  | 600 ꜩ                     |
+---------------------------------------------------------+---------------------------+
| "MINIMAL_PARTICIPATION_RATIO"                           | 2/3                       |
+---------------------------------------------------------+---------------------------+
| "PERCENTAGE_OF_FROZEN_DEPOSITS_SLASHED_PER_DOUBLE_BAKI  | 5%                        |
| NG"                                                     |                           |
+---------------------------------------------------------+---------------------------+
| "SLASHING_DELAY"                                        | 1 cycle                   |
+---------------------------------------------------------+---------------------------+
| "CONSENSUS_KEY_ACTIVATION_DELAY" [1]                    | 2 cycles                  |
+---------------------------------------------------------+---------------------------+
| "ISSUANCE_MODIFICATION_DELAY" [1]                       | 2 cycles                  |
+---------------------------------------------------------+---------------------------+
| "UNSTAKE_FINALIZATION_DELAY" [2]                        | 3 cycles                  |
+---------------------------------------------------------+---------------------------+

The above list of protocol parameters is a subset of the protocol
constants.

[1] These derived constants are automatically set to the same value as
    "CONSENSUS_RIGHTS_DELAY".

[2] This derived constant is automatically set to
    "CONSENSUS_RIGHTS_DELAY + SLASHING_DELAY".


Further External Resources
==========================

* Tenderbake report

* Tenderbake blog post.

* Tenderbake tzip.

The Amendment (and Voting) Process
**********************************

In the Tezos blockchain, the *economic protocol* can be amended.
Specifically, there is an on-chain mechanism to propose changes to the
economic protocol, to vote for-or-against these proposed changes, and,
depending on the result of the vote, to activate these changes or not.

Note that the proposal, voting and activation processes are part of
the economic protocol itself. Consequently the amendment rules
themselves are subject to amendments.

The rest of this page gives more details about the amendment and
voting process.


Periods
=======

The amendment process consists of five *periods*. Each period lasts
for "CYCLES_PER_VOTING_PERIOD" = 14 cycles (that is, around 14 days).
The periods (listed below) typically succeed one to another for a
total duration of approximately 2 months and a half, after which the
whole amendment process starts again.

The five periods are as follows:

* *Proposal period*: During this period, delegates can

  * submit *protocol amendment proposals* (or, simply, *proposals*)
    using the "Proposals" operation (see below);

  * upvote one or several proposals, using the same "Proposals"
    operation.

  Each delegate can submit a maximum of 20 proposals. Duplicates count
  towards this total.

  At the end of a **proposal period**, if participation reaches a
  proposal quorum, the proposal with most support is selected and we
  move to an **exploration period**. Note that support is measured in
  the voting power that delegates supporting the proposal have. E.g.,
  a proposal supported by a single delegate with 600,000 tz of voting
  power has more support than a proposal supported by two delegates
  with 100,000 tz each of voting power.

  If there are no proposals, or a tie between two or more proposals,
  or if participation did not reach the proposal quorum, the process
  moves back to a new **proposal period**.

* *Exploration period*: During this period delegates can cast one Yea,
  Nay, or Pass ballot on the selected proposal. They do so using the
  "Ballot" operation.

  If the voting participation reaches *quorum* and there is a *super-
  majority* of Yea, the process moves to the **cooldown period**. (See
  below for details on participation, quorum, and super-majority.)

  Otherwise the process moves back to the **proposal period**.

* *Cooldown period*: On-chain nothing specific happens during this
  period. Off-chain the delegates can read the proposal with more
  scrutiny, the community can discuss finer points of the proposal,
  the developers can perform additional tests, etc.

  At the end of this period, the process moves to the **promotion
  period**.

* *Promotion period*: During this period, delegates can cast a Yea,
  Nay, or Pass ballots using the "Ballot" operation.

  If the voting participation reaches *quorum* and there is a super-
  majority of Yea, the process moves to the **adoption period**.

  Otherwise the process moves back to the **proposal period**.

* *Adoption period*: On-chain nothing specific happens during this
  period except on the very last block (see below).

  Off-chain the developers release tools that include support for the
  soon-to-be activated protocol, other actors (bakers, indexers, etc.)
  update their infrastructure to support the newly released tools,
  smart-contract developers start working with soon-to-be-available
  features, etc.

  At the very end of the period, the proposal is *activated*. This
  means that the last block of the period is still interpreted by the
  current economic protocol, but the first block after the period is
  interpreted by the new economic protocol (the one that was voted
  in).

  And a new **proposal period** starts.

Warning:

  Thanks to an old spelling mistake, the most enthusiastic voters can
  cast "Yay" ballots. Enthusiasm does however not impact the result of
  the vote and the tools accepting "Yay" consider it as a deprecated
  synonym for "Yea".


Activation
==========

In the last block of the last cycle of an Adoption period (called
activation block), the current Protocol learns about the Protocol into
which the next block should be validated. Once the activation block is
applied, the Octez shell initializes the new protocol and performs the
protocol-stitching process. The first block of the cycle after the
Adoption period belongs to the new Protocol. It does not contain
attestations for the previous block.

After the activation step, the blocks added to the chain are
interpreted in the newly activated protocol. As a result gas costs may
differ, new operations may be available, contracts using new opcodes
may be injected, etc.

Because the amendment process is also part of the economic protocol,
the amendment process now unfolds according to the rules of the newly
activated protocol. As a result the periods may be lengthened or
shortened, a new period might be introduced, a different selection
mechanism may be used, the quorum requirement might differ, etc.


Voting Power
============

When supporting a proposal or casting a Yea, Nay, or Pass ballot, each
delegate has a **voting power** equal to the amount of tez owned by
all its delegators (including the delegate itself of course), no
matter whether they are staked or not.

More precisely, the voting power of a delegate during a voting period
is the sum of its total_staked and total_delegated amounts, measured
in *mutez*, and snapshotted at the beginning of the voting period.
(Note the differences from the validator selection for consensus,
which is based on the more complex baking power instead, and sampled
at the end of an earlier cycle.)


Super-majority and Quorum
=========================

As mentioned above, during either of the **exploration** or
**promotion** periods, delegates can cast ballots using the "Ballot"
operation (see below). In both cases, delegates can cast a single Yea,
Nay, or Pass ballot. A ballot has a weight equal to the delegate's
voting power as detailed above.

For either of these two periods, the process continues to the next
period if the *vote participation* reaches *quorum* and there is a
*super-majority* of Yea.

The *vote participation* is the ratio of all the cumulated stake of
cast ballots (including Pass ballots) to the total stake.

For the first vote, the *quorum* started at 80% of stake. The quorum
is adjusted after each vote as detailed below. This adjustment is
necessary to ensure that the amendment process can continue even if
some delegates stop participating. After each vote the new quorum is
updated based on the exponential moving average of the **vote
participation**:

   new_participation_ema = 0.8 * old_participation_ema + 0.2 * participation

with the following formula:

   new_quorum = 0.2 + (0.7 - 0.2) * old_participation_ema

This formula avoids establishing quorums close to 100% that would be
very difficult to attain, or, conversely, low quorums close to 0%
making little participation chronic, ensuring that the quorums are
lower- and upper-bounded by Quorum caps (0.2 and 0.7, respectively).

The *super-majority* is reached if the cumulated stake of Yea ballots
is greater than 8/10 of the cumulated stake of Yea and Nay ballots.

Note that Pass ballots do not count towards or against the super-
majority; they still counts towards participation and quorum.

More details can be found in the file
src/proto_023_PtSeouLo/lib_protocol/amendment.ml.


The Hash and the Protocol
=========================

On the one hand, the voting part of the process revolves around the
**hash of a protocol**. Specifically, a delegate submits a hash of a
protocol, and all the delegates cast ballots on the proposed hash. The
*hash of a protocol* is the hash of the files that constitute the
source code of the protocol.

On the other hand, the **protocol activation** (at the end of the
**adoption period**) revolves around the compiled sources of the
protocol.

Basically, the voting process works on an identifier of the protocol
whilst the activation step works on the protocol itself. Consequently,
if a protocol hash is voted in and the protocol it identifies is
invalid, the activation step fails.


Checking a hash is of a valid protocol
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When a hash is proposed by a delegate, it is usually accompanied by
some blogposts and forum threads on community websites. These should
include directions for testing the proposed protocols. If you cannot
find such directions, do not hesitate to ask.


Localised failures
^^^^^^^^^^^^^^^^^^

It is possible that the activation step fails on a single node or a
few nodes of the network, but succeed on the others. In this case the
nodes with the failure are stuck, but the network as a whole
continues.

The most likely cause for this is nodes that have not been updated and
do not include a new protocol environment version.

If your node becomes stuck, you should start a fresh up-to-date node.

A protocol is *invalid* if its code cannot be compiled (e.g., if the
code is not valid source code), if its code uses functions not present
in the protocol environment, or if it downgrades the protocol
environment version.

If an invalid protocol is voted in, then the activation fails for all
the nodes, and then the chain becomes stuck. This is why it is
important to vote for hashes that designate valid protocols: ones with
sources that are available and that can be compiled.


Operations
==========

There are two operations used by the delegates: **proposals** and
**ballot**.

A *proposals* operation can only be injected during a proposal period.

   Proposals : {
     source: Signature.Public_key_hash.t ;
     period: Voting_period_repr.t ;
     proposals: Protocol_hash.t list ; }

The "source" is the public key hash of the delegate, "period" is the
unique identifier of each voting period and "proposals" is a non-empty
list of maximum 20 protocol hashes. The operation can be submitted
more than once but only as long as the cumulative length of the
proposals lists is less than 20. Duplicate proposals from the same
delegate are accounted for in the maximum number of proposals for that
delegate. However duplicates from the same delegate are not tallied at
the end of the proposal period.

For example, a delegate submits a *proposals* operation for protocols
A and B early in the proposal period, later a new protocol C is
revealed and the delegate submits another *proposals* operation for
protocols B and C. The list of submissions that will be tallied is
[A,B,C].

A *ballot* operation can only be submitted during periods where a vote
happens (e.g. exploration, promotion), and only once per period.

   Ballot : {
     source: Signature.Public_key_hash.t ;
     period: Voting_period_repr.t ;
     proposal: Protocol_hash.t ;
     ballot: Vote_repr.ballot ; }

The fields "source" and "period" are the same as above, while
"proposal" is the currently selected proposal and "ballot" is one of
"Yea", "Nay" or "Pass". The "Pass" vote allows a delegate to
contribute towards the quorum without contributing towards the super-
majority. This is important because, as detailed above, the quorum is
adaptive and that low participation would lower the quorum of the next
vote.

More details on the operations can be found in
src/proto_023_PtSeouLo/lib_protocol/operation_repr.ml. The binary
format is described by "octez-client describe unsigned operation".


Client Commands
===============

The Octez client, "octez-client", provides commands for basic
exploration and interaction with the amendment and voting process.


Show
----

The client provides a command to show the status of a voting period.
It displays different information for different kind of periods, as in
the following samples:

   $ octez-client show voting period
   Current period: "proposal"
   Blocks remaining until end of period: 59
   Current proposals:
   PsNa6jTtsRfbGaNSoYXNTNM5A7c3Lji22Yf2ZhpFUjQFC17iZVp 2,400,000 ꜩ

   $ octez-client show voting period
   Current period: "exploration"
   Blocks remaining until end of period: 63
   Current proposal: PsNa6jTtsRfbGaNSoYXNTNM5A7c3Lji22Yf2ZhpFUjQFC17iZVp
   Ballots:
     Yay: 2,400,000 ꜩ
     Nay: 0 ꜩ
     Pass: 0 ꜩ
   Current participation 20.00%, necessary quorum 80.00%
   Current in favor 2,400,000 ꜩ, needed supermajority 1,920,000 ꜩ

   $ octez-client show voting period
   Current period: "cooldown"
   Blocks remaining until end of period: 64
   Current proposal: PsNa6jTtsRfbGaNSoYXNTNM5A7c3Lji22Yf2ZhpFUjQFC17iZVp

The proposal has a total stake of 2,400,000 ꜩ, which may come from a
single ballot from a delegate having a staking balance of 2,400,000 ꜩ
or it may come from multiple ballots from delegates with a combined
stake of 2,400,000 ꜩ.


Submit proposals
----------------

During a proposal period, a list of proposals can be submitted with:

   octez-client submit proposals for <delegate> <proposal1> <proposal2> ...

Remember that each delegate can submit a maximum of 20 protocol hashes
and that duplicates count towards this total. Moreover each proposal
is accepted only if it meets one of the following two conditions:

* the protocol hash was already proposed on the network. In this case
  we can submit an additional proposal that "upvotes" an existing one
  and our voting power are added to the ones already supporting the
  proposal.

* the protocol is known by the node. In particular the first proposer
  of a protocol should be able to successfully inject the protocol in
  its node which performs some checks, compiles and loads the
  protocol.

These are protection measures that the Octez client takes to prevent
the accidental injection of invalid protocols. As mentioned above, it
is still important to check the validity of the protocols that you
vote for as they may have been injected via different means.


Submit ballots
--------------

During either of the **exploration** or **promotion** periods, ballots
can be submitted once with:

   octez-client submit ballot for <delegate> <proposal> <yea|nay|pass>


Further External Resources
==========================

Further details and explanations on the voting procedure can be found
at:

* Governance on-chain on Open Tezos

* Tezos Governance on Tezos Agora.

For more details on the client commands refer to the manual at Client
manual.

For vote related RPCs check the Seoul RPCs - Reference under the
prefix "votes/".

For Ledger support refer to Obsidian Systems' documentation.

Accounts and addresses
**********************

The Tezos ledger currently supports two types of accounts that can
hold tokens (and be the destinations of transactions), identified by
distinct addresses:

* A user account (or implicit account) is a non-programmable account,
  whose tokens are spendable and delegatable by the owner of a private
  key. Its address is the hash of the public key, prefixed by "tz1",
  "tz2", "tz3" or "tz4".

* A smart contract is a programmable account, associated with some
  Michelson code and a piece of data storage. The address of a smart
  contract is a unique hash that depends on the operation that led to
  its creation, called origination, prefixed by "KT1". This is why
  smart contracts are also called originated accounts.

Finally, addresses prefixed with "sr1" identify Smart Rollups.


User accounts
=============

From the economic protocol's point of view, user accounts are
considered as a particular case of smart contracts that always succeed
in receiving tokens or tickets, and do nothing else. Transferring
toward a user account succeeds even if the address of the account has
never appeared on the chain before. This is why user accounts are also
called implicit accounts.

Transactions that are signed by the private key corresponding to the
public key hash, *i.e.* address of the account can spend its tokens.
Each prefix for addresses denotes a different cryptographic signing
scheme. They are briefly described below from a user point of view.

The sizes of public keys, secret keys and signatures may differ
between the different schemes but addresses are always 20 bytes long.


"tz1": Ed25519
--------------

Addresses that start with the "tz1" prefix are hashes of Ed25519
public keys and signatures must be produced by using the EdDSA
signature scheme with the Curve25519 curve. This is the default scheme
of Octez when, *e.g.*, generating key pairs. It is also the
recommended cryptographic scheme to use because it offers better
security guarantees than EcDSA and has good performance on most
hardware. It may not be available in all wallets or on all dedicated
chips which is why Tezos supports multiple schemes.


"tz2": Secp256k1
----------------

Addresses that start with the "tz2" prefix are hashes of Secp256k1
public keys and signatures must be produced by using the EcDSA
signature scheme with the Secp256k1 curve. Secp256k1 is notably the
cryptographic scheme used by Bitcoin and Ethereum. This means that
private keys and addresses used on Bitcoin can also be used on Tezos.


"tz3": P-256
------------

Addresses that start with the "tz3" prefix are hashes of P-256 public
keys and signatures must be produced by using the EcDSA signature
scheme with the P-256 curve, also known as Secp256r1. This is one of
the curves for EcDSA recommended by NIST. It is also often the only
cryptographic scheme supported by HSMs (Hardware Security Modules) of
cloud providers.


"tz4": BLS
----------

Addresses that start with the "tz4" prefix are hashes of BLS public
keys and signatures must be produced by using the BLS signature scheme
with the BLS12-381 curve. One particularity of BLS signatures is that
they are aggregatable. This means that multiple signatures can be
aggregated into one, and later verified as having been produced for
the correct expected public keys. This allows for numerous
applications like multi-signatures schemes, multi-party key exchanges,
signatures compaction, etc. BLS is notably used by Zcash and Ethereum
2.0.

Starting in protocol S, "tz4" addresses can be used by bakers, to
forge blocks and consensus operations. The aggregatable property of
BLS signatures is used with consensus operations to reduce the size
signatures take in each block. Such bakers that also wish to
participate in the DAL are required to register a companion key, that
is used to sign the DAL specific attestation and aggregate it with the
regular attestation.


Smart contracts
===============

A transaction to a smart contract address can provide data and
optionally some tokens, and triggers the execution of the code, which
may read and update the storage. The transaction can succeed or fail,
according to the transaction semantics.

Native Multisig accounts
************************

Note:

  This page provides reference documentation for native multisig
  accounts, available starting with protocol S. For step-by-step
  instructions on how to stake and bake with multisig accounts, see
  User stories for staking and baking multisigs .

Tezos accounts are defined by a pair of secret and public keys. Users
use their secret keys to sign operations and blocks (or any other
message), and the resulting signature is validated against the
account's public key. While this model is simple and effective, it
presents two main challenges:

* The entire security of the account relies on a single secret key. If
  this key is compromised, an attacker can gain full control of the
  account.

* The model is not well-suited for organizations or shared accounts,
  as it does not allow for multiple users to manage the account
  collectively.

Both problems are usually solved by using a multisig **contract**,
that allows to share its ownership and its associated balance or
assets, between several participants. Tezos provides specific support
for multisig contracts in the form of a builtin multisig contract and
even a set of related client commands to interact with it. However,
this solution is not applicable in all use cases, and most importantly
in the case of collectively staking or baking. This is because in
Tezos, a smart contract cannot be a delegate, and not even a staker.

Native multisig **accounts** address the above limitations with a
cryptographic solution based on BLS multi-signature schemes. BLS
signatures are particularly well-suited for this purpose due to their
*aggregation* properties. Tezos supports BLS signatures with tz4
address accounts, and starting with protocol S, these adopt a proof-
of-possession (PoP) scheme. As a result, protocols starting with S can
benefit from faster verification of multiple signatures of the same
message without breaking compatibility with the existing support for
BLS (and "tz4" accounts).

The multisig accounts implementation offers RPCs and client commands
to facilitate the signing of operations in a native multisig setup. As
a result, the protocol does not need to differentiate between a "tz4"
address belonging to a single user account and one associated with a
multisig account.


Enshrined multisig signing scenarios
====================================

A (native) **multisig account** is a "tz4" address managed
collectively by N participants (also called members).

Multisig accounts support two different multi-signature schemes, which
provide different scenarios based on the requirements for signing an
operation:

* **N-of-N scenario (signature aggregation)**: All N members must sign
  every operation for it to be valid. In this scenario, users derive a
  multisig account by aggregating the public keys of their existing
  "tz4" addresses without the need for sharing secrets or trusted
  parties.

* **M-of-N scenario (threshold signature)**: A *threshold* number of
  participants is required to sign the operation. Here, M denotes the
  minimum number of signatures required to create a valid signature,
  out of the total of N members. In this scenario, users generate a
  new unencrypted secret for the multisig account, which is then split
  into secret shares that need to be distributed among the
  participants along with a public identifier (a natural number
  between 1 and N). This entails that all participants must trust the
  honesty of the party in charge of splitting the secret key and
  distributing the secrets.

The workflow for signing operations is as follows:

* The participants decide *a priori* which scheme to use: either
  N-of-N or M-of-N.

* The participants reveal the resulting public key of the multisig
  account on-chain.

* The participants inject operations signed with the multisig
  accounts’s signature scheme.

For example, a multisig account can stake 10,000 tez following almost
the same process as if it were carried out by a single staker. The
only difference is the signing process: signatures (for each
operation) must first be collected from the required members, and then
aggregated into a single signature based on the chosen scenario. The
resulting operation is identical to a staking operation signed by a
single implicit "tz4" account. For instance, the JSON output of a
multisig "stake" operation would look like this:

   [image]

(the source and destination are both the address of the multisig
account, because staking is implemented by a pseudo-operation
consisting in sending a transaction to oneself).


Octez CLI commands and RPC endpoints
====================================

Multisig accounts introduce the following Octez client commands and/or
associated RPC endpoints to facilitate the creation of multisig
accounts and the aggregation of signatures.


Setting up a multisig account
-----------------------------

+------------+-----------------------------------------------+-----------------------------------------------+
|            | N-of-N (signature aggregation)                | M-of-N (threshold signature)                  |
|============|===============================================|===============================================|
| Client     | "octez-client aggregate bls public keys '[{"  | "octez-client share bls secret key <sk>       |
| command    | public_key":"<pk_1>","proof":"<proof_1>"},    | between <N> shares with threshold <M>"        |
|            | ..., {"public_key":"<pk_N>","proof":"<proof_  |                                               |
|            | N>"}]'"                                       |                                               |
+------------+-----------------------------------------------+-----------------------------------------------+
| RPC        | "rpc post /bls/aggregate_public_keys with '[  | No RPC as it would require sending a secret   |
|            | {"public_key":"<pk_1>","proof":"<proof_1>"},  | key.                                          |
|            | ..., {"public_key":"<pk_N>","proof":"<proof_  |                                               |
|            | N>"}]'"                                       |                                               |
+------------+-----------------------------------------------+-----------------------------------------------+
| Parameters | * Public keys "pk_i" of each participant with | * Unencrypted secret key "sk" of the multisig |
|            | their proof-of- possession "proof_i"          | account  * total number of participants "N"   |
|            |                                               | * threshold number "M" (minimum number of     |
|            |                                               | signatures required)                          |
+------------+-----------------------------------------------+-----------------------------------------------+
| Output     | * Public key of the multisig account          | * Public key of the multisig account          |
|            | "public_key" and its hash "public_key_hash"   | "public_key" and its hash "public_key_hash"   |
|            |                                               | * Proof of possession "proof" required for    |
|            |                                               | revealing a multisig account's public key     |
|            |                                               | "public_key"  * Secret shares                 |
|            |                                               | "secret_shares": a unique identifier "id" in  |
|            |                                               | the range [1; "N"] and a secret key           |
|            |                                               | "secret_key" for each participant of the      |
|            |                                               | multisig                                      |
+------------+-----------------------------------------------+-----------------------------------------------+
| Example    | { "public_key": "BLpk...",                    | { "public_key": "BLpk...",                    |
| output     | "public_key_hash": "tz4..." }                 | "public_key_hash": "tz4...",   "proof":       |
|            |                                               | "BLsig...",   "secret_shares":     [ { "id":  |
|            |                                               | 1, "secret_key": "BLsk..." },       { "id":   |
|            |                                               | 2, "secret_key": "BLsk..." },       { "id":   |
|            |                                               | 3, "secret_key": "BLsk..." },       // ...    |
|            |                                               | ] }                                           |
+------------+-----------------------------------------------+-----------------------------------------------+

When aggregating public keys, the N-of-N scenario must also check a
proof-of-possession (PoP) to mitigate rogue key attacks. For creating
a PoP, the dedicated client command is:

+------------+--------------------------------------------------------------------------------------------+
|            | N-of-N (signature aggregation)                                                             |
|============|============================================================================================|
| "proof_i"  | "octez-client create bls proof for <pk_i>"                                                 |
+------------+--------------------------------------------------------------------------------------------+

**On sharing secrets.** The current implementation of M-of-N Scenario
relies on Shamir’s Secret Sharing algorithm to share an unencrypted
master secret key between participants -- i.e. it returns a public key
and unencrypted secret keys for each of the multisig account
participants. We envision implementing support for further key-
generation mechanisms in future, notably Distributed Key Generation,
if there is sufficient ecosystem interest.


Proof of possession
-------------------

For revealing "tz4" accounts a proof of possession is required.

+------------+-----------------------------------------------+-----------------------------------------------+
|            | N-of-N (signature aggregation)                | M-of-N (threshold signature)                  |
|============|===============================================|===============================================|
| Client     | "octez-client aggregate bls proofs            | "proof" is returned by the "share bls secret  |
| command    | '{"public_key": "<pk_multisig>", "proofs":    | key" client command.                          |
|            | ["<proof_1_pk_multisig>", ...,                |                                               |
|            | "<proof_N_pk_multisig>"]}'"                   |                                               |
+------------+-----------------------------------------------+-----------------------------------------------+
| RPC        | "rpc post /bls/aggregate_proofs with          | No RPC as it would require sending a secret   |
|            | '{"public_key": "<pk_multisig>", "proofs":    | key.                                          |
|            | ["<proof_1_pk_multisig>", ...,                |                                               |
|            | "<proof_N_pk_multisig>"]}'"                   |                                               |
+------------+-----------------------------------------------+-----------------------------------------------+
| Parameters | * Public key of the multisig account          | *N/A*                                         |
|            | "public_key"  * Proof of possession           |                                               |
|            | "proof_i_pk_multisig" of each multisig        |                                               |
|            | participant created for a multisig account's  |                                               |
|            | public key "public_key"                       |                                               |
+------------+-----------------------------------------------+-----------------------------------------------+
| Output     | Proof of possession "proof" required for      | *N/A*                                         |
|            | revealing a multisig account's public key     |                                               |
|            | "public_key"                                  |                                               |
+------------+-----------------------------------------------+-----------------------------------------------+
| Example    | "BLsig..."                                    | *N/A*                                         |
| output     |                                               |                                               |
+------------+-----------------------------------------------+-----------------------------------------------+

For N-of-N Scenario, it requires to create "proof_i_pk_multisig" for a
multisig account's public key "pk_multisig":

+------------+--------------------------------------------------------------------------------------------+
|            | N-of-N (signature aggregation)                                                             |
|============|============================================================================================|
| "proof_i_  | "octez-client create bls proof for <pk_i> --override-public-key <pk_multisig>"             |
| pk_multis  |                                                                                            |
| ig"        |                                                                                            |
+------------+--------------------------------------------------------------------------------------------+


Signing operations with multisig accounts
-----------------------------------------

+------------+-----------------------------------------------+-----------------------------------------------+
|            | N-of-N (signature aggregation)                | M-of-N (threshold signature)                  |
|============|===============================================|===============================================|
| Client     | "octez-client aggregate bls signatures        | "octez-client threshold bls signatures        |
| command    | '{"public_key": "<pk_multisig>", "message":   | '{"public_key": "<pk_multisig>", "message":   |
|            | "<msg>", "signature_shares":                  | "<msg>", "signature_shares":                  |
|            | ["<signature_1>", ..., "<signature_N>"]}'"    | [{"id":1,"signature":"<signature_1>"},        |
|            |                                               | {"id":5,"signature":"<signature_5>"},...]}'"  |
+------------+-----------------------------------------------+-----------------------------------------------+
| RPC        | "rpc post /bls/aggregate_signatures with      | "rpc post /bls/threshold_signatures with      |
|            | '{"public_key": "<pk_multisig>", "message":   | '{"public_key": "<pk_multisig>", "message":   |
|            | "<msg>", "signature_shares":                  | "<msg>", "signature_shares":                  |
|            | ["<signature_1>", ..., "<signature_N>"]}'"    | [{"id":1,"signature":"<signature_1>"},        |
|            |                                               | {"id":5,"signature":"<signature_5>"},...]}'"  |
+------------+-----------------------------------------------+-----------------------------------------------+
| Parameters | * Public key of the multisig account          | * Public key of the multisig account          |
|            | "public_key"  * Message "message"  *          | "public_key"  * Message "message"  *          |
|            | Signatures "signature_i" for "message" from   | Signatures "signature_i" from threshold       |
|            | each multisig participant                     | number of multisig participants with their    |
|            |                                               | identifier.  * Signature "signature_i" for    |
|            |                                               | "message" is produced by a multisig           |
|            |                                               | participant "id_i"                            |
+------------+-----------------------------------------------+-----------------------------------------------+
| Output     | Valid aggregated signature of a message       | *idem*                                        |
|            | "message" under a multisig account's public   |                                               |
|            | key "public_key"                              |                                               |
+------------+-----------------------------------------------+-----------------------------------------------+
| Example    | "BLsig..."                                    | *idem*                                        |
| output     |                                               |                                               |
+------------+-----------------------------------------------+-----------------------------------------------+

In both scenarios, multisig participants may use the existing "sign
bytes" client command to produce their own signature "signature_i":

   octez-client sign bytes '<unsigned_operation>' for <pk_i>

Signature shares can also be verified using the existing "check bytes"
client command:

   octez-client check that bytes '<unsigned_operation>' were signed by <pk_i> to produce <signature_i>

Michelson: the language of Smart Contracts in Tezos
***************************************************

This specification gives a detailed formal semantics of the Michelson
language and a short explanation of how smart contracts are executed
and interact in the blockchain.

The language is stack-based, with high level data types and
primitives, and strict static type checking. Its design cherry-picks
traits from several language families. Vigilant readers will notice
direct references to Forth, Scheme, ML and Cat.

A Michelson program is a series of instructions that are run in
sequence: each instruction receives as input the stack resulting from
the previous instruction, and rewrites it for the next one. The stack
contains both immediate values and heap allocated structures. All
values are immutable and garbage collected.

The types of the input and output stack are fixed and monomorphic, and
the program is typechecked before being introduced into the system. No
smart contract execution can fail because an instruction has been
executed on a stack of unexpected length or contents.

This specification gives the complete instruction set, type system and
semantics of the language. It is meant as a precise reference manual,
not an easy introduction. Even though, some examples are provided at
the end of the document and can be read first or at the same time as
the specification. The document also starts with a less formal
explanation of the context: how Michelson code interacts with the
blockchain.


Semantics of smart contracts and transactions
=============================================

The Tezos ledger currently has two types of accounts that can hold
tokens. Accounts can be used in transactions as senders, to send
tokens, or as destinations, to receive tokens.

* User account: non-programmable account whose address is the public
  key hash, prefixed by "tz" and one digit. User accounts are
  sometimes called implicit accounts. A transaction to such an address
  cannot provide data, except tickets.

* Smart contract: programmable account associated to some Michelson
  code, whose address is a unique hash, prefixed by "KT1". A
  transaction to such an address can provide data, and can fail for
  reasons detailed below. Smart contracts are sometimes called
  originated accounts.

Finally, addresses prefixed with "sr1" identify Smart Rollups, which
cannot hold tokens but can be the destination of transactions.

See Accounts and addresses for more details.


Smart contract call semantics
-----------------------------

The only way to call a smart contract is to perform a transaction
operation whose destination is the address of the smart contract.

Alongside their tokens, smart contracts keep a piece of storage. Both
are ruled by a specific logic specified by a Michelson program. A
transaction to a smart contract will provide an input value and in
option some tokens, and in return, the smart contract can modify its
storage and transfer its tokens.

The Michelson program receives as input a stack containing a single
pair whose first element is an input value and second element the
content of the storage space. It must return a stack containing a
single pair whose first element is the list of operations that it
wants to emit, and second element is the new contents of the storage
space. Alternatively, a Michelson program can fail, explicitly using a
specific opcode, or because something went wrong that could not be
caught by the type system (e.g. gas exhaustion).

A bit of polymorphism can be used at contract level, with a
lightweight system of named entrypoints: instead of an input value,
the contract can be called with an entrypoint name and an argument,
and these two components are transformed automatically in a simple and
deterministic way to an input value. This feature is available both
for users and from Michelson code. See the dedicated section.


Transaction semantics
---------------------

On one hand, a smart call may result from an external operation of
kind Transaction. External operations are blockchain operations
included in a block, signed by a user account.

On the other hand, a smart contract call may be emitted as an internal
operation of kind Transaction, either when executing another smart
contract call or when triggering the execution of an outbox message
from a smart rollup.

This section explains the whole semantics of an external transaction,
including the execution of the smart contract called directly by it
and the others smart contracts called indirectly via emitted internal
operations.

The subset of blockchain operations that can be emitted by Michelson
programs as internal operations are of the following kinds:

* Transaction transferring:

  * tokens and optionally tickets to a user account, or

  * tokens and parameters to a smart contract (or, optionally, to a
    specified entrypoint of a smart contract), or

  * parameters to a smart rollup.

* Origination creating a new smart contract from its Michelson source
  code, an initial amount of tokens transferred from the source, and
  an initial storage content.

* Delegation assigning the tokens of the sender account to the stake
  of a user account (without transferring any tokens).

* Contract event delivering live information from a smart contract to
  external applications.

Internal operations are not included in any block, and are not signed.

Internal operations are run in an atomic sequence with the external
operation who triggered the contract call, *after* the contract
execution, in depth-first order. This implies in particular that any
contract calls emitted by a contract C are executed after contract C
successfully finishes execution returning its result (the list of
internal operations and the new storage).

Note that manager operations batches contain a sequence of external
operations signed as a whole by a source user account, which are
executed atomically. For example, in case of a batch of two external
operations, execution proceeds as follows:

   +------+----------------+-------+----------------+
   | op 1 | internal ops 1 |  op 2 | internal ops 2 |
   +------+----------------+-------+----------------+

Smart contracts called by internal operations can in turn also emit
internal operations. The interpreter uses a stack of internal
operations to perform them in depth-first order, as in the following
more detailed example, corresponding to the two external operations
above.

   +-----------+---------------+--------------------------+
   | executing | emissions     | resulting stack          |
   +-----------+---------------+--------------------------+
   | op 1      | 1a, 1b, 1c    | 1a, 1b, 1c               |
   | op 1a     | 1ai, 1aj      | 1ai, 1aj, 1b, 1c         |
   | op 1ai    |               | 1aj, 1b, 1c              |
   | op 1aj    |               | 1b, 1c                   |
   | op 1b     | 1bi           | 1bi, 1c                  |
   | op 1bi    |               | 1c                       |
   | op 1c     |               |                          |
   | op 2      | 2a, 2b        | 2a, 2b                   |
   | op 2a     | 2ai           | 2ai, 2b                  |
   | op 2ai    | 2ai1          | 2ai1, 2b                 |
   | op 2ai1   |               | 2b                       |
   | op 2b     | 2bi           | 2bi                      |
   | op 2bi    | 2bi1          | 2bi1                     |
   | op 2bi1   | 2bi2          | 2bi2                     |
   | op 2bi2   |               |                          |
   +-----------+---------------+--------------------------+


Failures
--------

All transactions can fail for a few reasons, mostly:

* Not enough tokens in the source to spend the specified amount.

* The script took too many execution steps.

* The script failed programmatically using the "FAILWITH" instruction.

External transactions can also fail for these additional reasons:

* The signature of the external operations was wrong.

* The code or initial storage in an origination did not typecheck.

* The parameter in a transfer did not typecheck.

* The destination did not exist.

* The specified entrypoint did not exist.

All these errors cannot happen in internal transactions, as the type
system catches them at operation creation time. In particular,
Michelson has two types to talk about other addresses: "address" and
"contract t". The "address" type merely gives the guarantee that the
value has the form of a Tezos address. The "contract t" type, on the
other hand, guarantees that the address is indeed a valid destination
of transfers whose parameter type is "t". To make a transaction from
Michelson, a value of type "contract t" must be provided, and the type
system checks that the argument to the transaction is indeed of type
"t". Hence, all transactions made from Michelson are well formed by
construction.

In any case, when a failure happens, either total success or total
failure is guaranteed. If a transaction (internal or external) fails,
then the whole sequence fails and all the effects up to the failure
are reverted. These transactions can still be included in blocks, and
the transaction fees are given to the user account who baked the
block.


Language semantics
==================

This specification explains in a symbolic way the computation
performed by the Michelson interpreter on a given program and initial
stack to produce the corresponding resulting stack. The Michelson
interpreter is a pure function: it only builds a result stack from the
elements of an initial one, without affecting its environment. This
semantics is then naturally given in what is called a big step form: a
symbolic definition of a recursive reference interpreter. This
definition takes the form of a list of rules that cover all the
possible inputs of the interpreter (program and stack), and describe
the computation of the corresponding resulting stacks.


Rules form and selection
------------------------

The rules have the main following form.

   > (syntax pattern) / (initial stack pattern)  =>  (result stack pattern)
       iff (conditions)
       where (recursions)
       and (more recursions)

The left hand side of the "=>" sign is used for selecting the rule.
Given a program and an initial stack, one (and only one) rule can be
selected using the following process. First, the toplevel structure of
the program must match the syntax pattern. This is quite simple since
there are only a few non-trivial patterns to deal with instruction
sequences, and the rest is made of trivial patterns that match one
specific instruction. Then, the initial stack must match the initial
stack pattern. Finally, some rules add extra conditions over the
values in the stack that follow the "iff" keyword. Sometimes, several
rules may apply in a given context. In this case, the one that appears
first in this specification is to be selected. If no rule applies, the
result is equivalent to the one for the explicit "FAILWITH"
instruction. This case does not happen on well-typed programs, as
explained in the next section.

The right hand side describes the result of the interpreter if the
rule applies. It consists in a stack pattern, whose parts are either
constants, or elements of the context (program and initial stack) that
have been named on the left hand side of the "=>" sign.


Recursive rules (big step form)
-------------------------------

Sometimes, the result of interpreting a program is derived from the
result of interpreting another one (as in conditionals or function
calls). In these cases, the rule contains a clause of the following
form.

   where (intermediate program) / (intermediate stack)  =>  (partial result)

This means that this rule applies in case interpreting the
intermediate state on the left gives the pattern on the right.

The left hand sign of the "=>" sign is constructed from elements of
the initial state or other partial results, and the right hand side
identify parts that can be used to build the result stack of the rule.

If the partial result pattern does not actually match the result of
the interpretation, then the result of the whole rule is equivalent to
the one for the explicit "FAILWITH" instruction. Again, this case does
not happen on well-typed programs, as explained in the next section.


Format of patterns
------------------

Code patterns are of one of the following syntactical forms.

* "INSTR" (an uppercase identifier) is a simple instruction (e.g.
  "DROP").

* "INSTR (arg) ..." is a compound instruction, whose arguments can be
  code, data or type patterns (e.g. "PUSH nat 3").

* "{ (instr) ; ... }" is a possibly empty sequence of instructions,
  (e.g. "IF { SWAP ; DROP } { DROP }"), nested sequences can drop the
  braces.

* "name" is a pattern that matches any program and names a part of the
  matched program that can be used to build the result.

* "_" is a pattern that matches any instruction.

Stack patterns are of one of the following syntactical forms.

* "[FAILED]" is the special failed state.

* "[]" is the empty stack.

* "(top) : (rest)" is a stack whose top element is matched by the data
  pattern "(top)" on the left, and whose remaining elements are
  matched by the stack pattern "(rest)" on the right (e.g. "x : y :
  rest").

* "name" is a pattern that matches any stack and names it in order to
  use it to build the result.

* "_" is a pattern that matches any stack.

Data patterns are of one of the following syntactical forms.

* integer/natural number literals, (e.g. "3").

* string literals, (e.g. ""contents"").

* raw byte sequence literals (e.g. "0xABCDEF42").

* "Tag" (capitalized) is a symbolic constant, (e.g. "Unit", "True",
  "False").

* "(Tag (arg) ...)" tagged constructed data, (e.g. "(Pair 3 4)").

* a code pattern for first class code values.

* "name" to name a value in order to use it to build the result.

* "_" to match any value.

The domain of instruction names, symbolic constants and data
constructors is fixed by this specification. Michelson does not let
the programmer introduce its own types.

Be aware that the syntax used in the specification may differ from the
concrete syntax. In particular some instructions are annotated with
types that are not present in the concrete language because they are
synthesized by the typechecker.


Shortcuts
---------

Sometimes, it is easier to think (and shorter to write) in terms of
program rewriting than in terms of big step semantics. When it is the
case, and when both are equivalents, we write rules of the form:

   p / S  =>  S''
   where   p' / S'  =>  S''

using the following shortcut:

   p / S  =>  p' / S'

The concrete language also has some syntax sugar to group some common
sequences of operations as one. This is described in this
specification using a simple regular expression style recursive
instruction rewriting.


Introduction to the type system and notations
=============================================

This specification describes a type system for Michelson. To make
things clear, in particular to readers that are not accustomed to
reading formal programming language specifications, it does not give a
typechecking or inference algorithm. It only gives an intentional
definition of what we consider to be well-typed programs. For each
syntactical form, it describes the stacks that are considered well-
typed inputs, and the resulting outputs.

The type system is sound, meaning that if a program can be given a
type, then if run on a well-typed input stack, the interpreter will
never apply an interpretation rule on a stack of unexpected length or
contents. Also, it will never reach a state where it cannot select an
appropriate rule to continue the execution. Well-typed programs do not
block, and do not go wrong.


Type notations
--------------

The specification introduces notations for the types of values, terms
and stacks. Apart from a subset of value types that appear in the form
of type annotations in some places throughout the language, it is
important to understand that this type language only exists in the
specification.

A stack type can be written:

* "[]" for the empty stack.

* "(top) : (rest)" for the stack whose first value has type "(top)"
  and queue has stack type "(rest)".

Instructions, programs and primitives of the language are also typed,
their types are written:

   (type of stack before) -> (type of stack after)

The types of values in the stack are written:

* "identifier" for a primitive data-type (e.g. "bool").

* "identifier (arg)" for a parametric data-type with one parameter
  type "(arg)" (e.g. "list nat").

* "identifier (arg) ..." for a parametric data-type with several
  parameters (e.g. "map string int").

* "[ (type of stack before) -> (type of stack after) ]" for a code
  quotation, (e.g. "[ int : int : [] -> int : [] ]").

* "lambda (arg) (ret)" is a shortcut for "[ (arg) : [] -> (ret) : []
  ]".


Meta type variables
-------------------

The typing rules introduce meta type variables. To be clear, this has
nothing to do with polymorphism, which Michelson does not have. These
variables only live at the specification level, and are used to
express the consistency between the parts of the program. For
instance, the typing rule for the "IF" construct introduces meta
variables to express that both branches must have the same type.

Here are the notations for meta type variables:

* "'a" for a type variable.

* "'A" for a stack type variable.

* "_" for an anonymous type or stack type variable.


Typing rules
------------

The system is syntax directed, meaning that it defines a single typing
rule for each syntax construct. A typing rule restricts the type of
input stacks that are authorized for this syntax construct, links the
output type to the input type, and links both of them to the
subexpressions when needed, using meta type variables.

Typing rules are of the form:

   (syntax pattern)
   :: (type of stack before) -> (type of stack after) [rule-name]
      iff (premises)

Where premises are typing requirements over subprograms or values in
the stack, both of the form "(x) :: (type)", meaning that value "(x)"
must have type "(type)".

A program is shown well-typed if one can find an instance of a rule
that applies to the toplevel program expression, with all meta type
variables replaced by non variable type expressions, and of which all
type requirements in the premises can be proven well-typed in the same
manner. For the reader unfamiliar with formal type systems, this is
called building a typing derivation.

Here is an example typing derivation on a small program that computes
"(x+5)*10" for a given input "x", obtained by instantiating the typing
rules for instructions "PUSH", "ADD" and for the sequence, as found in
the next sections. When instantiating, we replace the "iff" with "by".

   { PUSH nat 5 ; ADD ; PUSH nat 10 ; MUL }
   :: [ nat : [] -> nat : [] ]
      by { PUSH nat 5 ; ADD }
         :: [ nat : [] -> nat : [] ]
            by PUSH nat 5
               :: [ nat : [] -> nat : nat : [] ]
                  by 5 :: nat
           and ADD
               :: [ nat : nat : [] -> nat : [] ]
     and { PUSH nat 10 ; MUL }
         :: [ nat : [] -> nat : [] ]
            by PUSH nat 10
               :: [ nat : [] -> nat : nat : [] ]
                  by 10 :: nat
           and MUL
               :: [ nat : nat : [] -> nat : [] ]

Producing such a typing derivation can be done in a number of manners,
such as unification or abstract interpretation. In the implementation
of Michelson, this is done by performing a recursive symbolic
evaluation of the program on an abstract stack representing the input
type provided by the programmer, and checking that the resulting
symbolic stack is consistent with the expected result, also provided
by the programmer.


Type normalization
------------------

For convenience, Michelson developers can use some compact type
notations for types that do not exist in the language implementation.
These are just shorthands that are expanded into existing types during
the so-called "type normalization".

Currently, the only such notation concerns N-ary pairs (also called
tuples), which are expanded into "right combs", that is, nested binary
pairs with right associativity. See type pair for details.

The node RPC "/helpers/script/normalize_type" is available to
normalize a given Michelson type (see RPCs - OpenAPI reference, within
the protocol-dependent RPCs). This RPC is intended for tool developers
wanting to support the type shorthands in their tools without
reimplementing their normalization. However, one side effect of this
RPC is the stripping of annotations. As a consequence, a tool needing
to preserve annotations on shorthand data types should implement its
own type normalization instead of relying on this RPC.


Side note
---------

As with most type systems, it is incomplete. There are programs that
cannot be given a type in this type system, yet that would not go
wrong if executed. This is a necessary compromise to make the type
system usable. Also, it is important to remember that the
implementation of Michelson does not accept as many programs as the
type system describes as well-typed. This is because the
implementation uses a simple single pass typechecking algorithm, and
does not handle any form of polymorphism.


Types and instructions
======================

The complete sets of Michelson types and instructions are detailed in
the interactive Michelson reference page.

* Specifically, it contains synthesis tables for types and for
  instructions.

* Instructions are also organized by categories.

* Each instruction is precisely defined using typing and semantic
  inference rules.


Removed instructions and types
==============================

Protocol Babylon deprecated the following instructions. Because no
smart contract used these on Mainnet before they got deprecated, they
have been removed. The Michelson type-checker will reject any contract
using them.

* "CREATE_CONTRACT { parameter 'p ; storage 'g ; code ... }": Forge a
  new contract from a literal.

     Γ ⊢ CREATE_CONTRACT { parameter 'p ; storage 'g ; code ... }
     :: key_hash : option key_hash : bool : bool : mutez : 'g : 'S
     ⇒ operation : address : 'S

  There is a new version of this instruction, see its documentation.

* "CREATE_ACCOUNT": Forge an account creation operation.

     Γ ⊢ CREATE_ACCOUNT :: key_hash : option key_hash : bool : mutez : 'S
     ⇒ operation : address : 'S

  Takes as argument the manager, optional delegate, the delegatable
  flag and finally the initial amount taken from the currently
  executed contract. This instruction originates a contract with two
  entrypoints; "%default" of type "unit" that does nothing and "%do"
  of type "lambda unit (list operation)" that executes and returns the
  parameter if the sender is the contract's manager.

* "STEPS_TO_QUOTA": Push the remaining steps before the contract
  execution must terminate.

     Γ ⊢ STEPS_TO_QUOTA :: 'S ⇒ nat : 'S

Protocol Mumbai deprecated the following type. Because no smart
contract used it on Mainnet before it got deprecated, it has been
removed. The Michelson type-checker will reject any contract using it.

* "tx_rollup_l2_address": An address used to identify an account in a
  transaction rollup ledger. It is the hash of a BLS public key, used
  to authenticate layer-2 operations to transfer tickets from this
  account.


Macros
======

In addition to the instructions listed in the interactive Michelson
reference manual, several extensions have been added to the language's
concrete syntax. If you are interacting with the node via RPC,
bypassing the client, which expands away these macros, you will need
to desugar them yourself.

These macros are designed to be unambiguous and reversible, meaning
that errors are reported in terms of desugared syntax. Below you'll
see these macros defined in terms of other syntactic forms. That is
how these macros are seen by the node.


Compare
-------

Syntactic sugar exists for merging "COMPARE" and comparison
combinators, and also for branching.

* "CMP{EQ|NEQ|LT|GT|LE|GE}"

   > CMP(\op) / S  =>  COMPARE ; (\op) / S

* "IF{EQ|NEQ|LT|GT|LE|GE} bt bf"

   > IF(\op) bt bf / S  =>  (\op) ; IF bt bf / S

* "IFCMP{EQ|NEQ|LT|GT|LE|GE} bt bf"

   > IFCMP(\op) / S  =>  COMPARE ; (\op) ; IF bt bf / S


Fail
----

The "FAIL" macros is equivalent to "UNIT; FAILWITH" and is callable in
any context since it does not use its input stack.

* "FAIL"

   > FAIL / S  =>  UNIT; FAILWITH / S


Assertion macros
----------------

All assertion operations are syntactic sugar for conditionals with a
"FAIL" instruction in the appropriate branch. When possible, use them
to increase clarity about illegal states.

* "ASSERT"

   > ASSERT  =>  IF {} {FAIL}

* "ASSERT_{EQ|NEQ|LT|LE|GT|GE}"

   > ASSERT_(\op)  =>  IF(\op) {} {FAIL}

* "ASSERT_CMP{EQ|NEQ|LT|LE|GT|GE}"

   > ASSERT_CMP(\op)  =>  IFCMP(\op) {} {FAIL}

* "ASSERT_NONE"

   > ASSERT_NONE  =>  IF_NONE {} {FAIL}

* "ASSERT_SOME"

   > ASSERT_SOME @x =>  IF_NONE {FAIL} {RENAME @x}

* "ASSERT_LEFT"

   > ASSERT_LEFT @x =>  IF_LEFT {RENAME @x} {FAIL}

* "ASSERT_RIGHT"

   > ASSERT_RIGHT @x =>  IF_LEFT {FAIL} {RENAME @x}


Syntactic Conveniences
----------------------

These macros are simply more convenient syntax for various common
operations.

* "P(\left=A|P(\left)(\right))(\right=I|P(\left)(\right))R": A
  syntactic sugar for building nested pairs. In the case of right
  combs, "PAIR n" is more efficient.

   > PA(\right)R / S => DIP ((\right)R) ; PAIR / S
   > P(\left)IR / S => (\left)R ; PAIR / S
   > P(\left)(\right)R =>  (\left)R ; DIP ((\right)R) ; PAIR / S

A good way to quickly figure which macro to use is to mentally parse
the macro as "P" for pair constructor, "A" for left leaf and "I" for
right leaf. The macro takes as many elements on the stack as there are
leaves and constructs a nested pair with the shape given by its name.

Take the macro "PAPPAIIR" for instance:

   P A  P P A  I    I R
   ( l, ( ( l, r ), r ))

A typing rule can be inferred:

   PAPPAIIR
   :: 'a : 'b : 'c : 'd : 'S  ->  (pair 'a (pair (pair 'b 'c) 'd))

* "UNP(\left=A|P(\left)(\right))(\right=I|P(\left)(\right))R": A
  syntactic sugar for destructing nested pairs. These macros follow
  the same convention as the previous one.

   > UNPA(\right)R / S => UNPAIR ; DIP (UN(\right)R) / S
   > UNP(\left)IR / S => UNPAIR ; UN(\left)R / S
   > UNP(\left)(\right)R => UNPAIR ; DIP (UN(\right)R) ; UN(\left)R / S

* "C[AD]+R": A syntactic sugar for accessing fields in nested pairs.
  In the case of right combs, "CAR k" and "CDR k" are more efficient.

   > CA(\rest=[AD]+)R / S  =>  CAR ; C(\rest)R / S
   > CD(\rest=[AD]+)R / S  =>  CDR ; C(\rest)R / S

* "CAR k": Access the "k" -th part of a right comb of size "n > k +
  1". "CAR 0" is equivalent to "CAR" and in general "CAR k" is
  equivalent to "k" times the "CDR" instruction followed by once the
  "CAR" instruction. Note that this instruction cannot access the last
  element of a right comb; "CDR k" should be used for that.

   > CAR n / S  =>  GET (2n+1) / S

* "CDR k": Access the rightmost element of a right comb of size "k".
  "CDR 0" is a no-op, "CDR 1" is equivalent to "CDR" and in general
  "CDR k" is equivalent to "k" times the "CDR" instruction. Note that
  on a right comb of size "n > k >= 2", "CDR k" will return the right
  comb composed of the same elements but the "k" leftmost ones.

   > CDR n / S  =>  GET (2n) / S

* "IF_SOME bt bf": Inspect an optional value.

   > IF_SOME bt bf / S  =>  IF_NONE bf bt / S

* "IF_RIGHT bt bf": Inspect a value of a union.

   > IF_RIGHT bt bf / S  =>  IF_LEFT bf bt / S

* "SET_CAR": Set the left field of a pair. This is equivalent to
  "SWAP; UPDATE 1".

   > SET_CAR  =>  CDR ; SWAP ; PAIR

* "SET_CDR": Set the right field of a pair. This is equivalent to
  "SWAP; UPDATE 2".

   > SET_CDR  =>  CAR ; PAIR

* "SET_C[AD]+R": A syntactic sugar for setting fields in nested pairs.
  In the case of right combs, "UPDATE n" is more efficient.

   > SET_CA(\rest=[AD]+)R / S   =>
       { DUP ; DIP { CAR ; SET_C(\rest)R } ; CDR ; SWAP ; PAIR } / S
   > SET_CD(\rest=[AD]+)R / S   =>
       { DUP ; DIP { CDR ; SET_C(\rest)R } ; CAR ; PAIR } / S

* "MAP_CAR" code: Transform the left field of a pair.

   > MAP_CAR code  =>  DUP ; CDR ; DIP { CAR ; code } ; SWAP ; PAIR

* "MAP_CDR" code: Transform the right field of a pair.

   > MAP_CDR code  =>  DUP ; CDR ; code ; SWAP ; CAR ; PAIR

* "MAP_C[AD]+R" code: A syntactic sugar for transforming fields in
  nested pairs.

   > MAP_CA(\rest=[AD]+)R code / S   =>
       { DUP ; DIP { CAR ; MAP_C(\rest)R code } ; CDR ; SWAP ; PAIR } / S
   > MAP_CD(\rest=[AD]+)R code / S   =>
       { DUP ; DIP { CDR ; MAP_C(\rest)R code } ; CAR ; PAIR } / S


Concrete syntax
===============

The concrete language is very close to the formal notation of the
specification. Its structure is extremely simple: an expression in the
language can only be one of the five following constructs.

1. An integer in decimal notation.

2. A character string.

3. A byte sequence in hexadecimal notation prefixed by "0x".

4. The application of a primitive to a sequence of expressions.

5. A sequence of expressions.

This simple five cases notation is called Micheline.

In the Tezos protocol, the primitive "constant" with a single
character string applied has special meaning. See Global Constants.


Constants
---------

There are three kinds of constants:

1. Integers or naturals in decimal notation.

2. Strings, with some usual escape sequences: "\n", "\\", "\"".
   Unescaped line-breaks (both "\n" and "\r") cannot appear in a
   Michelson string. Moreover, the current version of Michelson
   restricts strings to be the printable subset of 7-bit ASCII, namely
   characters with codes from within "[32, 126]" range, plus the
   escaped characters mentioned above.

3. Byte sequences in hexadecimal notation, prefixed with "0x".


Differences with the formal notation
------------------------------------

The concrete syntax follows the same lexical conventions as the
specification: instructions are represented by uppercase identifiers,
type constructors by lowercase identifiers, and constant constructors
are capitalized.

All domain specific constants are Micheline constants with specific
formats. Some have two variants accepted by the data type checker: a
readable one in a string, and an optimized one using a more compact
encoding.

* "mutez" amounts are written as naturals.

* "timestamp"s are written either using "RFC3339" notation in a string
  (readable), or as the number of seconds since Epoch (when positive)
  or before Epoch (when negative) (optimized).

* "contract"s, "address"es, "key"s and "signature"s are written as
  strings, in their usual Base58 encoded versions (readable), or as
  their raw bytes (optimized).

* "bls12_381_g1"s and "bls12_381_g2"s are written as their raw bytes,
  using a big-endian point encoding, as specified here.

* "bls12_381_fr"s are written as their raw bytes, using a little-
  endian encoding.

The optimized versions should not reach the RPCs, the protocol code
will convert to optimized by itself when forging operations, storing
to the database, and before hashing to get a canonical representation
of a datum for a given type.

To prevent errors, control flow primitives that take instructions as
parameters require sequences in the concrete syntax.

   IF { instr1_true ; instr2_true ; ... }
      { instr1_false ; instr2_false ; ... }


Main program structure
----------------------

The toplevel of a smart contract file must be an un-delimited sequence
of three primitive applications (in no particular order) that provide
its "code", "parameter" and "storage" fields.

See the next section for a concrete example.


Annotations
===========

The annotation mechanism of Michelson provides ways to better track
data on the stack and to give additional type constraints. Except for
a single exception specified just after, annotations are only here to
add constraints, *i.e.* they cannot turn an otherwise rejected program
into an accepted one. The notable exception to this rule is for
entrypoints: the semantics of the "CONTRACT" and "SELF" instructions
vary depending on their constructor annotations, and some contract
origination may fail due to invalid entrypoint constructor
annotations.

Stack visualization tools like the Michelson's Emacs mode print
annotations associated with each type in the program, as propagated by
the typechecker as well as variable annotations on the types of
elements in the stack. This is useful as a debugging aid.

We distinguish three kinds of annotations:

* type annotations, written ":type_annot",

* variable annotations, written "@var_annot",

* and field or constructors annotations, written "%field_annot".

Note that all annotations are stripped during type normalization.


Type annotations
----------------

Each type can be annotated with at most one type annotation. They are
used to give names to types. For types to be equal, their unnamed
version must be equal and their names must be the same or at least one
type must be unnamed.

For instance, the following Michelson program which put its integer
parameter in the storage is not well typed:

   parameter (int :p) ;
   storage (int :s) ;
   code { UNPAIR ; SWAP ; DROP ; NIL operation ; PAIR }

Whereas this one is:

   parameter (int :p) ;
   storage int ;
   code { UNPAIR ; SWAP ; DROP ; NIL operation ; PAIR }

Inner components of composed typed can also be named.

   (pair :point (int :x_pos) (int :y_pos))

Push-like instructions, that act as constructors, can also be given a
type annotation. The stack type will then have on top a type with a
corresponding name.

   UNIT :t
   :: 'A -> (unit :t) : 'A

   PAIR :t
   :: 'a : 'b : 'S -> (pair :t 'a 'b) : 'S

   SOME :t
   :: 'a : 'S -> (option :t 'a) : 'S

   NONE :t 'a
   :: 'S -> (option :t 'a) : 'S

   LEFT :t 'b
   :: 'a : 'S -> (or :t 'a 'b) : 'S

   RIGHT :t 'a
   :: 'b : 'S -> (or :t 'a 'b) : 'S

   NIL :t 'a
   :: 'S -> (list :t 'a) : 'S

   EMPTY_SET :t 'elt
   :: 'S -> (set :t 'elt) : 'S

   EMPTY_MAP :t 'key 'val
   :: 'S -> (map :t 'key 'val) : 'S

   EMPTY_BIG_MAP :t 'key 'val
   :: 'S -> (big_map :t 'key 'val) : 'S

A no-op instruction "CAST" ensures the top of the stack has the
specified type, and change its type if it is compatible. In
particular, this allows to change or remove type names explicitly.

   CAST 'b
   :: 'a : 'S   ->   'b : 'S
      iff  'a = 'b

   > CAST t / a : S  =>  a : S


Variable annotations
--------------------

Variable annotations can only be used on instructions that produce
elements on the stack. An instruction that produces "n" elements on
the stack can be given at most "n" variable annotations.

The stack type contains both the types of each element in the stack,
as well as an optional variable annotation for each element. In this
sub-section we note:

* "[]" for the empty stack,

* "@annot (top) : (rest)" for the stack whose first value has type
  "(top)" and is annotated with variable annotation "@annot" and whose
  queue has stack type "(rest)".

The instructions which do not accept any variable annotations are:

   DROP
   SWAP
   DIG
   DUG
   IF_NONE
   IF_LEFT
   IF_CONS
   ITER
   IF
   LOOP
   LOOP_LEFT
   DIP
   FAILWITH

The instructions which accept at most one variable annotation are:

   DUP
   PUSH
   UNIT
   SOME
   NONE
   PAIR
   CAR
   CDR
   LEFT
   RIGHT
   NIL
   CONS
   SIZE
   MAP
   MEM
   EMPTY_SET
   EMPTY_MAP
   EMPTY_BIG_MAP
   UPDATE
   GET
   LAMBDA
   LAMBDA_REC
   EXEC
   ADD
   SUB
   CONCAT
   MUL
   OR
   AND
   XOR
   NOT
   ABS
   ISNAT
   INT
   NEG
   EDIV
   LSL
   LSR
   COMPARE
   EQ
   NEQ
   LT
   GT
   LE
   GE
   ADDRESS
   CONTRACT
   SET_DELEGATE
   IMPLICIT_ACCOUNT
   NOW
   LEVEL
   AMOUNT
   BALANCE
   HASH_KEY
   CHECK_SIGNATURE
   BLAKE2B
   SOURCE
   SENDER
   SELF
   SELF_ADDRESS
   CAST
   RENAME
   CHAIN_ID
   NAT
   BYTES

The instructions which accept at most two variable annotations are:

   UNPAIR
   CREATE_CONTRACT

Annotations on instructions that produce multiple elements on the
stack will be used in order, where the first variable annotation is
given to the top-most element on the resulting stack. Instructions
that produce "n" elements on the stack but are given less than "n"
variable annotations will see only their top-most stack type elements
annotated.

   UNPAIR @first @second
   :: pair 'a 'b : 'S
      ->  @first 'a : @second 'b : 'S

   UNPAIR @first
   :: pair 'a 'b : 'S
      ->  @first 'a : 'b : 'S

A no-op instruction "RENAME" allows to rename variables in the stack
or to erase variable annotations in the stack.

   RENAME @new
   :: @old 'a ; 'S -> @new 'a : 'S

   RENAME
   :: @old 'a ; 'S -> 'a : 'S


Field and constructor annotations
---------------------------------

Components of pair types, option types and or types can be annotated
with a field or constructor annotation. This feature is useful to
encode records fields and constructors of sum types.

   (pair :point
         (int %x)
         (int %y))

The previous Michelson type can be used as visual aid to represent the
record type (given in OCaml-like syntax):

   type point = { x : int ; y : int }

Similarly,

   (or :t
       (int %A)
       (or
          (bool %B)
          (pair %C
                (nat %n1)
                (nat %n2))))

can be used to represent the algebraic data type (in OCaml-like
syntax):

   type t =
     | A of int
     | B of bool
     | C of { n1 : nat ; n2 : nat }

Field annotations are part of the type (at the same level as type name
annotations), and so types with differing field names (if present) are
not considered equal.

Instructions that construct elements of composed types can also be
annotated with one or multiple field annotations (in addition to type
and variable annotations).

   PAIR %fst %snd
   :: 'a : 'b : 'S -> (pair ('a %fst) ('b %snd)) : 'S

   LEFT %left %right 'b
   :: 'a : 'S -> (or ('a %left) ('b %right)) : 'S

   RIGHT %left %right 'a
   :: 'b : 'S -> (or ('a %left) ('b %right)) : 'S

To improve readability and robustness, instructions "CAR" and "CDR"
accept one field annotation. For the contract to type check, the name
of the accessed field in the destructed pair must match the one given
here.

   CAR %fst
   :: (pair ('a %fst) 'b) : S -> 'a : 'S

   CDR %snd
   :: (pair 'a ('b %snd)) : S -> 'b : 'S


Syntax
------

Primitive applications can receive one or many annotations.

An annotation is a sequence of characters that matches the regular
expression "@%|@%%|%@|[@:%][_0-9a-zA-Z][_0-9a-zA-Z\.%@]*". Note
however that "@%", "@%%" and "%@" are special annotations and are not
allowed everywhere.

Annotations come after the primitive name and before its potential
arguments.

   (prim @v :t %x arg1 arg2 ...)

Ordering between different kinds of annotations is not significant,
but ordering among annotations of the same kind is. Annotations of the
same kind must be grouped together.

For instance these two annotated instructions are equivalent:

   PAIR :t @my_pair %x %y

   PAIR %x %y :t @my_pair

An annotation can be empty, in this case it will mean *no annotation*
and can be used as a wildcard. For instance, it is useful to annotate
only the right field of a pair instruction "PAIR % %right" or to
ignore field access constraints, *e.g.* in the macro "UNPPAIPAIR %x1 %
%x3 %x4".


Annotations and macros
----------------------

Macros also support annotations, which are propagated on their
expanded forms. As with instructions, macros that produce "n" values
on the stack accept "n" variable annotations.

   DUU+P @annot
   > DUU(\rest=U*)P @annot / S  =>  DIP (DU(\rest)P @annot) ; SWAP / S

   C[AD]+R @annot %field_name
   > CA(\rest=[AD]+)R @annot %field_name / S  =>  CAR ; C(\rest)R @annot %field_name / S
   > CD(\rest=[AD]+)R @annot %field_name / S  =>  CDR ; C(\rest)R @annot %field_name / S

   CMP{EQ|NEQ|LT|GT|LE|GE} @annot
   > CMP(\op) @annot / S  =>  COMPARE ; (\op) @annot / S

The variable annotation on "SET_C[AD]+R" and "MAP_C[AD]+R" annotates
the resulting toplevel pair while its field annotation is used to
check that the modified field is the expected one.

   SET_C[AD]+R @var %field
   > SET_CAR @var %field =>  CDR %field ; SWAP ; PAIR @var
   > SET_CDR @var %field =>  CAR %field ; PAIR @var
   > SET_CA(\rest=[AD]+)R @var %field / S   =>
     { DUP ; DIP { CAR ; SET_C(\rest)R %field } ; CDR ; SWAP ; PAIR @var } / S
   > SET_CD(\rest=[AD]+)R  @var %field/ S   =>
     { DUP ; DIP { CDR ; SET_C(\rest)R %field } ; CAR ; PAIR @var } / S

   MAP_C[AD]+R @var %field code
   > MAP_CAR code  =>  DUP ; CDR ; DIP { CAR %field ; code } ; SWAP ; PAIR @var
   > MAP_CDR code  =>  DUP ; CDR %field ; code ; SWAP ; CAR ; PAIR @var
   > MAP_CA(\rest=[AD]+)R @var %field code / S   =>
     { DUP ; DIP { CAR ; MAP_C(\rest)R %field code } ; CDR ; SWAP ; PAIR @var} / S
   > MAP_CD(\rest=[AD]+)R @var %field code / S   =>
    { DUP ; DIP { CDR ; MAP_C(\rest)R %field code } ; CAR ; PAIR @var} / S

Macros for nested "PAIR" accept multiple annotations. Field
annotations for "PAIR" give names to leaves of the constructed nested
pair, in order.  This next snippet gives examples instead of generic
rewrite rules for readability purposes.

   PAPPAIIR @p %x1 %x2 %x3 %x4
   :: 'a : 'b : 'c : 'd : 'S
      -> @p (pair ('a %x1) (pair (pair ('b %x) ('c %x3)) ('d %x4))) : 'S

   PAPAIR @p %x1 %x2 %x3
   :: 'a : 'b : 'c : 'S  ->  @p (pair ('a %x1) (pair ('b %x) ('c %x3))) : 'S

Annotations for nested "UNPAIR" are deprecated.


Automatic variable and field annotations inferring
--------------------------------------------------

When no annotation is provided by the Michelson programmer, the
typechecker infers some annotations in specific cases. This greatly
helps users track information in the stack for bare contracts.

For unannotated accesses with "CAR" and "CDR" to fields that are named
will be appended (with an additional "." character) to the pair
variable annotation.

   CDAR
   :: @p (pair ('a %foo) (pair %bar ('b %x) ('c %y))) : 'S ->  @p.bar.x 'b : 'S

If fields are not named but the pair is still named in the stack then
".car" or ".cdr" will be appended.

   CDAR
   :: @p (pair 'a (pair 'b 'c)) : 'S ->  @p.cdr.car 'b : 'S

If the original pair is not named in the stack, but a field annotation
is present in the pair type the accessed value will be annotated with
a variable annotation corresponding to the field annotation alone.

   CDAR
   :: (pair ('a %foo) (pair %bar ('b %x) ('c %y))) : 'S ->  @bar.x 'b : 'S

A similar mechanism is used for context dependent instructions:

   ADDRESS  :: @c contract _ : 'S   ->   @c.address address : 'S

   CONTRACT 'p  :: @a address : 'S   ->   @a.contract contract 'p : 'S

   BALANCE :: 'S   ->   @balance mutez : 'S

   SOURCE  :: 'S   ->   @source address : 'S

   SENDER  :: 'S   ->   @sender address : 'S

   SELF  :: 'S   ->   @self contract 'p : 'S

   SELF_ADDRESS  :: 'S   ->   @self address : 'S

   AMOUNT  :: 'S   ->   @amount mutez : 'S

   NOW  :: 'S   ->   @now timestamp : 'S

   LEVEL :: 'S  ->   @level nat : 'S

Inside nested code blocks, bound items on the stack will be given a
default variable name annotation depending on the instruction and
stack type (which can be changed). For instance the annotated typing
rule for "ITER" on lists is:

   ITER body
   :: @l (list 'e) : 'A  ->  'A
      iff body :: [ @l.elt e' : 'A -> 'A ]


Special annotations
-------------------

The special variable annotations "@%" and "@%%" can be used on
instructions "CAR", "CDR", and "UNPAIR". It means to use the accessed
field name (if any) as a name for the value on the stack. The
following typing rule demonstrates their use for instruction "CAR".

   CAR @%
   :: @p (pair ('a %fst) ('b %snd)) : 'S   ->   @fst 'a : 'S

   CAR @%%
   :: @p (pair ('a %fst) ('b %snd)) : 'S   ->   @p.fst 'a : 'S

The special field annotation "%@" can be used on instructions "PAIR",
"LEFT" and "RIGHT". It means to use the variable name annotation in
the stack as a field name for the constructed element. Two examples
with "PAIR" follows, notice the special treatment of annotations with
".".

   PAIR %@ %@
   :: @x 'a : @y 'b : 'S   ->   (pair ('a %x) ('b %y)) : 'S

   PAIR %@ %@
   :: @p.x 'a : @p.y 'b : 'S   ->  @p (pair ('a %x) ('b %y)) : 'S
   :: @p.x 'a : @q.y 'b : 'S   ->  (pair ('a %x) ('b %y)) : 'S


Entrypoints
===========

The specification up to this point has been mostly ignoring existence
of entrypoints: a mechanism of contract level polymorphism. This
mechanism is optional, non intrusive, and transparent to smart
contracts that don't use them. This section is to be read as a patch
over the rest of the specification, introducing rules that apply only
in presence of contracts that make use of entrypoints.


Defining and calling entrypoints
--------------------------------

Entrypoints piggyback on the constructor annotations. A contract with
entrypoints is basically a contract that takes a disjunctive type (a
nesting of "or" types) as the root of its input parameter, decorated
with constructor annotations. An extra check is performed on these
constructor annotations: a contract cannot define two entrypoints with
the same name.

An external transaction can include an entrypoint name alongside the
parameter value. In that case, if there is a constructor annotation
with this name at any position in the nesting of "or" types, the value
is automatically wrapped into the according constructors. If the
transaction specifies an entrypoint, but there is no such constructor
annotation, the transaction fails.

For instance, suppose the following input type.

"parameter (or (or (nat %A) (bool %B)) (or %maybe_C (unit %Z) (string
%C)))"

The input values will be wrapped as in the following examples.

   +------------+-----------+---------------------------------+
   | entrypoint | input     | wrapped input                   |
   +------------+-----------+---------------------------------+
   | %A         | 3         | Left (Left 3)                   |
   | %B         | False     | Left (Right False)              |
   | %C         | "bob"     | Right (Right "bob")             |
   | %Z         | Unit      | Right (Left Unit)               |
   | %maybe_C   | Right "x" | Right (Right "x")               |
   | %maybe_C   | Left Unit | Right (Left Unit)               |
   +------------+-----------+---------------------------------+
   | not given  | value     | value (untouched)               |
   | %BAD       | _         | failure, contract not called    |
   +------------+-----------+---------------------------------+


The "default" entrypoint
------------------------

A special semantics is assigned to the "default" entrypoint. If the
contract does not explicitly declare a "default" entrypoint, then it
is automatically assigned to the root of the parameter type.
Conversely, if the contract is called without specifying an
entrypoint, then it is assumed to be called with the "default"
entrypoint. This behaviour makes the entrypoint system completely
transparent to contracts that do not use it.

This is the case for the previous example, for instance. If a value is
passed to such a contract specifying entrypoint "default", then the
value is fed to the contract untouched, exactly as if no entrypoint
was given.

A non enforced convention is to make the entrypoint "default" of type
unit, and to implement the crediting operation (just receive the
transferred tokens).

A consequence of this semantics is that if the contract uses the
entrypoint system and defines a "default" entrypoint somewhere else
than at the root of the parameter type, then it must provide an
entrypoint for all the paths in the toplevel disjunction. Otherwise,
some parts of the contracts would be dead code.

Another consequence of setting the entrypoint somewhere else than at
the root is that it makes it impossible to send the raw values of the
full parameter type to a contract. A trivial solution for that is to
name the root of the type. The conventional name for that is "root".

Let us recapitulate this by tweaking the names of the previous
example.

"parameter (or %root (or (nat %A) (bool %B)) (or (unit %default)
string))"

The input values will be wrapped as in the following examples.

   +------------+---------------------+-----------------------+
   | entrypoint | input               | wrapped input         |
   +------------+---------------------+-----------------------+
   | %A         | 3                   | Left (Left 3)         |
   | %B         | False               | Left (Right False)    |
   | %default   | Unit                | Right (Left Unit)     |
   | %root      | Right (Right "bob") | Right (Right "bob")   |
   +------------+---------------------+-----------------------+
   | not given  | Unit                | Right (Left Unit)     |
   | %BAD       | _                   | failure, contract not |
   +------------+---------------------+-----------------------+


Calling entrypoints from Michelson
----------------------------------

Michelson code can also produce transactions to a specific entrypoint.

For this, both types "address" and "contract" have the ability to
denote not just an address, but a pair of an address and an
entrypoint. The concrete notation is ""address%entrypoint"". Note that
""address"" is strictly equivalent to ""address%default"", and for
clarity, the second variant is forbidden in the concrete syntax.

When the "TRANSFER_TOKENS" instruction is called, it places the
entrypoint provided in the contract handle in the transaction.

The "CONTRACT t" instruction has a variant "CONTRACT %entrypoint t",
that works as follows. Note that "CONTRACT t" is strictly equivalent
to "CONTRACT %default t", and for clarity, the second variant is
forbidden in the concrete syntax.

   +---------------+---------------------+------------------------------------------+
   | input address | instruction         | output contract                          |
   +---------------+---------------------+------------------------------------------+
   | "addr"        | CONTRACT t          | (Some "addr") if contract exists, has a  |
   |               |                     | default entrypoint of type t, or has no  |
   |               |                     | default entrypoint and parameter type t  |
   +---------------+---------------------+------------------------------------------+
   | "addr%name"   | CONTRACT t          | (Some "addr%name") if addr exists and    |
   +---------------+---------------------+ has an entrypoint %name of type t        |
   | "addr"        | CONTRACT %name t    |                                          |
   +---------------+---------------------+------------------------------------------+
   | "addr%_"      | CONTRACT %_ t       | None                                     |
   +---------------+---------------------+------------------------------------------+

Similarly, the "SELF" instruction has a variant "SELF %entrypoint",
that is only well-typed if the current contract has an entrypoint
named "%entrypoint".

* "SELF %entrypoint"

   :: 'S   ->   contract 'p : 'S
      where   contract 'p is the type of the entrypoint %entrypoint of the current contract

User accounts are considered to have a single "default" entrypoint of
type "Unit".


JSON syntax
===========

Micheline expressions are encoded in JSON like this:

* An integer "N" is an object with a single field ""int"" whose value
  is the decimal representation as a string.

  "{ "int": "N" }"

* A string ""contents"" is an object with a single field ""string""
  whose value is the decimal representation as a string.

  "{ "string": "contents" }"

* A sequence is a JSON array.

  "[ expr, ... ]"

* A primitive application is an object with two fields ""prim"" for
  the primitive name and ""args"" for the arguments (that must contain
  an array). A third optional field ""annots"" contains a list of
  annotations, including their leading "@", "%" or ":" sign.

     "{ "prim": "pair", "args": [ { "prim": "nat", "args": [] }, {
     "prim": "nat", "args": [] } ], "annots": [":t"] }"

As in the concrete syntax, all domain specific constants are encoded
as strings.


Development tools
=================

To ease the development of Michelson scripts, some tools are provided
to Michelson developers.


Emacs mode
----------

Emacs can be used as a practical environment for writing, editing and
debugging Michelson programs. Install it and follow the configuration
instructions in the Michelson Emacs README here.


Interactive toplevel
--------------------

An interactive Michelson toplevel (also known as a REPL) built on the
Mockup mode mode of Octez client is available in
"scripts/michelson_repl.sh", the typical usage is:

   $ octez-client --mode mockup --base-dir /tmp/mockup create mockup
   $ rlwrap scripts/michelson_repl.sh
   > UNIT
     { Stack_elt unit Unit }
   > UNIT
     { Stack_elt unit Unit ; Stack_elt unit Unit }
   > COMPARE
     { Stack_elt int 0 }


Examples
========

Contracts in the system are stored as a piece of code and a global
data storage. The type of the global data of the storage is fixed for
each contract at origination time. This is ensured statically by
checking on origination that the code preserves the type of the global
data. For this, the code of the contract is checked to be of  type
"lambda (pair 'arg 'global) -> (pair (list operation) 'global)" where
"'global" is the type of the original global store given on
origination. The contract also takes a parameter and returns a list of
internal operations, hence the complete calling convention above. The
internal operations are queued for execution when the contract
returns.


Empty contract
--------------

The simplest contract is the contract for which the "parameter" and
"storage" are all of type "unit". This contract is as follows:

   code { CDR ;           # keep the storage
          NIL operation ; # return no internal operation
          PAIR };         # respect the calling convention
   storage unit;
   parameter unit;


Example contract with entrypoints
---------------------------------

The following contract maintains a number in its storage. It has two
entrypoints "add" and "sub" to modify it, and the default entrypoint,
of type "unit" will reset it to "0".

   { parameter (or (or (nat %add) (nat %sub)) (unit %default)) ;
     storage int ;
     code { AMOUNT ; PUSH mutez 0 ; ASSERT_CMPEQ ; UNPAIR ;
            IF_LEFT
              { IF_LEFT { ADD } { SWAP ; SUB } }
              { DROP ; DROP ; PUSH int 0 } ;
            NIL operation ; PAIR } }


Example contract with recursive lambda
--------------------------------------

The following contract computes the factorial of the given parameter
using a recursive function and then saves the result in the storage.

In Michelson regular functions start with a stack containing a single
value, the function argument. If the function is of type "lambda int
int", when calling the function the stack will have just an "int".
Recursive functions start with two values, the argument and the
function itself. Therefore, if the recursive function is of type
"lambda int int" then, when it is being called, the stack will have an
"int" at the top and a "lambda int int" at the bottom.

In this recursive factorial we can see the first branch of the "IF",
this is the base case. The second one performs the recursive call. To
do that, we need to access the function. This is what the "DUP 3"
instruction does. Then we decrement the argument and finally make the
recursive call with "EXEC".

   { parameter int;
     storage int;
     code { CAR ;
            LAMBDA_REC  int int
                        { DUP;
                          EQ;
                          IF { PUSH int 1 }
                             { DUP;
                               DUP 3;
                               PUSH int 1;
                               DUP 4;
                               SUB;
                               EXEC;
                               MUL};
                          DIP { DROP 2 }};
            SWAP;
            EXEC;
            NIL operation;
            PAIR}}


Multisig contract
-----------------

The multisig is a typical access control contract. The ownership of
the multisig contract is shared between "N" participants represented
by their public keys in the contract's storage. Any action on the
multisig contract needs to be signed by "K" participants where the
threshold "K" is also stored in the storage.

To avoid replay of the signatures sent to the contract, the signed
data include not only a description of the action to perform but also
the address of the multisig contract and a counter that gets
incremented at each successful call to the contract.

The multisig commands of Octez command line client use this smart
contract. Moreover, functional correctness of this contract has been
verified using the Coq proof assistant.

   parameter (pair
                (pair :payload
                   (nat %counter) # counter, used to prevent replay attacks
                   (or :action    # payload to sign, represents the requested action
                      (pair :transfer    # transfer tokens
                         (mutez %amount) # amount to transfer
                         (contract %dest unit)) # destination to transfer to
                      (or
                         (option %delegate key_hash) # change the delegate to this address
                         (pair %change_keys          # change the keys controlling the multisig
                            (nat %threshold)         # new threshold
                            (list %keys key)))))     # new list of keys
                (list %sigs (option signature)));    # signatures

   storage (pair (nat %stored_counter) (pair (nat %threshold) (list %keys key))) ;

   code
     {
       UNPAIR ; SWAP ; DUP ; DIP { SWAP } ;
       DIP
         {
           UNPAIR ;
           # pair the payload with the current contract address, to ensure signatures
           # can't be replayed across different contracts if a key is reused.
           DUP ; SELF ; ADDRESS ; CHAIN_ID ; PAIR ; PAIR ;
           PACK ; # form the binary payload that we expect to be signed
           DIP { UNPAIR @counter ; DIP { SWAP } } ; SWAP
         } ;

       # Check that the counters match
       UNPAIR @stored_counter; DIP { SWAP };
       ASSERT_CMPEQ ;

       # Compute the number of valid signatures
       DIP { SWAP } ; UNPAIR @threshold @keys;
       DIP
         {
           # Running count of valid signatures
           PUSH @valid nat 0; SWAP ;
           ITER
             {
               DIP { SWAP } ; SWAP ;
               IF_CONS
                 {
                   IF_SOME
                     { SWAP ;
                       DIP
                         {
                           SWAP ; DIIP { DIP { DUP } ; SWAP } ;
                           # Checks signatures, fails if invalid
                           CHECK_SIGNATURE ; ASSERT ;
                           PUSH nat 1 ; ADD @valid } }
                     { SWAP ; DROP }
                 }
                 {
                   # There were fewer signatures in the list
                   # than keys. Not all signatures must be present, but
                   # they should be marked as absent using the option type.
                   FAIL
                 } ;
               SWAP
             }
         } ;
       # Assert that the threshold is less than or equal to the
       # number of valid signatures.
       ASSERT_CMPLE ;
       DROP ; DROP ;

       # Increment counter and place in storage
       DIP { UNPAIR ; PUSH nat 1 ; ADD @new_counter ; PAIR} ;

       # We have now handled the signature verification part,
       # produce the operation requested by the signers.
       NIL operation ; SWAP ;
       IF_LEFT
         { # Transfer tokens
           UNPAIR ; UNIT ; TRANSFER_TOKENS ; CONS }
         { IF_LEFT {
                     # Change delegate
                     SET_DELEGATE ; CONS }
                   {
                     # Change set of signatures
                     DIP { SWAP ; CAR } ; SWAP ; PAIR ; SWAP }} ;
       PAIR }


Views
-----

Here is an example using views, consisting of two contracts. The first
contract defines two views at toplevel that are named "add_v" and
"mul_v".

   { parameter nat;
     storage nat;
     code { CAR; NIL operation ; PAIR };
     view "add_v" nat nat { UNPAIR; ADD };
     view "mul_v" nat nat { UNPAIR; MUL };
   }

The second contract calls the "add_v" view of the above contract and
obtains a result immediately.

   { parameter (pair nat address) ;
     storage nat ;
     code { CAR ; UNPAIR; VIEW "add_v" nat ;
            IF_SOME { } { FAIL }; NIL operation; PAIR }; }


Full grammar
============

   <data> ::=
     | <int constant>
     | <string constant>
     | <byte sequence constant>
     | Unit
     | True
     | False
     | Pair <data> <data> ...
     | Left <data>
     | Right <data>
     | Some <data>
     | None
     | Lambda_rec <instruction>
     | { <data> ; ... }
     | { Elt <data> <data> ; ... }
     | instruction
   <natural number constant> ::=
     | [0-9]+
   <int constant> ::=
     | <natural number constant>
     | -<natural number constant>
   <string constant> ::=
     | "<string content>*"
   <string content> ::=
     | \"
     | \r
     | \n
     | \t
     | \b
     | \\
     | [^"\]
   <byte sequence constant> ::=
     | 0x[0-9a-fA-F]+
   <instruction> ::=
     | { <instruction> ... }
     | DROP
     | DROP <natural number constant>
     | DUP
     | DUP <natural number constant>
     | SWAP
     | DIG <natural number constant>
     | DUG <natural number constant>
     | PUSH <type> <data>
     | SOME
     | NONE <type>
     | UNIT
     | NEVER
     | IF_NONE { <instruction> ... } { <instruction> ... }
     | PAIR
     | PAIR <natural number constant>
     | CAR
     | CDR
     | UNPAIR
     | UNPAIR <natural number constant>
     | LEFT <type>
     | RIGHT <type>
     | IF_LEFT { <instruction> ... } { <instruction> ... }
     | NIL <type>
     | CONS
     | IF_CONS { <instruction> ... } { <instruction> ... }
     | SIZE
     | EMPTY_SET <comparable type>
     | EMPTY_MAP <comparable type> <type>
     | EMPTY_BIG_MAP <comparable type> <type>
     | MAP { <instruction> ... }
     | ITER { <instruction> ... }
     | MEM
     | GET
     | GET <natural number constant>
     | UPDATE
     | UPDATE <natural number constant>
     | IF { <instruction> ... } { <instruction> ... }
     | LOOP { <instruction> ... }
     | LOOP_LEFT { <instruction> ... }
     | LAMBDA <type> <type> { <instruction> ... }
     | LAMBDA_REC <type> <type> { <instruction> ... }
     | EXEC
     | APPLY
     | DIP { <instruction> ... }
     | DIP <natural number constant> { <instruction> ... }
     | FAILWITH
     | CAST
     | RENAME
     | CONCAT
     | SLICE
     | PACK
     | UNPACK <type>
     | ADD
     | SUB
     | MUL
     | EDIV
     | ABS
     | ISNAT
     | INT
     | NEG
     | LSL
     | LSR
     | OR
     | AND
     | XOR
     | NOT
     | COMPARE
     | EQ
     | NEQ
     | LT
     | GT
     | LE
     | GE
     | SELF
     | SELF_ADDRESS
     | CONTRACT <type>
     | TRANSFER_TOKENS
     | SET_DELEGATE
     | CREATE_CONTRACT { <instruction> ... }
     | IMPLICIT_ACCOUNT
     | VOTING_POWER
     | NOW
     | LEVEL
     | AMOUNT
     | BALANCE
     | CHECK_SIGNATURE
     | BLAKE2B
     | KECCAK
     | SHA3
     | SHA256
     | SHA512
     | HASH_KEY
     | SOURCE
     | SENDER
     | ADDRESS
     | CHAIN_ID
     | TOTAL_VOTING_POWER
     | PAIRING_CHECK
     | SAPLING_EMPTY_STATE <natural number constant>
     | SAPLING_VERIFY_UPDATE
     | TICKET
     | READ_TICKET
     | SPLIT_TICKET
     | JOIN_TICKETS
     | OPEN_CHEST
     | BYTES
     | NAT
   <type> ::=
     | <comparable type>
     | option <type>
     | list <type>
     | set <comparable type>
     | operation
     | contract <type>
     | ticket <comparable type>
     | pair <type> <type> ...
     | or <type> <type>
     | lambda <type> <type>
     | map <comparable type> <type>
     | big_map <comparable type> <type>
     | bls12_381_g1
     | bls12_381_g2
     | bls12_381_fr
     | sapling_transaction <natural number constant>
     | sapling_state <natural number constant>
     | chest
     | chest_key
   <comparable type> ::=
     | unit
     | never
     | bool
     | int
     | nat
     | string
     | chain_id
     | bytes
     | mutez
     | key_hash
     | key
     | signature
     | timestamp
     | address
     | option <comparable type>
     | or <comparable type> <comparable type>
     | pair <comparable type> <comparable type> ...


Reference implementation
========================

The language is implemented in OCaml as follows:

* The lower internal representation is written as a GADT whose type
  parameters encode exactly the typing rules given in this
  specification. In other words, if a program written in this
  representation is accepted by OCaml's typechecker, it is guaranteed
  type-safe. This is of course also valid for programs not handwritten
  but generated by OCaml code, so we are sure that any manipulated
  code is type-safe.

  In the end, what remains to be checked is the encoding of the typing
  rules as OCaml types, which boils down to half a line of code for
  each instruction. Everything else is left to the venerable and well
  trusted OCaml.

* The interpreter is basically the direct transcription of the
  rewriting rules presented above. It takes an instruction, a stack
  and transforms it. OCaml's typechecker ensures that the
  transformation respects the pre and post stack types declared by the
  GADT case for each instruction.

  The only things that remain to be reviewed are value dependent
  choices, such as we did not swap true and false when interpreting
  the IF instruction.

* The input, untyped internal representation is an OCaml ADT with only
  5 grammar constructions: "String", "Int", "Bytes", "Seq" and "Prim".
  It is the target language for the parser, since not all parsable
  programs are well typed, and thus could simply not be constructed
  using the GADT.

* The typechecker is a simple function that recognizes the abstract
  grammar described in section X by pattern matching, producing the
  well-typed, corresponding GADT expressions. It is mostly a checker,
  not a full inferrer, and thus takes some annotations (basically the
  input and output of the program, of lambdas and of uninitialized
  maps and sets). It works by performing a symbolic evaluation of the
  program, transforming a symbolic stack. It only needs one pass over
  the whole program.

  Here again, OCaml does most of the checking, the structure of the
  function is very simple, what we have to check is that we transform
  a "Prim ("If", ...)" into an "If", a "Prim ("Dup", ...)" into a
  "Dup", etc.


TZT, a Syntax extension for writing unit tests
==============================================

This section describes the TZT format, an extension of the Michelson
language allowing to run Michelson unit tests at a finer level than a
full smart contract script. This extension adds syntax to specify an
instruction (or sequence of instructions) to test, a concrete input
stack and the expected output stack.

These unit tests can be useful for both smart contract developers who
need to independently test various parts of the smart contracts they
develop and to the developers of new implementations of the Michelson
interpreter who need to check that their new implementations behave as
the reference implementation by passing a conformance test suite.

Similarly to Michelson scripts, the concrete syntax of TZT unit tests
is Micheline.

TZT unit test files usually have the extension ".tzt". A unit test
file describes a single unit test. It consists of a Micheline sequence
of primitive applications (see Micheline), in no particular order.
This is similar to Michelson scripts but the set of primitives allowed
at the toplevel differ; in Michelson scripts, the allowed toplevel
primitives are "parameter" (mandatory), "storage" (mandatory), "code"
(mandatory), and "view" (optional and repeated). For TZT unit tests,
the toplevel primitives which can be used are:

* "input",

* "code",

* "output",

* "now",

* "sender",

* "source",

* "chain_id",

* "self",

* "parameter",

* "amount",

* "balance",

* "other_contracts", and

* "big_maps".


Mandatory primitives
--------------------

Each of the mandatory primitives "input", "code", and "output" must
occur exactly once in a unit test file in no particular order.

The "input" primitive is used to declare the input stack (see the
syntax of concrete stacks).

The "code" primitive is used to declare the instruction or sequence of
instructions to execute.

The "output" primitive is used to declare if the execution is expected
to succeed or fail and what result is expected from the execution. For
executions expected to succeed, the argument of the "output" primitive
is simply the expected output stack (see the syntax of errors). For
executions expected to fail, the argument is the expected error. In
both cases, the wildcard pattern can be used to omit part of the
expected output.

The simplest test which can be written asserts that executing no
instruction on the empty stack successfully returns the empty stack:

   input {};
   code {};
   output {}

Here is a slightly more involved test which demonstrates the effect of
the SWAP instruction:

   input
     {
       Stack_elt nat 8 ;
       Stack_elt bool False
     };
   code SWAP;
   output
     {
       Stack_elt bool False ;
       Stack_elt nat 8
     }

It is possible to test the effect of several instructions by wrapping
them in a sequence:

   input
     {
       Stack_elt nat 8 ;
       Stack_elt bool False
     };
   code { SWAP ; SWAP };
   output
     {
       Stack_elt nat 8 ;
       Stack_elt bool False
     }

Here is an example showing how to test the "FAILWITH" instruction:

   input {Stack_elt nat 2};
   code FAILWITH;
   output (Failed 2)


Optional primitives
-------------------

Optional primitives are used to set the execution context for the
test. Each of the optional primitives can be used at most once, in no
particular order.

* "amount" (optional, defaults to 0): the amount, expressed in mutez,
  that should be pushed by the AMOUNT instruction

* "balance" (optional, defaults to 0): the balance, expressed in
  mutez, that should be pushed by the BALANCE instruction

* "now" (optional, defaults to ""1970-01-01T00:00:00Z""): the
  timestamp that should be pushed by the NOW instruction

* "sender" (optional, defaults to
  ""tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx""): the sender address that
  should be pushed by the SENDER instruction

* "source" (optional, defaults to
  ""tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx""): the source address that
  should be pushed by the SOURCE instruction

* "chain_id" (optional, defaults to ""NetXdQprcVkpaWU""): the chain
  identifier that should be pushed by the CHAIN_ID instruction

* "self" (optional, defaults to
  ""KT1BEqzn5Wx8uJrZNvuS9DVHmLvG9td3fDLi""): the address that should
  be pushed by the SELF and SELF_ADDRESS instructions

* "parameter" (optional, defaults to "unit"): the type of the
  parameter of the contract pushed by the SELF instruction

* "other_contracts" (optional, defaults to "{}"): mapping between the
  contract addresses that are assumed to exist and their parameter
  types (see the syntax of other contracts specifications)

* "big_maps" (optional, defaults to "{}"): mapping between integers
  representing "big_map" indices and descriptions of big maps (see the
  syntax of extra big maps specifications)

The following test example asserts that the default value for the NOW
instruction is the unix epoch:

   input {};
   code NOW;
   output { Stack_elt timestamp "1970-01-01T00:00:00Z" }

The following example shows how to use the "now" toplevel primitive to
make the NOW instruction return a chosen timestamp:

   input {};
   now "2020-01-08T07:13:51Z";
   code NOW;
   output { Stack_elt timestamp "2020-01-08T07:13:51Z" }


Syntax of concrete stacks
-------------------------

A concrete stack is written as a Micheline sequence whose elements are
of the form "Stack_elt <ty> <x>" where "<x>" is a Michelson value and
"<ty>" is its type. For example, "{ Stack_elt bool True ; Stack_elt
nat 42 }" is a concrete stack of length 2 whose top element is the
boolean "True" and the bottom element is the natural number "42".


Omitting parts of the output
----------------------------

Any part of the "output" specification can be replaced with the
wildcard pattern "_".

For example, let's consider the following test of the "PAIR"
instruction:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {Stack_elt (pair bool string) (Pair True "foo")}

Omitting the "True" argument to the "Pair" primitive can be done as
follows:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {Stack_elt (pair bool string) (Pair _ "foo")}

Omitting the "Pair" primitive:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {Stack_elt (pair bool string) (_ True "foo")}

Omitting the "pair bool string" type:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {Stack_elt _ (Pair True "foo")}

Omitting the resulting stack element:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output {_}

Omitting all of the output:

   input {Stack_elt bool True; Stack_elt string "foo"};
   code PAIR;
   output _

The difference between the last two examples is that "output {_}"
means that the instruction is expected to successfully return a stack
of length 1 while "output _" means that nothing in particular is
expected from the execution of the instruction, not even being
successful.

The wildcard pattern is typically used to omit unspecified aspects of
the Michelson language when writing portable tests; in particular the
cryptographic nonces in values of type "operation" (see the syntax of
concrete operations) or implementation-specific parts of error outputs
(see the syntax of errors).


Output normalization
--------------------

The input and output stacks can use the readable and optimized formats
for Michelson values and even mix the formats; for a test to pass, the
expected output does not need to syntactically match the result of the
execution but only to match up to conversion between optimized and
readable formats; the TZT test runner is responsible for normalizing
the actual output and the expected one to common format. This means in
particular that conversion between readable and optimized formats can
be tested by using "{}" as the "code" instruction sequence to test;
for example these two tests pass:

   input {Stack_elt address 0x0000e7670f32038107a59a2b9cfefae36ea21f5aa63c};
   code {};
   output {Stack_elt address "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"}

   input {Stack_elt address "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"};
   code {};
   output {Stack_elt address 0x0000e7670f32038107a59a2b9cfefae36ea21f5aa63c}

This normalization feature is however incompatible with using the
wildcard pattern in the output; when using wildcards the output must
be formatted using the readable format so the following test does not
pass:

   input {Stack_elt address "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"};
   code {};
   output {Stack_elt _ 0x0000e7670f32038107a59a2b9cfefae36ea21f5aa63c}

but the following test does pass:

   input {Stack_elt address 0x0000e7670f32038107a59a2b9cfefae36ea21f5aa63c};
   code {};
   output {Stack_elt _ "tz1gjaF81ZRRvdzjobyfVNsAeSC6PScjfQwN"}


Syntax of errors
----------------

To test that the execution of an instruction fails, the following
syntaxes can be used instead of the output stack as the argument of
the "output" toplevel primitive to specify which error the instruction
is expected to raise:

* "(StaticError <error description>)": an error occurred before the
  instruction was executed; the error description format is
  unspecified so consider using a wildcard such as "(StaticError _)"
  to write portable tests;

* "(Failed <value>)": the execution reached a "FAILWITH" instruction
  and the topmost element of the stack at this point was "<value>";

* "MutezUnderflow": a mutez subtraction resulted in a negative value.
  This should only happen in the case of the deprecated "mutez" case
  of the "SUB" instruction;

* "Overflow": an overflow was detected. This can happen when an
  addition or multiplication on type "mutez" produces a result which
  is too large to be represented as a value of type "mutez", or when
  the number of bits to shift using the "LSL" or "LSR" instruction is
  too large.

The following example shows how to test a runtime failure; it asserts
that the FAILWITH instruction produces a runtime error containing the
top of the stack.

   input { Stack_elt nat 4 ; Stack_elt bytes 0x };
   code FAILWITH;
   output (Failed 4)

The following example shows how to test type checking failure; it
asserts that the DUP instruction cannot be used on an empty stack.

   input {};
   code DUP;
   output (StaticError _)

The following example shows another kind of static failure: a string
cannot be passed as argument to the DUP instruction.

   input { Stack_elt nat 8 };
   code { DUP "foo" };
   output (StaticError _)


Syntax of concrete operations
-----------------------------

The operation type has no concrete syntax in Michelson. In order to
specify the result of the operation forging instructions
TRANSFER_TOKENS, CREATE_CONTRACT, and SET_DELEGATE , the following
data constructors are added:

* "Transfer_tokens",

* "Create_contract", and

* "Set_delegate".

They take as arguments the inputs of the corresponding operation
forging instructions plus a cryptographic nonce represented as a byte
sequence. The result of "TRANSFER_TOKENS", "CREATE_CONTRACT", and
"SET_DELEGATE" have respectively the following shapes:

* "Transfer_tokens <argument> <amount in mutez> <address of
  destination> <nonce>",

* "Create_contract { <script> } <optional delegate> <initial balance
  in mutez> <initial storage> <nonce>", and

* "Set_delegate <optional delegate> <nonce>".

The computation of the cryptographic nonce is not specified. To write
portable tests, the nonces appearing in output stack expectations
should be replaced by a wildcard pattern.

Here is an example unit test for the "SET_DELEGATE" instruction used
to set the delegate of the current contract to the account at address
"tz1NwQ6hkenkn6aYYio8VnJvjtb4K1pfeU1Z":

   input { Stack_elt (option key_hash) (Some "tz1NwQ6hkenkn6aYYio8VnJvjtb4K1pfeU1Z") } ;
   code SET_DELEGATE ;
   output { Stack_elt operation (Set_delegate (Some "tz1NwQ6hkenkn6aYYio8VnJvjtb4K1pfeU1Z") _) }


Syntax of other contracts specifications
----------------------------------------

The behaviour of the CONTRACT instruction depends on whether or not
its input is the address of a smart contract accepting the expected
type as parameter. To test it, the "other_contract" toplevel primitive
can be used to specify which contracts are assumed to be originated
and which type they accept as parameter.

The mapping given to the "other_contract" toplevel primitive is a
Micheline sequence whose elements have the form "Contract "KT1..."
<ty>" where ""KT1..."" is a valid smart contract address and "<ty>" is
the type of its parameter. Each address should appear at most once and
the order is irrelevant.


Syntax of extra big maps specifications
---------------------------------------

The behaviour of the instructions operating on type big_map depend on
the contents of big maps stored in the context. To test them, the
"big_maps" toplevel primitive can be used to specify the types and
contents of the big maps which are assumed to be present.

The mapping given to the "big_maps" toplevel primitive is a Micheline
sequence whose elements have the form "Big_map <i> <kty> <vty> { Elt
<k1> <v1>; Elt <k2> <v2>; ...}" where "<i>" is an integer (the
identifier of the big map), "<kty>" is the comparable type of keys,
"<vty>" is the type of values, each "<ki>" is of type "<kty>" and each
"<vi>" is of type "<vty>". Each identifier should appear at most once
and the order in which big maps are specified is irrelevant but each
"{ Elt <k1> <v1>; Elt <k2> <v2>; ...}" description of big map contents
should be given in increasing order of keys.

The following example tests the GET instruction in the big_map case:

   big_maps { Big_map 4 string nat { Elt "bar" 42 } };
   input { Stack_elt (big_map string nat) 4 };
   code { PUSH string "foo"; GET };
   output { Stack_elt (option nat) None }

On-chain Views
**************

Views are a mechanism that allows smart contracts to call smart
contracts, and that:

* are read-only: they may depend on the storage of the contract
  declaring the view but cannot modify it nor emit operations (but
  they can call other views),

* take arguments as input in addition to the contract storage,

* return results as output,

* are synchronous: the result is immediately available on the stack of
  the caller contract.


Homonyms
========

Beware that the term *view* has also been used for other kinds of
mechanisms to access some data from the blockchain.

* Off-chain views cannot be called from smart contracts but only by
  off-chain tools such as wallets.

* Legacy on-chain views are a conventional way of making some contract
  data available. Their purpose is subsumed by that of the on-chain
  views defined here, but they are still referenced in FA1.2 and FA2
  standards which predate the introduction of native on-chain views in
  the Hangzhou amendment.

In the following, the term "view" denotes the modern notion of on-
chain views, described in this page.


Execution flow
==============

The execution of a view is included in the operation of the caller’s
contract, but accesses the storage of the declarer’s contract, in
read-only mode. Thus, in terms of execution, views are more like
lambda functions rather than contract entrypoints, Here is an example:

   code {
     ...;
     TRANSFER_TOKENS;
     ...;
     VIEW "view_ex" unit;
     ...;
   }

This contract calls a contract "TRANSFER_TOKENS", and, later on, a
view called "view_ex". No matter if the callee "view_ex" is defined in
the same contract with this caller contract or not, this view will be
executed immediately in the current operation, while the operations
emitted by "TRANSFER_TOKENS" will be executed later on. As a result,
although it may seem that "view_ex" receives the storage modified by
"TRANSFER_TOKENS", this is not the case. In other words, the storage
of the view is the same as when the execution of the contract calling
the view started. In particular, in case of re-entrance, i.e., if a
contract "A" calls a contract "B" that calls a view on "A", the
storage of the view will be the same as when "B" started, not when "A"
started.


Declaration
===========

Views are **declared** at the toplevel of the script of the contract
on which they operate, alongside the contract parameter type, storage
type, and code. To declare a view, the "view" keyword is used; its
syntax is "view name 'arg 'return { instr; ... }" where:

* "name" is a string of at most 31 characters matching the regular
  expression "[a-zA-Z0-9_.%@]*"; it is used to identify the view,
  hence it must be different from the names of the other views
  declared in the same script;

* "'arg" is the type of the argument of the view;

* "'return" is the type of the result returned by the view;

* "{ instr; ... }" is a sequence of instructions of type "lambda (pair
  'arg 'storage_ty) 'return" where "'storage_ty" is the type of the
  storage of the current contract. Certain specific instructions have
  different semantics in views: "BALANCE" represents the current
  amount of mutez held by the contract where the view is declared;
  "SENDER" represents the contract which is calling the view;
  "SELF_ADDRESS" represents the contract declaring the view; "AMOUNT"
  is always 0 mutez.


Forbidden types
===============

Note that in both view input (type "'arg") and view output (type
"'return"), the following types are forbidden: "ticket", "operation",
"big_map" and "sapling_state".

Time-lock
*********

Time-lock is a cryptographic primitive that can be used as part of a
commit & reveal scheme, to provide a guarantee that the information
associated to the commit phase is eventually revealed.

For more background information on time-locks and their uses, see
Timelocks in the Tezos documentation.


Cryptographic design
====================

The time-lock features are supported in Octez by the
Tezos_crypto.Timelock library .

Users first generate a RSA modulus and a symmetric encryption key.
They use authenticated encryption to encrypt a packed Michelson value
(an array of bytes computed with "PACK") and encrypt that encryption
key using a time-lock puzzle. They then combine the RSA modulus, the
time-locked symmetric key, the constant "T" and the encrypted value as
a single value as well (called "chest" in our library).

A proof of decryption can be the symmetric key itself. However, a
malicious user could propose an authenticated ciphertext that does not
yield a valid value even when decrypted with the symmetric key that
was indeed time locked. To avoid this threat, an opening (called
"chest_key" in our library) includes the symmetric key and a proof
that the symmetric key proposed is indeed the one hidden in the time-
lock puzzle. In this way one can differentiate whether the chest or
the chest_key was proposed by a malicious user.


Opcode and types
================

To expose the features of this library, the Michelson language
provides the following types:

* "chest", which represents time-locked arbitrary bytes with the
  necessary public parameters to open it.

* "chest_key", which represents the decryption key, alongside with a
  proof that the key is correct.

and the following opcode:

"open_chest ::  chest_key → chest → time → bytes option"

"open_chest" takes a "chest" and a "chest_key", and produces either
the underlying plaintext or indicates that the "chest_key" is
malicious. If the "chest" is not well-formed (which is possible since
we use authenticated encryption), the empty Byte is returned.


Implementation of the time-lock puzzle
======================================

The implementation of the time-lock puzzle and proof scheme is located
in src/lib_crypto/timelock.ml.

To facilitate the use of time-locks,  commands have also been
implemented in Octez client to generate a "chest" and "chest_key" as
well as to open and verify them. An additional command "precompute"
was implemented to fasten the time-lock "chest" generation.

For more information on the client commands, see cli-commands.


Example
=======

The built-in coin flip contract  gives an example of using time-lock.
Beware this contract is for educational purpose only and is not
secure.

Sapling support
***************

This page first give some details on Sapling and its underlying
concepts, and then describes the support for Sapling in Octez and in
the Michelson language (part of the Tezos protocol). For support in
high-level smart contract languages, see the Sapling documentation in
OpenTezos.


Sapling
=======

Sapling is a protocol enabling privacy-preserving transactions of
fungible tokens in a decentralised environment. It was designed and
implemented by the Electric Coin Company as the last iteration over a
series of previous protocols and academic works starting with the
Zerocoin seminal paper.


Keys
----

Sapling offers a rich set of keys, each allowing different operations.
A *spending key* allows to spend tokens so if it is lost or
compromised the tokens could remain locked or be stolen. From a
spending key it is possible to derive a corresponding *viewing key*
which allows to view all incoming and outgoing transactions. The
viewing key allows the owner of the tokens to check their balance and
transaction history so if compromised there is a complete loss of
privacy. On the other hand a viewing key can willingly be shared with
a third party, for example with an auditor for regulatory compliance
purposes.

A viewing key can also derive several diversified addresses. An
address can be used to receive funds, much like the address of a user
account.

Additionally *proving keys* can be used to allow the creation of
proofs, thus revealing private information, without being able to
spend funds. They are useful for example in case the spending key is
stored in a hardware wallet but we'd like to use our laptop to craft
the transaction and produce the zero-knowledge proofs, which are
computationally too intensive for an embedded device.

More details can be found in the specification document.


Shielded transactions
---------------------

Transactions use Bitcoin's UTXO model with the important difference
that each input and output, instead of containing an amount and an
address, are just cryptographic *commitments*. In order to avoid
double spends, it's important to be able to check that a commitment
has not already been spent. In Bitcoin we just need to check if an
output is also later used as an input to verify if it's already spent.
In Sapling however we can't know because inputs are not linked to
outputs. For this reason for each input of a transaction, the owner
must also publish a *nullifier*, which invalidates it. The nullifier
can only be produced by the owner of a commitment and it's
deterministic so that everybody can check that it hasn't been already
published. Note however that it is not possible to infer which
commitment has been nullified. Transactions of this form are privacy
preserving and are referred to as *shielded*, because they reveal
neither the amount, the sender nor the receiver.

The existing set of transactions is referred to as the *shielded
pool*. Unlike Bitcoin, where everybody can compute the set of unspent
outputs of every user, in Sapling only the owner of a viewing key can
find their outputs and verify that they are not already spent. For
this reason, to an external observer, the shielded pool is always
increasing in size and the more transactions are added the harder it
is to pinpoint the commitments belonging to a user.

When we spend a commitment there is some additional information that
we need to transmit to the recipient in order for them to spend the
corresponding output. This data is encrypted under a symmetric key
resulting from a Diffie-Hellman key exchange using the recipient
address and an ephemeral key. In principle this *ciphertext* can be
transmitted off-chain as it's not needed to verify the integrity of
the pool. For convenience, in Tezos, it is stored together with the
commitment and the nullifier on chain.

For reasons of efficiency the commitments are stored in an incremental
Merkle tree which allows for compact proofs of membership. The root of
the tree is all that is needed to refer to a certain state of the
shielded pool.

In order to ensure the correctness of a transaction, given that there
is information that we wish to remain secret, the spender must also
generate proofs that various good properties are true. Thanks to the
use of SNARKs these proofs are very succinct in size, fast to verify
and they don't reveal any private information.

This model of transaction adapts elegantly to the case when we need to
mint or burn tokens, which is needed to shield or unshield from a
transparent token. It suffices to add more values in the outputs than
in the inputs to mint and to have more in inputs than outputs to burn.


Preventing Malleability
-----------------------

A Sapling transaction contains a "bound_data" field with arbitrary
bytes that gets signed by the user's Sapling key. This field can be
used to bind the transaction to another application's logic. For
example during an unshield operation it is important to include in the
"bound_data" the Tezos account that will receive the unshielded
tokens. Without any "bound_data", a Sapling unshield operation
authorizes any party that submits it to claim the unshielded tokens.
An adversary can intercept the Tezos operation containing the unshield
and resubmit it using its own Tezos account. This malleability attack
is prevented by including the recipient Tezos account in the
"bound_data", which is signed by the owner of the Sapling keys, and
that can be used by the Smart Contract to transfer the tokens.

**All Smart Contracts managing a Sapling shielded pool must include a
`bound_data` field in their unshield operations.**

Note that the "bound_data" field is not encrypted and could leak
information about a transaction, therefore for a transfer (e.g. an
operation with balance 0) it should be left empty. This is not a
concern for shielding and unshielding which are already linking a
shielded and a transparent address.


Privacy guarantees
------------------

We explained that the shielded pool contains one commitment for each
input (spent or not), and one nullifier for each spent input. These
cryptographic commitments hide the amount and the owner of the tokens
they represent. Additionally commitments are unlinkable meaning that
we can not deduce which input is spent to create an output.

It should be noted that the number of inputs and outputs of a
transaction is public, which could help link a class of transactions.
This problem can be mitigated by adding any number of dummy inputs or
outputs at the cost of wasting some space.

The shielded pool communicates with the public ledger by minting and
burning shielded tokens in exchange for public coins. Therefore going
in and out of the shielded pool is public: we know which address
shielded or unshielded and how much. We can among other things infer
the total number of shielded coins.

Timing and network information can also help to deduce some private
information. For example by observing the gossip network we might
learn the IP address of somebody that is submitting a shielded
transaction. This can be mitigated by using TOR.


Good practices
--------------

When blending in a group of people, one should always pay attention to
the size and the variety of the group.

We recommend two good practices. First, do not originate a second
contract if another one has the same functionalities, it will split
the anonymity set.

Second, remember that shielding and unshielding are public operations.
A typical anti-pattern is to shield from tz1-alice 15.3 tez, and then
unshield 15.3 tez to tz1-bob. It's fairly clear from timing and
amounts that Alice transferred 15.3 tez to Bob. To decorrelate the two
transfers it is important to change the amounts, let some time pass
between the two and perform the transactions when there is traffic in
the pool. Similar problems exist in Zcash and they are illustrated in
this introductory blog post.

There are a number of more sophisticated techniques to deanonymise
users using timing of operations, network monitoring, side-channels on
clients and analysis of number of inputs/outputs just to mention a few
(A fistful of Bitcoins is a good first read). We advice users to be
familiar with the use of the TOR network and to use clients developed
specifically to protect their privacy.


Sapling integration
===================

The reference implementation of Sapling, librustzcash, has been
integrated in the Octez codebase, and Sapling features were made
available for production use with the Edo protocol amendment in early
2021.

Librustzcash and the Octez integration implement the protocol
described in this specification, version 2020.1.0.


Michelson support
-----------------

Sapling support uses two new Michelson types "sapling_state" and
"sapling_transaction", and two instructions called
"SAPLING_VERIFY_UPDATE" and "SAPLING_EMPTY_STATE" (see the Michelson
reference for more details).

"SAPLING_EMPTY_STATE" pushes an empty "sapling_state" on the stack.

"SAPLING_VERIFY_UPDATE" takes a transaction and a state and returns an
option type which is "Some(bound_data, balance and updated state)" if
the transaction is correct, "None" otherwise. A transaction has a list
of inputs, outputs, a balance, the root of the Merkle tree containing
its inputs, some bound data and a signature.

The verification part checks:

* the zero-knowledge proofs of all inputs and outputs of the
  transaction, which guarantee several properties of correctness

* a (randomised) signature associated with each input (which
  guarantees that the owner forged the transaction), and the signature
  that binds the whole transaction together and guarantees the
  correctness of the balance

* that the root of the Merkle tree appears in one of the past states
  and that the nullifiers are not already present (i.e. no double
  spending is happening).

All the signatures are over the hash of the data that we wish to sign
and the hash function used is Blake2-b, prefixed with the anti-replay
string. The anti-replay string is the the concatenation of the chain
id and the smart contract address. The same string has to be used by
the client for signing.

If one of the checks fails, the instruction returns None. Otherwise
the function adds to the new state the nullifiers given with each
input and adds the outputs to the Merkle tree, which will produce a
new root. It should be noted that it is possible to generate
transactions referring to an old root, as long as the inputs used were
present in the Merkle tree with that root and were not spent after. In
particular, the protocol keeps 120 previous roots and guarantees that
roots are updated only once per block. Considering 2 blocks per minute
and that each block contains at least one call to the same contract, a
client has 1 hour to have its transaction accepted before it is
considered invalid.

The nullifiers are stored in a set. The ciphertexts and other relevant
information linked to the commitment of the Merkle tree are stored in
a map indexed by the position of the commitment in the Merkle tree.

Lastly the instruction pushes on the stack an option with the bound
data, the balance, and the updated state.

A smart contract typically shields or unshields tokens if the balance
is positive or negative, and simply updates the state if the balance
is zero. Additionally in case of an unshield, it must use the bound
data to authorize the transfer of unshielded tokens. For example it
could convert the bound data to a public-key hash and use it as
recipient address of a Tezos transfer.


Example contracts
-----------------


Shielded tez
~~~~~~~~~~~~

An example contract implementing a shielded pool of tokens with a 1 to
1 conversion rate to mutez is available in the tests of the protocol
at src/proto_023_PtSeouLo/lib_protocol/test/integration/michelson/con
tracts/sapling_contract.tz.


Simple Voting Contract
~~~~~~~~~~~~~~~~~~~~~~

One might envision using Sapling to implement private voting (i.e., a
secret ballot). It is possible to adapt shielded transactions to
express and quantify preferences. **However, this is not what Sapling
is designed for and it does not provide the same properties as an
actual private voting protocol.**

A naive idea is the following.

Suppose we want a set of users to express a preference for option A or
B. We can generate two Sapling keys with two addresses that are
published and represent the two options. The contract lets each user
create a token which represents one vote that can then be transferred
to address A or B. Using the published viewing keys everyone can check
the outcome of the vote.

However note that:

* A transaction can be replayed and we can see the balance of A or B
  going up.

* This system does not offer ballot privacy.

Therefore, this idea may be usable in some practical situations, but
**we recommend in general not to use it for any important vote.**

Note that using a random elliptic curve element as incoming viewing
key (IVK) allows to generate a dummy address that cannot be spent.
This eases the counting of the votes. To ensure that the IVK does not
correspond to a normal address with spending key, one can use the
Fiat-Shamir heuristic.


Fees issue
----------

The Sapling integration in Tezos exhibits a privacy issue that Z-cash
doesn't have. When interacting with a shielded pool one interacts with
a smart contract via a normal transaction and therefore have to pay
fees from a user account. One could guess that private transactions
whose fees are paid by the same user account are from the same user.
This can be mitigated by making a service that act as a proxy by
forwarding the user transactions and paying its fees. The user would
then include in the transaction a shielded output for the service that
covers the fees plus a small bonus to pay the service. This output can
be open by the service before sending the transaction to check that
there is enough money to cover its fees.

As usually done for mitigating other Z-cash privacy issues, users
interacting with the proxy should use TOR or mitigate network analysis
as they wish.


RPCs
----

There are two Sapling RPCs under the prefix "context/sapling".
"get_size" returns a pair with the size of the set of commitments and
the size of the set of nullifiers. "get_diff" takes two optional
starting offsets "cm_from" and "nf_from" and returns the sapling state
that was added from the offsets to the current size. In particular it
returns three lists:

* commitments,

* ciphertexts from position "cm_from" up to the last one added, and

* nullifiers from "nf_from" to the last one added.

Additionally it returns the last computed root of the Merkle tree so
that a client updating its tree using the diff can verify the
correctness of the result.


Client support
--------------


Wallet
~~~~~~

"octez-client" supports Sapling keys and can send shielded
transactions to smart contracts.

The client supports two ways to generate a new Sapling spending key.
It can be generated from a mnemonic using BIP39, so that it can be
recovered in case of loss using the mnemonic. Alternatively it is
possible to derive new keys from existing ones using ZIP32, a Sapling
variant of BIP32 for hierarchical deterministic wallets. As usual, in
this case it is important to note the derivation path of the key to be
able to recover it in case of loss. At the moment there is no hardware
wallet support, keys are stored in "~/.tezos-client/sapling_keys" by
default encrypted with a password. **Users should take care to backup
this file.**

The client can also derive addresses from viewing keys. By default
addresses are generated using an increasing counter called the address
index. Not all indexes correspond to valid addresses for each key so
it is normal to see an increasing counter that occasionally skips a
few positions.

The client binds each newly generated key to a specific smart contract
address.


Operations
~~~~~~~~~~

The client also facilitates the creation of shielded transactions and
their transfer as arguments of smart contracts. For now there is
seamless integration to send transactions to the reference shielded-
tez contract and there are plans to support a larger class of
contracts.

For the shielded-tez smart contract, the client supports shielding,
unshielding and shielded transactions. In the case of shielded
transactions there are two commands, one to forge a transaction and
save it to file and one to submit it to the smart contract. The idea
is that a user should not use their own transparent tz{1,2,3} address
to submit a shielded address but rather have a third party inject it.


Code base
---------

The current code-base is organized in three main components. There is
a core library called "lib_sapling" which binds "librustzcash", adds
all the data structures necessary to run the Sapling protocol and
includes a simple client and baker. Under the protocol directory there
is a "lib_client_sapling" library which implements a full client
capable of handling Sapling keys and forging transactions. Lastly in
the protocol there is a efficient implementation of the Sapling
storage, in the spirit of "big_map"s, and the integration of
"SAPLING_VERIFY_UPDATE" in the Michelson interpreter.


Protocol
~~~~~~~~

In order to make the Sapling library available to the protocol is has
been exposed through the environment that sandboxes the protocol. The
changes to src/lib_protocol_environment were delivered as part of
version V1 of the environment.

There are two main forms of support in the economic protocol: the
storage for Sapling and the addition of "SAPLING_VERIFY_UPDATE" to the
Michelson interpreter.

Given that the storage of a Sapling contract can be substantially
large, it is important to provide an efficient implementation.
Similarly to what is done for big_maps, the storage of Sapling can't
be entirely deserialized and modified in memory but only a diff of the
changes is kept by the interpreter and applied at the end of each
smart contract call.

The Michelson language offers two special-purpose types:
"sapling_state" and "sapling_transaction", and one instruction:
"SAPLING_VERIFY_UPDATE".


Client
~~~~~~

Under "lib_client_sapling" there is the client integration with the
support for Sapling keys and forging of transactions. The main
difference from the "lib_client" library is the need for the Sapling
client to keep an additional state, for each contract. Because Sapling
uses a UTXO model it is necessary for a client to compute the set of
unspent outputs in order to forge new transactions. Computing this set
requires scanning all the state of a contract which can be expensive.
For this reason the client keeps a local state of the unspent outputs
after the last synchronization and updates it before performing any
Sapling command. The update is done using the RPCs to recover the new
updates since the last known position.

The state of all Sapling contracts is stored in "~/.tezos-
client/sapling_states". This file can be regenerated from the chain in
case of loss. However disclosure of this file will reveal the balance
and the unspent outputs of all viewing keys.


Memo
~~~~

Sapling offers the possibility to add an arbitrary memo to any created
output. The memo is encrypted and available to anyone owning the
outgoing viewing key or the spending key. For privacy reasons the size
of the memo is fixed per contract and it is chosen at origination
time. A transaction containing an output with a different memo-size
will be rejected. Our client adds a default zero-filled message of the
right length. If a message is provided with the "--message" option,
the client will pad it or truncate it if necessary. A warning message
is printed only if the user's message is truncated.


Sandbox tutorial
----------------

Let us show how to test the system end-to-end using the Sandboxed
mode. After having set up the sandbox and originated the contract, a
good way to get familiar with the system is to generate keys and then
perform the full cycle of shielding, shielded transfer and
unshielding.

   # set up the sandbox
   ./src/bin_node/octez-sandboxed-node.sh 1 --connections 0 &
   eval `./src/bin_client/octez-init-sandboxed-client.sh 1`
   octez-activate-alpha

   # originate the contract with its initial empty sapling storage,
   # bake a block to include it.
   # { } represents an empty Sapling state.
   octez-client originate contract shielded-tez transferring 0 from bootstrap1 \
   running src/proto_023_PtSeouLo/lib_protocol/test/integration/michelson/contracts/sapling_contract.tz \
   --init '{ }' --burn-cap 3 &
   octez-client bake for bootstrap1

   # if necessary, you can get information from the octez-client manual
   octez-client sapling man

   # generate two shielded keys for Alice and Bob and use them for the shielded-tez contract
   # the memo size has to be indicated
   octez-client sapling gen key alice
   octez-client sapling use key alice for contract shielded-tez --memo-size 8
   octez-client sapling gen key bob
   octez-client sapling use key bob for contract shielded-tez --memo-size 8

   # generate an address for Alice to receive shielded tokens.
   octez-client sapling gen address alice
   zet1AliceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # Alice's address


   # shield 10 tez from bootstrap1 to alice
   octez-client sapling shield 10 from bootstrap1 to zet1AliceXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX using shielded-tez --burn-cap 2 &
   octez-client bake for bootstrap1
   octez-client sapling get balance for alice in contract shielded-tez

   # generate an address for Bob to receive shielded tokens.
   octez-client sapling gen address bob
   zet1BobXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # Bob's address

   # forge a shielded transaction from alice to bob that is saved to a file
   octez-client sapling forge transaction 10 from alice to zet1BobXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX using shielded-tez

   # submit the shielded transaction from any transparent account
   octez-client sapling submit sapling_transaction from bootstrap2 using shielded-tez --burn-cap 1 &
   octez-client bake for bootstrap1
   octez-client sapling get balance for bob in contract shielded-tez

   # unshield from bob to any transparent account
   octez-client sapling unshield 10 from bob to bootstrap1 using shielded-tez --burn-cap 1
   ctrl+z # to put the process in background
   octez-client bake for bootstrap1
   fg # to put resume the transfer

Liquidity Baking
****************

Liquidity baking incentivizes large amounts of decentralized liquidity
provision between tez and tzBTC by minting a small amount of tez every
block and depositing it inside of a constant product market making
smart-contract.


Contracts
=========

During activation of Granada protocol, a constant product market
making (CPMM) Michelson contract has been deployed on the chain with
address "KT1TxqZ8QtKvLu3V3JH7Gx58n7Co8pgtpQU5" as well as an
associated liquidity token contract (LQT) with address
"KT1AafHA1C1vk959wvHWBispY9Y2f3fxBUUo".

Warning:

  While the CPMM and LQT contract originations provide an
  "Origination_result", the LQT contract contains two big maps not
  included in a "lazy_storage_diff" field. Indexers and other tooling
  may need manual updates to include these.

The CPMM maintains a balance of "a" tez and "b" tzBTC, where tzBTC is
the FA1.2 token  found at address
"KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn". The smart contract accepts
deposits of "da" tez and returns "db" tzBTC (or vice versa) where the
invariant "(a + da * (1 - f - n)) * (b - db) = a b" is preserved, and
"f" and "n" are a fee and burn, set at 0.1% each. Calculations are
done with precision of 1000, rounding down on division.

To implement this contract, we use a fork of the open source code base
used by version two of the "Dexter" project. The implementation of
this contract has been formally verified against its functional
specification. The contract code is modified in the following way:

1. The fee is set to 0.1% only (the fee in Dexter v2 is set to 0.3%).
   Rationale: given the subsidy it is not necessary to charge a large
   fee and better to improve liquidity.

2. An additional 0.1% of every trade is burned by being transferred to
   the null user account. Rationale: this mechanism offsets inflation
   from the subsidy. The inflation is exactly balanced at a daily
   trade volume of 7.2 million tez.

3. The ability to set a delegate has been removed. Rationale: the
   subsidy means there is no need for a baker for that contract and
   having one would create an imbalance.

4. The ability to set a manager has been removed. Rationale: the only
   privilege of the Dexter manager is to set Dexter's delegate so this
   role is now unnecessary.

The LIGO and Michelson code for these contracts, as well as detailed
documentation, can be found on the liquidity baking branch of the
Dexter 2 repository.


Subsidy
=======

At every block in the chain, a small amount of tez is minted and
credited to the CPMM contract, and the CPMM's "%default" entrypoint is
called to update the "xtz_pool" balance in its storage. The amount
that is minted and sent to the CPMM contract for each block is a
protocol constant ("LIQUIDITY_BAKING_SUBSIDY"), defining a fixed
target rate of 5 tez/minute. One can easily compute the value of the
per block subsidy by taking into account the minimal block time, under
the assumption that all blocks are produced at round 0.

So the credits to the CPMM contract can be accounted for by indexers,
they are included in block metadata as a balance update with a new
constructor for "update_origin", "Subsidy".


Toggle vote
===========

The subsidy can be paused by a mechanism called the Liquidity Baking
Toggle Vote. At every block, the baker producing the block includes a
flag that requests ending the subsidy or on the contrary continuing or
restarting it. The context maintains an exponential moving average of
that flag. The baker has three options for this flag: "Off" to request
ending the subsidy, "On" to request continuing or restarting the
subsidy, and "Pass" to abstain.

"e[n+1] = e[n]" if the flag is set to "Pass". "e[n+1] = (1999 * e[n]
// 2000) + 1_000_000" if the flag is set to "Off". "e[n+1] = (1999 *
e[n] // 2000)" if the flag is set to "On". When computing "e[n+1]",
the division is rounded toward "1_000_000_000".

If at any block "e[n] >= LIQUIDITY_BAKING_EMA_THRESHOLD" (currently
"1_000_000_000") then it means that an exponential moving average with
a window size on the order of two thousand non-abstaining blocks has
had roughly at least a half of the blocks demanding the end of the
subsidy. If that is the case, the subsidy is halted but can be
reactivated if for some later block "e[n] <
LIQUIDITY_BAKING_EMA_THRESHOLD".

For indicative purposes, if among the non-abstaining blocks a fraction
"f" of blocks use it to request ending the subsidy, the threshold is
reached after roughly "2*(log(1-1/(2f)) / log(0.999))" non-abstaining
blocks, about 1386 blocks if everyone signals, 1963 blocks if 80% do,
3583 blocks if 60% do etc. Recall for comparison that assuming six
blocks per minute there are 8640 blocks per day.

When producing blocks using Octez baking daemon "octez-baker", there
are two command-line options affecting toggle vote. The "--liquidity-
baking-toggle-vote <on|off|pass>" option sets a static value to be
used in each block. Note that this option must be placed **after**
"run" on the command-line. Moreover, the path of a JSON file can be
given to the "--votefile <path>" option e.g. "octez-baker-<protocol
codename> run with local node ~/.tezos-node alice --liquidity-baking-
toggle-vote on --votefile "per_block_votes.json"", or placed in a
default location: "per_block_votes.json" in the current working
directory **or** in the client data directory (e.g. "~/.tezos-
client/per_block_votes.json"); the former location takes precedence.
The content of the JSON file will be repeatedly submitted on each
baked block, where "per_block_votes.json" contains just
"{"liquidity_baking_toggle_vote": "pass"}" (to abstain),
"{"liquidity_baking_toggle_vote": "off"}" (to request ending the
subsidy), or "{"liquidity_baking_toggle_vote": "on"}" (to request
continuing the subsidy). When the "--votefile" option is present it
takes precedence over "--liquidity-baking-toggle-vote". If the JSON
file is deleted or becomes malformed while the baker is running, the
last valid value read is used. If neither a valid vote file is
provided nor a CLI value given, the baker will fail on the first block
after it was started. See also the baker man page.

Global Constants
****************

The size limit for Michelson contracts is quite small, limited to 60
kilobytes as of Granada protocol. Global constants are a feature added
in Hangzhou protocol that enables the re-use of user-defined Micheline
chunks in Michelson scripts, allowing for larger and more complex
contracts on the chain. It works in the following way:

* Fragments of Michelson code (written in the Micheline format) are
  registered on the chain via a new operation
  "register_global_constant". An example expression might be the
  integer "999" or the lambda expression "{ PUSH int 999; ADD }"

* Included in the receipt of the operation is a hash of the expression
  registered. For example the hash "999" is
  "expruQN5r2umbZVHy6WynYM8f71F8zS4AERz9bugF8UkPBEqrHLuU8".

* Constants can be referenced inside a Michelson script with the new
  primitive "constant". For example, we could write a lambda
  equivalent to the one above like so: "{ PUSH int (constant
  "expruQN5r2umbZVHy6WynYM8f71F8zS4AERz9bugF8UkPBEqrHLuU8"); ADD }"


Global Constant Registration
============================

The new "register_global_constant" operation includes an object with a
single key ""value"", the value of which is the Micheline expression
to be registered.

You can submit this operation conveniently through a new octez-client
command. For example, the command:

   octez-client register global constant "999" from bootstrap1 --burn-cap 0.017

would result in the output:

   Node is bootstrapped.
   Estimated gas: 1440 units (will add 100 for safety)
   Estimated storage: 68 bytes added (will add 20 for safety)
   Operation successfully injected in the node.
   Operation hash is 'onsFknW5iWa6eiTYqAghY4peQZ7JYQUJg5fR8MwAQkMKjXfNqGf'
   NOT waiting for the operation to be included.
   Use command
     octez-client wait for onsFknW5iWa6eiTYqAghY4peQZ7JYQUJg5fR8MwAQkMKjXfNqGf to be included --confirmations 5 --branch BLockGenesisGenesisGenesisGenesisGenesisCCCCCeZiLHU
   and/or an external block explorer to make sure that it has been included.
   This sequence of operations was run:
     Manager signed operations:
       From: tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx
       Fee to the baker: ꜩ0.000385
       Expected counter: 1
       Gas limit: 1540
       Storage limit: 88 bytes
       Balance updates:
         tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx ................ -ꜩ0.000385
         fees(the baker who will include this operation,0) ... +ꜩ0.000385
       Register Global:
         Value: 999
         This global constant registration was successfully applied
         Balance updates:
           tz1KqTpEZ7Yob7QbPE4Hy4Wo8fHG8LhKxZSx ... -ꜩ0.017
         Consumed gas: 1440
         Storage size: 68 bytes
         Global address: expruQN5r2umbZVHy6WynYM8f71F8zS4AERz9bugF8UkPBEqrHLuU8

As you can see, the address of the constant is returned in the
operation receipt in the field "Global address". This address is the
Base58-encode Blake2b hash of the binary serialization of the
registered Micheline expression. This means constants are content-
addressable - given a particular Micheline expression, you can always
calculate its on-chain address and check if it’s registered.

A few points about registering global constants:

* Global constants may contain references to other constants; however,
  any referenced constants must already be registered on the chain. As
  a corollary, you cannot have cyclic references.

* Global constants are not type-checked before registration - any
  valid Micheline expression may be registered. That said, attempting
  to originate a contract that uses a constant in an ill-typed way
  will fail.

* The total depth of the expression registered as a constant (after
  expanding all constant references) may not exceed 10,000.

* The total number of nodes in the Micheline expression being
  registered (after expanding all constant references) may not exceed
  the "max_micheline_node_count" protocol constant. As of Hangzhou
  this is 50,000.

* The total number of bytes in the Micheline expression being
  registered (after expanding all constant references) may not exceed
  the "max_micheline_bytes_limit" protocol constant. As of Hangzhou
  this is 50,000.


Originating a Contract that uses Global Constants
=================================================

A global constant can be referenced in Michelson scripts via the
primitive "constant", which accepts a single string argument, being
the hash of the expression to be referenced at runtime. This primitive
can be used to replace any Micheline node in the bodies of the
"parameter", "storage", "code", or "view" fields of a Michelson
script. For example, we replace every instance of the type "lambda
unit unit" and value 999 with their respective hashes:

   parameter (constant "exprtYirrFwYKm6yKLzJNtYRbq49zedYq16BonRvMzHiwSbUekB9YL");
   storage (big_map int (constant "exprtYirrFwYKm6yKLzJNtYRbq49zedYq16BonRvMzHiwSbUekB9YL"));
   code {
     PUSH int (constant "expruQN5r2umbZVHy6WynYM8f71F8zS4AERz9bugF8UkPBEqrHLuU8");
     # <rest of code>
   };

The full expansion of this contract would be:

   parameter (lambda unit unit);
   storage (big_map int (lambda unit unit));
   code {
     PUSH int 999;
     # <rest of code>
   };

During origination, all constants are expanded recursively. The
operation will fail if the resulting contract is ill-typed. Global
constant expansion consumes gas; thus, the operation may also fail due
to gas exhaustion.


Global Constants at Runtime
===========================

Contracts that use global constants are semantically equivalent to the
contract with all constants expanded.

Note that using the UNPACK operation to deserialize a lambda which
contains a reference to a global constant is not supported. Similarly,
originating a contract which contains a reference to a global constant
using the CREATE_CONTRACT instruction will also fail.

Michelson Anti-Patterns
***********************

Even though Michelson is designed to make it easy to write secure
contracts and difficult to write vulnerable ones, it is still possible
to write buggy contracts that leak data and funds. This is a list of
mistakes that you can make when writing or interacting with contracts
on the Tezos blockchain and alternative ways to write code that avoid
these problems.

This list is not exhaustive and will never be. The following resources
partially complement it:

* https://opentezos.com/smart-contracts/avoiding-flaws/

* https://ligolang.org/docs/tutorials/security/

* https://github.com/InferenceAG/TezosSmartContractDetails

* https://medium.com/protofire-blog/recommendations-to-enhance-
  security-of-tezos-smart-contracts-d14c0e53a6d3


Storing unbounded data
======================

The gas costs for serializing and deserializing a contract storage are
proportional to its size. Contracts allowing arbitrary users to add
data, or allowing authenticated users to add data of unbounded size,
are vulnerable to malicious users increasing the storage size to make
legitimate interactions with the contract consume a lot of gas or even
deadlocking the contract.


Possible issues:
----------------

* Malicious users may increase the storage size by adding large chunks
  of data.

* Even if each account can only store a bounded amount of data, a
  malicious user may create many accounts to bypass the limit.


Alternatives/Solutions:
-----------------------

* Store unbounded data offchain. When some data is not required for
  the execution of a smart contract, the contract does not need to
  store it. Typically, in most cases there is no need to store the
  full contents of an NFT in a smart contract, but rather some
  metadata or pointer. Even when some data is genuinely required, it
  can often be replaced in storage by its hash and revealed by the
  user when it is required for the contract execution.

* Store user data in big maps. Since big map are lazy data structures,
  instead of deserializing the full content of a stored big map before
  the execution, deserialization happens during the execution and only
  for the accessed keys. The size of the part of a big map which is
  not accessed during the execution has no impact on the gas costs of
  the execution (with two exceptions: transferring or deleting a
  complete big map containing tickets leads to an update of the ticket
  table which is linear in the number of keys). By storing user data
  under a big map whose keys are linked to the user addresses having
  added each key, it is possible to guarantee that the gas costs for
  legitimate users is not impacted by the interactions of malicious
  users.


Refunding to a list of contracts
================================

One common pattern in contracts is to refund a group of people’s funds
at once. This is problematic if you accepted arbitrary contracts as a
malicious user can do cause various issues for you.


Possible issues:
----------------

* One contract swallows all the gas through a series of callbacks

* One contract writes transactions until the block is full

* Reentrancy bugs. Michelson intentionally makes these difficult to
  write, but it is still possible if you try.

* A contract calls the `FAIL` instruction, stopping all computation.


Alternatives/Solutions:
-----------------------

* Create a default account from people’s keys. Default accounts cannot
  execute code, avoiding the bugs above. Have people submit keys
  rather than contracts.

* Have people pull their funds individually. Each user can break their
  own withdrawal only. **This does not protect against reentrancy
  bugs.**


Avoid batch operations when users can increase the size of the batch
====================================================================

Contracts that rely on linear or super-linear operations are
vulnerable to malicious users supplying values until the contract
cannot finish without running into fuel limits. This can deadlock your
contract.


Possible issues:
----------------

* Malicious users can force your contract into a pathological worst
  case, stopping it from finishing with available gas. Note that in
  the absence of hard gas limits, this can still be disabling as node
  operators may not want to run contracts that take more than a
  certain amount of gas.

* You may hit the slow case of an amortized algorithm or data
  structure at an inopportune time, using up all of your contract’s
  available gas.


Alternatives/Solutions:
-----------------------

* Avoid data structures and algorithms that rely on amortized
  operations, especially when users may add data.

* Restrict the amount of data your contract can store to a level that
  will not overwhelm the available gas.

* Write your contract so that it may pause and resume batch
  operations. This would complicate these sequences and require
  constant checking of available gas, but it prevents various attacks.

*Do not assume an attack will be prohibitively expensive*
Cryptocurrencies have extreme price fluctuations frequently and an
extremely motivated attacker may decide that an enormous expense is
justified. Remember, an attack that disables a contract is not just
targeted at the authors, but also the users of that contract.


Signatures alone do not prevent replay attacks
==============================================

If your contract uses signatures to authenticate messages, beware of
replay attacks. If a user ever signs a piece of data, you *must* make
sure that that piece of data is never again a valid message to the
contract. If you do not do this, anyone else can call your contract
with the same input and piggyback on the earlier approval.


Possible issues:
----------------

* A previously approved action can be replayed.


Alternatives/Solutions
----------------------

* Use an internal counter to make the data you ask users to sign
  unique. This counter should be per key so that users can find out
  what they need to approve. This should be paired with a signed hash
  of your contract to prevent cross-contract replays.

* Use the "SENDER" instruction to verify that the expected sender is
  the source of the message.


Do not assume users will use a unique key for every smart contract
------------------------------------------------------------------

Users should always use a different key for every contract with which
they interact. If this is not the case, a message the user signed for
another contract can be sent to your contract. An internal counter
alone does not protect against this attack. It *must* be paired with a
hash of your contract. You must verify the source of the message.


Storing/transferring private data
=================================

Once data is published to anyone, including broadcasting a
transaction, that data is public. Never transmit secret information
via any part of the blockchain ecosystem. As soon as you have
broadcast a transaction including that piece of information, anyone
can see it. Furthermore, malicious nodes in the system can manipulate
unsigned transactions by delaying, modifying, or reordering them.


Possible Issues
---------------

* If data is not signed, it can be modified

* Transactions can be delayed

* Secret information will become public


Alternatives/Solutions
----------------------

* Do not store private information on the blockchain or broadcast it
  in transactions.

* Sign all transactions that contain information that, if manipulated,
  could be abused.

* Use counters to enforce transaction orders.

This will at least create a logical clock on messages sent to your
contract.


Not setting all state before a transfer
=======================================

Reentrancy is a potential issue on the blockchain. When a contract
makes a transfer to another contract, that contract can execute its
own code, and can make arbitrary further transfers, including back to
the original contract. If state has not been updated before the
transfer is made, a contract can call back in and execute actions
based on old state.


Possible Issues
---------------

* Multiple withdrawals/actions

* Generating illegal state if state is updated twice later


Alternatives/Solutions
----------------------

* Forbid reentrancy by means of a flag in your storage, unless you
  have a good reason to allow users to reenter your contract, this is
  likely the best option.

* Only make transfers to trusted contracts or default accounts.
  Default accounts cannot execute code, so it is always safe to
  transfer to them. Before trusting a contract, make sure that its
  behavior cannot be modified and that you have an extremely high
  degree of confidence in it.

Token transfers and balance updates
***********************************

This page describes the reporting of tokens transfers in block
metadata, as a sequence of balance updates. They serve among others to
the Octez client to derive and print the "Balance updates" sections of
the receipts reported when operations are included in blocks.


Overview
========

Minting, transferring or burning tokens is handled by the Token
module. The module provides functions ("Token.transfer" and
"Token.transfer_n") to transfer tokens from one, respectively from
more accounts, to another account. Balance updates found in block
metadata are generated by these functions as a trace of the token
movements having taken place.

Balance updates indicate that either: tokens have been minted and
deposited into an account, transferred from or to another account, or
taken from an account and burned. In the Json format, a balance update
consists of three parts:

* an account identification indicated by a combination of fields such
  as: "kind", "category", "contract", ...

* the amount transferred (in mutez) indicated by the field "change". A
  positive amount indicates that the account has been credited, and a
  negative amount indicates that the account has been debited.

* the cause of the update given by the field "origin" which may have
  the following values:

  * ""block"" means that the balance update originates from the
    application of a block

  * ""migration"" means that the balance update originates from
    migration

  * ""subsidy"" means that the balance update originates from
    subsidies for liquidity baking

  * ""delayed_operation"" means that the balance update originates
    from the delayed application of the operation whose hash is given
    in the additional field ""delayed_operation_hash"".

A transfer of tokens is represented by a continuous and ordered
sequence of (balance) updates. That sequence starts with a series of
debits, and ends with a credit matching those debits. In block
metadata, the field ""balance updates"" contains one or more
transfers, represented by their sequences of updates, concatenated in
a flat list. Consider for example the following list:

   [ {"kind": "...", ..., "change": "-100", "origin": "block"},
     {"kind": "...", ..., "change": "100", "origin": "block"},
     {"kind": "...", ..., "change": "-125", "origin": "block"},
     {"kind": "...", ..., "change": "-75", "origin": "block"},
     {"kind": "...", ..., "change": "200", "origin": "block"} ]

This list reports that two transfers have occurred: a transfer of
"100" mutez from one account to another, and a transfer of "200" mutez
from two accounts to a third. There is one exception to this though:
migration balance updates are a bit compressed to save space, so the
list of balance updates in the corresponding metadata does not
necessarily allow to correlate debits to a credit by analyzing the
sequence. However, for those balance updates, the "delegate" and
"contract" fields can be used to establish correlations between debits
and credits.


Source, container, and sink accounts
====================================

There are three kinds of accounts: source accounts, container
accounts, and sink accounts. Tokens can be transferred from source or
container accounts, to container or sink accounts. All balance updates
contain the field "kind" which allows to determine the kind of account
it refers to. The possible values of this field are described in the
following sections (they are not simply ""source"", ""container"", and
""sink""). Depending on the kind of account, more fields such as the
"category" field may be used to identify accounts more specifically.


Source accounts
---------------

Source accounts are debited whenever new tokens are minted. A balance
update refers to a source account if and only if the field "kind" has
the value ""minted"". The value of the additional field "category"
designates one of the following fictitious accounts, each containing a
virtually unlimited number of tokens:

* ""nonce revelation rewards"" is the source of tokens minted to
  reward delegates for revealing their nonces

* ""double signing evidence rewards"" is the source of tokens minted
  to reward delegates for injecting a double signing evidence

* ""attesting rewards"" is the source of tokens minted to reward
  delegates for attesting blocks

* ""baking rewards"" is the source of tokens minted to reward
  delegates for creating blocks

* ""baking bonuses"" is the source of tokens minted to reward
  delegates for validating blocks and including extra attestations

* ""DAL attesting rewards"" is the source of tokens minted to reward
  delegates for attesting DAL slots

* ""subsidy"" is the source of tokens minted to subsidize the
  liquidity baking CPMM contract

* ""invoice"" is the source of tokens minted to compensate some users
  who have contributed to the betterment of the chain

* ""commitment"" is the source of tokens minted to match commitments
  made by some users to supply funds for the chain

* ""Sc_rollup_refutation_rewards"" is the source of tokens minted to
  reward an account for winning a smart-contract rollup refutation
  game

* ""bootstrap"" is analogous to ""commitment"" but is for internal use
  or testing. It will not be used during normal operation on mainnet,
  but may be used on test networks or in sandboxed mode

* ""minted"" is only for internal use and may be used to mint tokens
  for testing. It will not be used during normal operation on mainnet,
  but may appear on test networks or in sandboxed mode.


Container accounts
------------------

Container accounts are regular (user and smart contract) accounts, or
convenience accounts that hold tokens temporarily (e.g. when parts of
a delegate's funds are frozen). The field "kind" allows to identify
the type of container account, it can have one of the following
values:

* ""contract"" represents a user account or smart contract, and comes
  with the additional field (also called) "contract" whose value is
  the public key hash of the user account or the contract hash of the
  smart contract.

* ""freezer"" represents frozen accounts, and comes with the
  additional field "category" that can have one of the following
  values:

  * ""deposits"" represents the accounts of frozen deposits. Accounts
    in this category are further identified by the additional field
    ""staker"" whose value is either

    * a ""contract"" field, that contains the public key hash of the
      staker owning the funds and a ""delegate"" (public key hash)
      which gets staking power from the deposit,

    * just a ""delegate"" to designate collectively the deposits of
      all stakers delegating to the provided user account.

    * a ""baker_own_stake"" field to designate the delegate's own
      deposits received from its own stake rewards.

    * a ""baker_edge"" field to designate the delegate's own deposits
      received from the edge on its staker's rewards.

  * ""unstaked_deposits"" represents the accounts of unstaked frozen
    tokens. Accounts in this category are further identified by the
    following additional fields:

    * the field ""staker""  whose value is either

      * a ""contract"" field, that contains the public key hash of the
        staker owning the funds and a ""delegate"" (public key hash)
        which gets staking power from the deposit,

      * just a ""delegate"" to designate collectively the deposits of
        all stakers and the delegate itself.

    * the field ""cycle"" contains either the cycle at which the funds
      have been unstaked or the last unslashable cycle
      ("UNSTAKE_FINALIZATION_DELAY + SLASHING_DELAY + 1" before
      current cycle) if it is greater than the unstaking cycle.

  * ""bonds"" represents the accounts of frozen bonds. Bonds are like
    deposits. However, they can be associated to user accounts or
    smart contracts, unlike deposits that only apply to user accounts
    that are also delegates. Accounts in this category are further
    identified by the following additional fields:

    * the field "contract" contains the public key hash of the user
      account, or the contract hash of the smart contract

    * the field "bond_id" contains the identifier of the bond (e.g. a
      rollup hash if the bond is associated to a transaction or a
      smart contract rollup).

* ""accumulator"" represents accounts used to store tokens for some
  short period of time. This type of account is further identified by
  the additional field "category" whose (only possible) value ""block
  fees"" designates the container account used to collect manager
  operation fees while block's operations are being applied. Other
  categories may be added in the future.

* ""commitment"" represents the accounts of commitments awaiting
  activation. This type of account is further identified by the
  additional field "committer" whose value is the encrypted public key
  hash of the user who has committed to provide funds.

* ""staking"" represents abstractions used for accounting staking by
  delegators, and comes with the additional field "category" that can
  have one of the following values:

  * ""delegator numerator"" abstracts the delegator's stake, and comes
    with the additional field ""delegator"" whose value is the public
    key hash of the delegator.

  * ""delegate denominator"" abstracts the total stake of delegate's
    delegators, and comes with the additional field ""delegate"" whose
    value is the public key hash of the delegate.


Sink accounts
-------------

Sink accounts are credited whenever tokens are burned. A balance
update refers to a sink account if and only if the field "kind" has
the value ""burned"". The value of the additional field "category"
allows to identify more specifically a fictitious account able to
receive a virtually unlimited number of tokens. The field "category"
of a sink account may have one of the following values:

* ""storage fees"" is the destination of storage fees burned for
  consuming storage space on the chain

* ""punishments"" is the destination of tokens burned as punishment
  for a delegate that has double baked or double attested

* ""lost attesting rewards"" is the destination of rewards that were
  not distributed to a delegate. This category comes with the
  following additional fields:

  * the field "delegate" contains the public key hash of the delegate

  * the field "participation" has the value ""true"" if participation
    was not sufficient and has the value ""false"" otherwise

  * the field "revelation" has the value ""true"" if the delegate has
    not revealed his nonce and has the value ""false"" otherwise.

* ""lost DAL attesting rewards"" is the destination of DAL rewards
  that were not distributed to a delegate. This category comes with
  the additional field "delegate" containing the public key hash of
  the delegate.

* ""Sc_rollup_refutation_punishments"" is the destination of tokens
  burned as punishment for submitting bad commitments that have been
  refuted

* ""burned"" is only for internal use and testing. It will not appear
  on mainnet, but may appear on test networks or in sandboxed mode.


Token transfers and metadata
============================

Balance updates in block metadata give a complete account of all token
transfers that have occurred when a block is applied. A few cases of
token transfers and the associated metadata are illustrated below. All
other cases of token transfers in the protocol follow the same
pattern. The only differences are the accounts involved.


Origination and transaction
---------------------------

When an origination or transaction operation is applied, tokens are
transferred from one contract to another. Depending on whether or not
storage space has been allocated on the chain by the application of
the operation, storage fees may also be burned. For example, a
transaction of "100" mutez from address "tz1a..." to address "KT1b..."
that allocates storage space for a cost of "10" mutez produces the
following list of balance updates:

   [ {"kind": "contract", "contract": "tz1a...", "change": "-100", "origin": "block"},
    {"kind": "contract", "contract": "KT1b...", "change": "100", "origin": "block"}
    {"kind": "contract", "contract": "tz1a...", "change": "-10", "origin": "block"}
    {"kind": "burned", "category": "storage fees", "change": "10", "origin": "block"} ]


Baking fees, rewards and bonuses
--------------------------------

When a contract pays the baking fees associated to an operation it has
emitted, those fees are temporarily collected (during the processing
of the block) into the container account ""block fees"". For example,
when a manager operation is applied, the account of the payer contract
is debited with the amount of fees and the ""block fees"" account is
credited with the same amount. Hence, for "100" mutez in fees, the
following balance updates are generated :

   [ {"kind": "contract", "contract": "tz1x...", "change": "-100", ...},
     {"kind": "accumulator", "category": "block fees", "change": "100", ...} ]

When all operations of a block have been applied baking fees rewards
and bonuses are distributed. The total amount of fees collected and
the baking rewards are transferred from the container account ""block
fees"" and the source account ""baking rewards"", respectively, to the
contract of the payload producer that selected the transactions to be
included in the block. So, for a total amount of "1000" mutez in fees
collected and an amount of "500" mutez in baking rewards, assuming
that the staking parameter of the delegate are such that 50 mutez are
frozen -- with 5 mutez being the delegates edge, 10 mutez being the
delegates share, and 35 mutez going to the stakers -- and 450 are
spendable, the following balance updates are generated:

   [ {"kind": "accumulator", "category": "block fees", "change": "-1000", ...},
     {"kind": "contract", "contract": "tz1a...", "change": "1000", ...}
     {"kind": "minted", "category": "baking rewards", "change": "-5", ...},
     {"kind": "freezer", "category": "deposits", "staker": { "baker_edge": "tz1a..."}, "change": "5", ...},
     {"kind": "minted", "category": "baking rewards", "change": "-10", ...},
     {"kind": "freezer", "category": "deposits", "staker": { "baker_own_stake": "tz1a..."}, "change": "10", ...},
     {"kind": "minted", "category": "baking rewards", "change": "-35", ...},
     {"kind": "freezer", "category": "deposits", "staker": { "delegate": "tz1a..."}, "change": "35", ...},
     {"kind": "minted", "category": "baking rewards", "change": "-450", ...},
     {"kind": "contract", "contract": "tz1a...", "change": "450", ...} ]

The baking bonus go to the block proposer that signed and injected the
block. Hence the amount of the bonus is transferred from the source
account ""baking bonuses"" to the contract of the block producer
and/or to its frozen balance. For example, the balance updates
generated for an amount of "100" mutez in baking bonus with 90% sent
to spendable balance and 10% to bakers frozen deposit (case with no
stakers and mainnet ratios) are:

   [ {"kind": "minted", "category": "baking bonus", "change": "-90", ...},
     {"kind": "contract", "contract": "tz1b...", "change": "90", ...},
     {"kind": "minted", "category": "baking bonus", "change": "-10", ...},
     {"kind": "freezer", "category": "deposits", "staker": { "baker_own_stake": "tz1b..."}, "change": "10", ...}]


Attesting, double signing evidence, and nonce revelation rewards
----------------------------------------------------------------

Attesting rewards are reflected in balance updates as a transfer of
tokens from the ""attesting rewards"" source account to the account of
the delegate that receives the reward. Hence, for a reward of "100"
mutez,  the following two balance updates are generated:

   [ {"kind": "minted", "category": "attesting rewards", "change": "-100", ...},
     {"kind": "contract", "contract": "tz1...", "change": "100", ...} ]

When attesting rewards are not distributed to the delegate due to
insufficient participation or for not revealing nonces, they are
transferred instead to the sink account identified by the quadruple
"("lost attesting rewards", delegate, participation, revelation)". For
example, for an amount of "100" mutez in rewards not distributed due
to insufficient participation, the following balance updates are
generated:

   [ {"kind": "minted", "category": "attesting rewards", "change": "-100", ...},
     {"kind": "burned",
      "category": "lost attesting rewards",
      "delegate": "tz1...",
      "participation": "true",
      "revelation": "false",
      "change": "100", ...} ]

DAL attesting rewards are analogous to attesting rewards, except that
the source account used is "DAL attesting rewards".

Depending on the staking parameters set by the delegate, some portion
of the (DAL) attesting rewards will go to the freezer container, as
for baking rewards and bonuses.

Double signing evidence rewards and nonce revelation rewards are
analogous to attesting rewards, except that the source accounts used
are ""double signing evidence rewards"" and ""nonce revelation
rewards"".

Mempool
*******

The economic protocol provides a protocol-side mempool module data
structure intended for use by the shell prevalidator (see The
Prevalidator) and by "octez-baker" to incrementally accumulate
operations that can be safely used to bake a new block.

It ensures that :

* Every operation contained are valid;

* every operation are co-valid: they can be safely included in a block
  in any arbitrary order, meaning operations commute;

* the merging of two mempools also maintains the aforementioned
  properties.

The protocol leverages the partial construction mode to incrementally
validate new operations while maintaining the aforementioned
invariants.

During validation, operations are never actually applied, as it is
unnecessary for asserting their validity.


Merging Mempools
================

The protocol-side mempool data structure is designed to allow the
merging of two mempools. This is useful in practice when injecting an
external mempool into the current one. The design also considers the
future possibility of parallelizing operations validation and then
merging mempools in a divide-and-conquer style, an approach that could
be implemented later to accelerate validation.


Conflict
========

Of course, two mempools containing conflicting operations cannot be
merged while maintaining the desired invariants. Similarly, if an
injected operation conflicts with existing ones, the properties would
not hold. To solve conflicts, the structure requires a
"conflict_handler", a function called upon encountering conflicts to
determine which operation remains and which is discarded.

Smart Optimistic Rollups
************************

A **rollup** is a processing unit that receives, retrieves and
interprets input messages to update its local state and to produce
output messages targetting the Tezos blockchain. In this
documentation, we will generally refer to the rollup under
consideration as the Layer 2 on top of the Tezos blockchain,
considered as the Layer 1.


Overview
========

Smart rollups are a permissionless scaling solution for the Tezos
blockchain.  Indeed, anyone can originate and operate one or more
rollups, allowing to increase the throughput of the Tezos blockchain,
(almost) arbitrarily.

The purpose of this documentation is to provide an overview of the
terminology and basic principles of smart rollups. In the Smart rollup
node, we provide a complete tour of smart rollups related workflows
and reference documentation for the development of a WASM kernel.

The integration of these rollups in the Tezos protocol is
*optimistic*: this means that when a participant publishes a claim
about the state of the rollup, this claim is *a priori* trusted.
However, a refutation mechanism allows anyone to economically punish
someone who has published an invalid claim. Therefore, thanks to the
refutation mechanism, a single honest participant is enough to
guarantee that the input messages are correctly interpreted.


Rollup kernel
=============

In the Tezos protocol, the subsystem of smart rollups is generic with
respect to the syntax and the semantics of the input messages. More
precisely, the originator of a smart rollup provides a program named a
**kernel** (in one of the languages supported by Tezos) responsible
for interpreting input messages. During the refutation mechanism, the
execution of this kernel is handled by a **Proof-generating Virtual
Machine (PVM)** for this language, provided by the Tezos protocol,
which allows to prove that the result of applying an input message to
the rollup context is correct. The rest of the time, any VM
implementation of the chosen language can be used to run the smart
rollup kernel, provided that it is compliant with the PVM.

The smart rollup infrastructure currently supports the WebAssembly
language. A WASM rollup runs a kernel expressed in WASM. The role of
the kernel is to process input messages, to update a state, and to
output messages targeting the Layer 1 following a user-defined logic.
Anyone can develop a kernel or reuse existing kernels. A typical use
case of WASM rollups is to deploy a kernel that implements the
Ethereum Virtual Machine (EVM) and to get as a result an EVM-
compatible Layer 2 running on top of the Tezos blockchain. WASM
rollups are not limited to this use case though: they are fully
programmable, hence their names, smart optimistic rollups, as they are
very close to smart contracts in terms of expressiveness.


User roles
==========

Just like smart contracts, smart rollups are decentralized software
components. However, contrary to smart contracts that are processed by
the network validators automatically, a smart rollup requires a
dedicated *rollup node* to function.

Any user can originate, operate, and interact with a rollup. For the
sake of clarity, we will distinguish three kinds of users in this
documentation: operators, kernel developers, and end-users. An
operator deploys the rollup node to make the rollup progress. A kernel
developer writes a kernel to be executed within a rollup. An end-user
interacts with the rollup through Layer 1 operations or Layer 2 input
messages.


Address
=======

When a Smart Rollup is originated on the Layer 1, a unique address is
generated to uniquely identify it. A smart rollup address starts with
the prefix "sr1" (see also Accounts and addresses).


Inputs
======

There are two channels of communication to interact with smart
rollups:

1. a global **rollups inbox** allows the Layer 1 to transmit
   information to all the rollups.

2. a **reveal data channel** allows each rollup to retrieve data
   coming from data sources external to the Layer 1. Rollups request
   data through that channel to the runner of that rollup kernel (i.e.
   the smart rollup node).


Rollups inbox
-------------

A single global inbox serves all rollups and contains two kinds of
messages: *external* messages are pushed through a Layer 1 manager
operation while *internal* messages are pushed by Layer 1 smart
contracts or by the protocol itself. All messages (external and
internal) pushed to the inbox also contain the Layer 1 level of their
insertion and a counter. The counter is the index of the message and
it is reset at each Layer 1 level.


External messages
~~~~~~~~~~~~~~~~~

Anyone can push a message to the rollups inbox. This message is a mere
sequence of bytes following no particular underlying format. The
interpretation of this sequence of bytes is the responsibility of each
kernel.

There are two ways for end-users to push an external message to the
rollups inbox: first, they can inject the dedicated Layer 1 operation
using the Octez client (see command "send smart rollup message
<messages> from <src>"); second, they can use the batcher of a smart
rollup node. More details can be found in the Sending an external
inbox message.


Internal messages
~~~~~~~~~~~~~~~~~

Contrary to external messages, which are submitted by the end users,
internal messages are constructed by the Layer 1.

At the beginning of every Tezos block, the Layer 1 pushes two internal
messages: “Start of level”, and “Info per level”. “Start of level”
does not have any payload associated to it, while “Info per level”
provides to the kernel the timestamp and block hash of the predecessor
of the current Tezos block. If the Tezos block is the first block of a
protocol, then the Layer 1 pushes another message “Protocol migration”
just after the “Info per level” that provides the new protocol version
(i.e. "<proto-name>_<NNN>").

A rollup is identified by an address and has an associated Michelson
type (defined at origination time). Any Layer 1 smart contract can
perform a transfer to this address with a payload of this type. This
transfer is realized as an internal message pushed to the rollups
inbox.

Finally, after the application of the operations of the Tezos block,
the Layer 1 pushes one final internal message “End of level”.
Similarly to “Start of level“, this internal message does not come
with any payload.


Reveal data channel
-------------------

The reveal data channel is a communication interface that allows the
rollup to request data from sources that are external to the inbox and
can be unknown to the Layer 1. The rollup node has the responsibility
to answer the rollup requests.

A rollup can do the following requests through the reveal data
channel:

1. **preimage requests**: The rollup can request arbitrary data of at
   most 4kBytes, provided that it knows its (blake2b) hash. The
   request is fulfilled by the rollup node, see Populating the reveal
   channel.

2. **metadata requests** The rollup can request information from the
   protocol, namely the address and the origination level of the
   rollup itself. The rollup node retrieves this information through
   RPCs to answer the rollup.

Information passing through the reveal data channel does not have to
be considered by the Layer 1: for this reason, the volume of
information is not limited by the bandwidth of the Layer 1. Thus, the
reveal data channel can be used to upload large volumes of data to the
rollup.


Origination
===========

A smart rollup is characterized by: - the kind of Proof-generating
Virtual Machine (PVM), - the kernel written in a language that the PVM
can interpret, - the Michelson type of the entrypoint used by Layer 1
smart contracts to send internal messages to it, and - an optional
list of addresses used as a white-list of allowed committers (see
Private rollups).

All these characteristics are provided when originating a new smart
rollup.


Processing
==========

Each time a Tezos block is finalized, a rollup reacts to three kinds
of events: the beginning of the block, the input messages possibly
contained in that block, and the end of the block. A **rollup node**
implements this reactive process: it downloads the Tezos block and
interprets it according to the semantics of the PVM. This
interpretation can require updating a state, downloading data from
other sources, or performing some cryptographic verifications. The
state of the rollup contains an **outbox**, which is a sequence of
latent calls to Layer 1 contracts.

The behavior of the rollup node is deterministic and fully specified
by a reference implementation of the PVM embedded in the protocol.
Notice that the PVM implementation is meant for verification, not
performance: for this reason, a rollup node does not normally run a
PVM to process inputs but a **fast execution engine** (e.g., based on
the Wasmer runtime for the WASM PVM in the case of the rollup node
distributed with Octez). This fast execution engine implements the
exact same semantics as the PVM. The PVM is only ever used by the
rollup node when it needs to produce a proof during the last step of
the refutation mechanism.


Commitments
===========

Starting from the rollup origination level, levels are partitioned
into **commitment periods** of a number of consecutive blocks
corresponding to about 15 minutes (currently 112 blocks).

A **commitment** claims that the interpretation of all inbox messages
published during a given commitment period, and applied on the state
of a parent commitment, led to a given new state by performing a given
number of execution steps of the PVM. Execution steps are called
**ticks** in Smart Rollups terminology.

A commitment must be published on the Layer 1 any time after each
commitment period, to have the rollup progress. A new commitment
period starts right after the previous commitment period, no matter if
commitments were published or not for the previous commitment
period(s). For example, if an operator rollup node stops running for
one day long, when it comes back, it will be able to resume publishing
commitments for the passed periods, in chronological order. Indeed, a
commitment is always based on a parent commitment (except for the
genesis commitment that is automatically published at origination
time), so publishing a commitment fails if the parent commitment has
not yet been published.

Since the PVM is deterministic and the inputs are completely
determined by the Layer 1 rollups inbox and the reveal channel, there
is only one honest commitment. In other words, if two distinct
commitments are published for the same commitment period, one of them
must be wrong.

An operator publishing a commitment is called a **committer**. Notice
that, in order to publish a commitment, the operator must freeze a
deposit of 10,000 tez, called **frozen bonds**. For this reason, the
committer is sometimes called a (smart rollup) *staker*. However, in
order to avoid confusion with the staker role in Tezos Layer 1's
Proof-of-Stake mechanism, we prefer to use the term "committer"
throughout this documentation.

Several committers can publish (and thus stake on) the same
commitment. When a committer *C* publishes a new commitment based on a
commitment that *C* has published, *C* does not have to provide a new
deposit: the deposit also applies to this new commitment.

There is no need to synchronize between operators: if two honest
operators publish the same commitment for a given commitment period,
the commitment will be published with two stakes on it.

A commitment is optimistically trusted but it can be refuted until it
is said to be **cemented** (i.e., final, unchangeable). Indeed, right
after a commitment is published, a two-week refutation period starts.
During the refutation period, anyone noticing that a commitment for a
given commitment period is invalid can post a concurrent commitment
for the same commitment period to force the removal of the invalid
commitment. If no one posts such a concurrent commitment during the
refutation period, the commitment can be cemented with a dedicated
operation injected in Layer 1, and the outbox messages can be executed
by the Layer 1 by an explicit Layer 1 operation (see Smart rollup
node), typically to transfer assets from the rollup to the Layer 1.

The outbox messages can follow three different formats. Firstly, the
Layer 1 operations contained in the outbox messages can be left
untyped, meaning only the Micheline expression is provided by the
kernel. Before executing the transaction, the Layer 1 typechecks said
expression against the expected type of the targeted entrypoint. Since
Nairobi, it is also possible for the kernel to provide its expected
type of the targeted entrypoint. This additional safety mechanism is
to avoid type confusion: namely, a kernel transferring a tuple that
the Layer 1 interprets as a ticket. Lastly, the outbox message can
contain a white-list update. This message can only be executed for a
rollup that is private since its origination (see Private rollups).


Bonds
=====

The committer's bond of 10,000 tez is frozen automatically when the
committer makes their first commitment. The bond remains frozen until
the committer is no longer staking on any uncemented commitment.

Because nodes have the length of the refutation period to challenge
another node's commitment, the bond stays locked until the end of the
refutation period for the last commitment that the node posted.
Recovering the bond safely takes a few steps; in general, node
operators follow these steps:

1. Switch the rollup node to "bailout" mode, which defends existing
   commitments but does not post new commitments.

2. Wait until the last commitment is cemented. If operators shut down
   the node before the last commitment is cemented, they risk losing
   their bond if another node challenges their commitments.

3. Recover the bond by running the "recover bond" operation, which
   unlocks their tez. Nodes running in "bailout" mode attempt to run
   this command automatically when the last commitment is cemented.
   For more information, see Deploying a rollup node.


Refutation
==========

Because of concurrent commitments, a rollup is generally related to a
**commitment tree** where branches correspond to different claims
about the rollup state.

By construction, only one view of the rollup state is valid (as the
PVM is deterministic). When two concurrent branches exist in the
commitment tree, the cementation process is stopped at the first fork
in the tree. To unfreeze the cementation process, a **refutation
game** must be started between *two concurrent committers* of these
branches. Refutation games are automatically played by rollup nodes to
defend their stakes: honest participants are guaranteed to win these
games. Therefore, an honest participant should not have to worry about
refutation games. Finally, a running refutation game does not prevent
new commitments to be published on top of the disputed commitments.

A refutation game is decomposed into two main steps: a dissection
mechanism and a final conflict resolution phase. During the first
phase, the two committers exchange hashes about intermediate states of
the rollups in a way that allows them to converge to the very first
tick on which they disagree. The exact number of hashes exchanged at a
given step is PVM-dependent. During the final phase, the committers
must provide a proof that they correctly interpreted this conflicting
tick.

The Layer 1 PVM then determines whether these proofs are valid. There
are only two possible outcomes: either one of the committers, that we
dub *C* in the sequel, has provided a valid proof, then *C* wins the
game, and is rewarded with half of the opponent's deposit (the other
half being burnt); or, both committers have provided an invalid proof
and they both lose their deposit. In the end, at most one stake will
be kept in the commitment tree. When a commitment has no more stake on
it (because all committers have lost the related refutation games), it
is removed from the tree. An honest player *H* must therefore play as
many refutation games as there are stakes on the commitments in
conflict with *H*'s own commitment.

Finally, notice that each player is subject to a timer similar to a
chess clock, allowing each player to play only up to one week: after
this time is elapsed, a player can be dismissed by any Layer 1 user
playing a timeout operation. Thus, the refutation game played by the
two players can last at most 2 weeks.

There is no timeout for starting a refutation game after having
published a concurrent commitment. However, assuming the existence of
an honest participant *H*, then *H* will start the refutation game
with all concurrent committers to avoid the rollup getting stuck.


Private rollups
===============

A **private** Smart Rollup guarantees that private data cannot be
leaked by any means, whereas in a public rollup, one can force a
rollup to leak part of the data by starting a refutation game. This is
achieved by restricting the set of allowed committers using a
*whitelist*. With that restriction, only addresses on the whitelist
can publish commitments and therefore participate in a refutation
game.

The whitelist is optionally defined at origination. The rollup is
considered public if no white-list is defined, private otherwise. The
whitelist can be updated with a specific outbox message. This message
contains an optional list, the new list completely replaces the stored
whitelist in layer 1. If the message contains no list, then the rollup
becomes public. In turn, it is forbidden to make a public rollup
private by sending an outbox message with a non-empty whitelist.

It is the responsibility of the kernel to maintain the white-list by
submitting outbox messages. Kernels must therefore implement their own
access control list logic to add and remove addresses.

Also, it is important to remember that because of the refutation
logic, an outbox message can only be executed when the associated
commitment has been cemented (see Smart rollup node).

Protocol Plugins
****************

This document describes the protocol plugins implemented in Octez.

Protocol-specific shell plugins, simply called protocol plugins,
implement extra APIs needed by the shell in order to interact with the
economic protocol, beyond the one provided by the protocol
environment. This code is not strictly speaking part of the protocol
code base, so this is not subject to on-chain governance (see voting
procedure), but it is still protocol-dependent, which means that it
may vary with different protocols. For instance, the plugin code for
protocol Alpha is located in file
src/proto_023_PtSeouLo/lib_plugin/plugin.ml. Thus, a specific version
is included in the Octez node for each protocol version (recall that a
new release of Octez is usually delivered for each new protocol
proposal, see Release System)

In contrast to the Octez code, plugins are protocol-specific and as
such, know how to read the content of an operation.

So what kind of features may a protocol plugin provide? For instance,
protocol plugins do not define the context, or restrict the validity
of operations. In turn protocol plugins may, for example:

* perform protocol-dependent computations that require data not
  available in the amendable part of the protocol like accessing the
  current time to reason on timestamps (see Consensus filter);

* preserve the opacity/abstraction barrier of the protocol's internal
  data by performing computations on internal data without revealing
  it: e.g., there are some RPCs that can introspect the protocol-
  dependent content for certain operations;

* implement some common operations that are customized for each
  protocol (e.g., Prevalidator filters).


Prevalidator filters
====================

**Prevalidator filters**, which are detailed in the rest of this page,
enable the node to discard some operations (that cannot be included in
the next block) faster and restrict the operations it propagates to
the network. Filters are implemented as a node plugin and a specific
filter is delivered with each protocol version. When the chain
switches to a new protocol, the node installs its corresponding
filters, *in lieu of* the filters of the previous protocol. Notice
that prevalidator filters are not mandatory, their absence does not
break the Tezos blockchain protocol.

The interface of the prevalidator plugin is described at the mempool
plugin API . The following filtering strategies are implemented in the
pre_filter.


Fees filter
-----------

A very affordable technique for attackers is based on flooding the
network with, valid but useless, zero-fees operations. This is why the
prevalidator filter currently restricts operations based on their
associated fees, to reject "too cheap" or "zero-fees" operations. This
can be configured via the "minimal_fees",
"minimal_nanotez_per_gas_unit" and "minimal_nanotez_per_byte" (see
filter RPCs) parameters of the filter configuration of your node.


Consensus filter
----------------

To prevent the network from being flooded by valid (pre)attestations
regarding fanciful future levels and rounds, the plugin provides a
filter that narrows down the number of consensus operations declared
as valid based on a timestamp heuristic.

This filter will classify a consensus operation as "Branch_refused" if
the operation concerns a level and round combination that is far-
fetched in the future in regard to the latest proposal predecessor and
the current timestamp. It can be configured via the "clock_drift" (see
filter RPCs) parameter of the filter configuration of your node.


Operations prioritization and ordering
--------------------------------------

In addition to quick filtering of undesired operations, the
"prefilter" provides a priority for each successfully filtered
operation. Concretely, the priority is either "High", "Medium" or
"Low" in the current implementation, depending on the validation pass.
Some extra information (like the fees, or the gas/fees ratio of
manager operations) are also provided along the priorities to enable
fine-grained operations ordering. This extra information is similar to
the one used by the baker's operations selection mechanism, that
decides which operations will be included in the next block.


Filters RPCs
============

Filters are tunable by several parameters, whose values can be
retrieved and changed by users via the following RPC calls,
respectively:

* "rpc get /chains/<chain>/mempool/filter"

* "rpc post /chains/<chain>/mempool/filter"

The following parameters can be thus inspected and modified:

* "minimal_fees": type "int", default "100"

* "minimal_nanotez_per_gas_unit": type "int", default "100"

* "minimal_nanotez_per_byte": type "int", default "1000"

* "allow_script_failure": type "bool", default "true"

* "clock_drift" : type "Period.t option", default "None"

* "max_operations" : type "int", default "10_000"

* "max_total_bytes" : type "int", default "10_000_000"

* "replace_by_fee_factor" : type "rational", default "21/20" (ie.
  "1.05%")

For example, each command below modifies the provided parameter and
resets all the others to their default values:

   octez-client rpc post /chains/main/mempool/filter with '{ "minimal_fees": "42" }'
   octez-client rpc post /chains/main/mempool/filter with '{ "replace_by_fee_factor": [ "23", "20" ] }'
   octez-client rpc post /chains/main/mempool/filter with '{ "max_operations": 7500 }'


Changing filters default configuration
--------------------------------------

Changing filters configuration may not have the expected outcome.
Because the configuration is only changed on your node, and not on all
the gossip network nodes, this will only impact how operations of your
nodes are propagated. For example, assuming there are three nodes "A",
"B" and "C" such that "A" is connected to "B" and "B" is connected to
"C". Assume that "A" modifies its filter configuration so that
"minimal_fees" is now "0". Then "A" may propagate an operation with
"0" fee to "B". However, because "B" has the default filter
configuration, this operation will not be propagated to "C" (so "C"
may never see it).

Contract events
***************

Contract events provide a way for smart contracts to deliver live
information to external applications. This mechanism allows off-chain
applications to react to Tezos contracts execution.

Note:

  Contract events should not be confused with log events, which are
  generated by the various Octez tools to include information about
  their execution in their own logs.

This page explains how events can be generated using specific support
in the Michelson language. For using events in higher-level languages
for smart contracts, consult their own documentation.


Emitting events
===============

Contracts can create event emission operations by invoking the
Michelson instruction "EMIT". "EMIT %tag ty" pops an item of type "ty"
off the stack and pushes an "operation" onto the stack. The type
ascription "ty" is special since annotations in "ty" are preserved and
eventually transmitted in the event so that indexers have a choice to
present the event body with appropriate labels. Both the "ty" type
ascription and "%tag" are optional. If "ty" is absent, this type is
inferred from the type of the top element of the stack. If "%tag" is
absent, no tag is attached to the event.

To actually send out the events, most importantly, the produced
"operation"s must be included in the list of operations returned by
the contract, along with "TRANSFER_TOKEN" operations for example.

Upon successful contract execution, the events appearing in the
resultant list of operations will be attached, in the order of
appearance, to the transaction receipt. There, the events are made
ready for consumption by services observing the chain.


Example
-------

Suppose a contract wants to emit events with the following type:

   or (nat %int) (string %str)

Then, this contract may generate an event emission operation with the
following instructions:

   PUSH string "right";
   RIGHT nat;
   EMIT %notify_client (or (nat %int) (string %str));

Note that the "EMIT" instruction will emit an event with a tag
"notify_client". The additional tags "int" and "str", allow indexers
to recognise the two variants and present the values with the
appropriate variant labels.


Retrieving events
=================

Events successfully emitted can be read off from transaction results.

This is typically achieved by making JSON RPCs to the block service.
It will return a list of operations, each including the event entries
with the information above.

Here is a sample result from a call, corresponding to the example
above.

   {
     "protocol": "ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK",
     "hash": "opNX59asPwNZGu2kFiHFVJzn7QwtyaExoLtxdZSm3Q3o4jDdSmo",
     // ... fields elided for brevity
     "contents": [
       {
         "kind": "transaction",
         // ... fields elided for brevity
         "metadata": {
           // ... fields elided for brevity
           "internal_operation_results": [
             {
               "kind": "event",
               // ... fields elided for brevity
               "source": "KT1M1ynE3YXkM7qLZoMppq6szMbBvxX9yQVL",
               "nonce": 0,
               "type": {
                  "prim": "or",
                  "args": [
                    {
                      "prim": "nat",
                      "annots": [
                        "%int"
                      ]
                    },
                    {
                      "prim": "string",
                      "annots": [
                        "%str"
                      ]
                    }
                  ]
               },
               "tag": "notify_client",
               "payload": {
                 "prim": "Right",
                 "args": [
                   {
                     "string": "right"
                   }
                 ]
               },
               "result": {
                 "status": "applied",
                 "consumed_milligas": "1000000"
               }
             }
           ]
         }
       }
     ]
   }

Note that the "operation" produced by "EMIT" does not constitute a
call to any other contract.

Blocks and Operations
*********************

The content of a Tezos block is made up of a block header and a
payload consisting of a list of operations.

This page first describes the protocol-specific part of the block
header, and then explains what operations are. For the protocol-
independent part of the block header, see Shell header.


Protocol-specific block header
==============================

Recall that, for the shell to interact with the economic protocol, two
notions are defined abstractly at the level of the shell and made
concrete at the level of the consensus protocol. Namely, these two
notions are the protocol-specific header and the fitness.

As in Emmy*, the protocol-specific header contains the fields:

* "signature": a digital signature of the shell and protocol headers
  (excluding the signature itself)

* "seed_nonce_hash": a commitment to a random number, used to generate
  entropy on the chain

* "proof_of_work_nonce": a nonce used to pass a low-difficulty proof-
  of-work for the block, as a spam prevention measure

* "liquidity_baking_toggle_vote": a vote to continue the Liquidity
  Baking Subsidy, stop it, or abstain.

There are two additional fields: "payload_hash" and "payload_round"
which are needed for establishing if a block is final.


Operations
==========

Operations implement and reify different functionalities provided by a
Tezos economic protocol: from reaching consensus on the state of the
Tezos blockchain, to performing smart contract calls and transactions.
Each Tezos economic protocol can specify different kinds of
operations.

This page only describes the operations supported by the economic
protocol that implement *enabled* features -- that is, those available
to end-users on Tezos Mainnet. The complete list of operations,
including those corresponding to features in development or available
only on test networks, is given in the OCaml Documentation .


Validation Passes
=================

The different kinds of operations are grouped into classes. Each class
has an associated index, a natural number, also known as a validation
pass. There are currently four classes of operations: consensus,
voting, anonymous, and manager operations. This order also specifies
the validation and application priority of each of these classes.
Consensus operations are considered the highest priority ones, and
manager operations the lowest.

Each kind of operation belongs to exactly one validation pass, except
for the Failing_noop operation which belongs to no validation pass and
therefore cannot be applied.

In the sequel, we describe the different classes of operations, and
the different kinds of operations belonging to each class.


Consensus Operations
====================

Consensus operations are administrative operations that are necessary
to implement the consensus algorithm. There are two kinds of consensus
operations, each belonging to the different voting phases required to
agree on the next block.

* A "Preattestation" operation implements a first vote for a candidate
  block with the aim of building a preattestation quorum.

* An "Attestation" operation implements a vote for a candidate block
  for which a preattestation quorum certificate (PQC) has been
  observed. These operations also hold information on DAL attestations
  when the attesting baker participates in the DAL.

Starting in protocol S, blocks are also able to include these
operations in an aggregated form, using operations
"Attestations_aggregate" and "Preattestations_aggregate". If the
attesting baker uses a tz4 consensus key, thanks to the BLS signature
scheme, its attestation can be aggregated with all the other tz4
pre/attestations, which helps reducing the size and validation time of
blocks without compromising on security. A valid block can include at
most one aggregated preattestation and at most one aggregated
attestation.


Voting Operations
=================

Voting operations are operations related to the on-chain Tezos
Amendment process. In this economic protocol, there are two voting
operations:

* The "Proposal" operation enables delegates to submit (also known as
  to "inject") protocol amendment proposals, or to up-vote previously
  submitted proposals, during the Proposal period.

* The "Ballot" operation enables delegates to participate in the
  Exploration and Promotion periods. Delegates use this operation to
  vote for ("Yea"), against ("Nay"), or to side with the majority
  ("Pass"), when examining a protocol amendment proposal.

Further details on each operation's implementation and semantics are
provided in the dedicated entry for on-chain governance.


Anonymous Operations
====================

This class groups all operations that do not require a signature from
a Tezos account (with an exception, detailed below). They implement
different functionalities of the protocol, and their common
characteristic is that they allow the account originating these
operations to remain anonymous in order to avoid censorship.

Two operations in this class implement functionality pertaining to the
protocol's random seeds generation mechanism:

* The "Seed_nonce_revelation" operation allows a baker to anonymously
  reveal the nonce seed for the commitment it had included in a
  previously baked block (in the previous cycle).

* The "Vdf_revelation" operation allows the submission of a solution
  to, and a proof of correctness of, the VDF challenge corresponding
  to the VDF revelation period of the randomness generation protocol.

Further details on the latter operation's implementation and semantics
are provided in the random seed generation protocol.

Two operations in this class are used to punish participants which
engage in Byzantine behaviour -- notably delegates which "double sign"
blocks, or emit conflicting consensus operations:

* The "Double_consensus_operation_evidence" operation allows for
  accusing a delegate of having *double-preattested* or *double-
  attested* -- i.e., of having preattested, or respectively attested,
  two different block candidates, at the same level and at the same
  round. The bulk of the evidence, the two arguments provided,
  consists of the two offending preattestations, respectively
  attestations.

  Beside the two offending operations, there is a "slot" field
  containing the delegate's slot used in the offending operations.
  More precisely, each offending operation may be either a standalone
  operation for this slot, or an aggregate operation containing this
  slot in its committee.

  In protocols older than S, there were two operations,
  "Double_preattestation_evidence" and "Double_attestation_evidence",
  instead of "Double_consensus_operation_evidence", which did not
  exist.

* The "Double_baking_evidence" allows for accusing a delegate of
  having "double-baked" a block -- i.e., of having signed two
  different blocks at the same level and at same round. The bulk of
  the evidence consists of the block headers of each of the two
  offending blocks.

See here for further detail on the semantics of evidence-providing
operations.

The "Activation" operation allows users which participated in the
Tezos fundraiser to make their accounts operational.

Finally, the "Drain_delegate" operation allows an active consensus-key
account, i.e., an account to which a baker delegated its consensus-
signing responsibility, to **empty** its delegate account. This
operation is used as a deterrent to ensure that a delegate secures its
consensus key as much as its manager (or main) key.

The DAL also adds the anonymous operation "DAL_entrapment_evidence",
see DAL integration.


Manager Operations
==================

Manager operations enable end-users to interact with the Tezos
blockchain -- e.g., transferring funds or calling smart contracts. A
manager operation is issued by a single *manager* account which signs
the operation and pays the fees to the baker for its inclusion in a
block. Indeed, manager operations are the only fee-paying and gas-
consuming operations.

* The "Reveal" operation reveals the public key of the sending
  manager. Knowing this public key is indeed necessary to check the
  signature of future operations signed by this manager. Additionally,
  when revealing a tz4 public key, the manager must also include a
  proof of possession, which is the signature of the public key
  itself.

* The "Transaction" operation allows users to transfer tez between
  accounts, to invoke a smart contract, or to invoke pseudo-operations
  on user accounts.

* The "Delegation" operation allows users to designate a delegate (a
  *baker*) for delegating or staking their coins, or to register
  themselves as delegates.

* The "Update_consensus_key" operation allows users to register a
  consensus key, which is a dedicated key for signing blocks and
  consensus-related operations. When the new consensus key is a tz4
  (BLS key), the optional "proof" field must contain a proof of
  possession, which is the signature of the public key itself.

* The "Update_companion_key" operation allows users to register a
  companion key, which is a dedicated key (introduced in protocol S)
  for signing the DAL specific part of consensus operations, when
  using a tz4 consensus key.

* The "Origination" operation is used to originate, that is to deploy,
  smart contracts in the Tezos blockchain.

* The "Set_deposits_limit" operation enables delegates to adjust the
  amount of stake a delegate has locked in bonds.

* Support for registering global constants is implemented with the
  "Register_global_constant" operation.

* The "Increase_paid_storage" operation allows a sender to increase
  the paid storage of some previously deployed contract.

* The "Event" operation enables sending event-like information to
  external applications from Tezos smart contracts -- see Contract
  Events for further detail.

* The "DAL_publish_commitment" operation (see DAL integration)

Moreover, all operations necessary to implement Tezos' *enshrined*
Layer 2 solutions into the economic protocol are also manager
operations.

In particular, smart rollups maintenance is handled with dedicated
manager operations.

* The "Smart_rollup_originate" operation is used to originate, that
  is, to deploy smart rollups in the Tezos blockchain.

* The "Smart_rollup_add_messages" operation is used to add messages to
  the inbox shared by all the smart rollups originated in the Tezos
  blockchain. These messages are interpreted by the smart rollups
  according to their specific semantics.

* The "Smart_rollup_publish" operation is used to regularly declare
  what is the new state of a given smart rollup in a so-called
  “commitment”. To publish commitments, a user account has to own at
  least ꜩ 10,000, which are frozen as long as at least one of their
  commitments is disputable.

* The "Smart_rollup_cement" operation is used to cement a commitment,
  if the following requirements are met: it has been published for
  long enough, and there is no concurrent commitment for the same
  state update. Once a commitment is cemented, it cannot be disputed
  anymore.

* The "Smart_rollup_recover_bond" operation is used by a user account
  to unfreeze their ꜩ 10,000. This operation only succeeds if and only
  if all the commitments published by the user account have been
  cemented.

* The "Smart_rollup_refute" operation is used to start or pursue a
  dispute. A dispute is resolved on the Tezos blockchain through a so-
  called refutation game, where two players seek to prove the
  correctness of their respective commitment. The game consists in a
  dissection phase, where the two players narrow down their
  disagreement to a single execution step, and a resolution, where the
  players provide a proof sustaining their claims. The looser of a
  dispute looses their frozen bond: half of it is burned, and the
  winner receives the other half in compensation.

* The "Smart_rollup_timeout" operation is used to put an end to a
  dispute if one of the two players takes too much time to send their
  next move (with a "Smart_rollup_refute" operation). It is not
  necessary to be one of the players to send this operation.

* The "Smart_rollup_execute_outbox_message" operation is used to enact
  a transaction from a smart rollup to a smart contract, as authorized
  by a cemented commitment. The targeted smart contract can determine
  if it is called by a smart rollup using the "SENDER" Michelson
  instruction.


Manager Operation Batches
-------------------------

Manager operations can be grouped, forming a so-called **batch**.
Batches enable the inclusion of several manager operations from the
same manager in a single block.

Batches satisfy the following properties:

* All operations in a batch are issued by the same manager, which
  provides a single signature for the entire batch.

* A batch is applied atomically: all its operations are executed
  sequentially, without interleaving other operations. Either all the
  operations in the batch succeed, or none is applied.


Failing_noop operation
======================

The "Failing_noop" operation is not executable in the protocol:

* it can only be validated in mempool mode, by the prevalidator
  component;

* consequently, this operation cannot be applied, and in fact will
  never be included into a block.

Rather, the "Failing_noop" operation allows to sign an arbitrary
string, without introducing an operation that could be misinterpreted
in the protocol.

The Octez client provides commands to sign and verify the signature of
input messages by a given key. These commands create a "failing_noop"
operation from the message that is being signed or checked.

   octez-client sign message "hello world" for <account>

   octez-client check that message "hello world" was signed by <account> to
   produce <signature>

Validation and Application
**************************

The economic protocol is responsible for providing the rules that
govern the Tezos network, and for enforcing that these rules implement
a correct blockchain protocol. However, it does so in coordination
with a Tezos shell, who ultimately implements these rules on its
behalf. To this end, a Tezos economic protocol must provide the shell
with an interface enabling roughly the following functionalities:

* deciding whether an operation is *valid*, and thus can be safely
  included into a new block in the blockchain;

* deciding whether a new block is *valid*, and thus can be safely
  appended to the blockchain;

* including an operation in a block, and executing it on the
  blockchain state, effectively modifying the ledger state; and,

* appending a new block to a Tezos blockchain, and computing the
  updated ledger state.

From a higher-level, *abstract* perspective, the validation system in
the Tezos protocol implements this business logic in a functional,
state-passing machine where:

* Its state is given by the context, the internal representation of
  the state of the Tezos ledger at a given blockchain level. For
  instance, the context contains the information of all activated
  accounts and contracts, and their balances. More generally, the
  context must provide enough information to determine the validity of
  operations and blocks.

* The *apply_operation* method determines whether an operation is safe
  to be executed in a given context. If so, it proceeds to execute it
  and outputs the context resulting from the applied operation's
  effects.

* The *apply* method takes as input a block and a context, determines
  whether the block can be safely and correctly appended to the
  current head of the chain. If so, it appends the block to the chain
  and outputs the resulting context, reflecting the state of the
  ledger after the block has successfully been applied. *apply* relies
  on (i.e., it should call) *apply_operation* to validate and apply
  each operation in the block, and compute intermediate states.

However, the *concrete* API exported from the Tezos economic protocol
does not implement this business logic *monolithically*, as described
above, but it rather presents a more fine-grained API. The rationale
is to provide specialized variations of the core *validation* and
*application* functionality, dubbed Validation modes. For example,
these modes enable the protocol to distinguish operations "in the
mempool", whose validation is triggered by the prevalidator, from
operations included in newly received blocks, whose validation is
triggered by the block validator, in order to localize validation
rules as needed. The resulting concrete API is specified by the
Protocol  module in the protocol environment "V15", and it is
implemented by this protocol in the Main module.

The rest of this document is organized as follows: we first describe
the different validation modes implemented by this Tezos economic
protocol, and then we delve deeper into the particulars of validation
and application for blocks and the operations supported.


Validation modes
================

The Tezos protocol provides different validation modes, intended to be
used by the Tezos *shell* and *baker* software implementations when
needing to apply (or to assert the validity of) blocks and operations
under different, or specialized, circumstances -- for example, in
order to *bake* a block. For each of these validation modes, the API
specified by the protocol environment offers an entry point so that
protocol-agnostic components, the Tezos shell for instance, are able
to use these different modes.


Full Application
----------------

The "Full application" mode is intended to be used to *fully* validate
and apply blocks. In particular, this mode is used to validate and
apply a **known** block, with a known operation trace. A Tezos shell
implementation should use the full application mode to decide whether
an incoming block can be safely included in the blockchain. That is,
all validity checks are enabled: the block's signature is correct, and
**all** operations included in the block are valid; the correct amount
of consensus operations have been included in order to satisfy the
consensus' threshold, etc.


Full Construction
-----------------

The "Full construction" mode is intended to be used when a *delegate*
is trying to bake a block and therefore needs to filter the validity
of the desired operation trace, to include only valid operations. This
mode is mostly similar to the "Full application" mode except that
*some* global block validity checks are disabled, and consensus
operations are validated with slightly different preconditions. For
instance, since a delegate cannot sign a block while it is being
built, the signature check is disabled, and it will be left to the
baker to correctly sign the resulting block after its construction is
finalized.

In Octez, this mode is mainly used by the baker daemon.


Partial Construction
--------------------

The "Partial construction" mode, also known as "Mempool mode" is used
by the prevalidator component of an Octez node to validate incoming
operations -- that is, those not-yet included into blocks. This mode's
business-logic is very close to the "Full construction" mode, and the
differences boil down to the intended usage. The partial construction
mode does not try to fully bake a block, but rather to inform the
Octez prevalidator on the potential validity of operations (and
whether they can safely included into a block), so that the latter can
**classify** incoming operations, and further decide how to process
them accordingly.

The protocol provides the shell with the following classification of
an operation, consisting of one valid kind -- "Applied" --, and four
error kinds defined by the protocol environment:

* "Applied": the operation is valid and can be included in a potential
  block in the current context.

* "Temporary": the operation is invalid in the current context, but it
  could *later* become valid -- in the context associated to a
  successor block of the current head. For instance, a manager
  operation whose counter value is greater than the one expected (a
  *"counter-in-the-future"* error), or the manager's balance is
  insufficient to pay the operation's fees, etc.

* "Branch": the operation is invalid in the current context and in any
  possible context from its future successors, but it might still be
  valid in an alternative branch. For example: a manager operation
  with a smaller counter than the one expected (a *"counter-in-the-
  past"* error), an unexpected attestation for the current level, etc.

* "Permanent": the operation is invalid in the current context, and
  there isn't any plausible context where it might be or become valid.
  For example, an operation carrying an invalid signature.

* "Outdated": the operation is *too old* to be included in a block.
  Furthermore, there might be still some value in the information
  provided by an "Outdated" operation. An example is the case of an
  attestation which was received *too late*, but that could still be
  used to form a consensus quorum.


Partial Application
-------------------

The "Partial application" mode is no longer used, but it subsists in
the code. Its aim is to provide Tezos shell implementations with a
light-weight (read "fast") block application mechanism, which can
determine whether a block has a *chance* of being valid or not, in a
situation when the provided context is *not a recent one*. That is,
when the block candidate succeeds neither the head of the chain, nor a
close ancestor.

This validation mode could be used when the node receives a
significantly large branch -- for instance, while bootstrapping. To
check whether this branch is plausibly valid or potentially malicious
spam, the shell retrieves the context from the most recent common
ancestor between its current head and the announced branch, and
proceeds to "partially apply" each block of this branch using the
common ancestor's context.

Indeed, by relying on the ancestor context, this mode can *only*
assert the validity of consensus-related preconditions (attesting
power, block fitness, etc.), as future consensus slots are known in
advance -- how much in advance being specified by the
"<CONSENSUS_RIGHTS_DELAY>" protocol constant. Thus, the "Partial
application" mode provides an over-approximation of the branch's
validity, and as a result intermediate results are not committed on
disk in order to prevent potential attacks.


Block Validation
================

The validity of a block depends on a set of precondition checks
implemented in different steps, which happen at different stages of
the application (and the construction) of a block.

The first step in the process is to decide whether a candidate block
is *well-formed*, that is, that it has the expected "shape" of a valid
block under the current Tezos economic protocol. Given a block
candidate, the block validation process will then verify that the
candidate block declares consistent level, round, and timestamp
values; that it carries a valid signature, etc. At this step, the
block validation process will also initialize the data-structures
required for subsequent steps.

The second step iterates over the block's operations and proceeds to
apply them sequentially. When at least one operation is found to be
invalid, under the conditions described in Operation Validation and
Application further below, the whole block is considered as invalid.

The last step in the block validation process, known as "block
finalization", aims to verify that the collected consensus operations
constitute a sufficiently large quorum. That is, it will verify that
the total attesting power present in the block is greater than the
"CONSENSUS_THRESHOLD" constant.

This sequence of three steps also yields a new context -- the
resulting state of the Tezos ledger after the application of the
candidate block. The shell may decide to commit this context to disk.

The Tezos economic protocol also offers a cheap (read "faster")
alternative to determine an over-approximation of the validity of a
block (see Partial Application above). This feature allows the shell
to propagate blocks faster without needing to fully validate them,
speeding-up block propagation over the network. Of course, as this is
an over-approximation, this feature cannot be considered to provide a
safe guarantee that a block will be valid: in particular, it does not
validate all kinds of operations.


Operation Validation and Application
====================================

In the Tezos economic protocol, we dissociate the notion of *validity*
from the notion of *applicability* for operations. A valid operation
is an operation that can be included safely in a block without
affecting the block's validity. Applying an operation, on the other
hand, actually performs the operation's side-effects which can be:
registering a new delegate, executing a smart contract, voting for a
new protocol amendment proposal, etc.

Note that an operation may fail during the application phase, even
though it has been checked as valid. For example, a smart contract
call that exceeds its gas limit can be included in a block even if an
error is raised at run-time. The application (that is, the operation's
side-effects) will not take effect, but fees will nonetheless be taken
from the account submitting the smart contract call.

In the sequel we refine the validity conditions and describe the
application process for each of the different validation passes.


Validity of operations
----------------------

For consensus, voting, and anonymous validation passes, validation is
sufficient to ensure successful application. For manager operations,
validation only ensures that the manager’s balance is sufficient to
cover the fees, in addition to verifying that the operation is well-
formed. It does not guarantee that the operation’s execution will be
error-free. If an operation does result in an error, all side effects
will be canceled, but the fees will be forfeited.


Validity of Manager Operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In this sub-section, we explain the conditions for manager operations
(and batches of managers operations) to be considered valid and hence
suitable for inclusion in a block.


Validity of Individual Manager Operations
"""""""""""""""""""""""""""""""""""""""""

Manager operations are a class of operations, issued by a single
*manager* account which signs the operation and pays their fees. The
different manager operation kinds share several common fields:

* "source": the public key's hash of the *source* account of the
  manager operation -- that is, the *manager*.

* "fee": the amount of tez paid to the baker which decides to include
  this operation;

* "counter": the manager account's counter, incremented each time this
  account executes a manager operation, to prevent replay-attacks.

* "gas_limit": the maximum amount of gas that this operation may
  consume before failing.

* "storage_limit": the maximum amount of storage that this operation
  may require before failing.

* "operation": the actual operation(s) -- e.g., transfers, smart-
  contract calls, originations, delegations, etc.

* "signature": the manager's signature used to prove its identity.

A manager operation is **valid** if and only if all of the following
conditions hold:

* The operation source's public key has been previously *revealed*,
  unless the operation is itself a "Reveal" operation.

* The operation's signature is correct with regard to the manager
  account's public key.

* The provided "counter" value is the expected one for the manager.

* Depending on the operation's kind, the "gas_limit" is high enough to
  cover the minimal cost of parsing the operation and further minimal
  treatment.

* The manager account is solvent to pay the announced fees.


Validity of Manager Operation Batches
"""""""""""""""""""""""""""""""""""""

A batch of manager operations includes one or more manager operations
for sequential and atomic execution. The atomicity property imposes
that the validity of a batch should entail the validity of each
individual operation in the batch, as defined above. However, it also
entails some additional *global* constraints on manager batches.

For each of the operations present in a batch, the validation process
must check that the individual constraints above are satisfied, *with
the exception of* the signature constraint. Given that the signature
concerns the whole batch, as all operations in the batch are signed by
the same manager, it suffices to verify the signature only once.

The **global batch validity** constraint for this economic protocol is
defined as the conjunction of the following conditions:

* When a "Reveal" operation is present, **it must only occur once**,
  and **it must be placed at the head** of the batch -- that is, the
  "Reveal" operation must be the first operation in the batch.

* Every operation in the batch should declare the same "source".

* Each of the individual operation counters must be incremented
  correctly and sequentially.

* The sum of each individual operation's declared fees must be lower
  than the account's balance. That is, the manager account must be
  solvent to pay the announced fees for all the operations in the
  batch.


Co-valid operations
"""""""""""""""""""

However, two valid manager operations (or batches) do not always form
a valid block together. For example, operations "op1" and "op2" might
both be valid in the context "C", but after applying "op1" to "C",
"op2" might no longer be valid in the resulting context because the
manager's balance may no longer cover the fees.

In order to enforce that a set of valid managers operations forms a
valib block, the protocol over-approximate this property by requiring
the property of *co-validity*:

Two manager operations are *compatible* if they correspond to distinct
managers.

Two manager operations are *co-valid* in a context if they are
compatible and valid in this context.

Considering two co-valid operations in a context, the application of
one of them preserves the solvability of the other.

This property extends to a set of operations if every pair of distinct
operations is co-valid. In this case, the operations could be included
in the next block in any order, modulo block limits (eg. maximum gas,
block size limit, etc).


Application of Manager Operations
---------------------------------

Once the validity of a manager operation (or, a batch of manager
operations) is established, the protocol proceeds to apply the
operation. This first step in this application phase is to transfer
the operation's fees to the baker that included this operation. Then,
the actual application depends on the operation kind. For instance,
this could be a smart contract execution, enacting a delegation, or
multiple actions executed as a batch. The application of a batch of
manager operations consists of the sequential application of each
operation in the batch, following their inclusion order -- the head of
the batch being the first manager operation being applied.

The application of each individual manager operation may either
succeed -- and therefore be reported as "Applied" --, or indeed fail
with an error. In both cases, the fees are taken and the counter for
the operation's manager is incremented.

When a manager operation fails, every side-effect which was previously
performed is backtracked. Moreover, the (rest of the) batch has to be
aborted.  Thus, depending on the position of the manager operation in
a batch, its failure has to be propagated accordingly:

* If there were other successfully applied operations in the batch
  prior to the offending one, the effect of each of them has to be
  reverted, and each of them will be reported as "Backtracked".

* If there were other operations pending application after the
  offending one, their application is aborted, and they are reported
  as "Skipped".

For example, let's consider a simple batch of three manager operations
"[op1, op2, op3]", if "op1" is successful but "op2" fails, the ticket
result for the application of the manager operation batch will report:

* "op1" -- "Backtracked", "op1" was applied successfully, but after
  "op2" failed, the operation was canceled;

* "op2" -- "Failed", the application of this particular operation
  failed;

* "op3" -- "Skipped", this operation was never executed because "op2"
  had previously failed.

Tickets
*******

Tickets are a special class of assets on Tezos that contracts can
construct and store, and that can then be transferred between
contracts, user accounts, or rollups. They are useful to model
ownership of some digital artefact, existing in a limited number of
copies (or instances); this can further serve as a basis for
restricting access to the artefact and permissions to use it. In this
document, we will explain the semantics and lifetime of tickets,
including creation, transfer, and destruction.


Semantics
=========

A ticket represents a digital artefact existing in a limited number of
copies (or instances). As such, a ticket has the following four
attributes:

1. "ticketer" is the creator of the ticket, which is always a Tezos
   smart contract.

2. "contents" is the Michelson expression attached to the ticket.

3. "content_ty" is the Michelson type of the "contents" data. It can
   be any comparable type.

4. "amount" is the amount of the ticket. It is a strictly positive
   whole number.

Furthermore, a ticket cannot be duplicated with the DUP or the DUP n
instructions.


Operations on tickets
=====================


Creating
--------

Tickets can be constructed by smart contracts. To construct tickets,
smart contracts use the "TICKET" instruction with some ticket amount,
ticket content type, and value as inputs. Smart contracts then may
store them in contract storage or transfer them along to either other
contracts, rollups, or user accounts.

Tickets cannot be constructed with instructions for duplication, such
as "DUP". Therefore, the issuance of tickets is completely within the
control of the ticketer contract by invoking "TICKET" instructions.


Splitting
---------

A ticket can be split into two tickets with the same "contents" and
"ticketer" values, as long as the sum of the "amount" of the two
tickets is equal to that of the original. This operation allows
tickets to be spent across several transactions by breaking tickets
into smaller tickets. Smart contracts may split tickets by invoking
the SPLIT_TICKET instruction. The ticket splitting is done
automatically when a user account transfers part of a ticket.


Joining
-------

Tickets issued by the same "ticketer" contract with the same
"contents" data of the same "contents_ty" type are considered of the
same kind. Therefore, two such tickets can be **joined** into one
ticket and the output "amount" will be the sum of those of the two
input tickets. Smart contracts can join tickets via the JOIN_TICKETS
instruction. Tickets of the same kind are automatically joined when
they belong to the same user account.


Transferring
------------

Once a ticket has been constructed by a smart contract, it may be
transferred to other contracts as follows:

* *Smart contract to user account*: Smart contracts can transfer a
  ticket to user accounts via TRANSFER_TOKENS. To do so, the contract
  needs to cast the address of the target user account to type
  "contract (ticket cty)" where "cty" is the type of the content of
  the ticket to be sent. This can be done using "CONTRACT (ticket
  cty)". The rest is the same as making a contract call. The following
  Michelson snippet is an example sending a ticket of amount "10" with
  a "string" content ""some ticket"" to a user account address made
  available at the top of the stack.

   # Stack: address :: S
   CONTRACT (ticket string) ;
   # Stack: option (contract (ticket string)) :: S
   ASSERT_SOME ;
   # Stack: contract (ticket string) :: S
   PUSH mutez 0 ;
   # Stack: mutez :: contract (ticket string) :: S
   PUSH nat 10 ;
   # Stack: nat :: mutez :: contract (ticket string) :: S
   PUSH string "some ticket" ;
   # Stack: string :: nat :: mutez :: contract (ticket string) :: S
   TICKET ;
   # Stack: option (ticket string) :: mutez :: contract (ticket string) :: S
   ASSERT_SOME ;
   # Stack: ticket string :: mutez :: contract (ticket string) :: S
   TRANSFER_TOKENS ;
   # Stack: operation :: S

* *Between smart contracts*: Contracts can send tickets to other
  contracts via regular contract calls using the instruction
  "TRANFSER_TOKENS", as long as target contracts accept tickets of
  matching content type in their "parameter"s.

* *Between user accounts*: User accounts can transfer existing tickets
  they own to other user accounts with "Transfer_ticket" operation
  from their wallets. For instance, "octez-client" can be invoked in
  the following way by a user account holder "alice" to transfer a
  ticket of amount "10", type "string", content ""some ticket"" and
  ticketer "ticketer" to another user account held by "bob".

   octez-client transfer 10 tickets from alice to bob with entrypoint default and type string and content '"some ticket"' and ticketer 'ticketer'

If the amount of the ticket ""some ticket"" owned by "alice" was
greater than 10, this transfer would cause an automatic ticket split,
after which "alice" would continue to own the remaining amount.

Note that the above command uses the expression "transfer 10 tickets"
of the given kind, instead of "transfer a ticket of amount 10".
Indeed, by virtue of automatic joining of tickets of the same kind
belonging to the same user account, this formulation does not
introduce any ambiguity: the user account cannot hold several distinct
tickets of this kind. Of course, these expressions would not be
interchangeable for tickets belonging to a smart contract.

* *User account to smart contract*: Using the same "Transfer_ticket"
  operation, user accounts can also send their tickets to smart
  contracts. In this case, the entrypoint, as defined by the
  specification of the target contract, must accept this kind of
  tickets. Here is an example using "octez-client" to transfer a
  ticket of amount "10", type "string", content ""some ticket"" and
  ticketer "ticketer" owned by "alice" to a contract "receiver"
  accepting tickets at the entrypoint "save".

   octez-client transfer 10 tickets from alice to receiver with entrypoint save and type string and content '"some ticket"' and ticketer 'ticketer'

In this case, too, automatic ticket splitting may happen, under the
same circumstances.

Note:

  This page does not cover transfers of tickets to/from rollups. For
  that, refer to the documentation pages of particular rollups (e.g.
  Smart Optimistic Rollups).


Destroying
----------

Only smart contracts can destroy tickets, by simply dropping them.
That is, by not storing them anymore in the contract storage and not
sending them to other contracts. User accounts, on the other hand,
cannot destroy any ticket in their possession.

For instance, a user account "A" may receive a "string" ticket of
amount "2" with content "Lorem ipsum" created by a smart contract "B".
"A" may send amount "1" of it to a smart contract "C". This
automatically splits the ticket into two tickets of amount "1". Now
"C" may destroy this ticket by dropping it and "A" will still hold a
ticket of amount "1" with a "string" content "Lorem ipsum" created by
"B". It will remain in the possession of "A" until "A" sends it to
another user account or smart contract.

\newcommand\F[2]{\mathit{#1}\left(#2\right)}
\newcommand\IL[1]{\normalsize{#1}} \newcommand\exp[1]{\F{exp}{#1}}


Adaptive Issuance
*****************

Adaptive Issuance is a novel mechanism regulating tez issuance in
Tezos.

The Tezos economic protocol issues new tez via:

* Participation rewards: incentives given to delegates for
  participation in consensus and random seed generation.

* The Liquidity Baking (LB) subsidy.

* Protocol "invoices": lump sums of tez issued and allocated during
  protocol migration.

Participation rewards and the LB subsidy are regularly issued by the
protocol, whereas the value and recipients of invoices are defined
discretionarily by the developers of a protocol proposal. The values
for participation rewards and the LB subsidy, if any, are currently
defined by the Tezos protocol using fixed constants.

Adaptive Issuance lets the amount of participation rewards depend on
the global **staked ratio** – that is, the ratio of staked tez to the
total supply. This lets issuance roughly match the *actual* security
budget the chain requires, the amount needed to encourage participants
to stake and produce blocks, but *no more*.

At the end of each blockchain cycle, the regular issuance is adjusted,
to nudge the staked ratio towards a protocol-defined target (set at
50% starting in the Paris protocol). Participation rewards are
recomputed to match that budget. When the staked ratio decreases and
diverges from the target, emission rates increase, incentivizing
participants to stake funds to re-approach the target. Conversely,
incentives decrease as the ratio increases beyond the target.


Adaptive issuance rate
======================

The adaptive issuance rate determines, at the end of cycle \IL{c}, the
issuance for cycle \IL{c + 3}. The adaptive issuance rate is the sum
of a static rate and a dynamic rate. This value is kept within a
minimal and a maximal value, to ensure nominal emissions remain within
reasonable bounds.


Staked ratio
------------

The **staked ratio** is the ratio of staked tez to the total supply.
It is computed at the end of a given "cycle":

   def staked_ratio(cycle):
     return total_frozen_stake(cycle + 1 + issuance_modification_delay) / total_supply(cycle)

Where:

* "issuance_modification_delay" is a derived protocol constant which
  is set to the same value as "CONSENSUS_RIGHTS_DELAY" (see Validator
  selection), that is, 2 cycles.

* "total_supply(cycle)" returns the total supply of tez at the end of
  the given "cycle".

* "total_frozen_stake(cycle)" returns the total frozen stake at the
  given "cycle".


Static rate
-----------

The **static rate** is a mechanism which approximates a Dutch auction
to compute a nominal issuance rate as a function of the staked ratio
for a given cycle. Its value decreases as the staked ratio increases.
The static rate is defined as follows:

   def static_rate(cycle):
     return 1 / 1600 * (1 / (staked_ratio(cycle) ** 2))

The choice of a scaling factor ensures that the curve takes reasonable
values for plausible staked ratios. Moreover, since Adaptive Issuance
is activated with a dynamic rate of 0, and at current staked ratio
(that is, ~7.5% of the total supply), this factor allows for a smooth
transition from the issuance rate (~4.6%) from the Oxford protocol
(before the activation of Adaptive Issuance).


Dynamic rate
------------

The **dynamic reward rate** adjusts itself over time based on the
distance between the staked ratio and the 50% (±2%) target ratio,
increasing when < 48% and decreasing when > 52%. The dynamic rate is
defined as follows:

   def dynamic_rate(cycle):
     seconds_per_cycle = blocks_per_cycle * minimal_block_delay
     days_per_cycle = seconds_per_cycle / 86400
     previous_dynamic = dynamic_rate(cycle - 1)
     staked_ratio = staked_ratio(cycle)
     if staked_ratio < 0.48:
       delta_d = (0.48 - staked_ratio) * growth_rate * days_per_cycle
     elif staked_ratio > 0.52:
       delta_d = (0.52 - staked_ratio) * growth_rate * days_per_cycle
     else:
       delta_d = 0
     return previous_dynamic + delta_d

Where:

* "blocks_per_cycle" denotes the number of blocks in a Tezos cycle.

* "minimal_block_delay" denotes the minimal duration of a block in
  seconds.

* "days_per_cycle" denotes the minimal duration in days of a Tezos
  cycle, assuming all blocks in the cycle are produced at the minimal
  allowed time – that is, every 8 seconds in Quebec.

* "growth_rate" controls the speed at which the dynamic rate adjusts.
  The value is set so that a one percentage point deviation of the
  staked ratio changes the dynamic rate by 0.01 percentage points per
  day.

In a nutshell, "dynamic_rate(c)" increases and decreases by an amount
proportional to the distance between the target rate and the interval
"[48%; 52%]". Note that to ensure that the issuance rate is kept
within the minimum and maximum bounds, the dynamic rate might be
adjusted accordingly. More precisely, if the issuance rate would
surpass the maximum issuance allowed for a given cycle, then
"dynamic_rate(c)" would be reduced to keep the issuance rate within
the bounds (this part of the formula has been omitted from the above
pseudocode for brevity).


Minimum and maximum rates
-------------------------

Starting in Paris, the minimum and maximum issuance rates will evolve
slowly over a set period of time, so that the range of possible
issuance rate values widens progressively.

The following figure describes the progressive maximum and minimum
values of Adaptive Issuance.

   [image]Figure 1. A gradual widening of the range ensures a smooth
   transition to Adaptive Issuance.

The schedule consists of three periods:

* an **initial** period, set to 1 month, where the minimum and maximum
  issuance rates are close to the issuance rate from the Oxford
  protocol (before the activation of Adaptive Issuance) and stay
  constant,

* a **transition** period, set to 5 months, where they evolve
  linearly, with a decreasing minimum, and an increasing maximum, and

* a **final** period where the minimum and maximum have reached their
  final values.

Formally, the functions for the minimum and maximum values are
piecewise linear functions of time, and can be generally defined as
follows:

   def compute_extremum(cycle, initial_value, final_value):
     trans = transition_period + 1
     initial_limit = ai_activation_cycle + initial_period
     trans_limit = initial_limit + trans
     if cycle <= initial_limit:
         return initial_value
     elif cycle >= trans_limit:
         return final_value
     else:
         t = cycle - initial_limit
         res = (t * (final_value - initial_value) / trans) + initial_value
         return res

Where:

* "ai_activation_cycle" is the first cycle with Adaptive Issuance
  active, which was cycle 748 on mainnet.

* "initial_period" is a predefined period of time, set to 1 month in
  Paris.

* "transition_period" is a predefined period of time, set to 5 months
  in Paris.

The issuance minimum rate for Adaptive Issuance curve is then defined
as follows.

   def minimum_rate(cycle):
     return compute_extremum(cycle, issuance_initial_min, issuance_global_min)

Where:

* "issuance_initial_min" (4.5%) is the initial minimum value. The
  issuance rate is kept above this bound for the initial period.

* "issuance_global_min" (0.25%) is the final value for the lower
  bound, reached at the end of the transition period.

The issuance maximum rate for Adaptive Issuance curve is then defined
as follows.

   def maximum_rate(cycle):
     return compute_extremum(cycle, issuance_initial_max, issuance_global_max)

Where:

* "issuance_initial_max" (5.5%) controls the initial maximum value.
  The issuance rate is kept below this bound for the initial period.

* "issuance_global_max" (10%) is the final value for the upper bound,
  reached at the end of the transition period.


Adaptive Maximum
----------------

In addition to the issuance bounds described above, the Quebec
protocol introduces another upper bound on the total issuance rate.
This new bound, called **adaptive maximum**, prevents the issuance
rate from taking undesirably high values when the staked ratio of the
chain is already close to the 50% target.

The adaptive maximum is defined as a function of the staked ratio that
decreases smoothly as the staked ratio grows closer to the 50% target,
down to a 1% maximal issuance rate when the staked ratio is at 50% or
above. See the blue curve on Figure 2, as well as a few sample values
in Figure 3. The exact formula is provided further below.

Thanks to the adaptive maximum, the total issuance rate (static rate +
dynamic rate) cannot be too far above the static rate (green dashed
curve). More precisely, the total issuance rate is forced to stay
between the static rate (green) and the adaptive maximum (blue). In
other words, the dynamic rate is effectively bounded to stay below the
red dotted curve, which plots the adaptive maximum minus the static
rate. Note that this no longer holds if the adaptive maximum gets
overridden by the minimum issuance, as discussed below.

   [image]Figure 2. Adaptive maximum compared to the static rate, in
   the range from 5% to 50% staked ratio.


Figure 3. Adaptive maximum: a few data points.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Staked ratio   | 5% and below   | 10%            | 20%            | 30%            | 40%            | 50% and above  |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Adaptive       | 10%            | 9.2%           | 5.6%           | 3%             | 1.5%           | 1%             |
| maximum        |                |                |                |                |                |                |
| issuance       |                |                |                |                |                |                |
| (approx.)      |                |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+

The function that defines the adaptive maximum is:

   def adaptive_maximum(r): # r is the staked ratio of the chain
     if r >= 0.5:
       return 0.01
     elif r <= 0.05:
       return 0.1
     y = (1 + 9 * ((50 - 100 * r) / 42 ) ** 2 ) / 100
     if y > 0.1:
       return 0.1
     elif y < 0.01:
       return 0.01
     else:
       return y

Note:

  Until the final value of the minimum rate is reached, it is
  possible, with a high enough staked ratio, for the corresponding
  adaptive maximum to be smaller than the minimum rate. If this
  happens, then the minimum rate takes priority, that is, the total
  issuance rate (static + dynamic) is set to the minimum rate exactly.
  This way, the total issuance rate is as close to the adaptive
  maximum as can be while fully satisfying the minimum rate.


Issuance rate
-------------

Finally, as mentioned before, the nominal adaptive issuance rate [1]
for a cycle "c + issuance_modification_delay + 1" is defined as the
sum of the static rate and the dynamic rate computed for the cycle
"c", bounded within the minimum and maximum rates, along with the
adaptive maximum, computed for the cycle "c + 1".

   def issuance_rate(cycle):
     adjusted_cycle = cycle - issuance_modification_delay
     static_rate = static_rate(adjusted_cycle - 1)
     dynamic_rate = dynamic_rate(adjusted_cycle - 1)
     minimum_rate = minimum_rate(adjusted_cycle)
     maximum_rate = min(maximum_rate(adjusted_cycle), adaptive_maximum(stake_ratio(adjusted_cycle)))
     total_rate = static_rate + dynamic_rate
     return max( min(total_rate, maximum_rate), minimum_rate )


Adaptive rewards
================

Before Adaptive Issuance activation, participation rewards were fixed
values defined by protocol constants. With the new mechanism, the
adaptive issuance rate provides instead a budget for the whole cycle,
which gets allocated equally to each block of the cycle and
distributed between the various rewards, in proportion to their
relative weights.


Reward weights
--------------

The weights for participation rewards are defined as:

* Attestation rewards: 10,240.

* Fixed baking reward: 5,120.

* Bonus baking reward: 5,120.

* DAL rewards: 2,275.

* Nonce revelation tip: 1.

* VDF tip: 1.

The total sum "sum_rewards_weight" of all weights is 22,757.

   sum_rewards_weight = (
     attestation_rewards +
     fixed_baking_rewards +
     bonus_baking_rewards +
     dal_rewards +
     nonce_revelation_tip +
     vdf_tip)

The coefficient to apply for reward computation is defined as follows.

   def reward_coeff(cycle):
     rate = issuance_rate(cycle)
     total_supply = total_supply(cycle - issuance_modification_delay - 1)
     return (rate / 525600) * total_supply / base_total_issued_per_minute

Where:

* "base_total_issued_per_minute" (80007812) is the expected amount of
  mutez issued per minute.

* 525600 is the number of minutes per year.

The issuance per block is then distributed amongst the different
rewards in proportion to their weight.

   def tez_from_weights(weight):
     num = weight * minimal_block_delay
     den = sum_rewards_weight * 60
     return base_total_issued_per_minute * num / den

   def reward_from_constants(cycle, weight):
     return tez_from_weights(weight) * reward_coeff(cycle)

**Consensus rewards.** Since the adoption of Tenderbake, Tezos
protocols before Paris have rewarded delegates for their participation
in consensus with the following rewards per block:

* A fixed **baking** reward, given to the delegate which produced the
  *payload* of the block (i.e. choosing transactions, and other non-
  consensus operations).

* A variable, baking **bonus** reward given to the delegate which
  produced the block included in the chain. This bonus is given for
  including attestations, if their combined attesting power exceeds
  the minimal threshold (two thirds of total slots).

* A *collective* **attestation** reward, for attesting block
  proposals, distributed at the end of the cycle to the delegates
  selected in the consensus committees for that cycle, proportionnaly
  to their expected participation.

We refer to the consensus page for further insight on the pre-
requisites and distribution of these rewards. Here, we derive the new
functions which compute their values per block.

   def baking_reward_fixed_portion(cycle):
     return reward_from_constants(cycle, fixed_baking_rewards)

   def baking_reward_bonus_per_slot(cycle):
     bonus_committee_size = consensus_committee_size - consensus_threshold
     return reward_from_constants(cycle, bonus_baking_rewards) / bonus_committee_size

   def attestation_reward_per_slot(cycle):
     return reward_from_constants(cycle, attestation_rewards) / consensus_committee_size

Where:

* "consensus_committee_size" (7000) is the number of attestation slots
  available in a block.

* "consensus_threshold" (4667) is the required number of attestations
  for a baker to propose a block.

**Nonce and VDF revelation tips.** The rewards allocated to delegates
for contributing to random seed generation (that is, for revealing
nonce seeds and posting VDF proofs) are not paid each block, but
rather every 192 blocks.

   def seed_nonce_revelation_tip(cycle):
     return reward_from_constants(cycle, nonce_revelation_tip * blocks_per_commitment)

   def vdf_revelation_tip(cycle):
     return reward_from_constants(cycle, vdf_tip * blocks_per_commitment)

Where:

* "blocks_per_commitment" (192) is the interval in blocks between each
  revelation, both VDF and nonce.

The RPC endpoint, "/issuance/expected_issuance" reports the
precomputed values of all participation rewards for the provided block
and the next "issuance_modification_delay" cycles.

[1] Note that if the nominal annual issuance rate is r, the annualized
    rate is close to \IL{\exp{r} - 1} as it is compounded at every
    cycle.

Note:

  For information about how to stake, see Staking on
  docs.tezos.com.For operational details about the staking mechanism
  and its configuration, see a staking mechanism tutorial.


Staking mechanism
*****************

Staking is an evolution of the existing Tezos Liquid Proof-of-Stake
mechanism. It introduces a new role for network participants, called
**staker**, complementary to the existing delegate (also known as
*baker*) and *delegator* roles. A staker must also be a *delegator* –
that is, they must first choose a delegate.

When stakers **stake** funds towards a delegate’s **staking**
**balance**, the associated **baking** and **voting powers** accrue to
that delegate. Similarly to how delegated funds work, staked funds
remain within the staker’s account at all times.

Staked and delegated funds **have different weights** in the
computation of delegates’ baking and voting powers: staked funds (both
external stakes by stakers and the delegate’s own) count **three
times** as much as delegated funds. This ratio is defined by the
protocol constant "EDGE_OF_STAKING_OVER_DELEGATION".

Unlike delegated funds, staked funds are considered to contribute to
the security deposit associated with their chosen delegate. Thus, they
are subject to slashing if the delegate misbehaves by double-signing
block proposals or consensus operations, and are subject to the same
withdrawal delays – colloquially, they are "frozen".

Stakers are slashed proportionally to their contribution to the
delegate’s staking balance. To simplify slashing, double-baking
penalties are now proportional to staked funds: instead of the
previous fixed sum of 640 tez they are now set to 5% of the delegate’s
stake. Moreover, denunciation rewards (both for double-baking and
double-attestations) are reduced from one half to one eleventh of the
slashed funds. The chosen value prevents adversarial delegates from
abusing the slashing mechanism for profit at the expense of their
stakers.

Participation rewards are automatically shared between delegates and
their stakers. Staker's rewards are proportional to their
participation in the delegate's total staked at the time the rewards
are given. This means that the staker gets rewards for staked tez as
soon as they are staked, and stops receiving rewards as soon as the
tez are unstaked, disregarding the fact that baking rights for the
delegate are computed with some delays. *Delegates* configure their
staking policy by setting staking parameters which regulate whether
they accept stakers (the default being to reject them), and if so, up
to which fraction of their total staking balance. They can also
configure which proportion of the staking rewards from other stakers
is set to accrue to their own staked balance instead, thereby
collecting an *edge* from the rewards attributable to their stakers.

Freezing and unfreezing of staked funds is controlled directly by
delegates and stakers. This entails that staked funds are frozen until
manually unfrozen by stakers. This is a two step process which spans
for at least "UNSTAKE_FINALIZATION_DELAY" cycles (cf. Staked funds
management).

A user interface is provided for delegates and stakers to interact
with the mechanism. It is based on four *pseudo-operations*: "stake",
"unstake", "finalize_unstake", and "set_delegate_parameters". Pseudo-
operations are self-transfers: a transfer operation where the
destination matches the source – each involving a special entry-point
of the same name introduced for user accounts. This approach was
chosen to minimize the work required by wallets, custodians,
exchanges, and other parties to support the functionality.

**NB** In the current implementation, only *user accounts* can become
stakers. In other words, smart contracts cannot stake funds (they can
of course still delegate them).


Staking policy configuration
============================

*Delegates* can configure their staking policy by setting the
following parameters:

* "edge_of_baking_over_staking": a ratio between 0 and 1, whose
  default value is 1. This parameter determines the fraction of the
  rewards that accrue to the delegate's frozen deposit – the remainder
  is shared among its stakers.

* "limit_of_staking_over_baking": a non-negative number, denoting the
  maximum portion of external stake by stakers over the delegate’s own
  staked funds. It defaults to 0 – which entails that delegates do not
  accept external stakes by default. It is moreover capped by a global
  constant, set to 9 starting in the Quebec protocol, which ensures
  the baker controls a significant part of the stake.

Delegates can modify these staking parameters at all times, using the
"set_delegate_parameters" pseudo-operation: that is, by transferring 0
tez to their own "set_delegate_parameters" entry-point. The chosen
values for both parameters need to be supplied. The new parameters are
then applied "DELEGATE_PARAMETERS_ACTIVATION_DELAY" (currently 5)
cycles later.

   octez-client transfer 0 from <delegate> to <delegate> --entrypoint set_delegate_parameters --arg "Pair <limit as int value in millionth> (Pair <edge as int value in billionth> Unit)"

or more conveniently:

   octez-client set delegate parameters for <delegate> --limit-of-staking-over-baking <value> --edge-of-baking-over-staking <value>

Once you have set your parameters, if you want to update only one
parameter you can use:

   octez-client update delegate parameters for <delegate> [--limit-of-staking-over-baking <value>] [--edge-of-baking-over-staking <value>]

The command will fill the missing parameter with the value of the last
pending parameters (or the active parameters if no set/update are
pending).

Please ensure to not send two updates in a row without waiting for the
first update operation to be included (i.e. do not use --wait none) or
you might end up with inconsistent parameters (The modification of one
parameter could be reverted by the subsequent modification of another
parameter).

**On overstaking and overdelegation.** Note that if a delegate’s
"limit_of_staking_over_baking" is exceeded (that is, the delegate is
*overstaked*), the exceeding stake is automatically considered as
*delegation* for the delegate’s baking and voting power calculation,
but it does remain slashable. The new mechanism does not alter
*overdelegation* (delegated funds beyond 9 times the delegate’s own
stake) nor its consequence on voting and baking powers. That is,
overdelegated funds are not counted towards a delegate baking power,
but they do increase their voting power.


Staked funds management
=======================

Stakers (and delegates) can use the "stake", "unstake", and
"finalize_unstake" pseudo-operations to control their stakes. Figure 1
illustrates their effect on a staker’s funds. Note that while these
pseudo-operations change the *state* of the involved funds, they
remain otherwise within the staker’s account at all times.

Note:

  Due to rounding that occurs at various stages of fund management,
  stakers' staked balances may deviate by a few mutez (millionth of
  tez) with respect to transferred amounts.

   [image]Figure 1: staked funds management using pseudo-operations.

To *stake* funds, a delegator uses the "stake" pseudo-operation,
transferring the chosen amount of **spendable** tez to their own
"stake" entry-point. The **staked** tez will then be frozen and
contribute to their chosen delegate’s staking balance. Note that the
"stake" pseudo-operation will fail if the sender account is not
*delegated*.

   octez-client transfer <amount> from <staker> to <staker> --entrypoint stake

or more conveniently:

   octez-client stake <amount> for <staker>

The staked amount and remaining spendable funds can then be checked
respectively with:

   octez-client get staked balance for <staker>
   octez-client get balance for <staker>

To *unstake* funds, a staker first submits an unstake request with the
"unstake" pseudo-operation. This is implemented by transferring the
chosen amount in tez to their "unstake" entry-point:

   octez-client transfer <amount> from <staker> to <staker> --entrypoint unstake

or more conveniently:

   octez-client unstake <amount|"everything"> for <staker>

The requested amount will be **unstaked** but will remain **frozen**,
a.k.a. **unfinalizable**.

After "UNSTAKE_FINALIZATION_DELAY + 1" cycles (more precisely, after
the cycle in which the unstake was requested has ended and then
another UNSTAKE_FINALIZATION_DELAY full cycles have passed), unstaked
frozen tokens are no longer considered at stake nor slashable. They
are said then to be both **unstaked** and **finalizable**.

A staker can retrieve all unstaked and finalizable tokens at any time,
making them spendable again. This is done using the "finalize_unstake"
entrypoint -– that is, by transferring 0 tez to their
"finalize_unstake" entry-point:

   octez-client transfer 0 from <staker> to <staker> --entrypoint finalize_unstake

or more conveniently:

   octez-client finalize unstake for <staker>

Note that starting with protocol S, not only the staker, but anyone
can trigger "finalize_unstake" (in any case, the unfrozen funds still
go to the staker).

In some circumstances, unstake and finalize can be done implicitly:
any call to "stake" or "unstake" will implicitly finalize all
currently finalizable pending unstake requests. Also, as we will see
next, change of delegate triggers an unstake operation.


Change of delegate
==================

When a staker changes its delegate, the operation will trigger an
implicit unstake request for the full frozen deposit of the staker.

As long as the unstake request is not finalized, the frozen tokens
will continue to be delegated to the old delegate, however the
spending balance of the account is accounted in the new delegate's
stake. It will not be possible to stake with the new delegate as long
as there are unfinalizable unstake request for token staked with the
old delegate.

Adaptive Slashing
*****************


Overview
========

In Oxford2, each instance of a baker's misconduct results in the
slashing of a fixed percentage of their staked funds. However, this
approach does not distinguish between innocent mistakes and deliberate
attacks. The rule conservatively punishes every misconduct with strong
penalties. The rationale for this model is its simplicity and the
observation that the cost of avoiding these mistakes is low anyway.
Nonetheless, it overlooks the impact of double signing of attestations
on the entire block committee, treating all cases uniformly and
ignoring collusion among bakers.

To better reflect this distinction, protocols starting with Paris
adjusts the amount of slashing based on the fraction of double-
attestation in a single block. A low fraction of misconduct incurs
moderate penalties, while a high fraction of misconduct is deemed to
be critical and faces more serious repercussions.

This document presents the definition of the adaptive slashing
function implementing this idea, as well as the new forbidden period.


Adaptive Slashing Function
==========================


Informal presentation
---------------------

For a given block, we call “fraction of double-attestations” the ratio
between the total weight of attestations for which a valid
denunciation has been included and the total weight of possible
attestations in a block.

The shape of the slashing function for double attestations is the
following:

   [image]

+--------------------------------------------+------------------------------------------+
| f: fraction of double attestations         | S: slashing ratio                        |
|============================================|==========================================|
| 1.00%                                      | 0.09%                                    |
+--------------------------------------------+------------------------------------------+
| 5.00%                                      | 2.25%                                    |
+--------------------------------------------+------------------------------------------+
| 10.00%                                     | 9.00%                                    |
+--------------------------------------------+------------------------------------------+
| 20.00%                                     | 36.00%                                   |
+--------------------------------------------+------------------------------------------+
| 23.57%                                     | 50.00%                                   |
+--------------------------------------------+------------------------------------------+
| 33.34%                                     | 100.00%                                  |
+--------------------------------------------+------------------------------------------+
| 100.00%                                    |                                          |
+--------------------------------------------+------------------------------------------+

Instead of using a constant function as in Oxford2, we use a convex
function that saturates at 100% when a critical fraction of doubled
attestations are issued. Accidental double attestations are unlikely
to cause a large amount of slashing to be applied, but concerted
attacks result in severe penalties.

**Remark 1.** Even though the baker does not attest to a weight
exactly equivalent to its staked funds at each block, the slashing
function is applied to bakers’ total staked funds for simplicity.

**Remark 2.** As in Oxford2, slashing happens at the end of each
cycle. Since the denunciation period for a block ranges over its cycle
and the next one, a baker can be punished for misbehaving during the
previous cycle, not only the one that has just ended.

**Remark 3.** If a baker wants to get back their at-stake funds, it
takes more than 2 cycles to complete the unstaking process. This
ensures that the baker can't decrease their at-stake funds after being
denunciated and before facing penalties.


A formal definition of slashing function for double-attestations
----------------------------------------------------------------

* \mathcal{W} denotes the maximal possible *weight* of attestations in
  a block, that is, the fixed number of available slots in any block.
  It is also known as CONSENSUS_COMMITTEE_SIZE.

* f(B) is the *fraction of double attestations* for block B, that is,
  the ratio of the total weight of double attestations in B, over
  \mathcal{W}.

* T is the *threshold* for the fraction of double attestations to be
  considered critical. A typical value for T is {1 \over 3}
  \mathcal{W}, which is the difference between \mathcal{W} and the
  CONSENSUS_THRESHOLD which is set to {2 \over 3} \mathcal{W}.

We define S(B) the percentage of slashed funds for all misbehaving
bakers at the block B as follows:

S(B) = \text{min} (100\%, {1 \over T^2} \cdot f(B)^2 \cdot 100\%)

Then, the percentage of slashed funds S(b,C) for a baker b at the end
of the cycle C is defined as follows:

S(b, C) = \text{min} (100\%, \sum_{(b, B) \in C} S(B))

where (b, B) \in C means that:

* the double attestation by baker b of block B has been denounced
  before the end of cycle C, and

* C is the last cycle of the denunciation period for B.


New definition for the forbidden period
=======================================

Given that slashing occurs with a delay, immediate action at
denunciation time is necessary upon clear evidence of a baker's
misbehavior to prevent further misconduct, or to protect the baker
against their own faulty setup. Any double-signing denunciation
immediately triggers the beginning of a **forbidden period** that
lasts at least 2 cycles, to make sure the slashing occurs before
accepting new attestations or blocks from the baker.

Note that it is still possible for one baker to commit multiple double
signings, but only if they all happen before any corresponding
denunciation gets included in a block.

This forbidding is lifted as soon as both following conditions are
met:

* all pending slashings for the delegate have occurred, and

* the current total frozen stake for the delegate (sum of the staking
  balances of the delegate itself and its stakers) is at least as high
  as the active stake that was used "CONSENSUS_RIGHTS_DELAY" cycles
  ago to compute the consensus rights for the next cycle.

The second condition may be fulfilled when the delegate and/or stakers
stake additional funds so that the total frozen stake grows back to
its pre-slashing value, thus matching the rights computed before the
slashing. Or it may be fulfilled "CONSENSUS_RIGHTS_DELAY" cycles after
the slashing, when the rights for the next cycle are finally based on
the post-slashing stake.

DAL integration
***************

The DAL's integration within the economic protocol relies on three
operations:

1. "DAL_publish_commitment": a manager operation, allowing anyone to
   publish a DAL commitment

2. "attestation": the existing consensus operation, allowing bakers to
   attach a DAL payload attesting the data seen on the DAL P2P network

3. "DAL_entrapment_evidence": an anonymous operation to denounce a
   baker that has attested a trap shard

and on an incentives scheme for the DAL.


DAL publish commitment
======================

"DAL_publish_commitment" is a manager operation that can be issued by
a user wishing to publish data onto the DAL. The payload of this
operation consists in the following fields:

* Slot index: Identifies the specific slot for which the data is being
  published. It is an integer between "0" and "NUMBER_OF_SLOTS - 1".

* Commitment: The KZG commitment over the data.

* Commitment proof: A proof that the commitment commits over data that
  does not exceed the size "SLOT_SIZE".

Users can create and manage these commitments and proofs through the
DAL node using these RPCs:

* To create a commitment: "POST /commitment"

* To retrieve a commitment’s proof: "GET
  /commitments/<commitment>/proof"


Concurrent operations
---------------------

If a block contains a valid "DAL_publish_commitment" operation, any
subsequent operations of the same kind within the same block (and the
same slot index) will be recognized as valid but will fail during
execution. They still incur transaction fees. More details about this
can be found in the validator documentation.


Economics
---------

Currently, the fees are estimated based on the execution cost of this
operation alone. There are no additional charges related to the
bandwidth required for bakers to download data from the DAL for this
commitment. However, this might be subject to changes in the future.


DAL attestation payloads
========================

The attestation operation includes an optional field "dal_content".
This field allows attesters participating to the DAL to announce
whether they were able to successfully download the shards assigned to
them. Concretely, this field is a bitset reflecting the status of each
slot. The size of the bitset corresponds to the total number of slots.
A value of 1 indicates successful retrieval of all assigned shards by
the baker for that slot, while 0 indicates an unsuccessful attempt.
The least significant bit corresponds to the smallest slot index.


Attestation timing
------------------

When a commitment is published at a certain level, say level "n", the
corresponding DAL payloads are expected to be included in the
attestations contained in the block at level "n + ATTESTATION_LAG".


Block metadata
--------------

In the block’s metadata, there is a specific field for the DAL, called
""dal_attestation"". This field reflects the availability of slots
based on the DAL payloads received. It is a bitfield with one bit per
slot (its format is the same as the attestation payload of the
"attestation" operation). The bit is set to 1 if the slot is declared
available. The smallest slot index corresponds to the least
significant bit. To consider a slot as available, there must be a
minimum number of shards, as defined by the "AVAILABILITY_THRESHOLD"
parameter, marked as available by the attesters for that slot (e.g. if
the number of shards is 2048 and the availability threshold is 50%,
then 1024 shards are required).

Therefore, for data committed (published) at level "n", the slot's
availability is determined by the metadata of the block at level "n +
ATTESTATION_LAG". Consequently, a smart rollup can only utilize this
data from level "n + ATTESTATION_LAG + 1" onward.


DAL incentives scheme
=====================


Overview
--------

Bakers must meet a 64% minimal participation threshold in a cycle to
earn a fixed percentage of the total participation rewards allocated
for them. As part of participation rewards, the DAL rewards are
subject to the adjustments done by Adaptive Issuance.

To ensure DAL attestations match the actual availability of data
shards, there are special shards, known as *traps*, which are designed
to test whether bakers have genuinely downloaded and processed their
assigned shards. Bakers must correctly identify these traps to avoid
losing the DAL rewards allocated to them.


Minimal participation
---------------------

Any baker that can take part in consensus is eligible for rewards.

Bakers meeting the minimum participation ratio
"MINIMAL_PARTICIPATION_RATIO" over a cycle, set to 64%, receive
rewards for that cycle, provided that they receive consensus
attestation rewards for that cycle.

The participation ratio of the baker is the proportion of slots the
baker attested over the slots that were attestable by this baker
during the cycle. Both slot numbers only count those slots that are
deemed available by the protocol.

A new RPC "GET
/chains/main/blocks/<block>/delegates/<pkh>/dal_participation" can be
used to obtain the DAL participation information for a given baker.
The output contains the following fields:

* "expected_assigned_shards_per_slot": the number of shards assigned
  to this baker (per slot) in the current cycle

* "delegate_attested_dal_slots": the number of DAL slots the baker has
  attested so far

* "delegate_attestable_dal_slots": the number of DAL slots that are
  attestable by the baker

* "expected_dal_rewards": the rewards (in mutez) the delegate will
  receive if it meets the participation threshold and is not denounced

* "sufficient_dal_participation": a Boolean flag indicating if the
  delegate’s current attestation rate meets the threshold (64%) to
  qualify for rewards

* "denounced": a Boolean flag indicating if the delegate has been
  denounced in the current cycle.

A delegate's participation is the ratio between
"delegate_attested_dal_slots" and "delegate_attestable_dal_slots".

All fields except "expected_assigned_shards_per_slot" and
"expected_dal_rewards" may vary depending on the moment of the call
during a cycle.

As an example (with made up numbers), let's consider the following
output:

   {
     "expected_assigned_shards_per_slot": 409,
     "delegate_attested_dal_slots": 2,
     "delegate_attestable_dal_slots": 5,
     "expected_dal_rewards": "1278125",
     "sufficient_dal_participation": false,
     "denounced": false
   }

corresponding to the following scenario. Suppose there are five
delegates with equal baking power, "8" blocks per cycle, and "256"
shards per slot. The delegate is expected to be assigned 409 shards
per slot (because "256 * 8 / 5 ≈ 409"). If the delegate reached "64%"
participation, it would receive "1,278,125" μꜩ. However, the delegate
has only attested "2" out of "5" available slots ("~40%"), which is
currently below the "64%" threshold and disqualifies it from rewards
unless it improves participation within this cycle.


DAL participation rewards
-------------------------

A fixed percentage, defined by a protocol parameter called
"REWARDS_RATIO", set to 10%, of the total participation rewards is
allocated to the DAL.

The DAL rewards per level are implicitly given by their weight,
"DAL_REWARDS_WEIGHT", as for the other types of participation rewards.
The value of "DAL_REWARDS_WEIGHT" is such that it represents
"REWARDS_RATIO" of all reward weights.

The rewards are distributed at the end of a cycle, and are computed in
the same manner as for the other participation rewards. For instance,
the stakers' share of these reward is proportional to the weight of
their stake in relation to their baker's baking power.

The metadata of the last block of a cycle contains the balance updates
corresponding to the allocated DAL rewards for that cycle. These
balance updates are identified by two categories for DAL rewards,
analogous to consensus attestation rewards, namely:

* DAL attesting rewards (kind: ""minted"", category: ""DAL attesting
  rewards"") and

* lost DAL attesting rewards (kind: ""burned"", category: ""lost DAL
  attesting rewards"").

The RPC "GET
/chains/main/blocks/<block>/context/issuance/expected_issuance" has a
new field ""dal_attesting_reward_per_shard"" indicating the DAL reward
allocated for a single shard. A delegate’s total potential DAL rewards
in a cycle are then: "expected_dal_rewards =
expected_assigned_shards_per_slot * dal_attesting_reward_per_shard",
where the "expected_dal_rewards" and
"expected_assigned_shards_per_slot" are the values given by the
"dal_participation" RPC (see above).


Trap mechanism
--------------

A deterministic function "trap(pkh, shard)" returning a boolean flag
indicates whether a shard is a trap for a specific baker identified by
its public key hash ("pkh").

The protocol parameter "TRAPS_FRACTION" controls the fraction of
shards marked as traps.

Bakers detect traps by retrieving shard content via their DAL node and
applying the trap function. A trap invalidates the corresponding
attestation: the baker should not attest a slot if one of the slot’s
shards assigned to him is a trap.

The "DAL_entrapment_evidence" accusation operation can be used to
accuse a baker of wrongly attesting a slot due to an undetected trap.
This accusation operation includes the offending attestation operation
(either individual or part of an aggregate), the offending baker's
consensus slot, the wrongly attested DAL slot, and the undetected
shard.

As for double-signing accusations, any baker can include a DAL
accusation in its block. Accusations can be included during a period
of "DENUNCIATION_PERIOD" cycles after the misbehavior event, which is
that of the corresponding attestation operation.


Penalties
---------

A baker that is correctly accused, through an accusation operation
included in a block, loses their DAL rewards for the cycle containing
the block.


Smart Rollups integration
=========================

The DAL is integrated with smart rollups so that kernels can request
pages from the DAL via the reveal data channel. A smart rollup can
fetch any page from the DAL node if the commitment respects some
conditions:

* The commitment should have been published after the rollup
  origination (this constraint will be leveraged so that the kernel
  can request any commitment in the past)

* The commitment should not have been published in a level in the
  future after the level of the next commitment of the state (at most
  30 levels in the future). This constraint might be leveraged thanks
  to the internal messages "Start of Level" and "End of Level"
  (abbreviated as SOL and EOL), which allow the kernel to track the L1
  level, so that the kernel cannot request a page for a commitment
  published after the current L1 level.

If the kernel requests a page that does not satisfy the mentioned
conditions, then the rollup node must answer with an empty page
indicating there is no data. Similarly, if the kernel requests for a
commitment that was not attested, the rollup node must also provide an
empty page. As a consequence, if a kernel requests a page for a
commitment for which its availability is still unknown, the rollup
node cannot answer and will have to wait for the availability status
to be confirmed by the L1.

Moreover, the rollup kernel has access to the protocol constants so
that the same kernel code can be used on different test networks.


DAL-related protocol constants
==============================

This section describes the protocol constants specific to the DAL as
well as their default values on mainnet (see Protocol constants on how
to find the values for tests networks):

* "FEATURE_ENABLE" (true): whether the DAL is available

* "INCENTIVES_ENABLE" (true): whether baker incentives are available

* "NUMBER_OF_SLOTS" (32): how many slots are available per block

* "ATTESTATION_LAG" (8 level): the timeframe for bakers to download
  shards between the published level of a commitment and the time they
  must attest the availability of those shards

* "ATTESTATION_THRESHOLD" (66%): the minimum percentage of shards
  attested for a given slot to declare the slot available

* "PAGE_SIZE" (3967B, ~4KiB): the size of a page (see DAL slots)

* "SLOT_SIZE" (126944B, ~128KiB): the size of a slot (see DAL slots)

* "REDUNDANCY_FACTOR" (8): the erasure-code factor (see DAL slots)

* "NUMBER_OF_SHARDS" (512): the number of shards per slot (see DAL
  slots)

* "MINIMAL_PARTICIPATION_RATIO" (64%): the minimum percentage of slots
  attested by a baker during a cycle (among all slots deemed
  available) that entitles them to rewards

* "REWARDS_RATIO" (10%): the ratio of the DAL rewards over the total
  participation rewards

* "DAL_REWARDS_WEIGHT" (2275): the weight of the DAL rewards (relative
  to other participation rewards)

* "TRAPS_FRACTION" (0.0005): the fraction of shards that are traps

Glossary
********

This glossary is divided in two sections:

* The first section contains Tezos generic terms (i.e., which are not
  related to a particular protocol), such as Tezos or block.

* The second section contains terms defined by the economic protocol,
  such as Tez/XTZ, account, smart contract, or delegate. The list of
  terms in this section and/or their definitions may be different for
  other protocol versions.


Tezos terms
===========

   The Tezos blockchain is a linked list of blocks (or actually, a
   tree when several competing branches exist). Blocks conceptually
   contain a header and a list of operations, which are specific to
   the economic protocol.

   The header itself decomposes into a shell header (common to all
   Tezos economic protocols), and a protocol-specific header. The
   shell header contains protocol-agnostic data such as the
   predecessor's block hash and the block's timestamp.

   The state of the blockchain. The context is defined by the economic
   protocol and typically includes information such as “this account
   is credited with this many tez” and “this is the code for that
   smart contract.”

   The context is modified by operations. For example, an operation
   can transfer tez from one account to another, which modifies the
   part of the context that tracks account credit.

   The economic protocol is the set of rules defining valid operations
   and blocks, how the network agrees on the next block to build (the
   consensus algorithm), and how operations update the blockchain
   state, also called context.

   In Tezos, the economic protocol can be upgraded without
   interruption or forking of the blockchain. This is because the
   procedure for an upgrade is also defined within the economic
   protocol, which can thus update itself.

 (a.k.a. score, weight)
   For each block, the consensus algotrithm can compute a score called
   fitness which determines the quality of the chain leading to that
   block. The shell changes the head of the chain to the valid block
   that has the highest fitness.

   See level.

 (a.k.a. block height)
   The position of a block in the blockchain, that is, the number of
   blocks since the genesis block, where the genesis block is at level
   0.

   A pool (set) of operations maintained by a node and not yet
   included in a block.

   A (block or operation) metadata is a piece of data computed as a
   result of the application of the block or operation on an
   associated context. The metadata consists of many pieces of
   information such as the operation receipts, rewards updates, voting
   period, etc.

   A block's metadata is the collections of operations metadata for
   all the operations included in the block (if the validation was
   successful).

   For a detailed metadata content check the Seoul RPCs - Reference
   under the prefix "../<block_id>/metadata".

   A peer in the P2P network. It maintains a local state and
   propagates blocks and operations.

   An operation transforms the context; this is what makes the state
   of the chain change. Operations are grouped into blocks; thus, the
   chain progresses in batches. For the different kinds of operations
   defined by the protocol, see operation kinds.

   See fitness.

   The shell is a software component of the node. It is parameterized
   by a specific economic protocol. It serves as the bridge between
   the P2P layer (handling communication between nodes) and the
   economic protocol layer (handling the context, operation
   application, scoring, etc.).

   Used either as a noun or an adjective to designate:

   * An open-source project and software (as in, "contributing to the
     Tezos protocol")

   * A peer-to-peer network of nodes maintaining a blockchain (as in
     "a Tezos node")

   * The specific Tezos chain with the most economic relevance (as in
     "the Tezos chain"). In particular, the chain whose millionth
     block had hash
     BKtC4QCWoF73kxLj773vFpQuuwrnye6PS7T1aM3XEPvFXiQbNu7

   See fitness.


Protocol terms
==============

   When a delegate attempts double signing (or when it tries to abuse
   the network in another similar way), another delegate can make an
   accusation, by providing evidence of the offense. The delegate
   injecting the accusation in a newly baked block is called the
   accuser.

   The accuser is awarded some funds from the security deposit of the
   accused.

   When using Octez, accusation operations are emitted by the accuser
   daemon. Note that this daemon is not associated to a delegate:
   accusation operations are anonymous, and any delegate can include
   them in a block.

   An account is an address managed by the protocol. In the context,
   each account is associated with a balance (an amount of tez
   available).

   An account can be either a user account or a smart contract.

   When a block is created and propagated on the network, delegates
   that have attesting rights for the matching block level and round
   can emit an attestation operation. Attestation operations are
   included in the next block.

   See baking rights.

   When a delegate creates a new block, it is called the baker of this
   block. Baking rights are distributed to different delegates based
   on their baking power. Only a delegate with baking rights is
   allowed to bake. The baker selects transactions from the mempool to
   be included in the block it bakes.

   When using Octez, baking and other consensus actions are handled by
   the baker daemon, on behalf of one or more delegate accounts. By
   extension, a baker designates the owner of such a delegate account,
   typically running the baker daemon on its behalf.

   The act of creating a new block by a baker.

   The amount of tokens that determines a delegate's weight in the
   selection of its baking and attesting rights. A delegate's baking
   power is computed from the delegate's own tokens and the sums of
   tokens delegated and staked to it (where staked tokens weigh more
   than tokens just delegated). See Baking Power for details.

   Baking/attesting a block can only be done by a delegate who holds
   the baking/attesting right for that block level and round. At the
   start of a cycle, baking and attesting rights are computed for all
   the block levels and rounds in the cycle, based on the baking power
   of each delegate.

   For each block level and round, there is exactly one account that
   is allowed to bake, but several accounts are allowed to attest.

   To ensure responsible use of the storage space on the public
   blockchain, there are some costs charged to users for consuming
   storage. These costs are burnt (i.e., the amount of tez is
   destroyed). For example, a per-byte storage cost is burnt for
   increasing the storage space of a smart contract; a fixed amount is
   burnt for allocating a new contract (which consumes space by
   storing its address on the blockchain).

   See also fee.

   Protocols are parameterized by several parameters called protocol
   constants, which may vary from one protocol to another or from one
   network to another.

   A cycle is a sequence of consecutive blocks of fixed length (given
   by a protocol constant). E.g., cycle 12 started at block level
   49152 and ended at block level 53248.

   Cycles are used as a unit of “time” in the block chain. For
   example, the different phases in the amendment voting procedures
   are defined based on numbers of cycles.

   The length of a cycle is a (parametric) protocol constant, and thus
   might change across different Tezos protocols.

   A user account that can participate in consensus and in governance.
   Actual participation is under further provisions, like having a
   minimal stake. A user account becomes a delegate by registering as
   such. Through delegation, other accounts can delegate their rights
   to a delegate account.

   An operation in which an account designates a delegate. The
   delegating account's balance increases the delegate's baking power
   and consequently its baking rights and attesting rights; it also
   increases its voting power. However, the delegate does not control
   the funds of the delegating account, e.g., it can not spend them.

   The situation when a baker signs two different blocks at the same
   level and same round, is called double baking. Double baking is
   detrimental to the network and might be indicative of an attempt to
   double spend. The same goes for signing two different attestations
   at the same level and the same round. As such, double signing
   (i.e., double baking or double attesting) is punished by the
   network: an accuser can provide proof of the double signing to be
   awarded part of the double signer's deposit -- see Slashing.

   The "Failing_noop" operation implements a *No-op*, which always
   fails at application time, and should never appear in applied
   blocks. This operation allows end-users to sign arbitrary messages
   which have no computational semantics.

   To ensure responsible use of computation resources of other nodes,
   and also to encourage active participation in the consensus
   protocol, users pay fees to bakers for including their operations
   in blocks. For example, fees are paid to a baker for operations
   such as a transaction or a revelation of a public key.

   Currently, only manager operations require collecting fees from its
   sender account.

   See also burn.

   A measure of the number of elementary steps performed during the
   execution of a smart contract. Gas is used to measure how much
   computing power is used to execute a smart contract.

   See user account.

   The primary blockchain i.e. the Tezos chain. Within any blockchain
   ecosystem, Layer 1 (L1) refers to the main chain to which side
   chains, rollups, or other protocols connect and settle to. The
   Layer 1 chain is deemed to be most secure, since it has the most
   value (or stake) tied to it, and be most decentralized and
   censorship resistant. However, transaction space is limited leading
   to low throughput and possibly high transaction costs. See Layer 2.

   Layer 2 (L2) includes sidechains, rollups, payment channels, etc.
   that batch their transactions and write to the Layer 1 chain. By
   processing transactions on layer 2 networks, greater scalability in
   speed and throughput can be achieved by the ecosystem overall,
   since the number of transactions the Layer 1 can process directly
   is limited. By cementing transactions from a L2 to L1, the security
   of the L1 chain backs those operations. Currently, Layer 2
   solutions on Tezos are built as smart rollups.

   The built-in language used by a smart contract.

   An amount of tez (e.g., 6000ꜩ) serving as a minimal amount for a
   delegate to have baking rights and voting rights in a cycle.

   The main kinds of operations in the protocol are transactions (to
   transfer funds or to execute smart contracts), accusations,
   activations, delegations, attestations, and originations. For the
   full list of operations, see Blocks and Operations.

   See smart contract.

   A manager operation whose purpose is to create -- that is, to
   deploy -- a smart contract on the Tezos blockchain.

   A PVM (Proof-generating Virtual Machine) is a reference
   implementation for a device on top of which a smart rollup can be
   executed. This reference implementation is part of the economic
   protocol and is the unique source of truth regarding the semantics
   of rollups. The PVM is able to produce proofs enforcing this truth.
   This ability is used during the final step of a refutation game.

   A process by which the economic protocol solves a conflict between
   two rollup committers. Note that the refutation mechanism used in
   Tezos smart rollups corresponds to the notion of fraud proofs used
   in other blockchain/Layer 2 ecosystems.

   When the first rollup commitment for a rollup commitment period is
   published, a refutation period of two weeks starts to allow this
   commitment to be challenged.

   deprecated; see minimal stake.

   A claim that the interpretation of all rollup inbox messages
   published during a given period, and applied on the state of a
   parent rollup commitment, led to a given new state by performing a
   given number of execution steps of the PVM.

   A period of roughly 15 minutes during which all rollup inbox
   messages must be processed by the rollup node state to compute a
   rollup commitment. A commitment must be published for each
   commitment period.

   A user account that has published and made a deposit on a rollup
   commitment.

   A sequence of messages from the Layer 1 to all the smart rollups.
   The contents of the inbox are determined by the consensus of the
   economic protocol.

   A daemon required for deploying and operating smart rollups. The
   rollup node is responsible for making the rollup progress by
   publishing rollup commitments and by playing refutation games.

   A sequence of messages from a smart rollup to the Layer 1. Messages
   are smart contract calls, potentially containing tickets. These
   calls can be triggered only when the related rollup commitment is
   cemented (hence, at least two weeks after the actual execution of
   the operation).

   An attempt to reach consensus on a block at a given level. A round
   is represented by an index, starting with 0. Each round corresponds
   to a time span. A baker with baking rights at a given round is only
   allowed to bake during the round's corresponding time span. Baking
   outside of one's designated round results in an invalid block.

   Account which is associated to a Michelson script. They are created
   with an explicit origination operation and are therefore sometimes
   called originated accounts. The address of a smart contract always
   starts with the letters "KT1".

   Smart rollups constitute a Layer 2 solution that can be used to
   deploy either a general-purpose polyvalent Layer 2 blockchain
   (e.g., an EVM-compatible one), or an application-specific DApp. See
   Smart Optimistic Rollups.

   A user account that made a security deposit. The user account must
   have set a delegate. The security deposit accrues to the stake of
   the user account's delegate and is subject to slashing in case the
   delegate misbehaves -- see Slashing.

   A unit of the cryptocurrency native to a Tezos chain, such as in "I
   sent you 2 tez." Tez is invariable. It is not capitalized except at
   the beginning of a sentence or when you would otherwise capitalize
   a noun. See also XTZ.

   An operation to transfer tez between two accounts, or to run the
   code of a smart contract.

   An account that is linked to a public key. Contrary to a smart
   contract, a user account cannot include a script and it cannot
   reject incoming transactions. User accounts are sometimes called
   "implicit accounts".

   If *registered*, a user account can act as a delegate.

   The address of a user account always starts with the letters "tz"
   followed by "1", "2", "3", or "4" (depending on the signature
   scheme) and finally the hash of the public key. See Accounts and
   addresses for a more detailed explanation on addresses.

   An index (a natural number) associated with a particular kind of
   operations, allowing to group them into classes. Validation passes
   enable prioritizing the validation and application of certain
   classes of operations.

   Any of the "proposal", "exploration", "cooldown", "promotion" or
   "adoption" stages in the voting procedure when amending the
   economic protocol.

   The amount of tokens that determines a delegate's weight in the
   voting process. A delegate's voting power is computed from the
   delegate's own tokens and the sum of tokens delegated to it. See
   Voting Power for details.

   The list calculated at the beginning of each voting period that
   contains the staking balance (in number of mutez) of each delegate
   that owns more than the minimal stake at that moment. For each
   delegate, the voting listings reflect the weight of the vote
   emitted by the delegate when amending the economic protocol.

   XTZ, tez, or ꜩ ("\ua729", "Latin small letter tz") is the native
   currency of Tezos.

   "XTZ" is an ISO-4217-compatible code for representing tez on the
   most economically relevant Tezos chain. Unless there is a very
   specific reason to use an ISO code for it, the term tez is
   preferred. Situations where the ISO code might be useful typically
   involve accounting systems, exchange rates with other currencies,
   and anything that might need some sort of standardized code.

Seoul RPCs - Reference
**********************

This page describes the RPCs specific to the particular version of the
Tezos protocol specified above. (The protocol-independent RPCs are
described in this other page.)

Warning:

  This list could be missing RPC endpoints. The OpenAPI specification
  may be used to retrieve the complete list of protocol RPCs and their
  associated schemas (search the ".json" files that are linked from
  that page).

Note that the RPCs served under a given prefix can also be listed
using the client, e.g.:

   octez-client rpc list /chains/main/blocks/head/context/constants

Any RPC endpoint may also be described, using the "describe" RPC to
retrieve all JSON and binary schemas, e.g.:

   octez-client rpc get /describe/chains/main/blocks/head/context/constants

Note:

  The protocol-specific RPCs are all served under the prefix
  "/chains/<chain_id>/blocks/<block_id>/". To make the RPC reference
  more readable, this prefix is not repeated every time below, but
  instead shortened as "../<block_id>".

The "<block_id>" can be:

1. Block-hash - Hash in Base58Check notation.

2. Alias - One of the following: "genesis" / "head" / "caboose" /
   "savepoint" / "checkpoint".

3. Block-level - as an integer

   One can also specify the relative positions of the block with
   respect to the above block identifiers. For ex. "head~N" or
   "<Block-hash>+N", where N is an integer, denotes the Nth block
   before (~) or after (+) the given block identifier, respectively.


Index
=====

* ../<block_id> (GET)

  * ../<block_id>/context

    * ../<block_id>/context/adaptive_issuance_launch_cycle (GET)

    * ../<block_id>/context/big_maps

      * ../<block_id>/context/big_maps/<big_map_id> (GET)

        * ../<block_id>/context/big_maps/<big_map_id>/<script_expr>
          (GET)

          * ../<block_id>/context/big_maps/<big_map_id>/<script_expr>
            /normalized (POST)

    * ../<block_id>/context/cache

      * ../<block_id>/context/cache/contracts

        * ../<block_id>/context/cache/contracts/all (GET)

        * ../<block_id>/context/cache/contracts/rank (POST)

        * ../<block_id>/context/cache/contracts/size (GET)

        * ../<block_id>/context/cache/contracts/size_limit (GET)

    * ../<block_id>/context/constants (GET)

      * ../<block_id>/context/constants/errors (GET)

      * ../<block_id>/context/constants/parametric (GET)

    * ../<block_id>/context/contracts (GET)

      * ../<block_id>/context/contracts/<contract_id> (GET)

        * ../<block_id>/context/contracts/<contract_id>/all_ticket_ba
          lances (GET)

        * ../<block_id>/context/contracts/<contract_id>/balance (GET)

        * ../<block_id>/context/contracts/<contract_id>/balance_and_f
          rozen_bonds (GET)

        * ../<block_id>/context/contracts/<contract_id>/big_map_get
          (POST)

        * ../<block_id>/context/contracts/<contract_id>/counter (GET)

        * ../<block_id>/context/contracts/<contract_id>/delegate (GET)

        * ../<block_id>/context/contracts/<contract_id>/entrypoints
          (GET)

          * ../<block_id>/context/contracts/<contract_id>/entrypoints
            /<entrypoint> (GET)

        * ../<block_id>/context/contracts/<contract_id>/estimated_own
          _pending_slashed_amount (GET)

        * ../<block_id>/context/contracts/<contract_id>/frozen_bonds
          (GET)

        * ../<block_id>/context/contracts/<contract_id>/full_balance
          (GET)

        * ../<block_id>/context/contracts/<contract_id>/manager_key
          (GET)

        * ../<block_id>/context/contracts/<contract_id>/script (GET)

          * ../<block_id>/context/contracts/<contract_id>/script/norm
            alized (POST)

        * ../<block_id>/context/contracts/<contract_id>/single_saplin
          g_get_diff (GET)

        * ../<block_id>/context/contracts/<contract_id>/spendable
          (GET)

        * ../<block_id>/context/contracts/<contract_id>/spendable_and
          _frozen_bonds (GET)

        * ../<block_id>/context/contracts/<contract_id>/staked_balance
          (GET)

        * ../<block_id>/context/contracts/<contract_id>/staking_numer
          ator (GET)

        * ../<block_id>/context/contracts/<contract_id>/storage (GET)

          * ../<block_id>/context/contracts/<contract_id>/storage/nor
            malized (POST)

          * ../<block_id>/context/contracts/<contract_id>/storage/pai
            d_space (GET)

          * ../<block_id>/context/contracts/<contract_id>/storage/use
            d_space (GET)

        * ../<block_id>/context/contracts/<contract_id>/ticket_balance
          (POST)

        * ../<block_id>/context/contracts/<contract_id>/unstake_reque
          sts (GET)

        * ../<block_id>/context/contracts/<contract_id>/unstaked_fina
          lizable_balance (GET)

        * ../<block_id>/context/contracts/<contract_id>/unstaked_froz
          en_balance (GET)

    * ../<block_id>/context/dal

      * ../<block_id>/context/dal/commitments_history (GET)

      * ../<block_id>/context/dal/published_slot_headers (GET)

      * ../<block_id>/context/dal/shards (GET)

      * ../<block_id>/context/dal/skip_list_cells_of_level (GET)

    * ../<block_id>/context/delegates (GET)

      * ../<block_id>/context/delegates/<pkh> (GET)

        * ../<block_id>/context/delegates/<pkh>/active_staking_parame
          ters (GET)

        * ../<block_id>/context/delegates/<pkh>/baking_power (GET)

        * ../<block_id>/context/delegates/<pkh>/companion_key (GET)

        * ../<block_id>/context/delegates/<pkh>/consensus_key (GET)

        * ../<block_id>/context/delegates/<pkh>/current_baking_power
          (GET)

        * ../<block_id>/context/delegates/<pkh>/current_frozen_deposi
          ts (GET)

        * ../<block_id>/context/delegates/<pkh>/current_voting_power
          (GET)

        * ../<block_id>/context/delegates/<pkh>/dal_participation
          (GET)

        * ../<block_id>/context/delegates/<pkh>/deactivated (GET)

        * ../<block_id>/context/delegates/<pkh>/delegated_balance
          (GET)

        * ../<block_id>/context/delegates/<pkh>/delegated_contracts
          (GET)

        * ../<block_id>/context/delegates/<pkh>/delegators (GET)

        * ../<block_id>/context/delegates/<pkh>/denunciations (GET)

        * ../<block_id>/context/delegates/<pkh>/estimated_shared_pend
          ing_slashed_amount (GET)

        * ../<block_id>/context/delegates/<pkh>/external_delegated
          (GET)

        * ../<block_id>/context/delegates/<pkh>/external_staked (GET)

        * ../<block_id>/context/delegates/<pkh>/frozen_deposits (GET)

        * ../<block_id>/context/delegates/<pkh>/frozen_deposits_limit
          (GET)

        * ../<block_id>/context/delegates/<pkh>/full_balance (GET)

        * ../<block_id>/context/delegates/<pkh>/grace_period (GET)

        * ../<block_id>/context/delegates/<pkh>/is_forbidden (GET)

        * ../<block_id>/context/delegates/<pkh>/min_delegated_in_curr
          ent_cycle (GET)

        * ../<block_id>/context/delegates/<pkh>/own_delegated (GET)

        * ../<block_id>/context/delegates/<pkh>/own_full_balance (GET)

        * ../<block_id>/context/delegates/<pkh>/own_staked (GET)

        * ../<block_id>/context/delegates/<pkh>/participation (GET)

        * ../<block_id>/context/delegates/<pkh>/pending_staking_param
          eters (GET)

        * ../<block_id>/context/delegates/<pkh>/stakers (GET)

        * ../<block_id>/context/delegates/<pkh>/staking_balance (GET)

        * ../<block_id>/context/delegates/<pkh>/staking_denominator
          (GET)

        * ../<block_id>/context/delegates/<pkh>/total_delegated (GET)

        * ../<block_id>/context/delegates/<pkh>/total_delegated_stake
          (GET)

        * ../<block_id>/context/delegates/<pkh>/total_staked (GET)

        * ../<block_id>/context/delegates/<pkh>/total_unstaked_per_cy
          cle (GET)

        * ../<block_id>/context/delegates/<pkh>/unstaked_frozen_depos
          its (GET)

        * ../<block_id>/context/delegates/<pkh>/voting_info (GET)

        * ../<block_id>/context/delegates/<pkh>/voting_power (GET)

    * ../<block_id>/context/denunciations (GET)

    * ../<block_id>/context/issuance

      * ../<block_id>/context/issuance/current_yearly_rate (GET)

      * ../<block_id>/context/issuance/current_yearly_rate_details
        (GET)

      * ../<block_id>/context/issuance/current_yearly_rate_exact (GET)

      * ../<block_id>/context/issuance/expected_issuance (GET)

      * ../<block_id>/context/issuance/issuance_per_minute (GET)

    * ../<block_id>/context/liquidity_baking

      * ../<block_id>/context/liquidity_baking/cpmm_address (GET)

    * ../<block_id>/context/merkle_tree (GET)

    * ../<block_id>/context/merkle_tree_v2 (GET)

    * ../<block_id>/context/nonces

      * ../<block_id>/context/nonces/<block_level> (GET)

    * ../<block_id>/context/protocol

      * ../<block_id>/context/protocol/first_level (GET)

    * ../<block_id>/context/raw

      * ../<block_id>/context/raw/bytes (GET)

      * ../<block_id>/context/raw/json (<dyn>)

    * ../<block_id>/context/sapling

      * ../<block_id>/context/sapling/<sapling_state_id>

        * ../<block_id>/context/sapling/<sapling_state_id>/get_diff
          (GET)

    * ../<block_id>/context/seed (POST)

    * ../<block_id>/context/seed_computation (GET)

    * ../<block_id>/context/smart_rollups

      * ../<block_id>/context/smart_rollups/all (GET)

        * ../<block_id>/context/smart_rollups/all/inbox (GET)

      * ../<block_id>/context/smart_rollups/smart_rollup

        * ../<block_id>/context/smart_rollups/smart_rollup/<smart_rol
          lup_address>

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/commitment

            * ../<block_id>/context/smart_rollups/smart_rollup/<smart
              _rollup_address>/commitment/<smart_rollup_commitment_ha
              sh> (GET)

              * ../<block_id>/context/smart_rollups/smart_rollup/<sma
                rt_rollup_address>/commitment/<smart_rollup_commitmen
                t_hash>/can_be_cemented (GET)

              * ../<block_id>/context/smart_rollups/smart_rollup/<sma
                rt_rollup_address>/commitment/<smart_rollup_commitmen
                t_hash>/stakers_indexes (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/consumed_outputs

            * ../<block_id>/context/smart_rollups/smart_rollup/<smart
              _rollup_address>/consumed_outputs/<block_level> (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/genesis_info (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/inbox_level

            * ../<block_id>/context/smart_rollups/smart_rollup/<smart
              _rollup_address>/inbox_level/<block_level>

              * ../<block_id>/context/smart_rollups/smart_rollup/<sma
                rt_rollup_address>/inbox_level/<block_level>/commitme
                nts (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/kind (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/last_cemented_commitment_hash_with_level
            (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/last_whitelist_update (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/staker

            * ../<block_id>/context/smart_rollups/smart_rollup/<smart
              _rollup_address>/staker/<pkh>

              * ../<block_id>/context/smart_rollups/smart_rollup/<sma
                rt_rollup_address>/staker/<pkh>/conflicts (GET)

              * ../<block_id>/context/smart_rollups/smart_rollup/<sma
                rt_rollup_address>/staker/<pkh>/games (GET)

              * ../<block_id>/context/smart_rollups/smart_rollup/<sma
                rt_rollup_address>/staker/<pkh>/index (GET)

              * ../<block_id>/context/smart_rollups/smart_rollup/<sma
                rt_rollup_address>/staker/<pkh>/staked_on_commitment
                (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/staker1

            * ../<block_id>/context/smart_rollups/smart_rollup/<smart
              _rollup_address>/staker1/<staker1_pkh>

              * ../<block_id>/context/smart_rollups/smart_rollup/<sma
                rt_rollup_address>/staker1/<staker1_pkh>/staker2

                * ../<block_id>/context/smart_rollups/smart_rollup/<s
                  mart_rollup_address>/staker1/<staker1_pkh>/staker2/
                  <staker2_pkh>

                  * ../<block_id>/context/smart_rollups/smart_rollup/
                    <smart_rollup_address>/staker1/<staker1_pkh>/stak
                    er2/<staker2_pkh>/timeout (GET)

                  * ../<block_id>/context/smart_rollups/smart_rollup/
                    <smart_rollup_address>/staker1/<staker1_pkh>/stak
                    er2/<staker2_pkh>/timeout_reached (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/stakers (GET)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/ticket_balance (POST)

          * ../<block_id>/context/smart_rollups/smart_rollup/<smart_r
            ollup_address>/whitelist (GET)

    * ../<block_id>/context/total_currently_staked (GET)

    * ../<block_id>/context/total_frozen_stake (GET)

    * ../<block_id>/context/total_supply (GET)

  * ../<block_id>/hash (GET)

  * ../<block_id>/header (GET)

    * ../<block_id>/header/protocol_data (GET)

      * ../<block_id>/header/protocol_data/raw (GET)

    * ../<block_id>/header/raw (GET)

    * ../<block_id>/header/shell (GET)

  * ../<block_id>/helpers

    * ../<block_id>/helpers/attestation_rights (GET)

    * ../<block_id>/helpers/baking_rights (GET)

    * ../<block_id>/helpers/complete

      * ../<block_id>/helpers/complete/<prefix> (GET)

    * ../<block_id>/helpers/consecutive_round_zero (GET)

    * ../<block_id>/helpers/current_level (GET)

    * ../<block_id>/helpers/forge

      * ../<block_id>/helpers/forge/bls_consensus_operations (POST)

      * ../<block_id>/helpers/forge/operations (POST)

      * ../<block_id>/helpers/forge/protocol_data (POST)

      * ../<block_id>/helpers/forge/signed_operations (POST)

    * ../<block_id>/helpers/forge_block_header (POST)

    * ../<block_id>/helpers/levels_in_current_cycle (GET)

    * ../<block_id>/helpers/parse

      * ../<block_id>/helpers/parse/block (POST)

      * ../<block_id>/helpers/parse/operations (POST)

    * ../<block_id>/helpers/preapply

      * ../<block_id>/helpers/preapply/block (POST)

      * ../<block_id>/helpers/preapply/operations (POST)

    * ../<block_id>/helpers/round (GET)

    * ../<block_id>/helpers/scripts

      * ../<block_id>/helpers/scripts/entrypoint (POST)

      * ../<block_id>/helpers/scripts/entrypoints (POST)

      * ../<block_id>/helpers/scripts/normalize_data (POST)

      * ../<block_id>/helpers/scripts/normalize_script (POST)

      * ../<block_id>/helpers/scripts/normalize_stack (POST)

      * ../<block_id>/helpers/scripts/normalize_type (POST)

      * ../<block_id>/helpers/scripts/pack_data (POST)

      * ../<block_id>/helpers/scripts/run_code (POST)

      * ../<block_id>/helpers/scripts/run_instruction (POST)

      * ../<block_id>/helpers/scripts/run_operation (POST)

      * ../<block_id>/helpers/scripts/run_script_view (POST)

      * ../<block_id>/helpers/scripts/run_view (POST)

      * ../<block_id>/helpers/scripts/script_size (POST)

      * ../<block_id>/helpers/scripts/simulate_operation (POST)

      * ../<block_id>/helpers/scripts/trace_code (POST)

      * ../<block_id>/helpers/scripts/typecheck_code (POST)

      * ../<block_id>/helpers/scripts/typecheck_data (POST)

    * ../<block_id>/helpers/total_baking_power (GET)

    * ../<block_id>/helpers/validators (GET)

  * ../<block_id>/live_blocks (GET)

  * ../<block_id>/metadata (GET)

  * ../<block_id>/metadata_hash (GET)

  * ../<block_id>/operation_hashes (GET)

    * ../<block_id>/operation_hashes/<list_offset> (GET)

      * ../<block_id>/operation_hashes/<list_offset>/<operation_offse
        t> (GET)

  * ../<block_id>/operation_metadata_hashes (GET)

    * ../<block_id>/operation_metadata_hashes/<list_offset> (GET)

      * ../<block_id>/operation_metadata_hashes/<list_offset>/<operat
        ion_offset> (GET)

  * ../<block_id>/operations (GET)

    * ../<block_id>/operations/<list_offset> (GET)

      * ../<block_id>/operations/<list_offset>/<operation_offset>
        (GET)

  * ../<block_id>/operations_metadata_hash (GET)

  * ../<block_id>/protocols (GET)

  * ../<block_id>/resulting_context_hash (GET)

  * ../<block_id>/votes

    * ../<block_id>/votes/ballot_list (GET)

    * ../<block_id>/votes/ballots (GET)

    * ../<block_id>/votes/current_period (GET)

    * ../<block_id>/votes/current_proposal (GET)

    * ../<block_id>/votes/current_quorum (GET)

    * ../<block_id>/votes/listings (GET)

    * ../<block_id>/votes/proposal_count

      * ../<block_id>/votes/proposal_count/<pkh> (GET)

    * ../<block_id>/votes/proposals (GET)

    * ../<block_id>/votes/successor_period (GET)

    * ../<block_id>/votes/total_voting_power (GET)


Full description
================

**GET ../<block_id>?[version=<version>]&[force_metadata]&[metadata=<m
etadata_rpc_arg>]**

**GET ../<block_id>/context/adaptive_issuance_launch_cycle**

**GET ../<block_id>/context/big_maps/<big_map_id>?[offset=<uint>]&[le
ngth=<uint>]**

**GET ../<block_id>/context/big_maps/<big_map_id>/<script_expr>**

**POST
../<block_id>/context/big_maps/<big_map_id>/<script_expr>/normalized**

**GET ../<block_id>/context/cache/contracts/all**

**POST ../<block_id>/context/cache/contracts/rank**

**GET ../<block_id>/context/cache/contracts/size**

**GET ../<block_id>/context/cache/contracts/size_limit**

**GET ../<block_id>/context/constants**

**GET ../<block_id>/context/constants/errors**

**GET ../<block_id>/context/constants/parametric**

**GET ../<block_id>/context/contracts**

**GET
../<block_id>/context/contracts/<contract_id>?[normalize_types]**

**GET
../<block_id>/context/contracts/<contract_id>/all_ticket_balances**

**GET ../<block_id>/context/contracts/<contract_id>/balance**

**GET ../<block_id>/context/contracts/<contract_id>/balance_and_froze
n_bonds**

**POST ../<block_id>/context/contracts/<contract_id>/big_map_get**

**GET ../<block_id>/context/contracts/<contract_id>/counter**

**GET ../<block_id>/context/contracts/<contract_id>/delegate**

**GET ../<block_id>/context/contracts/<contract_id>/entrypoints?[norm
alize_types]**

**GET ../<block_id>/context/contracts/<contract_id>/entrypoints/<entr
ypoint>?[normalize_types]**

**GET ../<block_id>/context/contracts/<contract_id>/estimated_own_pen
ding_slashed_amount**

**GET ../<block_id>/context/contracts/<contract_id>/frozen_bonds**

**GET ../<block_id>/context/contracts/<contract_id>/full_balance**

**GET ../<block_id>/context/contracts/<contract_id>/manager_key**

**GET ../<block_id>/context/contracts/<contract_id>/script**

**POST
../<block_id>/context/contracts/<contract_id>/script/normalized**

**GET ../<block_id>/context/contracts/<contract_id>/single_sapling_ge
t_diff?[offset_commitment=<uint63>]&[offset_nullifier=<uint63>]**

**GET ../<block_id>/context/contracts/<contract_id>/spendable**

**GET ../<block_id>/context/contracts/<contract_id>/spendable_and_fro
zen_bonds**

**GET ../<block_id>/context/contracts/<contract_id>/staked_balance**

**GET
../<block_id>/context/contracts/<contract_id>/staking_numerator**

**GET ../<block_id>/context/contracts/<contract_id>/storage**

**POST
../<block_id>/context/contracts/<contract_id>/storage/normalized**

**GET
../<block_id>/context/contracts/<contract_id>/storage/paid_space**

**GET
../<block_id>/context/contracts/<contract_id>/storage/used_space**

**POST ../<block_id>/context/contracts/<contract_id>/ticket_balance**

**GET ../<block_id>/context/contracts/<contract_id>/unstake_requests**

**GET ../<block_id>/context/contracts/<contract_id>/unstaked_finaliza
ble_balance**

**GET ../<block_id>/context/contracts/<contract_id>/unstaked_frozen_b
alance**

**GET ../<block_id>/context/dal/commitments_history**

**GET ../<block_id>/context/dal/published_slot_headers?[level=<block_
level>]**

**GET ../<block_id>/context/dal/shards?[level=<block_level>]&(delegat
es=<pkh>)***

**GET ../<block_id>/context/dal/skip_list_cells_of_level**

**GET ../<block_id>/context/delegates?[active]&[inactive]&[with_minim
al_stake]&[without_minimal_stake]**

**GET ../<block_id>/context/delegates/<pkh>**

**GET
../<block_id>/context/delegates/<pkh>/active_staking_parameters**

**GET ../<block_id>/context/delegates/<pkh>/baking_power**

**GET ../<block_id>/context/delegates/<pkh>/companion_key**

**GET ../<block_id>/context/delegates/<pkh>/consensus_key**

**GET ../<block_id>/context/delegates/<pkh>/current_baking_power**

**GET ../<block_id>/context/delegates/<pkh>/current_frozen_deposits**

**GET ../<block_id>/context/delegates/<pkh>/current_voting_power**

**GET ../<block_id>/context/delegates/<pkh>/dal_participation**

**GET ../<block_id>/context/delegates/<pkh>/deactivated**

**GET ../<block_id>/context/delegates/<pkh>/delegated_balance**

**GET ../<block_id>/context/delegates/<pkh>/delegated_contracts**

**GET ../<block_id>/context/delegates/<pkh>/delegators**

**GET ../<block_id>/context/delegates/<pkh>/denunciations**

**GET ../<block_id>/context/delegates/<pkh>/estimated_shared_pending_
slashed_amount**

**GET ../<block_id>/context/delegates/<pkh>/external_delegated**

**GET ../<block_id>/context/delegates/<pkh>/external_staked**

**GET ../<block_id>/context/delegates/<pkh>/frozen_deposits**

**GET ../<block_id>/context/delegates/<pkh>/frozen_deposits_limit**

**GET ../<block_id>/context/delegates/<pkh>/full_balance**

**GET ../<block_id>/context/delegates/<pkh>/grace_period**

**GET ../<block_id>/context/delegates/<pkh>/is_forbidden**

**GET
../<block_id>/context/delegates/<pkh>/min_delegated_in_current_cycle**

**GET ../<block_id>/context/delegates/<pkh>/own_delegated**

**GET ../<block_id>/context/delegates/<pkh>/own_full_balance**

**GET ../<block_id>/context/delegates/<pkh>/own_staked**

**GET ../<block_id>/context/delegates/<pkh>/participation**

**GET
../<block_id>/context/delegates/<pkh>/pending_staking_parameters**

**GET ../<block_id>/context/delegates/<pkh>/stakers**

**GET ../<block_id>/context/delegates/<pkh>/staking_balance**

**GET ../<block_id>/context/delegates/<pkh>/staking_denominator**

**GET ../<block_id>/context/delegates/<pkh>/total_delegated**

**GET ../<block_id>/context/delegates/<pkh>/total_delegated_stake**

**GET ../<block_id>/context/delegates/<pkh>/total_staked**

**GET ../<block_id>/context/delegates/<pkh>/total_unstaked_per_cycle**

**GET ../<block_id>/context/delegates/<pkh>/unstaked_frozen_deposits**

**GET ../<block_id>/context/delegates/<pkh>/voting_info**

**GET ../<block_id>/context/delegates/<pkh>/voting_power**

**GET ../<block_id>/context/denunciations**

**GET ../<block_id>/context/issuance/current_yearly_rate**

**GET ../<block_id>/context/issuance/current_yearly_rate_details**

**GET ../<block_id>/context/issuance/current_yearly_rate_exact**

**GET ../<block_id>/context/issuance/expected_issuance**

**GET ../<block_id>/context/issuance/issuance_per_minute**

**GET ../<block_id>/context/liquidity_baking/cpmm_address**

**GET
../<block_id>/context/merkle_tree(/<context_path>)*?[holey=<bool>]**

**GET ../<block_id>/context/merkle_tree_v2(/<context_path>)*?[holey=<
bool>]**

**GET ../<block_id>/context/nonces/<block_level>**

**GET ../<block_id>/context/protocol/first_level**

**GET
../<block_id>/context/raw/bytes(/<context_path>)*?[depth=<uint>]**

**GET ../<block_id>/context/sapling/<sapling_state_id>/get_diff?[offs
et_commitment=<uint63>]&[offset_nullifier=<uint63>]**

**POST ../<block_id>/context/seed**

**GET ../<block_id>/context/seed_computation**

**GET ../<block_id>/context/smart_rollups/all**

**GET ../<block_id>/context/smart_rollups/all/inbox**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/commitment/<smart_rollup_commitment_hash>**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/commitment/<smart_rollup_commitment_hash>/can_be_cemented**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/commitment/<smart_rollup_commitment_hash>/stakers_indexes**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/consumed_outputs/<block_level>**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/genesis_info**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/inbox_level/<block_level>/commitments**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/kind**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/last_cemented_commitment_hash_with_level**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/last_whitelist_update**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/staker/<pkh>/conflicts**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/staker/<pkh>/games**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/staker/<pkh>/index**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/staker/<pkh>/staked_on_commitment**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/staker1/<staker1_pkh>/staker2/<staker2_pkh>/timeout**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/staker1/<staker1_pkh>/staker2/<staker2_pkh>/timeout_reached**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/stakers**

**POST ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup
_address>/ticket_balance**

**GET ../<block_id>/context/smart_rollups/smart_rollup/<smart_rollup_
address>/whitelist**

**GET ../<block_id>/context/total_currently_staked**

**GET ../<block_id>/context/total_frozen_stake**

**GET ../<block_id>/context/total_supply**

**GET ../<block_id>/hash**

**GET ../<block_id>/header**

**GET ../<block_id>/header/protocol_data**

**GET ../<block_id>/header/protocol_data/raw**

**GET ../<block_id>/header/raw**

**GET ../<block_id>/header/shell**

**GET ../<block_id>/helpers/attestation_rights?(level=<block_level>)*
&[cycle=<block_cycle>]&(delegate=<pkh>)*&(consensus_key=<pkh>)***

**GET ../<block_id>/helpers/baking_rights?(level=<block_level>)*&[cyc
le=<block_cycle>]&(delegate=<pkh>)*&(consensus_key=<pkh>)*&[max_round
=<uint>]&[all]**

**GET ../<block_id>/helpers/complete/<prefix>**

**GET ../<block_id>/helpers/consecutive_round_zero**

**GET ../<block_id>/helpers/current_level?[offset=<int32>]**

**POST ../<block_id>/helpers/forge/bls_consensus_operations**

**POST ../<block_id>/helpers/forge/operations**

**POST ../<block_id>/helpers/forge/protocol_data**

**POST ../<block_id>/helpers/forge/signed_operations**

**POST ../<block_id>/helpers/forge_block_header**

**GET ../<block_id>/helpers/levels_in_current_cycle?[offset=<int32>]**

**POST ../<block_id>/helpers/parse/block**

**POST ../<block_id>/helpers/parse/operations?[version=<version>]**

**POST
../<block_id>/helpers/preapply/block?[sort]&[timestamp=<date>]**

**POST ../<block_id>/helpers/preapply/operations?[version=<version>]**

**GET ../<block_id>/helpers/round**

**POST ../<block_id>/helpers/scripts/entrypoint**

**POST ../<block_id>/helpers/scripts/entrypoints**

**POST ../<block_id>/helpers/scripts/normalize_data**

**POST ../<block_id>/helpers/scripts/normalize_script**

**POST ../<block_id>/helpers/scripts/normalize_stack**

**POST ../<block_id>/helpers/scripts/normalize_type**

**POST ../<block_id>/helpers/scripts/pack_data**

**POST ../<block_id>/helpers/scripts/run_code**

**POST ../<block_id>/helpers/scripts/run_instruction**

**POST
../<block_id>/helpers/scripts/run_operation?[version=<version>]**

**POST ../<block_id>/helpers/scripts/run_script_view**

**POST ../<block_id>/helpers/scripts/run_view**

**POST ../<block_id>/helpers/scripts/script_size**

**POST ../<block_id>/helpers/scripts/simulate_operation?[version=<ver
sion>]&[successor_level]**

**POST ../<block_id>/helpers/scripts/trace_code**

**POST ../<block_id>/helpers/scripts/typecheck_code**

**POST ../<block_id>/helpers/scripts/typecheck_data**

**GET ../<block_id>/helpers/total_baking_power**

**GET ../<block_id>/helpers/validators?(level=<block_level>)*&(delega
te=<pkh>)*&(consensus_key=<pkh>)***

**GET ../<block_id>/live_blocks**

**GET ../<block_id>/metadata?[version=<version>]**

**GET ../<block_id>/metadata_hash**

**GET ../<block_id>/operation_hashes**

**GET ../<block_id>/operation_hashes/<list_offset>**

**GET
../<block_id>/operation_hashes/<list_offset>/<operation_offset>**

**GET ../<block_id>/operation_metadata_hashes**

**GET ../<block_id>/operation_metadata_hashes/<list_offset>**

**GET ../<block_id>/operation_metadata_hashes/<list_offset>/<operatio
n_offset>**

**GET ../<block_id>/operations?[version=<version>]&[force_metadata]&[
metadata=<metadata_rpc_arg>]**

**GET ../<block_id>/operations/<list_offset>?[version=<version>]&[for
ce_metadata]&[metadata=<metadata_rpc_arg>]**

**GET ../<block_id>/operations/<list_offset>/<operation_offset>?[vers
ion=<version>]&[force_metadata]&[metadata=<metadata_rpc_arg>]**

**GET ../<block_id>/operations_metadata_hash**

**GET ../<block_id>/protocols**

**GET ../<block_id>/resulting_context_hash**

**GET ../<block_id>/votes/ballot_list**

**GET ../<block_id>/votes/ballots**

**GET ../<block_id>/votes/current_period**

**GET ../<block_id>/votes/current_proposal**

**GET ../<block_id>/votes/current_quorum**

**GET ../<block_id>/votes/listings**

**GET ../<block_id>/votes/proposal_count/<pkh>**

**GET ../<block_id>/votes/proposals**

**GET ../<block_id>/votes/successor_period**

**GET ../<block_id>/votes/total_voting_power**

JSON/RPC interface
******************


Overview
========

The Octez node provides a JSON/RPC interface. Note that it is an RPC
interface, and it is JSON based, but it does not follow the “JSON-RPC”
protocol.

Some RPC calls are computationally intensive. They can slow down the
node and even temporarily block synchronization with its peers and get
the node out of sync.

The JSON/RPC interface is not active by default mainly for security
reasons. The operator should be explicit about exposing endpoints and
possibly add a whitelist if necessary. See Default ACL for RPC for
limiting access to certain RPCs. Another reason is the potential
performance impact. See Activating RPC section.

To address the problem of heavy RPC requests, the RPC interface can be
activated in two different modes.

* Local RPC server: this server operates in the same process as the
  Octez node and serves data directly from the node.

* External RPC server: this server operates in a separate process and
  forwards requests to the Local RPC server. However, some RPC
  requests are handled directly by the External RPC server, which we
  refer to as being handled locally. This mode prevents the node from
  blocking.

   [image: Local and External RPC servers][image]

The RPC interface is typically used in one of the two use cases
referred as Baking setup and Service provider setup.

Baking setup:

* A private node

* Used by a Baker

* Time-critical setup; lagging means losing money as blocks cannot be
  built

* Only several RPC users

* Slow/heavy RPCs are very rare

Service provider setup:

* A fleet of public RPC nodes sharing the load

* Multiple external users, e.g., explorers, indexers, wallets, etc.

* Heavy load

* Slow/heavy individual RPC requests are probable and can
  significantly impact the service availability

In both cases, the total load generated by the RPC requests on the
Octez node is important. However, in baking setup the heavy individual
RPC requests are unlikely and thus enabling External RPC server will
have minor or no benefit. For a service provider, enabling the
External RPC server is essential to ensure the node remains in sync,
thereby avoiding the risk of serving outdated answers if the node gets
stuck.


Activating RPC
==============

As mentioned before, the RPC interface is not active by default and it
must be explicitly activated.

For activating the Local RPC server mode, use the "--rpc-addr" option.
In that case, a local RPC server is started by the node at port
"8732".

As an example, if you are not trying to run a public RPC node, but you
just want to explore the RPC interface on your own, you would run:

   ./octez-node run --rpc-addr localhost

To run an External RPC server, use instead "--external-rpc-addr" in
the command line. Then an External RPC server is started at port
"18731".

   ./octez-node run --external-rpc-addr localhost

The RPC interface is self-documented and the "octez-client" executable
is able to pretty-print the RPC API. For instance, to see the API
provided by the Octez Shell:

   ./octez-client rpc list

To get API attached to the “genesis” block, including the remote
procedures provided by the associated economic protocol version:

   ./octez-client rpc list /chains/main/blocks/genesis

You might also want the JSON schema describing the expected input and
output of a RPC. For instance:

   ./octez-client rpc schema get /chains/main/blocks/genesis/hash

Note: you can get the same information, but as a raw JSON object, with
a simple HTTP request:

   curl -s localhost:8732/chains/main/blocks/head~10
   wget -O - http://localhost:8732/describe?recurse=true
   wget -O - http://localhost:8732/describe/chains/main/blocks/genesis?recurse=true
   wget -O - http://localhost:8732/describe/chains/main/blocks/genesis/hash

An online index of RPC calls is also available.

The general call of an RPC from the client is "octez-admin-client rpc
(get|post) <url>". For instance, if you wish to request the current
balance of a given block and contract, you can call the associated RPC
via the command:

   octez-admin-client rpc get /blocks/<block_id>/proto/context/contracts/<contract_id>/balance

An RPC may take an *input* and generate an *output* both in JSON
format. For example, the previous RPC call, that does not require an
input, would display on the standard output:

   { "balance": "4000000000000" }

When calling an RPC that requires an input through command line, you
will be prompted to provide the JSON input in your default configured
text editor. Alternatively, you can provide the JSON input using
command

   octez-admin-client rpc post <url> with <JSON>

Don't forget to quote the JSON according to your shell rules.


External RPC server
===================

Thanks to this feature, the node won't experience slowdowns on
computationally intensive RPC calls. This significantly reduces the
load on the Octez node and prevents it from stopping synchronization
with its peers or becoming out of sync. Additionally, performance is
expected to be slightly increased in terms of requests handled per
second. The External RPC server increases the latency for the RPC
requests by a negligible amount (less than 1 ms).

A benchmark framework, implemented in the devtools/benchmarks-
tools/bench_RPS/rps.sh script, allows to run performance evaluations
easily. Along with this benchmark framework, former results are stored
in the dedicated devtools/benchmarks-tools/bench_RPS/results.json
file.


RPC versions
============

See Octez & Protocol versioning and RPC Versioning.


See also
========

* RPC handling architecture

RPCs - OpenAPI reference
************************

OpenAPI is a specification format for REST APIs. This format is
supported by several tools, such as Swagger UI which allows you to
browse a specification and perform API calls from your browser.
Several code generators also exist to generate API libraries for
various programming languages.


Octez Node
==========

The REST API served by the Octez node on the RPC port is described by
the union of several OpenAPI specifications:

* "rpc-openapi(-rc).json", containing the protocol-independent (or
  "shell") RPCs in the last (candidate) release

* "rpc-openapi-dev.json", containing the protocol-independent (or
  "shell") RPCs in the current development version (branch "master")

* For each protocol in use:

  * "$PROTOCOL-openapi.json" (served under the prefix: "/chains
    /<chain-id>/blocks/<block-id>")

  * "$PROTOCOL-mempool-openapi.json" (served under the prefix:
    "/chains/<chain-id>/mempool")

For instance, for an RPC listed as "GET /filter" in "$PROTOCOL-
mempool-openapi.json", its real endpoint is "GET /chains/<chain-
id>/mempool/filter".

These OpenAPI specifications, detailed below, can be generated by
running the Octez node as shown in section How to Generate. For
convenience, the files generated from the most recent release(s) are
provided in this page, annotated each time with the corresponding
release (in parentheses).

Note:

  There exists an alternative reference for the node RPCs, presented
  in more static pages (e.g. Rio RPCs - Reference for the active
  protocol). However, the static referece omits some RPCs, such as the
  ones related to the mempool.


Shell RPCs
----------

The node provides some RPCs which are independent of the protocol.
Their OpenAPI specification can be found at:

* rpc-openapi (version 22.1)

* rpc-openapi-rc (version 23.0~rc2)

* rpc-openapi-dev (version master)


Rio RPCs
--------

The OpenAPI specifications for RPCs which are specific to the Rio
("PsRiotum") protocol can be found at:

* rio-openapi (version 22.1)

* rio-openapi-rc (version 23.0~rc2)

The OpenAPI specifications for RPCs which are related to the mempool
and specific to the Rio protocol can be found at:

* rio-mempool-openapi (version 22.1)

* rio-mempool-openapi-rc (version 23.0~rc2)


Seoul RPCs
----------

The OpenAPI specifications for RPCs which are specific to the Seoul
("PtSeouLo") protocol proposal can be found at:

* seoul-openapi-rc (version 23.0~rc2)

The OpenAPI specifications for RPCs which are related to the mempool
and specific to the Seoul protocol proposal can be found at:

* seoul-mempool-openapi-rc (version 23.0~rc2)


Alpha RPCs
----------

The OpenAPI specifications for RPCs which are specific to the Alpha
protocol can be found at:

* alpha-openapi (version master)

The OpenAPI specifications for RPCs which are related to the mempool
and specific to the Alpha protocol can be found at:

* alpha-mempool-openapi (version master)


Smart Rollup Node
=================

The smart rollup node exposes different RPCs depending on the
underlying L1 protocol in use. Their specification is given in the
sections below. (The exact versions of the rollup node for which these
files are produced can be seen in the field ".info.version" within
each file.)


Rio RPCs
--------

The OpenAPI specifications for the RPCs of the smart rollup node for
the Rio ("PsRiotum") protocol can be found at:

* rio-smart-rollup-node-openapi (version 22.1)

* rio-smart-rollup-node-openapi-rc (version 23.0~rc2)


Seoul RPCs
----------

The OpenAPI specifications for the RPCs of the smart rollup node for
the Seoul ("PtSeoul") protocol proposal can be found at:

* seoul-smart-rollup-node-openapi-rc (version 23.0~rc2)


Alpha RPCs
----------

The OpenAPI specifications for the RPCs of the smart rollup node for
the Alpha protocol can be found at:

* alpha-smart-rollup-node-openapi (version master)


DAL Node
========

The DAL node also provides RPCs. Their OpenAPI specification can be
found at:

* dal-node-openapi (version 22.1)

* dal-node-openapi-rc (version 23.0~rc2)

* dal-node-openapi (version master)


How to Generate
===============

To regenerate the "*-dev.json" and "alpha-*.json" files above from the
current sources in your Octez repository, run "make -C docs openapi"
from the root of the Octez repository, which will run the generation
script and check if the JSON files above are up-to-date with respect
to their versions under Git (modulo the "version" fields inside). If
there are any significant differences, it will also render the
corresponding JSON files as HTML.

Note that:

* The generation script "src/bin_openapi/generate.sh" requires the
  Octez executables to be built, so you have to first run "make" from
  the repository root.

* The HTML rendering script docs/scripts/openapi2html.sh requires yq
  and redocly to be installed.

For platform contributors: if you want to update the OpenAPI and HTML
files in the official repository and documentation, you may create a
merge request to update the changed files under "docs/api/".

Instead of executing all at once with the makefile, you may execute
manually the generation or rendering, as follows.


OpenAPI generation
------------------

The OpenAPI generation script "src/bin_openapi/generate.sh" must be
run from the root of the Octez repository.

The generation script will start a sandbox node, activate the
protocol, get the RPC specifications from this node and convert them
to OpenAPI specifications.

To generate the OpenAPI specification for the RPCs provided by a
specific protocol instead of Alpha, update the following variables in
src/bin_openapi/generate.sh:

"`sh protocol_hash=ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK
protocol_parameters=src/proto_alpha/parameters/sandbox-parameters.json
protocol_name=alpha `"

For "protocol_hash", use the value defined in "TEZOS_PROTOCOL".


HTML rendering
--------------

For generated OpenAPI files that changed, you may derive the
corresponding statically-rendered HTML pages, typically using:

   docs/scripts/openapi2html.sh docs/api/*.json


How to Test
===========

You can test OpenAPI specifications using Swagger Editor to check for
syntax issues (just copy-paste "rpc-openapi.json" into it or open it
from menu "File > Import file").

You can run Swagger UI to get an interface to browse the API (replace
"xxxxxx" with the directory where "rpc-openapi.json" is, and "rpc-
openapi.json" by the file you want to browse):

   docker pull swaggerapi/swagger-ui
   docker run -p 8080:8080 -e SWAGGER_JSON=/mnt/rpc-openapi.json -v xxxxxx:/mnt swaggerapi/swagger-ui

Then open it in your browser.

How to contribute
*****************

Octez is an open source software developed and maintained by a wide
constellation of teams and contributors (collectively called platform
developers) within the larger Tezos ecosystem. Hence, it is open by
nature and provides many interesting challenges for existing and new
contributions.

Here you can find information about:

* the wide spectrum of possible contributions, and some corresponding
  workflows and guidelines

* the Octez Merge Team, which is the instance coordinating updates to
  the code base

* policies associated with the repository

etc.

* How to contribute to Octez

  * Introduction

  * Our git strategy

  * Workflow of an MR

  * Preparing a Merge Request

  * Creating the Merge Request

  * Special case: MRs that introduce a new dependency

  * Protocol development MRs

  * Code Review

* Octez Merge Team

  * Octez Merge Team Scope

  * Getting into the Octez Merge Team

  * Helping the Octez Merge Team

* Documentation and coding guidelines

  * Documentation guidelines

  * Coding guidelines

* Scope of the Octez repository

How to contribute to Octez
**************************

The purpose of this document is to help developers contribute to
Octez.


Introduction
============

There are several ways to get involved with the codebase and its
documentation, and you may want to start with some preliminary steps.


Providing feedback on documentation
-----------------------------------

As your journey will likely start by reading this documentation,
please consider giving us feedback using the "Send feedback" button
available in all entries. You can use it to mention aspects you like
or don't like. Many kinds of feedback are useful! For instance, there
are more objective aspects of documentation, such as:

* reliable: are the explanations accurate? up-to-date?

* consistent: are there any contradictions between the different
  parts?

* complete: are there important topics or aspects not described?

* precise: are the explanations detailed enough?

* grammatically correct: is the phrasing correct?

or more subjective aspects such as:

* clear: are the explanations easy to understand?

* easy to find: did you have troubles locating some information?

* useful: does the text help in some way?

However, if the feedback is about grammar, spelling, and other similar
aspects, please consider implementing your suggested changes by
submitting fixes to the Octez Docs' typo train -- explained next.


Fixing typos
------------

You may want to fix some typos and minor errors or incoherencies in
the documentation, which is situated in the "docs/" subfolder of the
code repository, or within *comments* or *docstrings* within the code.
Small tweaks like these can be contributed without creating a merge
request and commits can rather be pushed directly to the "typo-doc"
branch in the "tezos/tezos" repository. This branch is regularly
merged into the master branch, e.g., every one or two weeks. (If the
branch has been automatically deleted following a merge, just create
it again.) This periodic merging is implemented by a series of MRs
named "the Typo Train", created for you by a volunteer, and batching
the currently pending fixes. Of course, all these commits will be
reviewed before being integrated. The current edition of the typo
train MR can be found in meta-issue #2329.

If you don't have enough permissions to push to the branch above, you
can still make commits in your own fork of the Octez repository, and
ask for them to be cherry-picked on the typo/train on the
"#documentation" channel on the Tezos Dev Slack space. Alternatively,
you may of course create your own MRs on the tezos/tezos repository
for submitting documentations changes without using the Typo Train. In
that case, Doc owners might contact you on the MR to suggest changes,
including integrating these in a current open train.


Reporting issues
----------------

Another simple way to contribute to Octez is to report issues that you
may find with the software on GitLab. If you are unsure about an issue
consult the Tezos community first and always make sure to search the
existing issues before reporting a new one. Some information that are
probably important to include in the description: the architecture
(e.g. *ARM64*), the operating system (e.g. *Debian Stretch*), the
network you are connected to (e.g. *Ghostnet*), the binary or
component (e.g. *octez-node crashes* or *rpc X returns Y while Z was
expected*).

Other useful contributions consist in inspecting existing issues and
adding details, suggesting workarounds, etc.


Other non-code contributions
----------------------------

If you acquired a good expertise level in using, configuring, or
overcoming problems with the various Octez executables, you may bring
useful contributions by providing assistance to others via the
different channels (Slack,  StackExchange, etc.) in the Tezos
community.

If you want to shape the future of Octez, you may consider
contributing to Tezos Improvement Proposals (TZIPs), that specify new
features or define standards. These are located in the TZIP repo.


Going further
-------------

To directly contribute to the *code* itself, expertise in a few areas
is necessary.

First, make sure that you are proficient enough in OCaml. The OCaml
community website https://ocaml.org gives a few useful pointers for
that. In particular, we use a lot of functors, and a few GADTs in the
codebase, so you may want to make sure that you master these advanced
concepts. For a more specific explanation of GADT usage in Octez you
can check out Generalized Algebraic Data Types (GADTs).

Then, if you don’t know much about the Lwt library, that’s what you
want to learn next. This library is used extensively throughout the
code base: we use it to handle concurrency. You can use the online
documentation. The chapter on concurrency of the Real World OCaml book
has also been ported to Lwt.

After that, it is a good idea to read the tutorials for error_monad
and data_encoding, two homegrown libraries that we use pervasively.

While you familiarize yourself with the basics as suggested above, you
can have a look at the software architecture of Tezos. It will give
you the main components and their interactions, and links to the
documentation for the various parts.

You may also want to take a look to some developer tools that can make
protocol development more convenient.

Now, that you're ready to delve into the code, it is time to know how
contributions to the code are submitted, reviewed, and finally
accepted into the master branch.


Our git strategy
================

First of all, the repository is https://gitlab.com/tezos/tezos. So if
you want to contribute, simply create an account there.

There are many ways to use Git, here is ours.

We mostly use merge requests (aka MRs) for contributing to the master
branch, meaning that nobody should be pushing into the master branch
directly. Once a merge request is ready, it is reviewed and approved,
then merged with a merge commit.

Therefore, you can bring useful contributions both by creating your
own MRs and by reviewing MRs submitted by others.

We maintain a semi-linear history, which means that merge requests are
only merged if they are direct suffixes of the master branch. This
means that merge requests are rebased on top of "master" before they
are merged. This can only be done automatically if there is no
conflict though. So whenever "origin/master" changes, you should make
sure that your branch can still be rebased on it. In case of conflict,
you need to rebase manually (pull "master", checkout your branch and
run "git rebase master"). You may have to edit your patches during the
rebase. Then use "push -f" in your branch to rewrite the history.
Being proficient with interactive rebases is mandatory to avoid
mistakes and wasting time.

This Git strategy is a variant of the git rebase workflow.


Workflow of an MR
=================

This section presents a global view of our MR workflow. Details about
the individual steps in this workflow are described in the following
sections.

Our code review process uses GitLab. First a developer creates a new
branch for preparing the MR.  As this is a private new branch, the
developer is free to rebase, squash commits, rewrite history ("git
push --force"), etc. at will.

Once the code is ready to be shared with the rest of the team, the
developer opens a Merge Request.  It is useful to explain why the MR
is created, to add a precise description of the code changes, and to
check if those are in line with the initial requirements (if
responding to an issue), or to the stated reasons (otherwise).
Dependencies on other merge requests, other relationships to MRs, to
issues, etc, should also be mentioned.

While the code is still not ready to be peer reviewed, but it is
merely a work in progress, the developer assigns it to themselves and
prefixes the MR title with "Draft:". This will tell everybody they can
look at the code, comment, but there is still work to be done and the
branch can change and history be rewritten.

Finally, when the code is ready for the code review, the developer
assigns the MR to a reviewer and freezes the branch. From this moment
on, the developer will refrain from rewriting history, although they
can add new commits and rebase the branch for syncing it with master
(this can be done regularly to make sure the branch does not get
stale). At this point the developer interacts with the reviewers to
address their comments and suggestions.

GitLab allows both to comment on the code and to add general comments
on the MR.  Each comment should be addressed by the developer. They
can add additional commits to address each comment. This incremental
approach will make it easier for the reviewer to keep interacting till
each discussion is resolved. When the reviewer is satisfied, they will
mark the discussion resolved.

The Draft: status can be set at any time during the authoring/review
process to signal that the MR is not yet ready for merge (for
instance, the "git" history might not be clean). But, the "Draft"
status *does not* prohibit review -- this should be achieved by
assigning the MR to the developer.

You can refer to the following table that summarizes how assignee and
"Draft" status interact to decide whether an MR is open for review:

+---------+------------+-------------+--------------+
| Draft   | Assignee   | Reviewers   | Reviewable   |
|=========|============|=============|==============|
| -       | -          | ∅           | No           |
+---------+------------+-------------+--------------+
| Yes     | Author     | -           | No           |
+---------+------------+-------------+--------------+
| Yes     | Others     | Some        | Yes          |
+---------+------------+-------------+--------------+
| No      | Author     | Some        | Yes          |
+---------+------------+-------------+--------------+

When all discussions are resolved, and the MR has got at least two
approvals from Octez Merge Team members, the developer should squash
any fix-up commits that were applied (remembering to edit the commit
message appropriately). Then the developer (or anyone else when not
possible) can assign the MR to the Nomadic Marge-bot, which will check
the MR and automatically add it to a merge train.


Preparing a Merge Request
=========================

While working on your branch to prepare a Merge Request, make sure you
respect the following rules:

* Give a meaningful and consistent name to the branch

  * It is useful to prefix the name of the branch with the name of the
    developer to make it clear at a glance who is working on what:
    e.g. "john@new-feature".

  * Suffix the branch name by "-release" if it is a release branch.

* Prefer small atomic commits over a large one that does many things.

* Don’t mix refactoring, reindentation, whitespace deletion, or other
  style changes with new features or other real changes.

* No peneloping: don't do something in a commit just to undo it two
  commits later.

* We expect every commit to compile and pass tests. Obviously, we
  require tests to pass between each MR.

* Follow the format of commit names, "<Component>: <message>", with
  message in indicative or imperative present mood e.g. "Shell: fix
  bug #13" rather than "Shell: fixed bug #13". Use multiline commit
  messages for important commits.

* Adhere to the coding guidelines.

* Document your changes, in the MR description and commit messages.
  Imagine if somebody asked what your change was about in front of the
  coffee machine, write down your answer and put it in the MR.

* If there is a design description at the top of the file, consider
  updating it to reflect the new version. Additionally, if you feel
  that your design *changes* are worth mentioning to help upcoming
  contributors (e.g. justify a non-obvious design choice), you should
  document them in this file header, but in a separate "History"
  section.

* If you add new functions to an interface, don’t forget to document
  the function in the interface (in the corresponding ".mli" file; or,
  if there is no ".mli" file, directly in the ".ml" file)

* If you add a new RPC endpoint or modify an existing one, be sure to
  take into account the impact on RPC security.

* If you modify the user API (e.g. add or change a configuration
  parameter or a command-line option), update the corresponding
  documentation. In particular, for configuration parameters of the
  Octez node, update the node configuration documentation and the
  documentation of the modified component(s), usually referred by that
  page.

* If your MR introduces new dependencies, follow the additional
  instructions.

* Check whether your changes need to be reflected in changes to the
  corresponding README file (the one in the directory of the patched
  files). If your changes concern several directories, check all the
  corresponding README files.

* For parts that have specifications in the repository (e.g.,
  Michelson), make sure to keep them in sync with the implementation.

* If your contribution has an impact on users, e.g. if it fixes a bug,
  adds a new feature, changes an RPC or the command-line interface,
  add an item to the changelog in the CHANGES.rst file located at the
  root of the repository.


Creating the Merge Request
==========================

Your goal is to help the reviewers convince themselves that your patch
should be merged. Well-documented merge requests will receive feedback
faster. Complicated patches with no comments to help the reviewer will
cause the reviewer to make the wrong decision or will discourage the
reviewer to work on the MR.

Therefore, when creating your MR, observe the following rules:

* *Give it an appropriate title*.

* *Give context*: why was this patch written?

  * Does it fix a bug, add a feature or refactor existing code?

  * Is there an open issue on GitLab, or a post from an angry user
    somewhere?

  * Must it be merged before another merge request?

* *Test*:

  * Explain how you tested your patch (or why you didn't).

  * The description of merge requests must include instructions for
    how to manually test them, when applicable.

  * Merge requests should include automated tests for new
    functionality and bug fixes.

    * Refer to the testing guide for more information.

    * Bug fixes should include a test that demonstrates that the bug
      has been fixed (i.e. that fails before the supplied fix).

    * The test coverage can be used to guide testing of the proposed
      MR. If the modified code lacks coverage, then this indicates
      that tests should be added.

    * If no tests are included, a justification should be given in the
      description. Possible justifications include that testing is
      prohibitively difficult, or that the modified code is already
      well-exercised by the existing test suite. The point of the
      justification is to stress the importance of testing and to
      guide improvements of the test framework.

* *Divide and conquer*: it is easier to merge several simple commits
  than a big one.

  * Isolate complicated parts of your patch in their own commits.

  * Put simple, non-controversial commits first. For instance: commits
    that fix typos, improve documentation, or are simple enough that
    we may want to merge them even without the rest of the merge
    request. Even better put them in a separate MR which can be merged
    easily.

  * Split your commits so that each step is convincing on its own,
    like the proof of a big theorem which is split into several
    lemmas.

  * Avoid merge requests that are too large. They are harder to rebase
    and request a longer continuous time for reviewing, making them
    overall slower to merge. See favoring small merge requests below
    for more details.

* *Anticipate questions*: explain anything which may look surprising,
  as comments in the code itself if it has value to future readers, or
  in the MR description.

* *MR Labels*: Add GitLab labels to the MR, like "doc" or "protocol".

  * The following special labels can be used to trigger different
    parts of the CI pipeline. To take effect, the label must be added
    before any push action is made on the MR.

    * "ci--opam" is for triggering the opam packaging tests pipeline.

    * "ci--docs" is for testing some scripts in the documentation
      (e.g. Octez installation scenarios).

    * "ci--docker" is for publishing the Docker image of the MR.

    * "ci--arm64" is for building on the ARM64 architecture.

    * "ci--no-coverage" disables the job "unified_coverage" (but it
      does not disable coverage instrumentation).

* *MR Options*: When opening an MR you should probably tick the
  following options:

  * "Delete source branch when merge request is accepted." Helps
    keeping the repository clean of old branches.

  * "Squash commits when merge request is accepted." Sometimes it's
    useful to have many small commits to ease the review and see the
    story of a branch, but they are not relevant for the history of
    the project. In this case they can be squashed and replaced with a
    single meaningful commit. Nevertheless, you should squash yourself
    all fix-up commits when all discussions are resolved, as described
    above in the MR workflow, in order to ease the reviewers' task.

  * "Allow commits from members who can merge to the target branch."
    This option is useful to allow members of the Octez merge team,
    who are not developers in your project, to commit to your branch.
    It helps to rebase and propose fixes.

* *Find reviewers*: it is the responsibility of the author to find
  suitable reviewers. In this context, finding a reviewer means
  finding someone that agreed to review in the next days after the MR
  becomes ready. Opting for a reviewer that is not in the capacity to
  review your MR in the next days when others can is unfortunate,
  because the merge request will become unnecessarily blocked; which
  is bad for:

  * the author, as their work gets delayed, and

  * the health of the repository, as it gives the impression that a
    new contribution will land soon; while it is not the case.

  To find reviewers that will review promptly, we refer to the
  documentation of the reviewer field below.

* *Check progress*: It is important to maintain to a minimum the
  number of your open MRs, and to constantly check that the discussion
  is progressing.

Example of an MR with a good, clean history (each bullet is a commit,
any subitems represent the longer description of that commit):

   * Doc: mark bug #13 as fixed
   * Test_python: add test for p2p bug #13
   * Shell: fix p2p bug #13
     - fix bug in the shell
     - fix relative unit test
     - add docstrings

**Beware**: For MRs touching "src/proto_alpha/lib_protocol", see
protocol MRs.


Favoring Small Merge Requests
-----------------------------

Small merge requests are encouraged for multiple reasons:

* They are faster to review, which encourage reviewers to pick them.

* They are easier to rebase, hereby saving developers time.

* They are reviewed more thoroughly.

* If the merge request is not accepted, less work is lost; in
  particular less review time has been spent.

However, small merge requests also come with drawbacks:

* They make it more difficult for reviewers to get the global picture
  of the intended change.

* They may introduce intermediate states, during which a feature is
  not yet finished; or dead code is temporarily introduced.

* They have to be reverted if the entire feature is ultimately
  cancelled.

For "tezos/tezos" to evolve fast, however, we are convinced that the
advantages of small merge requests outweigh the drawbacks. If
possible, drawbacks must be mitigated as follows:

* Have the entire piece of work described or done somewhere. For
  example in an issue, or a branch containing the entire change, or a
  large (unsplit) work as a draft merge request. For complex works, an
  external document may be referred in the issue/MR, detailing the
  design/implementation rationale; if such documents are only targeted
  to reviewers and/or are only describing a *change*, they should not
  go in the online documentation.

* Include a link to the entire piece of work in the description of
  each small merge requests created by splitting the large piece of
  work. This will help reviewers get the big picture.

* Explain why the intermediate state is harmless, if applicable.

* To mitigate loss of work if the whole piece is not accepted, we
  advise to split the work so that improvements that are desirable on
  their own are the first ones to be merged in the sequence of small
  merge requests. A desirable standalone improvement is for example a
  refactoring that improves the quality of the code, or adds new
  tests, or fixes typos.


Merge Request "Assignees" Field
-------------------------------

Merge requests targeted at "tezos/tezos master" should in general have
exactly one assignee. The assignee is someone from which an action is
required to get the merge request moving. Example actions include:

* review;

* respond to a comment thread;

* update the code;

* rebase (in particular in case of conflicts);

* merge;

* find someone else who can get the merge request moving.

The assignee will thus often be one of the reviewers (if they needs to
review or respond to a comment) or one of the merge request authors
(if they need to update the code or respond to a comment).

Even though merge requests could require action from several people to
be merged, we avoid assigning more than one to avoid diluting
responsibility.


Merge Request "Reviewers" Field
-------------------------------

The reviewer field of GitLab is used to specify reviewers. Once the
merge request is ready for review, assign one of the reviewers that
you specified in the reviewers field. As mentioned previously, it is
the responsibility of authors to find reviewers. To find reviewers,
either:

* Advertize your merge request on the "#mr-advertizing" channel of the
  tezos-dev Slack. Good advertisement consists of a link to the MR and
  a one sentence summary.

* Look at authors of the code you are modifying using git blame.

Depending on your GitLab role you may or may not be able to use the
*Reviewers* field for specifying the reviewers. If you don't have the
right, mention the reviewers using their GitLab handle (username
prefixed with "@") in a comment. It causes GitLab to send a
notification to them.


Merge Request "Draft" Mode
--------------------------

A merge request that is not yet ready to be merged should be marked as
draft by prefixing its title with "Draft:". Typical reasons for this
is that the author still has some minor tasks to accomplish, such as
cleaning up the merge request's "git" history (e.g. squashing "fixup!"
commits), even if the request is already reviewed and approved (and in
this case, the draft status will prevent the request from being
prematurely merged).

Being in "Draft:" mode does not signal "not yet ready for review".
Instead, MRs that should not be reviewed should be in the "Draft"
state and assigned to the author, as it is the MR's assignee that
should take action next.

To decide the correct draft status with respect to whether the MR is
ready for review, consider the following table:

+------------+---------------+------------------+
| Assignee   | Has fixups?   | Correct status   |
|============|===============|==================|
| Other      | No            |                  |
+------------+---------------+------------------+
| Other      | Yes           | Draft            |
+------------+---------------+------------------+
| Author     | No            | Draft            |
+------------+---------------+------------------+
| Author     | Yes           | Draft            |
+------------+---------------+------------------+

In sum: the correct status for an MR assigned to another person than
the author should be "Draft" if its history contains "fixup!" or if
other development tasks remain. If the MR is assigned to the author,
then it should be "Draft".


Merge Request's owner
---------------------

Usually the person who has created a Merge Request is also responsible
for pushing it forward: finding reviewers, addressing their comments
and so on. Occasionally though it happens that the author has to move
to more pressing tasks and hands his MR over to another person to
finish. This is a problem, because an MR assigned to its author has
sort of a special status – it informs everyone that the branch is
likely to change at any moment. When an MR is handed over, we lose
that important information.

For this reason, when take-over happens, it's useful to make that
explicit. The person taking over should put a comment informing about
this so that everyone knows that the person attending to the MR
changes. Also the description should be updated to reflect that
information. It is suggested to prefix the description with a line
specifying the MR's owner if that's different to the author indicated
by GitLab.


Special case: MRs that introduce a new dependency
=================================================

In the special case where your MR adds a new opam dependency or
updates an existing opam dependency, you will need to follow this
additional dedicated guide:

* How to add or update opam dependencies

  * Background

  * Local work

  * Making an MR

  * TL;DR

In the special case where your MR adds a new Python, Rust, JavaScript,
or other dependency, additional steps must also be followed.

* for Python, you can refer to the related section in the python
  testing documentation.

* the Rust dependencies are located in the "src/rust_deps" folder

For others, there is currently no dedicated guide. Do not hesitate to
ask for help on the "#devteam" channel on the tezos-dev Slack.


Protocol development MRs
========================

Because of the amendment procedure that governs the protocol, the
workflow for protocol development is significantly different from
master.


Protocol snapshots
------------------

Before a proposal, a new directory, e.g. "proto-005-PsBabyM1/", is
created from "proto_alpha/" where the development continues. These
directories are called *protocol snapshots*.

The hash of each active or candidate protocol is computed from the
directory "src/proto_0*/lib_protocol/", so every change in these
directories is forbidden. However, the protocol snapshots contain
other subdirectories than the frozen "lib_protocol/", implementing
protocol-dependent code. For instance, "lib_client/" implements
commands in the Octez client that can invoke protocol-dependent
functionalities, such as transfers between accounts.

Thereby, if your MR changes code in the protocol snapshot directories
(outside the frozen "lib_protocol/"), you may consider patching the
corresponding code for the other protocols as well (at least the ones
which are still in use), when applicable.


The Migration
-------------

Right before the activation of a new protocol, there is a migration of
the context that takes place. This migration typically converts data
structures from the old to the new format. Each migration works
exclusively between two protocol hashes and it is useless otherwise.
For this reason after the activation of a protocol the first step to
start a new development cycle is to remove the migration code. In
order to facilitate this, *migration code is always in a different
commit* with respect to the protocol features it migrates. When
submitting an MR which contains migration code, **the author must also
have tested the migration** (see How to Test a Protocol Proposal) and
write in the description what was tested and how so that **reviewers
can reproduce it**.


Code Review
===========

All code is peer reviewed before getting committed in the master
branch by the Octez merge team. Briefly, a code review is a discussion
between two or more developers about changes to the code to address an
issue.


Merge Request Approvals
-----------------------

Two approvals from different Octez Octez merge team members are
required for merge requests to be merged. Both approvals must result
from independent, thorough reviews. After both reviews, the second
approver will also typically merge if the request is not a draft
(which otherwise means that the author has still some cleaning up to
do).

However, for less critical parts of the code, an Octez merge team
member may choose to trust the review of a developer who is not a
member of the Octez merge team. In that case, the Octez merge team
member may choose to count that review as their own, effectively
"converting" the other developer's passing review into an approval.
This may be done even when the Octez merge team member is the author
of the merge request. A second independent review from an Octez merge
team member is still required, of course.

For this reason, if you make a partial review, for instance if you
only reviewed part of the code, or only the general design, it is good
practice to say so in a comment, so that other reviewers know what is
left to review. If you manually tested the merge request or ran some
benchmarks, you can add a comment with the results.


Author Perspective
------------------

Code review is a tool among others to enhance the quality of the code
and to reduce the likelihood of introducing new bugs in the code base.
It is a technical discussion; not an exam, but rather a common effort
to learn from each other.

These are a few common suggestions we often give while reviewing new
code. Addressing these points beforehand makes the reviewing process
easier and less painful for everybody. The reviewer is your ally, not
your enemy.

* Commented code: Did I remove any commented out lines? Did I leave a
  TODO/FIXME comment without an issue number?

* Docstrings: Did I export a new function? Each exported function
  should be documented in the corresponding ".mli" (or directly in the
  ".ml" file if there is no ".mli").

* README: Did I check whether my changes impact the corresponding
  README file(s)?

* Readability: Is the code easy to understand? Is it worth adding a
  comment to the code to explain a particular operation and its
  repercussion on the rest of the code?

* Variable and function names: These should be meaningful and in line
  with the conventions adopted in the code base.

* Testing: Are the tests thoughtful? Do they cover the failure
  conditions? Are they easy to read? How fragile are they? How big are
  the tests? Are they slow?

* Are your commit messages meaningful? (see https://cbea.ms/git-
  commit/)

Review your own code before calling for a peer review from a
colleague.


Reviewer Perspective
--------------------

Code review can be challenging at times. These are suggestions and
common pitfalls a code reviewer should avoid.

* Ask questions: How does this function work? If this requirement
  changes, what else would have to change? How could we make this more
  maintainable?

* Discuss in person for more detailed points: Online comments are
  useful for focused technical questions. On many occasions it is more
  productive to discuss it in person rather than in the comments.
  Similarly, if discussion about a point goes back and forth, it will
  be often more productive to pick it up in person and finish out the
  discussion.

* Explain reasoning: Sometimes it is best to both ask if there is a
  better alternative and at the same time justify why a problem in the
  code is worth fixing. Sometimes it can feel like the changes
  suggested are nit-picky without context or explanation.

* Make it about the code: It is easy to take notes from code reviews
  personally, especially if we take pride in our work. It is best to
  make discussions about the code than about the developer. It lowers
  resistance and it is not about the developer anyway, it is about
  improving the quality of the code.

* Suggest importance of fixes: While offering many suggestions at
  once, it is important to also clarify that not all of them need to
  be acted upon and some are more important than others. It gives an
  important guidance to the developer to improve their work
  incrementally.

* When you consider that a fix is important but should not prevent the
  current MR to be merged (e.g., because it adds a sufficient amount
  of useful new features), you may suggest creating a follow-up issue.
  If the place in the code that needs to be fixed later is clear, you
  may also suggest marking it with a TODO/FIXME comment.

* Take the developer's opinion into consideration: Imposing a
  particular design choice out of personal preferences and without a
  real explanation will incentivize the developer to be a passive
  executor instead of a creative agent.

* Do not re-write, remove or re-do all the work: Sometimes it is
  easier to re-do the work yourself discarding the work of the
  developer. This can give the impression that the work of the
  developer is worthless and adds additional work for the reviewer
  that effectively takes responsibility for the code.

* Consider the person you are reviewing: Each developer is a person.
  If you know the person, consider their personality and experience
  while reviewing their code. Sometimes it is possible with somebody
  to be more direct and terse, while other people require a more
  thorough explanation.

* Avoid confrontational and authoritative language: The way we
  communicate has an impact on the receiver. If communicating a
  problem in the code or a suggestion is the goal, making an effort to
  remove all possible noise from the message is important. Consider
  these two statements to communicate about a problem in the code :
  "This operation is wrong. Please fix it." and "Doing this operation
  might result in an error, can you please review it?". The first one
  implies you made an error (confrontational), and you should fix it
  (authority). The second suggests to review the code because there
  might be a mistake. Despite the message being the same, the
  recipient might have a different reaction to it and impact on the
  quality of this work. This general remark is valid for any comment.

When reviewing MRs involving documentation, you may check the built
documentation directly within the GitLab interface, see Building the
documentation in the CI.


Developer Tools
---------------

Some tools to make protocol development more convenient can be found
in the src/tooling/ folder. In particular, it contains "tztop", a REPL
(interactive read-eval-print loop) based on "utop".

How to add or update opam dependencies
**************************************

When a merge request (MR) introduces a new dependency to an opam
package, or updates an existing dependency to a different version of
an opam package, additional steps must be taken in the development and
merge process. This document explains those steps.

If you have already read this guide and only need a refresher, skip to
the TL;DR.


Background
==========

The Octez project is built under a system that is somewhat stricter
than the default for OCaml projects. The goal is to make sure that
users, developers and the CI use the exact same dependencies, with the
exact same versions. To this end:

* the set of opam dependencies and their exact version number is
  stored in an opam lock file ;

* the hash of the commit to use from the public opam repository is
  stored in scripts/version.sh in the variable "opam_repository_tag";

* "make build-deps" and "make build-dev-deps" use the lock file and
  the hash to select dependencies;

* the CI builds and uses Docker images using this information.

Note:

  The CI Docker images contain additional dependencies such as "odoc"
  which are needed by the CI but not to build Octez.

The rest of this document explains the process of adding, removing or
updating dependencies from the point-of-view of a developer (you). The
instructions below assume you have already set up your work
environment but that you installed *development* dependencies ("make
build-dev-deps" instead of "make build-deps").


Local work
==========

The simplest way of using a new dependency on the Octez codebase when
working locally (i.e., on your own machine) is to install it using
"opam".

Because you have used "make build-dev-deps" in order to install the
Octez dependencies, you have access to the default opam repository.

**Install your dependency:** "opam install foo"

**Add dependencies to build files:** both opam files and dune files
must be updated. Add the dependency to the relevant declarations in
"manifest/". And then use "make -C manifest" to update the opam and
dune files accordingly.

For example, if you are modifying the Shell using the new dependency,
you must add an entry in the "~deps" list of the "let octez_shell ="
entry of the manifest/product_octez.ml and then run "make -C
manifest". You should see the changes propagated onto opam/octez-
libs.opam and src/lib_shell/dune, as well as opam/virtual/octez-
deps.opam.

You can work on your feature, using the types and values provided by
your new dependency.


Making an MR
============

Even though you can compile and run code locally, the CI will likely
fail. This is because its set of available dependencies is now
different from yours. You must follow the steps below in order to
produce the necessary Docker images, allowing your work to eventually
be merged.

First, in your local copy of Octez, **update the**
"opam_repository_tag" **variable in the** scripts/version.sh **file**.
You should set this variable to the commit hash of a recent version of
the "master" branch of the default opam repository. (Note: this is not
always necessary, but it is simpler for you to do so than to check
whether it is necessary to do so.)

Second, update the opam lock file. The safest way to do that is to
**execute the** scripts/update_opam_lock.sh **script**. It will ask
opam to upgrade all Octez dependencies, making sure that unwanted
package versions are not selected for dependencies, and will update
the lock file accordingly. Note that the diff may include a few more
changes than what you strictly need. Specifically, it might include
some updates of some other dependencies. This is not an issue in
general but it might explain some changes unrelated to your work.

Note:

  If you do not wish to upgrade all dependencies, you can also just
  run "opam lock opam/virtual/octez-deps.opam" followed by "mv octez-
  deps.opam.locked opam/virtual", or even edit the lock file manually.
  Neither of these guarantees that packages are available in the
  commit identified by "opam_repository_tag" of the public opam
  repository, and even so, you may end up with unwanted versions of
  dependencies; so you should review the resulting lock file even more
  carefully. Editing the lock file manually is even less safe than
  running "opam lock" as it does not guarantee that the set of
  dependencies is actually a valid solution that the opam solver could
  have chosen.

Third, commit the change of "scripts/version.sh" and the updated
lockfiles with a title along the lines of “CI: use dependency "foo"”.

Finally, **push these changes and open an MR**.


TL;DR
=====

As a developer:

* You have an Octez MR from "<your-organisation>/tezos:<your-branch>"
  onto "tezos/tezos:master" introducing a dependency to "foo".

* You amend the "manifest/" files to declare the dependency.

* You propagate the changes to "opam" and "dune" files by running
  "make -C manifest".

* You update the "opam_repository_tag" to the commit hash of a recent
  version of the public default opam repository.

* You update opam/virtual/octez-deps.opam.locked, for instance by
  executing scripts/update_opam_lock.sh.

* You push the changes to your Octez MR.

As a merger there are no special steps to take:

* You test, review, etc. the code.

* You assign the Octez MR to Marge Bot.

Octez Merge Team
****************

The Octez merge team is a group of developers from different teams of
the Tezos ecosystem. They are automatically selected as approvers when
you create a merge request.


Octez Merge Team Scope
======================

The role of the Octez Merge Team is limited to ensuring the quality of
what gets merged into Octez, i.e. all of tezos/tezos except
src/proto_alpha/lib_protocol/. By quality, we mean ensuring that the
code is as good as it can be, be it at the implementation level or the
architecture level.

The Octez merge team does not decide what contributions are included
in the next protocol ("src/proto_alpha/lib_protocol/"). Companies that
contribute to tezos/tezos reach a consensus to decide what gets in the
Alpha protocol, i.e. in the proposal of the next upgrade. In
particular, other companies can fork this repository and do their own
protocol proposals.


Getting into the Octez Merge Team
=================================

The Octez merge team is always looking for software engineers with at
least the following qualities:

* You are an active developer on the tezos/tezos repository.

* You have done numerous reviews and merge requests in the past, i.e.
  you have shown your expertise and interest in contributing both with
  code and reviews.

* You are paying attention to detail and are on the lookout for
  possible errors, security issues, and/or enhancements in your
  reviews. It's fine not knowing the entire codebase (no one does),
  but you should nevertheless actively try to make the code of others
  better when you do reviews.

* You are well capable in software design and can identify subpar
  design choices.

* One way to show attention to detail and to design is to show these
  qualities in your own merge requests. You should also be willing to
  amend your merge requests by taking into account the feedback of
  reviewers, be it for small changes or architectural changes.

* You know when your expertise is insufficient and you are keen to ask
  for support.

* You are relatively skilled in one of the technologies used in the
  "tezos/tezos" repository, i.e. "OCaml", "python", "CI", building,
  packaging, etc.

To apply for being included in the Octez merge team, contact an
existing member on the tezos-dev Slack. Your application will be
discussed during the next weekly meeting of the Octez merge team.


Helping the Octez Merge Team
============================

Even if you are not in the Octez merge team, you can help it! If you
review a merge request and you think it is ready to be merged, you
should click the *Approve* button even if your approval "doesn't
count".

If the merge request is not critical, or if you are knowledgeable on
the merge request topic; your approval will make it easier for an
Octez merge team member to approve too. This is also an excellent way
to show your skills and willingness to move development forward,
contributing to potentially making you an Octez merge team member in
the future (see *Getting into the Octez Merge Team* above).

Documentation and coding guidelines
***********************************

This document provides guidelines that should be observed by all the
contributors to the Octez codebase. It first presents documentation
guidelines, and then rules specific to coding.


Documentation guidelines
========================

Documentation guidelines are presented for different scales:
functions, modules, files, and libraries. General remarks applying to
all scales are presented afterwards.


Comments within the code
------------------------

The OCaml code should include comments facilitating comprehension and
maintenance.

Towards this goal, comments are useful in the code wherever they bring
important information that is not easily deducible from the code
itself, such as: explaining the high-level goal of a code fragment,
justifying an unusual pattern of code, exhibiting an  invariant, etc.
See also rule Don't repeat the code.

Additionally, the main syntactic constructs in the code should be
commented as follows.

Modules:

* One-line comment explaining the purpose of the module

* If needed: More detailed description

The comments on a module may describe the overall abstraction provided
by the module and its capabilities, without implementation details or
even specifics of the different functions. It may also describe
limitations of the module that are important to developers using it
(e.g. concurrency or performance limitations).

Types:

* One-line comment explaining what the type represents, typically the
  invariants satisfied by its inhabitants

Functions and methods:

* Purpose of the function, brief description of the arguments and
  returned value

* If needed: How and why to use this function

* If needed: Pre-conditions for calling the function, including
  constraints on arguments values or dependencies between arguments,
  if any

* If needed: Conditions under which the function will return an error
  or raise an exception

* If needed: Any special behavior that is not obvious

Constants, variables, variants, and struct fields:

* Purpose and definition of this data. Useful comments do not just
  expand a name using the same words, but rather describe the data
  including details that could not be included in its name.

* If there is an associated measurement unit, define it in the
  comment, unless it is contained in the name, e.g.,  "TIMEOUT_MS" for
  a timeout in milliseconds.


TODO/FIXME comments
~~~~~~~~~~~~~~~~~~~

During the code review process, follow-up issues may be created to
improve some piece of code that already implements its specification
(e.g., optimize, refactor, or bring a potentially useful
generalization). When the place of the future evolution is known in
advance (e.g. a given function), you should mark it with a "TODO"
comment of the form:

   (* TODO: <reference to issue>
      <one-line explanation>
      <Add long explanation if the issue description is not the right place.>
   *)

If the evolution is needed to fix some code that does *not* fully
implement its specification, (e.g., a known bug detected but not yet
fixed, or a temporary implementation handling only some particular
cases), rather than to merely improve the code, you should use a
"FIXME" tag instead of the "TODO", adhering to the same syntax for the
rest.

Thus, the difference between "TODO" and "FIXME" tags is a semantic
one,  reflecting the full/partial implementation of the specification.
Consequently, when the specification evolves to become more demanding,
some "TODO" tags corresponding to *potential* evolutions may have to
be recasted as "FIXME" tags, corresponding to *required* evolutions.

Another consequence of the fact that FIXME tags indicate an already-
specified remaining work, they can point to a merge request (MR)
instead of an issue, especially when the remaining work has already
been started. This may correspond to at least the following familiar
scenarios:

1. You are making a list of small-sized MRs to fix one big issue, and
   in one MR you want to point the reviewers to other MRs already
   started.

2. While working on some MR, you notice a problem with your code, by
   which it derives from the specification. But you already have one
   approval for the current MR and you don't want to invalidate it by
   adding a significant chunk of code. So you fix the problem in a
   separate MR from the one you are working on.

In the above cases, it would be possible to create a new (sub-)issue,
and reference it in the code (and also in the MR). But that would add
a level of indirection, both for the person doing the work and for the
person reviewing the work. Therefore, in such cases, it can be more
convenient to refer to an MR directly.

Note that the reference to an existing issue or MR on the first line
is mandatory, to facilitate searches of specific evolutions. These
tags might be checked automatically, e.g. to signal tags that refer to
non-existent issues or MRs, to closed issues, or to already merged
MRs. The reference to an issue may be one of:

* a URL such as "https://gitlab.com/tezos/tezos/-/issues/1377"

* a GitLab notation for an issue such as "#123" (implicitly under
  "tezos/tezos"), "michelson-reference#123" (implicitly under "tezos
  /michelson-reference"), or "oxheadalpha/merbocop#123" (fully
  qualified).

* a GitLab notation for a merge request such as "!10990".


License
-------

The Octez software is distributed under the MIT license. Every OCaml
source file should start with a header comment instantiating the
following template:

   (*****************************************************************************)
   (*                                                                           *)
   (* SPDX-License-Identifier: MIT                                              *)
   (* SPDX-FileCopyrightText: [year(s)] [Holder <email>]                        *)
   (*                                                                           *)
   (*****************************************************************************)

For other languages than OCaml, use the appropriate comment syntax.
For example, here is the Rust template:

   //
   // SPDX-License-Identifier: MIT
   // SPDX-FileCopyrightText: [year(s)] [Holder <email>]
   //

The following, full-text equivalent, template for OCaml is also valid
but deprecated:

   (*****************************************************************************)
   (*                                                                           *)
   (* MIT License                                                               *)
   (* Copyright (c) [year(s)] [Holder <email>]                                  *)
   (*                                                                           *)
   (* Permission is hereby granted, free of charge, to any person obtaining a   *)
   (* copy of this software and associated documentation files (the "Software"),*)
   (* to deal in the Software without restriction, including without limitation *)
   (* the rights to use, copy, modify, merge, publish, distribute, sublicense,  *)
   (* and/or sell copies of the Software, and to permit persons to whom the     *)
   (* Software is furnished to do so, subject to the following conditions:      *)
   (*                                                                           *)
   (* The above copyright notice and this permission notice shall be included   *)
   (* in all copies or substantial portions of the Software.                    *)
   (*                                                                           *)
   (* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR*)
   (* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,  *)
   (* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL   *)
   (* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER*)
   (* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING   *)
   (* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER       *)
   (* DEALINGS IN THE SOFTWARE.                                                 *)
   (*                                                                           *)
   (*****************************************************************************)

Note that:

* The holder, on the copyright line, is the name of the company which
  hires the employee or the sub-contractor.

* For sub-contractors, check your specific contract terms. They
  sometimes allow to include, as an additional copyright holder, the
  name of a particular developer, but consider that this may end up
  with bloated license headers.

* When adding a significant new contribution to a file (i.e. more like
  whole new features, rather than simple fixes), check whether there
  already is a copyright for your copyright holder (see above).

  * If there is one, mentioning any year, it is not required to add
    the current year (but this is allowed). In no case should you
    *replace* the existing year with the current one.

  * If there is no line for your copyright holder, you should add one,
    with the current year.


Documenting interfaces and implementations
------------------------------------------

At the granularity of OCaml files, it is essential to document the
interface implemented by each file. In many cases, it is useful to
also document the implementation, but *separately* from the interface.

Implementation (".ml") files:

* Document the interface:

  * In the common case where there is a corresponding interface
    (".mli") file, document the interface file instead, as detailed
    below.

  * In the less common case where there is no corresponding interface
    (".mli") file, document the exported elements directly in the
    implementation (".ml") file.

* Document the implementation: For many non-trivial implementations,
  it is most useful to document the design principles, code structure,
  internal invariants, and so on. Such information should be placed in
  a comment block at the top of the file.

Interface (".mli") file comments:

* One-line description

* Brief description of the library, introducing the needed concepts

* Brief description of each module, type, function, data, as described
  for comments in the code but using special comments

* If applicable, external invariants (i.e., visible to the user).

Interface comments should omit implementation details, to the extent
where these are invisible to users of the interface.


Using docstrings
~~~~~~~~~~~~~~~~

The documentation of OCaml interfaces is automatically generated by
"odoc" from *special comments* in the source code, of the form "(**
... *)", also called documentation comments, or "docstrings". Here are
a few tips and guidelines on using docstrings.

* A documentation page is generated for each module *interface* and
  module type, and for each class *interface* and class type, from the
  docstrings attached to them and to their contained elements.
  Docstrings inside a module *implementation* (i.e. "struct ... end"
  constructs) are not kept. You should use normal comments ("(* ...
  *)") inside the implementation.

* The page generated for each interface starts with a "preamble",
  consisting of: the docstring on its declaration, if any, and the
  "top-comment" inside the interface (the first docstring inside the
  interface not attached to a contained element), if any. The first
  paragraph of the preamble, if any, will serve as the "synopsis" of
  the interface, and will be used when the interface appears in a list
  (typically, in the page of its containing interface).

* When it makes sense, you should document not only interfaces at the
  top level of a file, but also those embedded in other (module or
  class) interfaces.

* Docstrings in ".mli" and ".ml" files are handled the same, so do not
  omit documenting the interfaces in the latter files.

For more information on using docstrings, see the "odoc" documentation
for library authors. In particular, it contains a cheatsheet that may
be useful.


Docstrings errors
"""""""""""""""""

When "odoc" is generating documentation from docstrings, it performs
various syntax and semantics checks and may thereby emit many kinds of
warnings. Some of these warnings are turned into errors when the flag
"ODOC_WARN_ERROR" is on. You have to fix at least these errors for
making the CI green.

When "odoc" returns a non-zero exit code, the list of errors is
displayed in the terminal but some details are abstracted away. The
full details of each error, and also the regular warnings, can be
found in the log produced by "odoc" in file "${TMPDOCDIR}/odoc.log",
where variable "TMPDOCDIR" is defined in file docs/Makefile. Search
for the string "Error:" in that file to find all the errors.

You may consult a list of typical error messages found by odoc in the
Tezos repository. These examples may help both to avoid common
pitfalls when writing docstrings,  and to better understand "odoc"
errors that you may encounter.


index.mld files
---------------

At the granularity of the library, you can optionally include an
"index.mld" file. This file is used to generate the landing page for
the online API of the library. If you do not include this file, the
landing page is automatically generated and only includes a list of
all top-level modules.

An "mld" file is written in the ocamldoc markup language.

Because it is used for the online API of the library it should contain
information that might be of interest to the users. This includes

* A general introduction to the library.

* Design decisions for the API.

* Usage recommendations including the security model and assumptions.

* Example uses.

* Links to related documents (such as RFC and ISO standards).

The file should also include a link to all of the modules that are
intended entry-points for the users. You can include those with
"{!module:Foo}" inline in an appropriate paragraph of the
documentation. You can also include those with a dedicated block:

   {!modules:
   Foo
   Bar
   Blah
   }

Warning:

  When you add an "index.mld" file, don't forget to add a
  "~documentation:[]" parameter to the package's manifest which adds a
  documentation stanza in the corresponding dune file. Otherwise, your
  "index.mld" file will be ignored by "odoc" and the API page will be
  the default one!


README files
------------

Also at the level of the library, you should include a "README.md" in
Markdown format. Such files are mandatory in top-level directories of
the Octez codebase (such as "src/" and "docs/"), and at least in
immediate sub-directories of the source directory ("src/*/"). Because
it is accessible only in the source tree, the file should contain
information that might be of interest to the maintainers and
contributors.

You must instantiate the "README.md" file with the following template:

   # Component Name
   <!-- Summary line: One sentence about this component. -->

   ## API Documentation
   <!-- Link to the external API. -->

   ## Installation
   <!-- Describe how this component can be installed (if applicable). -->

   ## Overview
   <!--
   - Describe the purpose of this component.
   - Describe the interaction of the code in this directory with the other
     components. This includes dependencies on other components, for instance.
   -->

   ## Implementation Details
   <!--
   - Describe the file structure and the location of the main components.
   - Other relevant implementation details (e.g., global invariants,
     implementation design rationale, etc.).
   - Testing specifics, build-system specifics, etc. as needed.
   -->

The rationale of this template is that a README file addresses the
developers that are not just using the library but also fixing or
modifying it. To avoid duplication between the "index.mld" and
"README.md" files, follow this simple rule-of-thumb: the "index.mld"
contains information for the users (how to use the API) whereas the
"README.md" file contains information for the maintainers (how are the
files organised, are there any build-system quirks, etc.).

When filling in the template, you should keep untouched the guidelines
within HTML comments (which are visible to the document maintainers
but invisible to end-users), so that any maintainer can check how well
the README instantiates the template, and address any gap if needed.


General documentation rules
---------------------------


Don't repeat the code
~~~~~~~~~~~~~~~~~~~~~

When writing comments, a common mistake is to repeat the code,
paraphrasing it with the same words, or at the same abstraction level.
Good comments bring information that is not obvious from the code
itself. They complement the code with details at a *different level of
abstraction*, thus typically using different words to provide details
on the meaning of the code.

* Lower-level comments add precision

  Such comments add precision by providing details not easily inferred
  from the code. For example, comments on a function may define its
  behaviour in particular cases, or provide defaults for the
  arguments; comments on a variable may add details, such as the
  measurement units, boundary conditions, invariants, or define the
  meaning of the different values (e.g. for a boolean variable).

* Higher-level comments enhance intuition

  Such comments are written at a higher level of discourse than the
  code, and provide a basis to understand the provided asbtractions
  and underlying concepts, and to reason about them. For example:

  * An implementation comment inside a module or function body may
    explain the overall goal and provide a conceptual framework in
    which the details of the code may be easily understood.

    Typically, the comment will explain *what* the code is doing, and
    possibly *why* (e.g., in which context the code is called), rather
    than *how* it is done --  this can be usually seen in the code
    itself.

  * A comment on the interface provided by a library may describe the
    solution provided for the given problem domain, how the different
    functions may be combined together, some limitations of the
    library (e.g. in terms of concurrency), etc.


Comments maintainability
~~~~~~~~~~~~~~~~~~~~~~~~

Maintaining comments up-to-date with the code is a crucial component
of their usefulness. Moreover, if many comments in a piece of software
become out of date, developers may start ignoring them altogether.
Fortunately, a few techniques may greatly increase the maintainability
of comments.

* Place comments near the concerned code.

  Comments are more likely to be updated if they are placed at the
  "natural" place where developers would look for them. For interface
  comments, the position of docstrings is already defined by Odoc: on
  the definition of the module, function, or variable. Implementation
  comments should be placed near the code they describe; avoid placing
  all the comments at the beginning of the file, unless their scope is
  the whole file.

* Stay at a high level.

  Usually, the farther comments are from the code, the more high-level
  they must be. Comments associated to a large scope of code may
  typically present a global view of the code in general terms, which
  makes them less likely to be invalidated by local changes; when
  changes concern a large part of that scope, they can be impacted but
  there are chances that developers will see them.

* Avoid duplicating comments.

  Some comments are needed in several places.

  * For instance, an externally-visble module exporting part of the
    functionality of some internal module(s) should describe each
    exported element, because users may not look to the internal
    module(s); but the internal modules should also describe their
    interface for their own users and maintainers.

    In such cases, duplication may be avoided by choosing a single
    place for the comments, and using cross-references to it wherever
    needed. Fortunately, Odoc provides such cross-references and
    checks for each reference that the target element exists, ensuring
    they do not get stale when code is moved or suppressed.

  * In other cases, the documentation needed near some piece of code
    is already available outside the code. For instance, a user-
    visible client command or node RPC should not duplicate
    explanations in the user reference.

    Most such cases are handled in Octez by automatically generating
    the reference manual from the code. When this is not the case,
    comments in the code should refer to the outside sources instead
    of duplicating them, by using: web links to the technical
    documentation, to GitLab repositories, or to third-party websites
    when appropriate (e.g. Wikipedia for known concepts).

  * Another important case is comments describing a cross-module
    design decision or property.

    Duplication can be avoided here by using cross-references, as
    described above. However, if all the modules belong to the same
    library, you could rather place the comments in a "index.mld" file
    describing the library, and refer to that file in all the needed
    places, using Odoc-checked cross-references.

* Envision cohesive documentation.

  * When writing online technical documentation, refer whenever needed
    to code artefacts: module APIs, GitLab entities in the Octez
    repository, or source files, using our Sphinx-checked custom
    roles.

  * More generally, including various kinds of checked cross-
    references in the documentation makes it more cohesive, more
    robust and easier to maintain. Indeed, the more checked links you
    include in the documentation (from comments to related code
    elements, from comments to ".mld" files and back, from external
    documentation to code and back, to external websites, etc.), the
    more are chances that the documentation will be updated along with
    the code, when some links break.

    Our documentation tools (Odoc and Sphinx, currently) provide
    support for such cohesive documentation which is probably more
    advanced than the average, so take advantage from it!


Logging vs comments
~~~~~~~~~~~~~~~~~~~

Sometimes, it may be preferable to use logging instead of commenting a
section of code. That is, consider using:

   Log.info "Does this and that..." ;

instead of:

   (* Does this and that... *)

and certainly instead of:

   (* Does this and that... *)
   Log.info "Does this and that..." ;

This form of "executable comments" is appropriate for instance when
writing tests, and more generally, whenever a dynamic execution trace
is useful, provided that the log message contains all the information
that you would put in a comment. Of course, if the log message is too
short to fully describe the code section, consider completing it
(rather than duplicating it) with comments.

For details on logging, see Logging Levels.


Comment first
~~~~~~~~~~~~~

This section is by no means a rule to comply with, but just some
arguments, advocated by John Ousterhout [1], for a documentation
practice consisting in writing most comments *before* writing the
code, or *while* writing the code. These arguments are left to the
appreciation of each developer.

Most comments, including all the docstrings that describe interfaces,
are more related to the software design phase than to the other phases
of software development such as coding, testing, releasing. Indeed,
these comments are currently the only way to fully describe the
abstractions provided by software and their underlying concepts.
Writing comments during the design phase (e.g. woven with the
definition of interfaces), may be an effective way to improve the
quality of both the design and the comments, because:

* During this phase, the developers' mindset is at the right level of
  abstraction to describe software components clearly and simply, not
  distracted by implementation details. This usually results in better
  comments.

* Moreover, by expressing the interface design concisely in words,
  comments are an effective design tool: they allow evaluating the
  simplicity of interfaces from the start, and thus producing better
  designs. For example, a module or function difficult to describe in
  a simple way may be a sign that the design does not effectively hide
  complexity or does not provide the right abstractions.

* Finally, writing comments before the code may be more fun and
  rewarding. Indeed, integrating comments in a useful way in the
  design process can make documentation more appealing, and a well-
  presented result may by more satisfying. As opposed to that, many
  developers are writing comments as some sort of menial work to do
  after code is complete, when the design has to be partly
  reconstituted from the code; this typically leads to more work, and
  to lower-quality, and hence less satisfactory comments.


Coding guidelines
=================

This section presents coding guidelines, related to aspects such as
logging levels, code formatting, and naming.


Logging Levels
--------------

The Octez libraries use a logging library with 5 different verbosity
*levels* defined in "src/lib_event_logging/internal_event.mli" for the
shell and "src/lib_protocol_environment/sigs/v3/logging.mli" for
protocol code.

It is important to choose the appropriate level for each event in the
code to avoid flooding the node administrator with too much
information.

These are the rules of thumb that we use in the code to decide the
appropriate level (here listed from most to least verbose) for each
event:

* "Debug" level, the most verbose, is used by developers to follow the
  flow of execution of the node at the lowest granularity. Feel free
  to add detailed info while staying reasonable, as emitting too many
  logs can be costly even if they are not printed anywhere.

* "Info" level contains additional information that you might want to
  have, but these logs are not important to have if your node is
  running OK (and definitely do not require any action). These logs
  are stored in the daily logs (by default) that devs may want to look
  for more technical info but should not be too verbose to prevent the
  log files from growing too large.

* "Notice" level (the default) should be about things that the node
  admin should be concerned, but do not require any action. These logs
  are printed on stdout and should be short, concise, noise-free, and
  easy to read and understand for a non-developer. They should
  indicate progress and surprising behavior.

The two following levels are used to alert the node administrator
about potential problems and errors:

* "Warning" level are all those events that might require the node
  administrator's attention and can reveal potential anomalies in the
  node's operation.

* "Error" level are all those events that require an intervention from
  the node administrator or signal exceptional circumstances.

There is another level "Fatal" with the highest priority but it is
rarely relevant. Specifically, "Fatal" should be reserved for errors
that absolutely cannot be recovered. All logging at the "Fatal" level
should be immediately followed by a call to "Lwt_exit.exit_and_raise".

Note that a library is never able to decide whether a certain
condition is fatal or not. Indeed, the application that calls into the
library may not consider the function call as essential to the
continuation of the application's main purpose. Consequently, "Fatal"
should never be used within libraries.


Logging Guidelines
------------------


Make logs for higher severity levels actionable
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Logs with levels "Fatal", "Error", and "Warning" should be actionable.
Ensure the log provides sufficient information for debugging and
reproduction if possible. Include object type, identity (variable and
value that identifies the context, such as "pid: 123"), state, error
code and other critical info if applicable.

Examples of actionable log:

   (error code E1007) file ~/.octez-node/config.json is not valid JSON, please remove it and reconfigure your node with 'octez-node config init'

   [pid:42825][rpc_middleware] : (16) got conn closed, but shutdown callback not found

Example of poor logs:

   unexpected error

   cannot parse configuration


Avoid impacting performance
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Carefully consider the balance between the performance impact and the
need for debug capabilities.

* Be cautious about logging in performance-critical sections of the
  code to avoid unnecessary overhead.

* Ensure that calculations done solely for logging are performed
  within the log statement itself, ensuring they are only executed if
  the logs are enabled.


Make logs readable
~~~~~~~~~~~~~~~~~~

Print out states and error codes as a plain text unless the state is
well known as a number. Good examples are "Enabled" and "404".


Maintain consistency in "Debug" logs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

"Debug" logs are often extensive, so prepare them for easier analysis:

* Maintain a consistent style for formatting states and identities,
  allowing easy log filtering. For example, if you choose to write
  "pid: 123", stick to it and avoid using "pid=123" elsewhere for the
  same context.

* Place identities in the same order, making it easier to filter logs
  by multiple parameters simultaneously. For example, use "conn: 123,
  pid: 456".

* Prefer a table style for logs. Group identities at the beginning of
  the line and follow with a clear description.

* Make the log originator easily identifyable. Enabling "Debug" logs
  typically triggers logs from multiple components and logical areas.
  Ensure it is easy to filter out logs belonging to one logical area
  or one log originator.

Example of consistent logs:

   Jun 07 18:11:41.604: [pid:95779][resto] server (18) got conn closed
   Jun 07 18:11:41.606: [pid:95662][node_run_cmd] (6) got conn closed
   Jun 07 18:11:41.606: [pid:95662][resto] server (6) got conn closed
   Jun 07 18:11:41.606: [pid:95662][resto] server (6) closed bound streamed connection

Example of poor logs:

   May 29 12:15:59.267: started
   May 29 12:15:59.267: requester worker timout
   May 29 12:16:31.467: connection closed 3
   May 29 12:16:31.468: (4) received request


Privacy and Security
~~~~~~~~~~~~~~~~~~~~

Avoid logging sensitive information. Ensure that logs do not expose
private data, keys, or personally identifiable information (PII).


Code formatting
---------------

To ensure that your OCaml code is well formatted, set up correctly
your editor:

* automatically run "ocamlformat" when saving a file

* no tabs, use whitespaces

* no trailing whitespaces

* indent correctly (e.g. use lisp-mode for dune files)

Some of these checks can be executed with a pre-commit hook (see Pre-
Commit Hook for more details).


Exposing internals
------------------

Sometimes you want to expose some internal functions, types or sub-
modules of a module: usually it is for testing part of the module
logic, but it may also be to let some power users access internals
(for performance, extra-functionality, or any other reason).

A first question you should ask yourself is: "Is this a hint that my
module has too many responsibilities?".

If the answer is yes: consider splitting your module per
responsibility and see if you still need to expose internals.

If the answer is no: the guideline is to expose an internal module as
part of your API:

* "Internal_for_tests" for internal types, functions and sub-modules
  that are only meant to be accessed by tests

* "Internal" for internal types, functions and sub-modules that are
  meant to be accessed by production code (but may also be accessed by
  tests)

Additionally you should add the "(**/**)" Stop special comment so that
this module is not displayed in the public module documentation.

The rationale of "Internal_for_tests" is to make it explicit both for
developers writing code and for reviewers that this module must not be
used in production code.

Example:

    (* mli *)
    type t

    val f : t -> u

    (**/**)

    module Internal_for_tests : sig
      (** The actual representation of [t] *)
      type t_raw = {bar : string; baz : int}

      (** Access inner fields of a value of type [t] *)
      val unwrap : t -> t_raw

      (** An internal function that we want to test in isolation *)
      val g : t -> v

      (** You can also expose internal modules, e.g. the inner cache *)
      module Mycache : Cache.S
    end

    (* ml *)
    module Mycache = Cache.Make ...

    type t = {bar : string; baz : int}

    (* [g] is an internal function, not part of the public API *)
    let g t = ...

    (* [f] uses internal function [g] *)
    let f t =
      let w = g t in
      ...

    module Internal_for_tests = struct
      type t_raw = t = {bar : string; baz : int}

      let unwrap t = t

      let g = g

      module Mycache = Mycache
    end


Exceptions and errors
---------------------

The following pieces of advice should be applied in general, although
exceptions apply (pun intended).

* Only use exceptions locally and don't let them escape: raise them
  and catch them within the same function or the same module.

  * If a function that is exported can fail, return a "result" or a
    "tzresult".

  * If you cannot (or for another reason do not) handle an exception
    and it may escape you **must** document it.

* Never catch "Stack_overflow" nor "Out_of_memory" which are
  exceptions from the OCaml runtime rather than the code itself. In
  other words, when one of these exception is raised in one process,
  the same exception may or may not be raised in another process
  executing the same code on other machines. When you catch this
  exception, you make a branching in the code that is decided not
  based on properties of the code, but properties of the process
  executing the code. Consequently, the same branching may differ on
  two distinct runs of the same code. This is, in essence, non-
  determinism.

  * If you are in one of the small cases where non-determinism is ok
    and you have a compelling reason to catch either "Stack_overflow"
    or "Out_of_memory", you **must** include a comment explaining why.

  * Note that catch-all patterns (such as wildcard ("| _ ->") and
    variable ("| exn ->") include "Stack_overflow" and
    "Out_of_memory".

* Do not let low-level, implementation-dependent exceptions and errors
  bubble up to high-level code. For example, you should catch
  "Unix_error" near the syscall sites (ideally, within the same
  module) and handle it there. If you cannot handle it (e.g., if the
  error is non-recoverable) you should translate it into an error that
  is more relevant to the high-level code.

  * E.g., If a file-writing call to a library function raises
    "Unix_error(ENOSPC, _, _)", the caller of that library function
    should

    * catch the exception,

    * attempt to recover (if possible; e.g., by removing other old
      files before attempting it again),

    * and if the recovery does not work (e.g., does not release
      sufficient space) or is impossible (e.g., there are no
      references to old files in scope) then it should fail in a more
      meaningful way than by forwarding the exception (e.g.,
      indicating what operation it was trying to carry).

  * In the rare case that the underlying exception/error is
    satisfactory to the higher level code, then you may propagate it
    as is.

The "Lwtreslib" and the "Error_monad" libraries provide functions that
can help you follow these guidelines. Notably, "traces" allow callers
to contextualise the errors produced by its callees.


RPC security
------------

During the development of the codebase a lot of RPC endpoints were
created, some of which are responsible for delicate or computationally
intense tasks like validating blocks or executing Michelson scripts.
While some of them are necessary for the node's users to interact with
the blockchain, others are there to expose API to processes
responsible for baking and attesting, for configuration or debugging
purposes or to facilitate development of smart contracts.

In order to mitigate risks related to exposing these endpoints, Access
Control Lists (ACL for short) were introduced to limit the scope of
the API exposed to public networks (see also RPC parameters). While
node administrators are free to configure these ACLs however they
like, there is the default ACL, which lists all the endpoints that are
**exposed by default**.

When adding a new RPC endpoint, please consider whether or not there
is a reason to call it over a public network. If the answer is yes,
you should probably consider adding the new endpoint to the ACL. If
there are also risks related to calling the endpoint by a potentially
malicious user, they should be weighed when making the decision too.
There are no simple answers here. Remember that all new endpoints are
**blocked by default** unless explicitly added to the ACL.

When changing an existing public RPC endpoint it is also important to
consider, how does the change impact possible risks related to calling
the endpoint. Should it be removed from the ACL?


RPC Versioning
--------------

General information about RPC versioning can be found in Octez &
Protocol versioning.


How to Version an RPC
~~~~~~~~~~~~~~~~~~~~~

If an RPC already has a query parameter "version", just add a variant
to the corresponding type "t_with_version" (see When to Add a New
Version to an RPC). Otherwise, the "version" query parameter should be
added (a natural number starting from "0"). See example here.

For versioning an RPC which returns a type "t", you have to write in
the service module of the RPC a type "t_with_version" which has one
constructor by version. The encoding of "t_with_version" is simply
constructed using the "Data_encoding.union" function.

To ensure that the implementation of the RPC (the directory module)
uses the version parameter we recommend that "t_with_version" is
abstract and a dispatcher is written in the service file. This way,
when a new version is added, only the dispatcher function needs to be
updated. In general, the type for this dispatcher will be:

   val t_dispatcher : t -> ~version:int -> t_with_version

A similar process can be followed to modify the input of an RPC.
However, in that case, the semantics of "union" makes the parameter
"version" optional (all versions are supported by default). It is
still interesting to version RPCs to allow removing older versions in
the future.

Notice that we use only one version number for both input and output
of an RPC.


When to Add a New Version to an RPC
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you modify the input or the output of an RPC, it is not always
necessary to create a new version for the RPC. A new version should be
brought in when a **breaking change** is introduced. Assume that an
RPC returns the following JSON value:

   {
     "foo": 5,
     "bar": {
       "baz": 10
     }
   }

If you introduce a new field "foobar" like this:

   {
     "foo": 5,
     "bar": {
       "baz": 10,
       "foobar" : 5
     }
   }

it should not be considered a breaking change. Indeed, many decoders
which accept the former value also accept the latter.

However, if you remove a field or change the encoding of a field in a
non-extensible way as above, it should be considered a breaking change
like the two examples below.

   {
     "foo": "bar",
     "bar": {
       "baz": 10
     }
   }

   {
     "bar": {
       "baz": 10
     }
   }


Linting
-------

The OCaml part of Octez code is analysed by a linter. You can check
more details in scripts/semgrep/README.md. Below are explanations for
the different rules that may trigger linting errors.


Comparing the length of two lists
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This rule detects inefficient comparisons between two list lengths and
suggests more efficient specialised functions.

When comparing the lengths of two lists it might be tempting to
compute the lengths and compare them. This seems the most
straightforward approach. Unfortunately, this approach is costly.
Specifically, "List.length xs > List.length ys" is
O("length(xs)``+``length(ys)") because each list is traversed in full.

The OCaml "Stdlib.List" module provides the function "compare_lengths
: 'a list -> 'b list -> int" which traverses both lists at once, and
only as much of it as is necessary to determine which is longer.
Consequently, the cost of "List.compare_lengths xs ys" is
O(min("length(xs)", "length(ys)") because the function stops when it
reaches the end of one list.

The value returned by "compare_lengths" is compatible with the
semantic of other comparison functions in the Stdlib. This means that
the naive comparison "List.length xs > List.length ys" can be
rewritten more efficiently as "List.compare_lengths xs ys > 0" (note
the same comparison operator is used).

In Octez, there is also "Compare.List_lengths" which provides infix
operators to compare the lengths of two lists directly. The same
example can be rewritten "Compare.List_lengths.(xs > ys)".


Comparing the length of a list to a constant
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This rule detects inefficient comparisons between a list length and a
constant and suggests more efficient specialised functions.

When comparing the length of a list to a constant it might be tempting
to compute the length and compare it to the constant. This seems the
most straightforward approach. Unfortunately, this approach is costly.
Specifically, "List.length xs > k" is O("length(xs)") because the
expression traverses the entirety of "xs".

The OCaml "Stdlib.List" module provides the function
"compare_length_with : 'a list -> int -> int" which only traverses as
much of the list as is necessary to determine if it is longer than the
constant. The cost of "List.compare_length_with xs k" is
O(min("length(xs)", "k") because it stops when it reaches the end of
"xs" or after traversing "k" elements.

The value returned by "compare_length_with" is compatible with the
semantic of other comparison functions in the Stdlib. This means that
the expression "List.length xs > k" can be rewritten more efficiently
as "List.compare_length_with xs k > 0" (note the same comparison
operator is used).

In Octez, there is also "Compare.List_length_with" which provides
infix operators to compare the length of a list to a constant
directly. The same example can be written
"Compare.List_length_with.(xs > k)".


Comparing the length of a list with zero
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A common special case of the former consists in testing if a list is
empty. While "List.compare_length_with xs 0 = 0" or
"Compare.List_length_with.(xs = 0)" test this property in constant
time, pattern-matching on "xs" or calling "List.is_empty xs" should
generally be preferred for readability.


Folding over a promise or a result
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This rule detects difficult-to-read patterns of code wherein you
traverse a list, with an Lwt promise or a result as accumulator.

When folding over a list (using "List.fold_left" or
"List.fold_right"), the accumulator can be of any type. In particular,
it can be an Lwt promise or a result, and the folding can perform some
additional control-flow.

This is valid code accepted by the compiler. But it often produces
code which is difficult to read. For example, when folding over a
promise, a small change can affect whether the traversal sequential
(one element at a time) or concurrent (all elements treated at the
same time).

To make the code more readable, you should use the functions provided
in the Octez support libraries. Specifically, the "List" module in
Octez includes Lwt-, Result-, and Lwt-Result-specific variants of all
the traversal functions ("map", "iter", "for_all", "exists", etc.)

Check the online documentation  for a full list of the content of the
"List" module.


Chaining List.concat and List.map
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This rule detects compositions of "List.concat" and "List.map" that
are suboptimal.

The specialised "List.concat_map" function is equivalent to, but more
efficient than, the composition of "List.concat" and "List.map". The
specialised traversor is even tail-recursive.

Lwtreslib provides additional combinators "List.concat_map_s",
"List.concat_map_e", and "List.concat_map_es" to replace the non-
vanilla compositions.

Check the online documentation .


Naming
------

Good names help maintainers quickly understand the code by concisely
suggesting what a variable represents or what a function computes.

This implies that good names precisely evoke the denoted value(s),
without being longer than necessary:

* *precision*: avoid using too generic names such as "buffer",
  "counter", or "result"; give instead some hint about what is stored,
  counted, or computed.

* *concision*: as you cannot put everything into a name, select the
  most important notion(s), and avoid redundant words (abbreviating or
  not is up to you; it's just about avoiding names longer than
  necessary). For instance, suffixing the name with its type such as
  "..._list" is not needed since the IDE can tell its type; just using
  plural is usually enough. On the contrary, it is useful to add
  details to the type, such as "..._dir" or "..._title" for a value of
  type string.

Beyond choosing good individual names, it is also important to ensure
*consistency* of naming. For example, if within a same module,
"operation" is used sometimes for Michelson operations and other times
for arithmetic operations, this introduces ambiguity, that can be
solved by refining the naming scheme.


Other coding conventions
------------------------

Other than the guidelines above, Octez developers should be aware of
general OCaml programming guidelines, which recommend formatting,
naming conventions, and more.

[1] 10. Ousterhout, A Philosophy of Software Design, Yaknyam Press,
        2021.

Scope of the Octez repository
*****************************

The Gitlab repository tezos/tezos contains the source code for Octez,
as well as the embedded economic protocols for Tezos.

Following widespread software engineering principles, a major part of
the code is structured in the form of libraries. Libraries foster code
reuse, by allowing to share code between the various Octez components
and/or the economic protocols, to build increasing levels of
abstractions, and so on.

Some of these libraries, such as "lib_rpc" or "lib_error_monad",
constitute an integral part of the project, in that they are directly
linked to Octez executables, while others, like "lib_benchmark" are
used in the development process as tools which allow developers to
make the software better.

Sometimes such a helper project is developed independently at first,
but later during its development it becomes significantly coupled to
one or more libraries developed as part of the Octez project.
Maintaining such a tool is frustrating, because it often breaks due to
changes in the Octez libraries it depends upon. A possible solution to
this problem would be to include the tool in "tezos/tezos" repository
and develop it together with Octez, which allows to discover breakages
more quickly and prevent or fix them immediately.

Because the number of helper projects in the Tezos ecosystem is still
growing, it's impossible to develop all of them within the Octez
repository. Therefore it is necessary to select which projects can be
merged into the Octez repository and which cannot. The currently
accepted rule is that to be merged into Octez, an external project
should fulfill *all* the following criteria:

1. The project should depend on internal Octez libraries, as opposed
   to public APIs such as inter-process communication via RPC.

2. The dependency mentioned above should be unavoidable. That is, if
   there is another way to build the project, avoiding depending on
   internals of Octez, adding such an optional dependency won't
   qualify a project to be included.

3. The dependency should be on a feature or part of the codebase that
   is unstable enough to make it impractical to depend on released
   versions of Octez.

4. The project should be used regularly as a helper in developing
   Octez (e.g. run in the CI pipeline).

Programming tutorials
*********************

This page groups documentation about some specific aspects of
programming within the Octez code base. This covers libraries and
frameworks available in Octez, but also guidelines for applying more
general programming techniques, such as GADTs, to Octez.

* Generalized Algebraic Data Types (GADTs)

  * Ensuring type safety of the Michelson interpreter

  * Building complex expressions

  * Type equality witness

* The Error Monad

  * Part 1: "result", Lwt, and Lwt-"result"

  * Part 2: "tzresult" and Lwt-"tzresult"

  * Part 3: Advanced topics

  * Part 4: Appendices

* The "clic" library

  * Wallet example

  * Conclusion

* The "data_encoding" library

  * Examples/Tutorial

  * How the Dataencoding module works

* Using The Event Logging Framework

  * Adding Events

  * Creating Sinks

Generalized Algebraic Data Types (GADTs)
****************************************

GADTs are a recent extension of OCaml, which is widely used in the
Octez code-base, especially in the protocol. They serve many important
purposes, most notably increasing type safety of the Michelson
interpreter. This page reviews some of the use cases of GADTs in Octez
codebase and explains what role do they play in the design.

For an explanation of the GADT concept itself, try one of the links
below:

* Algebraic Data Types (introduction to OCaml composite types)

* Generalized Algebraic Data Types (OCaml manual)

* A simple expression evaluator (example)


Ensuring type safety of the Michelson interpreter
=================================================

One of the most important use cases for GADTs in Octez is ensuring
type safety of the Michelson interpreter. Obviously, because Michelson
scripts manage possibly large amounts of funds, it is extremely
important that the interpreter is bug-free and reliable. The static
type checking of Michelson expressions assures that no operation will
ever be executed on a value of another type than the one this
operation expects. Moreover, the Michelson type checking is directly
implemented as OCaml type checking, so that the correctness of the
latter entails the correctness of the former.

The (simplified) definition of Michelson expressions type is presented
below (see the src/proto_alpha/lib_protocol/script_typed_ir.ml module
for the actual definition):

   type 't ty =
     | Unit_t : type_annot option -> unit ty
     | Int_t : type_annot option -> z num ty
     | String_t : type_annot option -> string ty
     | Mutez_t : type_annot option -> Tez.t ty
     | Bool_t : type_annot option -> bool ty
     | Pair_t :
       ('a ty * field_annot option * var_annot option)
       * ('b ty * field_annot option * var_annot option)
       * type_annot option
       -> ('a, 'b) pair ty
     | Union_t :
       ('a ty * field_annot option)
       * ('b ty * field_annot option)
       * type_annot option
       -> ('a, 'b) union ty

In the above type **values** of type "ty" represent **types** of
Michelson expressions. These types are ascribed to Michelson values of
corresponding types and then used to type check these expressions.
However, these values are expressed in terms of OCaml types of the
underlying representations of these expressions. This way the OCaml
compiler itself can verify that the Michelson interpreter only
evaluates well-typed expressions. The "'t" type variable here never
appears on the right-hand side of the definition. Indeed, the GADT
allows "'t" to be solely determined by the constructor used to create
a value of type "ty". This way a Michelson type represented by a "'t
ty" value is directly linked to OCaml type "'t".

However, in case of compound types like pairs, unions and functions,
"'t" depends on more than just the constructor. After all, these types
are polymorphic just like in OCaml. This is why the "Pair_t"
constructor for instance accepts (among other arguments) two values of
the same type "ty", but parameterized independently. These arguments
determine types of the components of the pair. Note that whatever
types that will be, they will be included in the final type of
"Pair_t". This way concrete types "propagate" through variables of
polymorphic types so that the OCaml compiler can always know what
types of values it works with when evaluating a Michelson expression.

Also note that on the type level, constructor "Union_t" is very
similar to "Pair_t". Indeed, the difference between these types lies
in how their values are constructed, but types themselves are very
similar.


Building complex expressions
============================

A syntactically correct Michelson expression is not necessarily well-
typed. Hence a need presents itself to create a representation where
all sub-expressions are assigned their proper types so that it can be
checked if they match. Such an internal representation of an
expression is usually called an *abstract syntax tree* (or AST for
short). This need for strongly-typed ASTs is already fulfilled by the
kind of type definitions above.

However, when one tries to gather a lot of typed expressions into a
single data structure, a problem immediately arises: with usual ADTs
and the kind of GADTs show above the type of such data structure must
be parametrized by all type variables appearing inside. On the other
hand, when dealing with abstract Michelson expressions, one would like
them to be just of type "expr" (i.e. monomorphic). Fortunately GADTs
allow to omit type variables appearing inside, thus making a type look
monomorphic from user's perspective:

   type ex_ty = Ex_ty : 'a ty -> ex_ty

This type is defined in
src/proto_alpha/lib_protocol/script_ir_translator.mli As one can see
it's just a wrapper around "'a ty". However, its function is to hide
the "'a" variable from the user (and the compiler), by including it
only on the right-hand side. This way it's possible to form a list (or
any data structure for that matter) consisting of arbitrary type
expressions. They don't have to match. On the other hand, thanks to
the construction of "'t ty" type, no type information is really lost.
Indeed, because of how "'t ty" is constructed, its precise type can be
retrieved by virtue of type inference. Since the compiler knows that
for instance "Unit_t" is **always** of type "unit ty", then in a
pattern matching branch on that constructor it will conclude that "'a
ty" must be indeed "unit ty".

While usual parametric polymorphism is often associated with the
**universal quantifier** (for all) in logic, this construction is
associated with the **existential quantifier** (hence the name
"ex_ty"). In type "'a ty" for all possible types "'a" there exists a
corresponding type "'a ty" (universal quantification), whereas in type
"ex_ty" there exists some "'a" which is contained by its value, but
it's unknown, which is it exactly (existential quantification).


Type equality witness
=====================

Consider the following GADT:

   type (_, _) eq = Refl : ('t, 't) eq

This is the type equality witness. Note that the value constructor
"Refl" does not require any arguments, so it can be created at will
just like "()". Indeed, both its type parameters are phantom (i.e.
unrelated to any components of the actual value, but carrying some
logical information about the type). Nonetheless, a value of type
"('a, 'b) eq" can be created if and only if "'a" and "'b" are the same
type. If a value of such type is present in a scope, the compiler will
happily unify "'a" and "'b".

This type is defined in the protocol environment, in
"Equality_witness" module and used throughout the protocol to
dynamically check for type equality of values. It is not possible to
write a general function checking equality of types; it's only
possible for particular examples. One such example can be found in
src/proto_alpha/lib_protocol/script_ir_translator.ml in function
"merge_types". It either produces a witness that two "'t ty" values
are of the same type or "None" if that's not the case.

The Error Monad
***************

There are multiple ways to deal with errors in OCaml: exceptions,
"result", "option", etc., each approach has strengths and weaknesses.
However, in Octez we settle on one, single, uniform way of dealing
with errors: the *error monad*.

This is a tutorial for the error monad. It makes the following
assumptions

* You can read English.

* You have a basic familiarity with OCaml.

* You have a basic familiarity with Lwt (if not, this tutorial links
  to Lwt resources when it becomes necessary).

Other than that, each section of this tutorial can be understood using
the information of previous sections, no additional information is
necessary. Note, however, that most sections also make forward
references along the lines of “more details on this later” or
“alternatives are presented later”. During a first read, this
foreshadowing is only useful in that it lets you know that the
information is coming and that you can move on with a partial
understanding. On subsequent reads, it helps putting things in
context.

Note that the core of this tutorial focuses on error management in
Octez. For a short section on error management in the protocol see
Part 3 Working within the protocol.

* Part 1: "result", Lwt, and Lwt-"result"

  * The "result" type

  * The binding operator

  * Aside: the "Error_monad" module is opened everywhere

  * Recovering from errors

  * Mixing different kinds of errors

  * The "Result_syntax" module

  * Lwt

  * The "Lwt_syntax" module

  * Promises of results: Lwt and "result" together

  * The "Lwt_result_syntax" module

  * Converting errors of promises

  * Lifting

  * Wait! There is too much! What module am I supposed to open locally
    and what operators should I use?

  * What’s an error?

  * Wait! It was supposed to be “one single uniform way of dealing
    with errors”! What is this?

  * META COMMENTARY

* Part 2: "tzresult" and Lwt-"tzresult"

  * "error", "trace", and "tzresult"

  * Generic failing

  * Declaring and registering "error"s

  * The "Result_syntax"'s "tz" extensions

  * The "Lwt_result_syntax"'s "tz" extensions

  * Lifting

  * Tracing

  * META COMMENTARY

* Part 3: Advanced topics

  * Working with standard data-structures: Lwtreslib

  * Working with traces and errors and such directly

  * Working within the protocol

  * Working below the error-monad

  * Working with external libraries

* Part 4: Appendices

  * In depth discussion: what even is a monad?

  * In depth discussion: "Error_monad", "src/lib_error_monad/",
    "Tezos_base__TzPervasives", etc.

  * In depth discussion: "result" as data and "result" as control-flow

  * In depth discussion: pros and cons of "result" compared to other
    error management techniques

  * Legacy code

Part 1: "result", Lwt, and Lwt-"result"
***************************************

This is Part 1 of 4 of the The Error Monad tutorial.


The "result" type
=================

The type "result" is part of the standard library of OCaml. It is
defined as

   type ('a, 'b) result =
   | Ok of 'a
   | Error of 'b

(See reference manual.)

The constructors of the "result" type have meaning: "Ok" is for
normal, successful cases that carry a value that is somewhat expected.
"Error" is for abnormal, failure cases that carry information about
what went wrong. E.g.,

   let get a index =
     if index < 0 then
       Error "negative index in array access"
     else if index >= Array.length a then
       Error "index beyond length in array access"
     else
       Ok a.(index)

You can see the "result" type as an opinionated "either": a left-or-
right sum type where the left and right side have distinct roles. Or
you can see the "result" type as a buffed-up "option" type: a type
that either carries a value or doesn’t (but in this case it carries an
error instead).


Exercises
---------

* Write the implementation for

     (** [to_option r] is [Some x] if [r] is [Ok x]. Otherwise it is [None]. *)
     val to_option : ('a, unit) result -> 'a option

* Write the implementation for

     (** [to_option ?log r] is [Some x] if [r] is [Ok x]. Otherwise it calls
         [log e] and returns [None] if [r] is [Error e]. By default [log] is
         [ignore]. *)
     val to_option : ?log:('e -> unit) -> ('a, 'e) result -> 'a option

* Write the implementation for

     (** [catch f] is [Ok (f ())] except if [f] raises an exception [exc] in
         which case it is [Error exc]. *)
     val catch : (unit -> 'a) -> ('a, exn) result


The binding operator
====================

Working directly with the "result" type can quickly become cumbersome.
Consider, for example the following code.

   (** [compose3 f g h x] is [f (g (h x))] except that it handles errors. *)
   let compose3 f g h x
     match h x with
     | Error e -> Error e
     | Ok y ->
       match g y with
       | Error e -> Error e
       | Ok z ->
         f z

The nested "match"-"with" constructs lead to further and further
indentation. The "Error" cases are all identical and simply add noise.

To circumvent this, in Octez we use a binding operator: a user-defined
"let"-binding. Specifically, you can open the "Result_syntax" module
which includes the binding operator "let*" dedicated to "result".

   (** [compose3 f g h x] is [f (g (h x))] except that it handles errors. *)
   let compose3 f g h x
     let open Result_syntax in (* adds [let*] in scope *)
     let* y = h x in
     let* z = g y in
     f z

An expression "let* x = e in e'" is equivalent to "Result.bind e (fun
x -> e')" and also to "match e with | Error err -> Error err | Ok x ->
e'". In all of these forms, the expression "e'" is evaluated if and
only if the expression "e" is successful (i.e., evaluates to "Ok").


Exercises
---------

* Rewrite the following code without "match"-"with"

     let apply2 (f, g) (x, y) =
       match f x with
       | Error e -> Error e
       | Ok u ->
         match g y with
         | Error e -> Error e
         | Ok v -> Ok (u, v)

  Did you remember to open the syntax module?

* Write the implementation for

     (** [map f [x1; x2; x3; ..]] is a list [[y1; y2; y3; ..]] where [y1 = f x1],
         [y2 = f x2], etc. except if [f] is fails on one of the inputs, in which
         case it is an [Error] carrying the same error as [f]'s. *)
     val map : ('a -> ('b, 'err) result) -> 'a list -> ('a list, 'err) result

  Note that this exercise is for learning only. You won’t need to
  write this function in Octez. Indeed, a helper function which does
  exactly that is provided in the extended standard library of Octez.


Aside: the "Error_monad" module is opened everywhere
====================================================

In Octez, the "Error_monad" module provides types and values for error
management. It is part of the "tezos-error-monad" package. And it is
opened in most of the source code. Apart from some specific libraries
(discussed later), the content of this module is already in scope.

The "Error_monad" module contains the "Result_syntax" module. This is
why you can use "let open Result_syntax in" directly in your own
functions.

The "Error_monad" module contains other modules and functions and
types which you will learn about later.


Recovering from errors
======================

When given a value of type "result", you can inspect its content to
determine if it is an "Ok" or an "Error". You can use this feature to
recover from the failures.

   match e with
   | Ok x -> do_something x
   | Error e -> do_something_else e

Recovering can mean different things depending on the task that failed
and the error with which it failed. Sometimes you just want to retry,
sometimes you want to retry with a different input, sometimes you want
to propagate the error, sometimes you want to log the error and
continue as if it hadn’t happened, etc.

   let rec write data =
     match write_to_disk data with
     | Ok () -> ()
     | Error EAGAIN -> write data (* again: try again *)
     | Error ENOSPC -> Stdlib.exit 1 (* no space left on device: escalate to exit *)

There is a correspondence between a "match"-"with" around a "result"
and a "try"-"with". Both are for recovering from failures. However, in
Octez, you will mostly use a "match"-"with" around a "result", because
we favour "result" over exceptions. You may use the "try"-"with"
construct when interfacing with an external library which uses
exceptions.

There are several ways to use the "match"-"with" recovery with the
binding operator from "Result_syntax". Depending on the size of the
expression you are recovering from, one may be more readable than the
other. Choose accordingly.

You can simply place the expression directly inside the
"match"-"with".

   match
     let* x = get_horizontal point in
     let* y = get_vertical point in
     Ok (x, y)
   with
   | Ok (x, y) -> ..
   | Error e -> ..

Alternatively, if the expression grows too much in size or in
complexity, you can move the expression inside a vanilla
"let"-binding: "let r = .. in match r with ..".

Alternatively, if the expression grows even more, or if the expression
may be re-used in other parts of the code, you may move the expression
inside a vanilla function which you can call inside the
"match"-"with".

You can also use the functions from the standard library’s Result
module. Note however, that some of these functions are shadowed in the
extended library of Octez, which you will learn more about later.


Mixing different kinds of errors
================================

Occasionally, you may have to call a function which returns a value of
type, say, "(_, unit) result" and one, say, "(_, string) result". In
this case, you cannot simply bind the two expressions as is.
Specifically, the type checker will complain. You can see the
constraint you would be breaking in the type of "let*" where the two
error types are the same ("'e"):

   val ( let* ) : ('a, 'e) result -> ('a -> ('b, 'e) result) -> ('b, 'e) result

When you need to mix those function, you have to either handle the
errors of each independently (see the section above about recovering
from errors) or you need to convert the errors so they have the same
type. You should use "Result.map_error" to do that.

   let* cfg =
     Result.map_error (fun () -> "Error whilst reading configuration")
     @@ read_configuration ()
   in
   ..


The "Result_syntax" module
==========================

You have already learned about the "let*" binding operator from the
"Result_syntax". But there are other values you can use from this
module.

The following functions form the core of the "Result_syntax" module.

* "let*": a binding operator to continue with the value in the "Ok"
  constructor or interrupt with the error in the "Error" constructor.

  "let* x = e in e'" is equivalent to "match e with Error err -> Error
  err | Ok x -> e'".

  (See above for examples and more details.)

* "return : 'a -> ('a, 'e) result": the expression "return x" is
  equivalent to "Ok x". The function is included for consistency with
  other syntax modules presented later. You can use either form.

* "fail : 'e -> ('a, 'e) result": the expression "fail e" is
  equivalent to "Error e". The function is included for consistency
  with other syntax modules presented later. You can use either form.

The following functions offer additional, less often used
functionalities.

* "both : ('a, 'e) result -> ('b, 'e) result -> ('a * 'b, 'e list)
  result": the expression "both e1 e2" is "Ok" if both expressions
  evaluate to "Ok" and "Error" otherwise.

  Note that the expression "both e1 e2" is different from the
  expression "let* x = e1 in let* y = e2 in return (x, y)". In the
  former ("both") version, both "e1" and "e2" are evaluated,
  regardless of success/failure of "e1" and "e2". In the latter
  ("let*"-"let*") version, "e2" is evaluated if and only if "e1" is
  successful.

  This distinction is important when the expressions "e1" and "e2"
  have side effects: "both (f ()) (g ())".

* "all : ('a, 'e) result list -> ('a list, 'e list) result": the
  function "all" is a generalisation of "both" from tuples to lists.

  Note that, as a generalisation of "both", in a call to the function
  "all", all the elements of the list are evaluated, regardless of
  success/failure of the elements: "all (List.map f xs)".

* "join : (unit, 'e) result list -> (unit, 'e list) result": the
  function "join" is a specialisation of "all" for lists of unit-typed
  expressions (typically, for side-effects).

  Note that, as a specialisation of "all", in a call to the function
  "join", all the elements of the list are evaluated, regardless of
  success/failure of the elements: "join (List.map f xs)".

The following functions do not provide new functionalities but they
are useful for small performance gains or for shorter syntax.

* "return_unit" is equivalent to "return ()" but it avoids one
  allocation. This is important in parts of the code that are
  performance critical, and it is a good habit to take otherwise.

     "return_nil" is equivalent to "return []" but it avoids one
     allocation.
     "return_true" is equivalent to "return true" but it avoids one
     allocation.
     "return_false" is equivalent to "return false" but it avoids
     one allocation.

     "return_none" is equivalent to "return None" but it avoids one
     allocation.
     "return_some x" is equivalent to "return (Some x)" and it is
     provided for completeness with "return_none".

* "let+" is a binding operator similar to "let*" but when the
  expression which follows the "in" returns a non-result value. In
  other words, "let+ x = e in e'" is equivalent to "let* x = e in
  return (e')".

  The "let+" is purely for syntactic conciseness (compared to the "*"
  variant), use it if it makes your code more readable.


Lwt
===

In Octez, I/O and concurrency are handled using the Lwt library. With
Lwt you use *promises* to handle I/O and concurrency. You can think of
promises as data structures that are empty until they *resolve*, at
which point they hold a value.

A promise for a value of type "'a" has type "'a Lwt.t". The function
"Lwt.bind : 'a t -> ('a -> 'b t) -> 'b t" waits for the promise to
resolve (i.e., to carry a value of type "'a") before applying the
provided function. The expression "bind p f" is a promise which
resolves only once the promise "p" has resolved *and then* the promise
returned by "f" has resolved too.

If you are not familiar with Lwt, you should check out the official
manual and this introduction before continuing. This is important. Do
it.

Unlike is mentioned in those separate resources on Lwt, in Octez, we
do not in general use the success/failure mechanism of Lwt. Instead,
we mostly rely on "result" (as mentioned above).

Thus, in the rest of this tutorial we only consider the subset of Lwt
without failures. In practice, you might need to take care of
exceptions in some cases, but this is discussed in the later, more
advanced parts of the tutorial.


The "Lwt_syntax" module
=======================

In Octez, because Lwt is pervasive, you need to bind promises often.
To make it easier, you can use the "Lwt_syntax" module. The
"Lwt_syntax" module is made available everywhere by the "Error_monad"
module. The "Lwt_syntax" module is similar to the "Result_syntax"
module but for the Lwt monad (more about monads later).

* "let*": a binding operator to wait for the promise to resolve before
  continuing.

  "let* x = e in e'" is a promise that resolves after "e" has resolved
  to a given value and then "e'" has resolved with that value carried
  by "x".

  Note that "Lwt_syntax" and "Result_syntax" (see above) both use
  "let*" for their main binding operator. Consequently, the specific
  meaning of "let*" depends on which module is open. This extends to
  other syntax modules introduced later in this tutorial.

  (What if you need to use both Lwt and "result"? Which syntax module
  should you use? You will learn about that in the next section!)

* "return : 'a -> 'a Lwt.t": the expression "return x" is equivalent
  to "Lwt.return x". It is a promise that is already resolved with the
  value of "x".

* "and*": a binding operator alias for "both" (see below). You can use
  it with "let*" the same way you use "and" with "let".

     let apply_triple f (x, y, z) =
       let open Lwt_syntax in
       let* u = f x
       and* v = f y
       and* w = f z
       in
       return (u, v, w)

  When you use "and*", the bound promises ("f x", "f y", and "f z")
  are evaluated concurrently, and the expression which follows the
  "in" ("return ..") is evaluated once all the bound promises have all
  resolved.

The following functions offer additional, less often used
functionalities.

* "both: 'a Lwt.t -> 'b Lwt.t -> ('a * 'b) Lwt.t": the expression
  "both p q" is a promise that resolves only once both promises "p"
  and "q" (which make progress concurrently) have resolved.

  In practice, you will most likely use "and*" instead of both.

* "all: 'a Lwt.t list -> 'a list Lwt.t": the function "all" is a
  generalisation of "both" from tuples to lists.

  Note that, as a generalisation of "both", in a call to the function
  "all", all the promises in the provided list make progress towards
  resolution concurrently.

* "join : unit Lwt.t list -> unit Lwt.t": the function "join" is a
  specialisation of "all" to lists of units (i.e., side-effects).

The following functions do not provide new functionalities but they
are useful for small performance gains or for shorter syntax.

* "return_unit" is equivalent to "return ()" but it avoids one
  allocation. This is important in parts of the code that are
  performance critical, and it is a good habit to take otherwise.

     "return_nil" is equivalent to "return []" but it avoids one
     allocation.
     "return_true" is equivalent to "return true" but it avoids one
     allocation.
     "return_false" is equivalent to "return false" but it avoids
     one allocation.

     "return_none" is equivalent to "return None" but it avoids one
     allocation.
     "return_some x" is equivalent to "return (Some x)" and it is
     provided for completeness.

     "return_ok x" is equivalent to "return (Ok x)" and it is
     provided for completeness.
     "return_error x" is equivalent to "return (Error x)" and it is
     provided for completeness.

* "let+" and "and+" are binding operators similar to "let*" and "and*"
  but when the expression which follows the "in" returns a non-promise
  value. In other words, "let+ x = e1 and+ y = e2 in e" is equivalent
  to "let* x = e1 and* y = e2 in return e".

  The "let+" and "and+" are purely for syntactic conciseness (compared
  to the "*" variants), use them if it makes your code more readable.


Promises of results: Lwt and "result" together
==============================================

In Octez, we have functions that perform I/O and also may fail. In
this case, the function returns a promise of a "result". This is the
topic of this section.

Note that Lwt and "result" are orthogonal concerns. On the one hand,
Lwt is for concurrency, for automatically scheduling code around I/O,
for making progress on different parts of the program side-by-side. On
the other hand, "result" is for aborting computations, for handling
success/failures. It is because Lwt and "result" are orthogonal that
we can use them together.

   'a  --------------> ('a, 'e) result
    |                           |
    |                           |
    V                           V
   'a Lwt.t ---------> ('a, 'e) result Lwt.t

When we combine Lwt and "result" for control-flow purpose we combine
both of the orthogonal behaviours. We can achieve this combined
behaviour “by hand”. However, doing so requires mixing "Lwt_syntax.(
let* )" and regular "match"-"with":

   let apply2 (f, g) (x, y) =
     let open Lwt_syntax in
     let* r = f x in
     match r with
     | Error e -> return (Error e)
     | Ok u ->
       let* r = g y in
       match r with
       | Error e -> return (Error e)
       | Ok v -> return (Ok (u, v))

This is interesting to consider because it shows the two orthogonal
features of control-flow separately: wait for the promise to resolve,
and check for errors. However, in practice, this becomes cumbersome
even faster than when working with plain "result" values.

To make this easier, in Octez we use a binding operator. Specifically,
you can open the "Lwt_result_syntax" (instead of the other syntax
modules) which includes a binding operator dedicated to promises of
"result".

   let apply2 (f, g) (x, y) =
     let open Lwt_result_syntax in
     let* u = f x in
     let* v = g y in
     return (u, v)

When a promise resolves to "Ok" we say that it resolves successfully.
When it resolves to "Error" we say that it resolves unsuccessfully or
that it fails.


Exercises
---------

* Rewrite the following code without "match"-"with"

     let compose3 f g h x
       let open Lwt_syntax in
       let* r = h x in
       match r with
       | Error e -> return (Error e)
       | Ok y ->
         let* s = g y in
         match s with
         | Error e -> return (Error e)
         | Ok z ->
           f z

  Did you remember to change the opened syntax module?


The "Lwt_result_syntax" module
==============================

Octez provides the "Lwt_result_syntax" module to help handle promises
of results.

* "let*": a binding operator to wait for the promise to resolve before
  continuing with the value in the "Ok" constructor or interrupting
  with the error in the "Error" constructor.

  Note how the "let*" binding operator combines the behaviour of
  "Lwt_syntax.( let* )" and "Result_syntax.( let* )". Also note that
  the different "let*"s are differentiated by context; specifically by
  what syntax module has been opened.

* "return: 'a -> ('a, 'e) result Lwt.t": the expression "return x" is
  a promise already successfully resolved to "x". More formally,
  "return x" is equivalent to "Lwt_syntax.return (Result_syntax.return
  x)".

* "fail: 'e -> ('a, 'e) result Lwt.t": the expression "fail e" is a
  promise already unsuccessfully resolved with the error "e". More
  formally, "fail e" is equivalent to "Lwt_syntax.return
  (Result_syntax.fail e)".

The following functions offer additional, less often used
functionalities.

* "both : ('a, 'e) result Lwt.t -> ('b, 'e) result Lwt.t -> ('a * 'b,
  'e list) result Lwt.t": the expression "both p1 p2" is a promise
  that resolves successfully if both "p1" and "p2" resolve
  successfully. It resolves unsuccessfully if either "p1" or "p2"
  resolve unsuccessfully.

  Note that in the expression "both p1 p2", both promises "p1" and
  "p2" are evaluated concurrently. Moreover, the returned promise only
  resolves once both promises have resolved, even if one resolves
  unsuccessfully.

  Note that this syntax module does not offer "and*" as a binding
  operator alias for "both". This is because, as with "Result_syntax",
  the type for errors in "both" is not stable (it is "'e" on the
  argument side and "'e   list" on the return side). This hinders
  practical uses of "and*".

* "all : ('a, 'e) result Lwt.t list -> ('a list, 'e list) result
  Lwt.t": the function "all" is a generalisation of "both" from tuples
  to lists.

  Note that, as a generalisation of "both", in a call to the function
  "all", all the promises in the provided list make progress towards
  resolution concurrently and continue to evaluate until resolution
  regardless of successes and failures.

* "join : (unit, 'e) result Lwt.t list -> (unit, 'e list) result
  Lwt.t": the function "join" is a specialisation of "all" for lists
  of unit-type expressions (typically, for side-effects).

The following functions do not provide new functionalities but they
are useful for small performance gains or for shorter syntax.

* "return_unit" is equivalent to "return ()" but it avoids one
  allocation. This is important in parts of the code that are
  performance critical, and it is a good habit to take otherwise.

     "return_nil" is equivalent to "return []" but it avoids one
     allocation.
     "return_true" is equivalent to "return true" but it avoids one
     allocation.
     "return_false" is equivalent to "return false" but it avoids
     one allocation.

     "return_none" is equivalent to "return None" but it avoids one
     allocation.
     "return_some x" is equivalent to "return (Some x)" and it is
     provided for completeness.

  Note that, like with "Result_syntax", this syntax module does not
  provide "return_ok" and "return_error". This is to avoid nested
  "result" types. If you need to nest "result"s you can do so by hand.

* "let+" is a binding operator similar to "let*" but when the
  expression which follows the "in" returns a non-promise value. In
  other words, "let+ x = e in e'" is equivalent to "let* x = e in
  return (e')".

  The "let+" is purely for syntactic conciseness (compared to the "*"
  variant), use it if it makes your code more readable.


Exercises
---------

* Write the implementation for

     (** [map f [x1; x2; ..]] is a promise for a list [[y1; y2; .. ]] where [y1]
         is the value that [f x1] successfully resolves to, etc. except if [f]
         resolves unsuccessfully on one of the input in which case it also
         resolves unsuccessfully with the same error as [f]. *)
     map : ('a -> ('b, 'e) result Lwt.t) -> 'a list -> ('b list, 'e) result Lwt.t

  How does your code compare to the one in the "result"-only variant
  of this exercise?

  Note that this exercise is for learning only. You won’t need to
  write this function in Octez. Indeed, a helper function which does
  exactly that is provided in the extended standard library of Octez.

* Make your "map" function tail-recursive.

* What type error is triggered by the following code?

     open Lwt_result_syntax ;;
     let ( and* ) = both ;;
     let _ =
       let* x = return 0 and* y = return 1 in
       let* z = return 2 in
       return (x + y + z) ;;

* Rewrite the following function without "match"-"with"

     let compose3 f g h x =
       let open Lwt_syntax in
       let* y_result = f x in
       match y_result with
       | Error e -> return (Error e)
       | Ok y ->
         let* z_result = g y in
         match z_result with
         | Error e -> return (Error e)
         | Ok z ->
           h z


Converting errors of promises
=============================

Remember that, with "Result_syntax", you cannot mix different types of
errors in a single sequence of "let*". This also applies to
"Lwt_result_syntax". Indeed, the type checker will prevent you from
doing so.

You can use the same "Result.map_error" function as for plain
"result"s. But when you are working with promises of "result", the
syntactic cost of doing so is high:

   let open Lwt_result_syntax in
   let* config =
     let open Lwt_syntax in
     let* config_result = read_configuration () in
     Lwt.return (Result.map_error (fun () -> ..) config_result)
   in
   ..

To avoid this syntactic weight, the "Lwt_result_module" provides a
dedicated function:

   lwt_map_error : ('e -> 'f) -> ('a, 'e) result Lwt.t -> ('a, 'f) result Lwt.t


Lifting
=======

Occasionally, whilst you are working with promises of "result" (i.e.,
working with values of the type "(_, _) result Lwt.t"), you will need
to call a function that returns a simple promise (a promise that
cannot fail, a promise of a value that’s not a "result", i.e., a value
of type "_ Lwt.t") or a simple result (an immediate value of a
"result", i.e., a value of type "(_, _) result"). This is common
enough that the module "Lwt_result_syntax" provides helpers dedicated
to this.


From "result"-only into Lwt-"result"
------------------------------------

The module "Lwt_result_syntax" includes the binding operator "let*?".
It is dedicated to binding Result-only expressions.

   let*? x = check foo bar in (* Result-only: checking doesn't yield *)
   ..


Mnemonic
^^^^^^^^

The "let*?" binding operator uses the question mark ("?") to represent
the uncertainty of the "result". Is it a success? Is it a failure?


From Lwt-only into Lwt-"result"
-------------------------------

The module "Lwt_result_syntax" includes the binding operator "let*!".
It is dedicated to binding Lwt-only expressions.

   let*! x = Events.emit foo bar in (* Lwt-only: logs can't fail *)
   ..


Mnemonic
^^^^^^^^

The "let*!" binding operator uses the exclamation mark ("!") to
represent the impossibility of errors: Thou shall not fail!


Wait! There is too much! What module am I supposed to open locally and what operators should I use?
===================================================================================================

If you are feeling overwhelmed by the different syntax modules, here
are some simple guidelines.

* If your function returns "(_, _) result Lwt.t" values, then you
  start the function with "let open Lwt_result_syntax in". Within the
  function you use

  * "let" for vanilla expressions,

  * "let*" for Lwt-"result" expressions,

  * "let*!" for Lwt-only expressions,

  * "let*?" for "result"-only expressions.

  And you end your function with a call to "return".

* If your function returns "(_, _) result" values, then you start the
  function with "let open Result_syntax in". Within the function you
  use

  * "let" for vanilla expressions,

  * "let*" for "result" expressions,

  And you end your function with a call to "return".

* If your function returns "_ Lwt.t" values, then you start the
  function with "let open Lwt_syntax in".

  * "let" for vanilla expressions,

  * "let*" for Lwt expressions,

  And you end your function with a call to "return".

These are rules of thumb and there are exceptions to them. Still, they
should cover most of your uses and, as such, they are a good starting
point.


What’s an error?
================

So far, this tutorial has considered errors in an abstract way. Most
of the types carried by the "Error" constructors have been parameters
("'e"). This is a common pattern for higher-order functions that
compose multiple "result" and Lwt-"result" functions together. But, in
practice, not every function is a higher-order abstract combinator and
you sometimes need to choose a concrete error. This section explores
common choices.

**A dedicated algebraic data type**

Often, a dedicated algebraic data type is appropriate. A sum type
represents the different kinds of failures that might occur. E.g.,
"type hashing_error = Not_enough_data | Invalid_escape_sequence". A
product type (typically a record) carries multiple bits of information
about a failure. E.g., "type parsing_error = { source: string;
location: int; token: token; }"

This approach works best when a set of functions (say, all the
functions of a module) have similar ways to fail. Indeed, when that is
the case, you can simply define the error type once and all calls to
these functions can match on that error type if need be.

E.g., binary encoding and decoding errors in data-encoding.

**Polymorphic variants**

In some cases, the different functions of a module may each fail with
different subsets of a common set of errors. In such a case, you can
use polymorphic variants to represent errors. E.g.,

   val connect_to_peer:
     address -> (connection, [ `timeout | `connection_refused ]) result Lwt.t
   val send_message:
     connection -> signing_key -> string ->
       (unit, [ `timeout | `connection_closed ]) result Lwt.t
   val read_message:
     connection ->
       (string, [ `timeout | `unknown_signing_key | `invalid_signature | `connection_closed ]) result Lwt.t
   val close_connection: connection -> (unit, [ `unread_messages of string list ]) result

The benefit of this approach is that the caller can compose the
different functions together easily and match only on the union of
errors that may actually happen. The type checker keeps track of the
variants that can reach any program point.

   let handshake conn =
     let open Lwt_result_syntax in
     let* () = send_message conn "ping" in
     let* m = read_message conn in
     if m = "pong" then
       return ()
     else
       `unrecognised_message m

   let handshake conn =
     let open Lwt_syntax in
     let* r = handshake conn in
     match r with
     | Ok () -> return_unit
     | Error (`unknown_signing_key | `invalid_signature) ->
       (* we ignore unread messages if the peer had signature issues *)
       let _ = close_connection conn in
       return_unit
     | Error (`timeout | `connection_closed) ->
       match close_connection with
       | Ok () -> return_unit
       | Error (`unread_messages msgs) ->
         let* () = log_unread_messages msgs in
         return_unit

**A human-readable string**

In some cases, there is nothing to be done about an error but to
inform the user. In this case, the error may just as well be the
message.

It is important to note that these messages are not generally meant to
be matched against. Indeed, such messages may not be stable and even
if they are, they probably don’t carry precise enough information to
be acted upon.

You should only use "string" as an error type when the error is not
recoverable and you should not try to recover from "string" errors (or
more precisely, your recovery should not depend on the content of the
string).

**An abstract type**

If the error is not meant to be recovered from, it is sometimes ok to
use an abstract type. This is generally useful at the interface of a
module, specifically when the functions within the module are meant to
inspect the errors and possibly attempt recovery, but the callers
outside of the modules are not.

If you do use an abstract type for errors, you should also provide a
pretty-printing function.

**A wrapper around one of the above**

Sometimes you want to add context or information to an error. E.g.,

   type 'a with_debug_info = {
     payload: 'a;
     timestamp: Time.System.t;
     position: string * int * int * int;
   }

   let with_debug_info ~position f =
     match f () with
     | Ok _ as ok -> ok
     | Error e -> Error { payload = e; timestamp = Time.System.now (); position }

This specific example can be useful for debugging, but other wrappers
can be useful in other contexts.

**Mixing error types**

It is difficult to work with different types of errors within the same
function. This most commonly happens if you are calling functions from
different libraries, which use different types of errors.

This is difficult because the errors on both sides of the binding
operator are the same.

   val ( let* ) : ('a, 'e) result -> ('a -> ('b, 'e) result) -> ('b, 'e) result

The error monad provides some support to deal with multiple types of
errors at once. But this support is limited. It is not generally an
issue because error-mixing is somewhat rare: it tends to happen at the
boundary between different levels of abstractions.

If you encounter one of these situations, you will need to convert all
the errors to a common type.

   type error = Configuration_error | Command_line_error

   let* config =
     match Config.parse_file config_file with
     | Ok _ as ok -> ok
     | Error Config.File_not_found -> Ok Config.default
     | Error Config.Invalid_file -> Error Configuration_error
   in
   let* cli_parameters =
     match Cli.parse_parameters () with
     | Ok _ as ok -> ok
     | Error Cli.Invalid_parameter -> Error Command_line_error
   in
   ..

You can also use the "Result.map_error" and "lwt_map_error" functions
introduced in previous sections.


Wait! It was supposed to be “one single uniform way of dealing with errors”! What is this?
==========================================================================================

The error management in Octez is a unified way (syntax modules with
regular, predictable interfaces) of handling different types of
errors.

The variety of errors is a boon in that it lets you use whatever is
the most appropriate for the part of the code that you are working on.
However, the variety of errors is also a curse in that stitching
together functions which return different errors requires boilerplate
conversion functions.

That’s where the global "error" type comes in: a unified type for
errors. And that’s for the next section to introduce.


META COMMENTARY
===============

The previous sections are not Octez-specific. True, the syntax modules
are defined within the Octez source tree, but they could be released
separately (and they will be) or they could easily be replicated in a
separate project.

The next sections are Octez-specific. They introduce types and values
that are used within the whole of Octez.

   You should take this opportunity to take a break.
   Come back in a few minutes.

Part 2: "tzresult" and Lwt-"tzresult"
*************************************

This is Part 2 of 4 of the The Error Monad tutorial.


"error", "trace", and "tzresult"
================================

The "error" type is an extensible variant type defined in the
"Error_monad" module.

Each part of the Octez source code can extend it to include some
errors relevant to that part. E.g., the p2p layer adds "type error +=
Connection_refused" whilst the store layer adds "type error +=
Snapshot_file_not_found of string". More information on errors is
presented later.

The "'error trace" type is a data structure dedicated to holding
"error"s. It is exported by the "Error_monad" module. It is only ever
instantiated as "error trace". It can accumulate multiple errors,
which helps present information about the context in which an error
occurs. More information on traces is presented later.

The "'a tzresult" type is an alias for "('a, error trace) result". It
is used in Octez to represent the possibility of failure in a generic
way. Using "tzresult" is a trade-off. You should only use it in
situations where the pros outweigh the cons.

Pros:

* A generic error that is the same everywhere means that the binding
  operator "let*" can be used everywhere without having to convert
  errors to a common type.

* Traces allow you to easily add context to an existing error (see how
  later).

Cons:

* A generic wrapper around generic errors that cannot be meaningfully
  matched against (although the pattern exists in legacy code).

* Type information about errors is coarse-grained to the point of
  being meaningless (e.g., there is no exhaustiveness check when
  matching).

* Registration is syntactically heavy and requires care (see below).

In general, the type "tzresult" is mostly useful at a high-enough
level of abstraction and in the outermost interface of a module or
even package (i.e., when exposing errors to callers that do not have
access to internal implementation details).


Generic failing
===============

The easiest way to return a "tzresult" is via functions provided by
the "Error_monad". These functions are located at the top-level of the
module, as such they are available, unqualified, everywhere in the
code. They do not even require the syntax modules to be open.

* "error_with" is for formatting a string and wrapping it inside an
  "error" inside a "trace" inside an "Error". E.g.,

     let retry original_limit (f : unit -> unit tzresult) =
       let rec retry limit f
         if limit < 0 then
           error_with "retried %d times" original_limit
         else
           match f () with
           | Error _ -> retry (limit - 1) f
           | Ok () -> Ok ()
       in
       retry original_limit f

  You can use all the formatting percent-escapes from the Format
  module. However, you should generally keep the message on a single
  line so that it can be printed nicely in logs.

  Formally, "error_with" has type "('a, Format.formatter, unit, 'b
  tzresult) format4 -> 'a". This is somewhat inscrutable. Suffice to
  say: it is used with a format string and returns an "Error".

* "error_with_exn : exn -> 'a tzresult" is for wrapping an exception
  inside an "error" inside a "trace" inside an "Error".

     let media_type_kind s =
       match s with
       | "json" | "bson" -> Ok `Json
       | "octet-stream" -> Ok `Binary
       | _ -> error_with_exn Not_found

* "failwith" is for formatting a string and wrapping it inside an
  "error" inside a "trace" inside an "Error" inside a promise. E.g.,

     failwith "Cannot read file %s (ot %s)" file_name (Unix.error_message error)

  "failwith" is similar to "error_with" but for the combined
  Lwt-"tzresult" monad. Again the type ("('a, Format.formatter, unit,
  'b tzresult Lwt.t) format4 -> 'a") is inscrutable, but again usage
  is as simple as a formatting.

* "fail_with_exn" is for wrapping an exception inside an "error"
  inside a "trace" inside an "Error" inside a promise. E.g.,

     fail_with_exn Not_found

  "fail_with_exn" is similar to "error_with_exn" but for the combined
  Lwt-"tzresult" monad.

These functions are useful as a way to fail with generic errors that
carry a simple payload. The next section is about using custom errors
with more content.


Exercises
---------

* Write a function "tzresult_of_option : 'a option -> 'a tzresult"
  which replaces "None" with a generic failure of your choosing.

* Write a function "catch : (unit -> 'a) -> 'a tzresult" which wraps
  any exception raised during the evaluation of the function call.


Declaring and registering "error"s
==================================

On many occasions, "error_with" and friends (see above) are not
sufficient: you may want your error to carry more information than can
be conveyed in a simple string or a simple exception. When you do, you
need a custom error. You must first *declare* and then *register* an
"error".

To declare a new error you simply extend the "error" type. Remember
that the "error" type is defined and exported to the rest of the code
by the "Error_monad" module. You extend it with the "+=" syntax:

   type error +=
     Invalid_configuration of { expected: string; got: string; line: int }

The registration function "register_error_kind" is also part of the
"Error_monad" module. You *register* a new error by calling this
function.

   let () =
     register_error_kind
       `Temporary
       ~id:"invalid_configuration"
       ~title:"Invalid configuration"
       ~description:"The configuration is invalid."
       ~pp:(fun f (expected, got, line) ->
         Format.fprintf f
           "When parsing configuration expected %s but got %s (at line %d)"
           expected got line
       )
       Data_encoding.(
         obj3
           (req "expected" string)
           (req "got" string)
           (req "line" int31))
       (function
         | Invalid_configuration {expected; got; line} ->
           Some (expected, got, line)
         | _ -> None)
       (fun (expected, got, line) -> Invalid_configuration {expected; got; line})

Note that you **MUST** register the errors you declare. Failure to do
so can lead to serious issues.

The arguments for the "register_error_kind" function are as follows: -
category ("`Temporary"): the category argument is meaningless in the
shell, just use "`Temporary".

* "id": a short string containing only characters that do not need
  escaping ("[a-zA-Z0-9._-]"), must be unique across the whole
  program.

* "title": a short human readable string.

* "description": a longer human readable string.

* "pp": a pretty-printing function carrying enough information for a
  full error message for the user. Note that the function does not
  receive the error, instead it receives the *projected payload of the
  error* (here a 3-tuple "(expected, got, line)".

* encoding: an encoding for the projected payload of the error.

* projection: a partial function that matches the specific error (out
  of all of them) and return its projected payload. This function
  always has the form "function | <the error you are returning> ->
  Some <projected payload> | _ -> None".

* injection: a function that takes the projected payload and
  constructs the error out of it.

For errors that do not carry information (e.g., "type error +=
Size_limit_exceeded"), the projected payload of the error is unit.

It is customary to either register the error immediately after the
error is declared or to register multiple errors immediately after
declaring them all. In some cases, the registration happens in a
separate module. Either way, registration of declared error is
compulsory.


Exercises
---------

* Register the following error

     (** [Size_limit_exceeded {limit; current_size; attempted_insertion}] is used
         when an insertion into the global table of known blocks would cause the
         size of the table to exceed the limit. The field [limit] holds the
         maximum allowed size, the field [current_size] holds the current size of
         the table and [attempted_insertion] holds the size of the element that
         was passed to the insertion function. *)
     type error += Size_limit_exceeded {
       limit: int;
       current_size: int;
       attempted_insertion: int
     }


The "Result_syntax"'s "tz" extensions
=====================================

Remember that "'a tzresult" is a special case of "('a, 'e) result".
Specifically, a special case where "'e" is "error trace".
Consequently, you can handle "tzresult" values using the
"Result_syntax" module.

The module "Result_syntax" exports a few functions dedicated to
handling "tzresult". These functions were omitted from Part 1.

* "tzfail: 'e -> ('a, 'e trace) result": the expression "tzfail e"
  wraps "e" in a "trace" inside an "Error". When "e" is of type
  "error" as is the case throughout Octez, "tzfail e" is of type "'a
  tzresult".

* "and*": a binding operator alias for "tzboth" (see below). You can
  use it with "let*" the same way you use "and" with "let".

     let apply_triple f (x, y, z) =
       let open Result_syntax in
       let* u = f x
       and* v = f y
       and* w = f z
       in
       return (u, v, w)

  When you use "and*", the bound results ("f x", "f y", and "f z") are
  all evaluated fully, regardless of the success/failure of the
  others. The expression which follows the "in" ("return ..") is
  evaluated if all the bound results are successful.

* "tzboth : ('a, 'e trace) result -> ('b, 'e trace) result -> ('a *
  'b, 'e trace) result": the expression "both a b" is "Ok" if both "a"
  and "b" are "Ok" and "Error" otherwise`.

  Note that unlike "both", the type of errors ("error trace") is the
  same on both the argument and return side of this function: the
  traces are combined automatically. This remark applies to the
  "tzall" and "tzjoin" (see below) as well.

  The stability of the return type is what allows this syntax module
  to include an "and*" binding operator.

* "tzall : ('a, 'e trace) result list -> ('a list, 'e trace) result":
  the function "tzall" is a generalisation of "tzboth" from tuples to
  lists.

* "tzjoin : (unit, 'e trace) result list -> (unit, 'e trace) result":
  the function "tzjoin" is a specialisation of "tzall" for list of
  unit-typed expressions (typically, for side-effects).

* "and+" is a binding operator similar to "and*" but for use with
  "let+" rather than "let*".


Exercises
---------

* What is the difference between the two following functions?

     let twice f =
       let open Result_syntax in
       let* () = f () in
       let* () = f () in
       return_unit

     let twice f =
       let open Result_syntax in
       let* () = f ()
       and* () = f ()
       in
       return_unit


The "Lwt_result_syntax"'s "tz" extensions
=========================================

In the same way "result" can be combined with Lwt, "tzresult" can also
be combined with Lwt. And in the same way that "Result_syntax" exports
a few "tz"-specific extensions, "Lwt_result_syntax" exports a few
Lwt+``tz`` specific extensions.

There are possibly too many parallels to keep track of, so the diagram
below might help.

   'a  -----------> ('a, 'e) result ------------> 'a tzresult
    |                        |                         |
    |                        |                         |
    V                        V                         V
   'a Lwt.t ------> ('a, 'e) result Lwt.t ------> 'a tzresult Lwt.t

Anyway, the "Lwt_result_syntax" module exports a few functions
dedicated to handling Lwt+``tzresult``. These functions were omitted
from Part 1.

* "tzfail: 'e -> ('a, 'e trace) result Lwt.t": the expression "tzfail
  e" wraps "e" in a "trace" inside an "Error" inside a promise. When
  "e" is of type "error" as is the case throughout Octez, "tzfail e"
  is of type "'a tzresult Lwt.t".

* "and*": a binding operator alias for "tzboth". You can use it with
  "let*" the same way you use "and" with "let".

     let apply_triple f (x, y, z) =
       let open Lwt_result_syntax in
       let* u = f x
       and* v = f y
       and* w = f z
       in
       return (u, v, w)

  When you use "and*", the bound promises ("f x", "f y", and "f z")
  are evaluated concurrently, and the expression which follows the
  "in" ("return   ..") is evaluated once all the bound promises have
  all resolved but only if all of them resolve successfully.

  Note how this "and*" binding operator inherits the properties of
  both "Lwt_syntax.( and* )" and "Result_syntax.( and* )".
  Specifically, the promises are evaluated concurrently and the
  expression which follows the "in" is evaluated only if all the bound
  promises have successfully resolved. These two orthogonal properties
  are combined. This remark also applies to "tzboth", "tzall",
  "tzjoin" and "and+" below.

* "tzboth : ('a, 'e trace) result Lwt.t -> ('b, 'e trace) result Lwt.t
  -> ('a * 'b, 'e trace) result Lwt.t": the expression "tzboth p q" is
  a promise that resolves once both "p" and "q" have resolved. It
  resolves to "Ok" if both "p" and "q" do, and to "Error" otherwise`.

  Note that unlike "Lwt_result_syntax.both", the type of errors
  ("error trace") is the same on both the argument and return side of
  this function: the trace are combined automatically. This remark
  applies to the "tzall" and "tzjoin" (see below) as well.

  The stability of the return type is what allows this syntax module
  to include an "and*" binding operator.

* "tzall : ('a, 'e trace) result Lwt.t list -> ('a list, 'e trace)
  result Lwt.t": the function "tzall" is a generalisation of "tzboth"
  from tuples to lists.

* "join : (unit, 'e trace) result Lwt.t list -> (unit, 'e trace)
  result Lwt.t": the function "tzjoin" is a specialisation of "tzall"
  for lists of unit-typed expressions (typically, for side-effects).

* "and+" is a binding operator similar to "and*" but for use with
  "let+" rather than "let*".


Exercises
---------

* Rewrite this function to use the "Lwt_result_syntax" module and no
  other syntax module.

     let apply_tuple (f, g) (x, y) =
       let open Lwt_syntax in
       let* u = f x
       and* v = g y
       in
       let r = Result_syntax.tzboth u v in
       return r

* Write the implementation for

     (** [map f [x1; x2; ..]] is [[y1; y2; ..]] where [y1] is the successful
         result of [f x1], [y2] is the successful result of [f x2], etc. If [f]
         fails on any of the inputs, returns an [Error] instead. Either way, all
         the calls to [f] on all the inputs are evaluated concurrently and all
         the calls to [f] have resolved before the whole promise resolves. *)
     val map : ('a -> 'b tzresult Lwt.t) -> 'a list -> 'b list tzresult


Lifting
=======

When you are working with promises of "tzresult" (i.e., within
Lwt-"tzresult"), you may occasionally need to call functions that
return a simple promise (i.e., within Lwt-only) or a simple "tzresult"
(i.e., within "tzresult"-only).

Because "tzresult" is a special case of "result", you can use the same
operators "let*!" and "let*?" as presented in Part 1.

   let*! x = plain_lwt_function foo bar in
   let*? x = plain_result_function foo bar in
   ..


Tracing
=======

Remember that a trace is a data structure specifically designed for
errors.

Traces have two roles:

* As a programmer you benefit from the traces' ability to combine
  automatically. Indeed, this feature of traces makes the "and*"
  binding operators possible which can simplify some tasks such as
  concurrent evaluation of multiple "tzresult" promises.

* For the user, traces combine multiple errors, allowing for high-
  level errors (e.g., "Cannot_bootstrap_node") to be paired with low-
  level errors (e.g., "Unix_error EADDRINUSE"). When used correctly,
  this can help create more informative error messages which, in turn,
  can help debugging.

Tracing primitives are declared in the "Error_monad" module. As such,
they are available almost everywhere in the code. They should be used
whenever an error passes from one component (say the p2p layer, or the
storage layer) into another (say the shell).

* "record_trace err r" leaves "r" untouched if it evaluates to "Ok".
  Otherwise, it adds the error "err" to the trace carried in "Error".

     let check_hashes head block operation =
       let open Result_syntax in
       let* () =
         record_trace (Invalid_hash { kind: "head"; hash: head}) @@
         check_hash chain
       in
       let* () =
         record_trace (Invalid_hash { kind: "block"; hash: block}) @@
         check_hash block
       in
       let* () =
         record_trace (Invalid_hash { kind: "operation"; hash: operation}) @@
         check_hash operation
       in
       return_unit

  In this example a failure from any of the calls to "check_hash" will
  be given context that helps with understanding the source of the
  error.

* "record_trace_eval" is a lazy version of "record_trace" in that the
  error added to the trace is only evaluated if needed. More formally
  "record_trace_eval make_err r" leaves "r" untouched if it evaluates
  to "Ok". Otherwise it calls "make_err" and adds the returned error
  onto the trace.

  You should use the strict "record_trace" version when the error you
  are adding to the trace is an immediate value (e.g., "Overflow") or
  a constructor with immediate values (e.g., "Invalid_file name"). You
  should use the lazy "record_trace_eval" version when the error you
  are adding to the trace requires computation to generate (e.g., if
  it requires formatting or querying).

* "trace" is the Lwt-aware variant of "record_trace". More formally,
  "trace err p" leaves "p" untouched if it resolves successfully,
  otherwise it adds the error "err" to the trace carried by the
  unsuccessful resolve.

     let get_data_and_gossip_it () =
       let open Lwt_result_syntax in
       let* data =
         trace Cannot_get_random_data_from_storage @@
         Storage.get_random_data ()
       in
       let* number_of_peers =
         trace Cannot_gossip_data @@
         P2p.gossip data
       in
       return (data, number_of_peers)

  In this example, low-level storage errors are given more context by
  the "Cannot_get_random_data_from_storage" error. Similarly, low-
  level p2p errors are given more context by the "Cannot_gossip_data"
  error. This is important because both the storage and the p2p layer
  may suffer from similar system issues (such as file-descriptor
  exhaustion).

* "trace_eval" is the lazy version of "trace", or, equivalently, the
  Lwt-aware version of "record_trace_eval".

  You should use the strict "trace" version when the error you are
  adding is immediate or a constructor with immediate values. You
  should use the lazy "trace_eval" version when the error you are
  adding requires computation to generate.

Do not hesitate to use the tracing primitives. Too much context is
better than too little context. Think of "trace" (and variants) as a
way to document Octez. Specifically, as making the error messages of
Octez more informative.


META COMMENTARY
===============

The previous sections had a practical focus: how to handle errors? how
to mix different syntaxes? how?! By contrast, the following sections
are in-depth discussions of advanced or tangential topics which you
should feel free to skim or even to skip.

   You should take this opportunity to take a break.
   Come back in a few minutes.

Part 3: Advanced topics
***********************

This is Part 3 of 4 of the The Error Monad tutorial.


Working with standard data-structures: Lwtreslib
================================================

Handling values within the Lwt, "result", and Lwt-"result" monads is
so common in Octez that you also have access to an extension of the
Stdlib dedicated to these monads: the "Lwtreslib" library.

The "tezos-lwt-result-stdlib" package exports an "Lwtreslib" module
which is made available, through "tezos-error-monad" and "tezos-base",
to the whole of the codebase. Specifically, within the codebase of
Octez the following modules of OCaml’s "Stdlib" are shadowed by
Lwtreslib’s:

* "List",

* "Result",

* "Option",

* "Seq",

* "Map",

* "Set", and

* "Hashtbl".

In all those modules, the underlying data structures are compatible
with those of the "Stdlib" and thus with the rest of the OCaml
ecosystem. However, the primitives in these modules are extended to
support Lwt, "result" and the combination of the two. Specifically,
for each function that traverses the data structure, the module also
contains variants that perform the same traversal within each of the
monad. E.g., for "List.map"

   (* vanilla map *)
   val map : ('a -> 'b) -> 'a list -> 'b list

   (* [result]-aware map: stops at the first error *)
   val map_e : ('a -> ('b, 'trace) result) -> 'a list -> ('b list, 'trace) result

   (* sequential Lwt map: treats each element after the previous one *)
   val map_s : ('a -> 'b Lwt.t) -> 'a list -> 'b list Lwt.t

   (* sequential Lwt-[result] map:
      - treats each element after the previous one
      - stops at the first error *)
   val map_es :
     ('a -> ('b, 'trace) result Lwt.t) ->
     'a list ->
     ('b list, 'trace) result Lwt.t

   (* concurrent Lwt map: treats all the elements concurrently *)
   val map_p : ('a -> 'b Lwt.t) -> 'a list -> 'b list Lwt.t

   (* concurrent Lwt-[result] map:
      - treats all the elements concurrently
      - treats the whole list no matter the success/errors *)
   val map_ep :
     ('a -> ('b, 'trace) result Lwt.t) ->
     'a list ->
     ('b list, 'trace list) result Lwt.t

Check out the online documentation of Lwtreslib  for a description of
the semantic and naming convention.

In addition to shadowing existing modules, "Lwtreslib" also exports
new modules:

* "Seq_e" for "result"-aware variant of "Seq",

* "Seq_s" for Lwt-aware variant of "Seq",

* "Seq_es" for Lwt-"result"-aware variant of "Seq", and

* "WithExceptions" for unsafe accesses such as "Option.get".

Whenever you need to traverse a standard data structure with some
"result" or Lwt or Lwt-"result" function, "Lwtreslib" should have that
function ready for you. **You should never fold over a data structure
with a promise or result accumulator.** E.g., you should do

   List.rev_map_es fetch keys

and you shouldn’t do

   let open Lwt_result_syntax in
   List.fold_left
     (fun resources key ->
       let* resources = resources in
       let* resource = fetch key in
       return (resource :: resources))
     (return [])
     keys

If you do not find the traversal function you need, do not hesitate to
contribute to Lwtreslib.


Working with traces and errors and such directly
================================================

Occasionally, you may need more interaction with traces than the
primitives presented thus far.

For error reporting or debugging purpose, you may need to show traces
to users. You can do so with the following values.

* "pp_print_trace": a "%a"-compatible formatter. Note that the trace
  formatting is unspecified and subject to change. Also be aware that
  it generally prints the trace over multiple lines.

* "pp_print_top_error_of_trace": a "%a"-compatible formatter that only
  shows the most recent error in the trace (or one of the most recent
  errors if there are several). This is useful to get shorter error
  messages. Most often used for the declaration of logging events in
  "Internal_event.Simple".

* "trace_encoding": an encoding for traces. Useful to combine into
  encoding of data structures that contain traces. Most often used for
  the declaration of logging events in "Internal_event.Simple".

If you are working with non-standard data structures and if you need
to define monad-aware traversors for these data structures, you may
need to build some traces by hand. You can do so with the following
values.

* "TzTrace.make : 'e -> 'e trace" useful to convert an "error" into an
  "error   trace". By extension, this is useful to convert an "('a,
  error) result" into an "'a tzresult".

* "TzTrace.cons : 'e -> 'e trace -> 'e trace" is the low-level
  combinators that builds-up traces. In most cases, you’ll want to use
  "trace" or "record_trace" instead, but you might need it when you
  are defining a low-level traversal function for some data structure.

     let iter_with_bounded_errors bound f xs =
       (* we rely on syntax for Lwt, we handle results by hand *)
       let open Lwt_syntax in
       let rec aux_all_ok = function
         | [] -> return_ok ()
         | x :: xs ->
           let* r = f x in
           match r with
           | Ok () -> aux_all_ok xs
           | Error e -> aux_some_error 1 (TzTrace.make e) xs
       and aux_some_error num_errors trace xs =
         if num_errors > bound then
           return_error (TzTrace.cons (Exceeded_error_limit bound) trace)
         else
           match xs with
           | [] -> return_ok ()
           | x :: xs ->
             let* r = f x in
             match r with
             | Ok () -> aux_some_error num_errors trace xs
             | Error e -> aux_some_error (num_errors + 1) (TzTrace.cons e trace) xs
       in
       aux_all_ok xs

* "TzTrace.conp : 'e trace -> 'e trace -> 'e trace" is the parallel
  composition of two traces. Unlike "cons", the traces composed by
  "conp" are not organised hierarchically. The errors are presented as
  having happened side-by-side.

  Note that currently there is little difference between cons and conp
  traces. But the difference will be more marked in the future.

  You should use "conp" (rather than "cons") when you are gathering
  errors and traces from two or more concurrent processes.


Working within the protocol
===========================

If you are working on the protocol, things are slightly different for
you. This is because the protocol has a restricted access to external
resources and libraries. You can find more details in the dedicated
documentation. This section focuses on the error-monad within the
protocol.

The protocol environment libraries evolve at a slightly different pace
than the underlying library. You need to check the "mli" files within
"src/lib_protocol_environment/sigs/".

Note that unlike in the shell, the traces in the protocol are already
abstract. As a result there is no matching of traces (and thus errors)
within the protocol: you can match "Ok" and "Error", but not the
payload of the "Error". This part of the legacy code has already been
removed.

The main difference between the protocol and the shell is that the
"category" parameter of the "register_error_kind" function is
meaningful. You must pass a category which is appropriate for the
error you are registering:

* "Branch": is for branch-specific failures, i.e., failures that
  happen in the current branch (of the chain) but maybe wouldn’t
  happen in a different branch. E.g., a reference to an unknown block
  is invalid, but it might become valid once the head block has
  changed. This category is then used by the shell to retry after the
  branch changes.

* "Temporary": is for transient failures, i.e., failures that happen
  but may not always happen. This category is used by the shell to
  retry at some later time.

* "Permanent": is for irremediable failures, i.e., failures that
  happen and will always happen whatever the context. E.g.,
  originating a contract that does not type-check is a permanent
  error. This is used by the shell to mark the data as invalid.

* "Outdated": is for failures that happen when some data is too old.

Another thing to consider is that errors from the protocol can reach
the shell. However, because the "error" type of the protocol is
distinct from that of the shell, the protocol errors are wrapped
inside a shell error constructor.

This has no impact within the protocol (where shell errors don’t
exist) nor within the shell (where protocol errors are automatically
wrapped inside a shell error). However, it can have an impact in the
spaces in between. Most typically, this matters in the unit-tests of
the protocol ("src/proto_alpha/lib_protocol/test/unit/") where you
call some protocol functions directly. In this case, you need to wrap
the errors yourself, using the wrapping functions provided by the
environment: "Environment.wrap_tzresult", "Environment.wrap_tztrace",
and "Environment.wrap_tzerror".


Working below the error-monad
=============================

If you are working on some low-level libraries (e.g.,
"src/lib_stdlib") or the external dependencies (e.g., "data-encoding")
you don’t have access to the error monad at all.

In this case, you can still use the "result" type but you need to
define your own "let*" binding operator: "let ( let* ) = Result.bind".

You can also use Lwt which provides its own Lwt.Syntax module.

Finally, the Lwt_result module (provided as part of Lwt) can help you
deal with result-Lwt combinations, including via its Lwt_result.Syntax
module.


Working with external libraries
===============================

This tutorial covers error-management techniques in Octez. However,
from within Octez, you may need to call external libraries for
cryptography or RPCs or data-encoding or what have you.

The first thing you do is to carefully read the documentation of the
external library you are using. You should check the overview
documentation with a look out for comments on error management.

Then, you also need to read the documentation of each function that
you are calling. This documentation may explain how errors are
handled: does the function return a "result"? does it raise and
exception? is it unspecified?

If the function you are calling may raise exceptions, you should catch
these exceptions. You can either do so at the level of the call itself
or, if you are calling multiple functions that can all raise similar
exceptions, around a whole block of calls.

When you catch an exception, the most common thing to do is to
translate it or wrap it into a "result" or a "tzresult".

   try
     let v1 = Data_encoding.Json.destruct e1 j1 in
     let v2 = Data_encoding.Json.destruct e2 j2 in
     Ok (v1, v2)
   with
     | exc -> Error (Cannot_destruct_json_value exc)

Note that if you are calling an Lwt function, you have to use
"Lwt.catch" or "Lwt.try_bind" rather than "try"-"with".

   Lwt.catch
     (fun () ->
       let open Lwt_syntax in
       let* () = Lwt_unix.mkdir d1 perm in
       let* () = Lwt_unix.mkdir d2 perm in
       Lwt_result_syntax.return_unit)
     (function
       | exc -> Lwt_result_syntax.fail (Cannot_destruct_json_value exc))

The error monad provides several helpers functions for catching
exceptions .

   val catch : ?catch_only:(exn -> bool) -> (unit -> 'a) -> 'a tzresult

If the function you are calling may raise exceptions only under well-
defined conditions on the parameters, then you can also check those
conditions yourself and ignore the exceptions. When doing so, please
add a comment to explain it.

   let get_or_defaults low_default high_default array offset =
     if offset < 0 then
       low_default
     else if offset >= Array.length array then
       high_default
     else
       (* This cannot raise because of checks on offset above *)
       Array.get array offset

If the function may fail with "result", you can map the error directly
or simply continue with it. If it may fail with "option", you can
translate "None" into an appropriate error.

   match find k kvs with
   | None -> Error "cannot find key"
   | Some v -> Ok v

If the function’s documentation specifies some pre-conditions but
doesn’t explain what happens if those aren’t met, then you must check
those pre-conditions.

Part 4: Appendices
******************

These appendices are Part 4 of 4 of the The Error Monad tutorial.


In depth discussion: what even is a monad?
==========================================

This tutorial has been pretty loose with the word monad. It has
focused on usage with very little explanations of fundamental
concepts. It is focused on the surface syntax instead of the
underlying mathematical model. This section goes into a bit more
details about the general principles of monads and such. It does not
claim to even attempt to give a complete description of monads, it
simply gives more details than the previous sections.

For coding purposes, a monad is a parametric type equipped with a set
of specific operators.

   type 'a t
   val bind : 'a t -> ('a -> 'b t) -> 'b t
   val return : 'a -> 'a t

The "return" operator injects a value into the monad and the "bind"
operator continues within the monad.

The set of operators must also follow the monad laws. For example
"bind (return x) f" must be equivalent to "f x".

Monads are used as a generic way to encode different abstractions
within a programming language: I/O, errors, collections, etc. For
example, the "option" monad is defined as

   module OptionMonad = struct
     type 'a t = 'a option
     let bind x f = match x with | None -> None | Some x -> f x
     let return x = Some x
   end

And it is useful when dealing with queries that may have no answer.
This can be used as a lighter form of error management than the
"result" monad.

Some programming languages also offer syntactic sugar for monads. This
is to avoid having to write "bind" within "bind" within "bind". E.g.,
Haskell relies heavily on monads and has the dedicated "do"-notation.
In OCaml, you can use one of the following methods:

* binding operators (since OCaml 4.08.0)

     let add x y =
       let ( let* ) = OptionMonad.bind in
       let* x = int_of_string_opt x in
       let* y = int_of_string_opt y in
       Some (string_of_int (x + y))

* infix operators

     let add x y =
       let ( >>= ) = OptionMonad.bind in
       int_of_string_opt x >>= fun x ->
       int_of_string_opt y >>= fun y ->
       Some (string_of_int (x + y))

  Note that mixing multiple infix operators is not always easy because
  of precedence and associativity.

* partial application and infix "@@"

     let add x y =
       OptionMonad.bind (int_of_string_opt x) @@ fun x ->
       OptionMonad.bind (int_of_string_opt y) @@ fun y ->
       Some (string_of_int (x + y))

  This is useful for the occasional application: you do not need to
  declare a dedicated operator nor open a dedicated syntax module.

Monads can have additional operators beside the required core. E.g.,
you can add "OptionMonad.join : 'a option option -> 'a option".


In depth discussion: "Error_monad", "src/lib_error_monad/", "Tezos_base__TzPervasives", etc.
============================================================================================

The different parts of the error monad (syntax modules, extended
stdlib, tracing primitives, etc.) are defined in separate files. Yet,
they are all available to you directly. This section explains where
each part is defined and how it reaches the scope of your code.

**From your code, working back to the definitions.**

In most of Octez, the "Error_monad" module is available. Specifically,
it is available in all the packages that depend on "tezos-base". This
covers everything except the protocols and a handful of low-level
libraries.

In those part of Octez, the build files include "-open
Tezos_base__TzPervasives".

The module "Tezos_base__TzPervasives" is defined by the compilation
unit "src/lib_base/TzPervasives.ml".

This compilation unit gathers multiple low-level modules together. Of
interest to us is "include Tezos_error_monad.Error_monad" (left
untouched in the "mli") and "include Tezos_error_monad.TzLwtreslib"
(not present in the "mli", used to shadow the Stdlib modules "List",
"Option", "Result", etc.).

The "Error_monad" module exports:

* the "error" type along with the "register_error_kind" function,

* the "'a tzresult" type,

* the "TzTrace" module,

* the "Result_syntax" and "Lwt_result_syntax" modules (from a
  different, more generic name),

* and exports a few more functions.

The rest of the "tezos-error-monad" package:

* defines the "'a trace" type (in "TzTrace.ml"), and

* instantiates "TzLwtreslib" by applying "Lwtreslib"’s "Traced"
  functor to "TzTrace".

The "Lwtreslib" module exports a "Traced (T: TRACE)" functor. This
functor takes a definition of traces and returns a group of modules
intended to shadow the Stdlib.

**From the underlying definitions, working all the way up to your
code.**

At the low-level is Lwtreslib.

* "src/lib_lwt_result_stdlib/bare/sigs": defines interfaces for basic,
  non-traced syntax modules and Stdlib-replacement modules.

* "src/lib_lwt_result_stdlib/bare/structs": defines implementations
  basic, non-traced syntax modules and Stdlib-replacement modules.

* "src/lib_lwt_result_stdlib/traced/sigs": defines interfaces for
  traced syntax modules and Stdlib-replacement modules. These
  interfaces are built on top of the non-traced interfaces, mostly by
  addition and occasionally by shadowing.

* "src/lib_lwt_result_stdlib/traced/structs": defines implementations
  for traced syntax modules and Stdlib-replacement modules. These
  implementations are built on top of the non-traced implementations,
  mostly by addition and occasionally by shadowing. These are defined
  as functors over some abstract tracing primitives.

* "src/lib_lwt_result_stdlib/lwtreslib.mli": puts together the traced
  implementations into a single functor "Traced" that takes a trace
  definition and returns fully instantiated modules to shadow the
  Stdlib.

Above Lwtreslib is the Error monad.

* "src/lib_error_monad/TzTrace.ml": defines the "'a trace" type along
  with the low-level trace-construction primitives.

* "src/lib_error_monad/TzLwtreslib.ml": instantiates
  "Lwtreslib.Traced" with "TzTrace".

* "src/lib_error_monad/monad_extension_maker.ml": provides a functor
  which, given a tracing module, provides some higher level functions
  for tracing as well as a few other functions.

* "src/lib_error_monad/core_maker.ml": provides a functor which, given
  a name, provides an "error" type, a "register_error_kind" function,
  and a few other related functions. This is a functor so we can
  instantiate it separately for the shell and for each of the
  protocols.

* "src/lib_error_monad/TzCore.ml": instantiates the "core_maker"
  functor for the shell.

* "src/lib_error_monad/error_monad.ml": puts together all of the above
  into a single module.

Above the Error monad is lib-base:

* "src/lib_base/TzPervasives.ml": exports the "Error_monad" module,
  includes the "Error_monad" module, exports each of the "TzLwtreslib"
  module.


In depth discussion: "result" as data and "result" as control-flow
==================================================================

Note that "result" (and similarly, "tzresult") is a data type.
Specifically

   type ('a, 'b) result =
   | Ok of 'a
   | Error of 'b

You can treat values of type "result" as data of that data-type. In
this case, you construct and match the values, you pass them around,
etc.

Note however that, in Octez, we also use the "result" type as a
control-flow mechanism. Specifically, in conjunction with the "let*"
binding operator, the "result" type has a continue/abort meaning.

Within your code, you can go from one use to the other. E.g.,

   let xs =
     List.rev_map
       (fun x ->
         (* [result] as control-flow *)
         let open Result_syntax in
         let* .. = .. in
         let* .. = .. in
         return ..)
       ys
   in
   let successes xs =
     (* [result] as data *)
     List.length (List.rev_filter_ok xs)
   in
   ..

Using "result" as sometimes data and sometimes control-flow is the
main reason to bend the guidelines about which syntax module to open.
E.g., if your function returns "(_, _) result Lwt.t" but the "result"
is data returned by the function rather than control-flow used within
the function, then you should open "Lwt_syntax" (rather then
"Lwt_result_syntax").

As a significant aside, note that in OCaml you can also use exceptions
for control-flow (with "raise" and "try"-"with" and
"match"-"with"-"exception") and as data (the type "exn" is an
extensible variant data-type).

   (** [iter_no_raise f xs] applies [f] to all the elements of [xs]. If [f] raises
       an exception, the iteration continues and [f] is still applies to other
       elements. The function returns pairs of the exceptions raised by [f] along
       the elements of [xs] that triggered these exceptions. *)
   let iter_no_raise f xs =
     List.fold_left
       (fun excs x ->
         match f x with
         | exception exc -> exc :: excs
         | () -> excs)
       []
       xs

You can find uses of exception as data within the error monad itself.
First, the generic failure functions ("error_with", "error_with_exn",
"failwith", and "fail_with_exn") are just wrapper around an "error"
which carries an exception (as data).

Second, Lwtreslib provides helpers to catch exceptions. E.g.,
"Result.catch : (unit -> 'a) -> ('a, exn) result" calls a function and
wraps any raised exception inside an "Error" constructor.


In depth discussion: pros and cons of "result" compared to other error management techniques
============================================================================================

In Octez, we use "result" and the specialised "tzresult". For this
reason, this tutorial is focused on "result"/"tzresult". However,
there are other techniques for handling errors. This section compares
them briefly.

In general you should use "result" and "tzresult" but in some specific
cases you can deviate from that. The comparisons below may help you
decide.

**Exceptions**

In exception-based error handling, you raise an exception (via
"raise") when an error occurs and you catch it (via "try"-"with") to
recover. Exceptions are fast because the OCaml compiler and runtime
provide the necessary mechanisms directly.

Whether a function can raise an exception or not cannot be determined
by its type. This means that it is easy to forget to recover from an
exception. An external library may change the set of exceptions that a
function raises and you need to update calls to this function, but the
type-checker cannot warn you about it. This places a heavy burden on
the developer who is responsible for checking the documentation of all
the functions they call.

Exception-raising functions should be documented as such using the
"@raise" documentation keyword.

   Pros: performance is good, used widely in the larger ecosystem.
   Cons: you cannot rely on the type-checker to help you at all, you
   depend on the quality of the documentation of your external and
   internal dependencies.

Note that within the protocol, you should not use exceptions at all.

**tzresult**

With "tzresult", errors are carried by the "Error" constructor of a
"result". In this way an "'a tzresult" represents the result of a
computation that normally returns an "'a" but may fail.

Because the type of errors is an abstract wrapper ("trace") around an
extensible variant ("error"), you can only recover from these errors
in a generic way.

   Pros: the type of a function indicates if it can fail or not, you
   cannot forget to check for success/failure.
   Cons: you cannot check which error was raised, registration is heavy
   and complicated.

**result**

With "result", errors are carried by the "Error" constructor. Each
function defines its own type of errors.

   Pros: the type of a function indicates if and how it can fail, you
   cannot forget to check for success/failure, you can check the payload
   of failures.
   Cons: different errors from different functions cannot be used
   together (need conversions), "and*" is unusable.

**option**

With "option", errors are represented by the "None" constructor.
Errors are completely void of payload.

Because there are no payloads attached to an error, you should
generally treat the error directly at the call site. Otherwise you
might lose track of the origin of the failure. E.g., what was not
found in the following code fragment?

   match
     let open Option_syntax in
     let* z = find "zero" in
     let* o = find "one" in
     Some (z, o)
   with
   | None -> ..
   | Some (z, o) -> ..

   Pros: the type of a function indicates if it can fail, you cannot
   forget to check for success/failure.
   Cons: a single kind of errors means it cannot be very informative.

Option is a common enough strategy that the "Option_syntax" and
"Lwt_option_syntax" modules are available in the Octez source.

**fallback**

Another approach to errors is to have a default or fallback value. In
that case, the function returns a default sensible value when it would
raise and exception or return an error. Alternatively, it can take
this fallback value as parameter.

   (** @raise Not_found if argument is [None] *)
   val get : 'a option -> 'a

   (** returns [default] if argument is [None] *)
   val value : default:'a -> 'a option -> 'a

   Pros: there is no error.
   Cons: doesn’t work for every function, works differently on different
   functions.


Legacy code
===========

The codebase contains only "let"-style binding operators. However, you
might encounter infix bindings in older protocols. If you do and you
are unsure about what the many infix operators do, read on.

The legacy code is written with infix bindings instead of "let"-style
binding operators. The binding ">>?" for "result" and "tzresult",
">>=" for Lwt, and ">>=?" for Lwt-"result" and Lwt-"tzresult". A full
equivalence table follows.

+----------------------------------------+---------------------------------+
| Modern                                 | Legacy                          |
+----------------------------------------+---------------------------------+
| let open Result_syntax in let* x = e   | e >>? fun x -> e'               |
| in e'                                  |                                 |
+----------------------------------------+---------------------------------+
| let open Result_syntax in let+ x = e   | e >|? fun x -> e'               |
| in e'                                  |                                 |
+----------------------------------------+---------------------------------+
| let open Lwt_syntax in let* x = e in   | e >>= fun x -> e'               |
| e'                                     |                                 |
+----------------------------------------+---------------------------------+
| let open Lwt_syntax in let+ x = e in   | e >|= fun x -> e'               |
| e'                                     |                                 |
+----------------------------------------+---------------------------------+
| let open Lwt_result_syntax in let* x = | e >>=? fun x -> e'              |
| e in e'                                |                                 |
+----------------------------------------+---------------------------------+
| let open Lwt_result_syntax in let+ x = | e >|=? fun x -> e'              |
| e in e'                                |                                 |
+----------------------------------------+---------------------------------+
| "and*", "and+" (any syntax module)     | No equivalent, uses "both_e",   |
|                                        | "both_p", or "both_ep"          |
+----------------------------------------+---------------------------------+
| let open Lwt_result_syntax in let*! x  | (e >>= ok) >>=? fun x -> e'     |
| = e in e'                              |                                 |
+----------------------------------------+---------------------------------+
| let open Lwt_result_syntax in let*? x  | e >>?= fun x -> e'              |
| = e in e'                              |                                 |
+----------------------------------------+---------------------------------+

In addition, instead of dedicated "return" and "fail" functions from a
given syntax module, the legacy code relied on global values.

+----------------------------------------+---------------------------------+
| Modern                                 | Legacy                          |
+----------------------------------------+---------------------------------+
| let open Result_syntax in return x     | ok x                            |
+----------------------------------------+---------------------------------+
| let open Result_syntax in fail e       | No equivalent, uses "Error e"   |
+----------------------------------------+---------------------------------+
| let open Lwt_syntax in return x        | No equivalent, uses "Lwt.return |
|                                        | x"                              |
+----------------------------------------+---------------------------------+
| let open Lwt_result_syntax in return x | return x                        |
+----------------------------------------+---------------------------------+
| let open Lwt_result_syntax in fail e   | No equivalent, uses             |
|                                        | "Lwt.return_error e"            |
+----------------------------------------+---------------------------------+
| let open Result_syntax in return x     | ok x                            |
+----------------------------------------+---------------------------------+
| let open Result_syntax in tzfail e     | error e                         |
+----------------------------------------+---------------------------------+
| let open Lwt_result_syntax in return x | return x                        |
+----------------------------------------+---------------------------------+
| let open Lwt_result_syntax in tzfail e | fail e                          |
+----------------------------------------+---------------------------------+

In addition to these syntactic differences, there are also usage
differences. You might encounter the following patterns which you
should not repeat:

* Matching against a trace:

     match f () with
     | Ok .. -> ..
     | Error (Timeout :: _) -> ..
     | Error trace -> ..

  This is discouraged because the compiler is unable to warn you if
  the matching is affected by a change in the code. E.g., if you add
  context to an error in one place in the code, you may change the
  result of the matching somewhere else in the code.

The "clic" library
******************

tezos-clic  is an OCaml combinator library for writing self-
documenting command-line argument parsers. Clic is similar to
cmdliner, but (unlike cmdliner) "tezos-clic" allows to define a
domain-specific language as a subset of a natural language, by mixing
keyword and positional arguments. For example, in "octez-client",
commands look like this, thanks to "tezos-clic":

   octez-client list understood protocols
   octez-client compute chain id from block hash <hash>
   octez-client originate contract <contract_alias> transferring <initial_balance> from <originator> running <script>

"clic" is used for most of the binaries distributed with Octez, such
as "octez-client" and "octez-codec". A notable exception is "octez-
node" which uses "cmdliner".

In this tutorial, we will give a gentle introduction to "clic" by
demonstrating how to implement a wallet command inspired by those of
"octez-client". Impatient readers will find the full example in this
file.


Wallet example
==============

Command-line parsing in "clic" is centered around *commands*. A
command roughly corresponds to one action of the command-line
application. For instance, "octez-client get balance" and "octez-
client run script <script>" are two different commands. To demonstrate
the use of "clic", we will add the following command to a dummy octez-
client: "list known contracts". As the name indicates, this command
outputs the contracts known to the wallet.

Commands execute in a user-supplied *context*. Typically, the context
serves as an abstraction barrier between the command and its execution
environment. This mechanism allows commands to be defined once, but
reused in different contexts. An example where this useful is for
swapping out the normal context for a *dry-run context*, allowing
users to simulate the effect of commands before running them for real.

Note that using contexts is not required: it can be side-stepped by
defining commands that execute over a context of type "unit".

In the running example, we define a context whose job is to contain
the actual implementation of the commands.

We first declare "CONTEXT" as the signature of modules that contain a
"list_known_contracts" function. As we add more commands to the
example, we will also extend the context. For convenience, we add a
type alias "context" for first-class "CONTEXT" modules.

   module type CONTEXT = sig
     val list_known_contracts : unit -> unit
   end

   type context = (module CONTEXT)


We define a "Dummy_context" that satisfies the "CONTEXT" signature,
with a place-holder function simply printing what the actual function
would do.

   module Dummy_context : CONTEXT = struct
     let list_known_contracts () =
       Format.printf "<Print the list of known contracts>\n"
   end


Commands are defined through the "Tezos_clic.command" function. It has
the following signature:

   Tezos_clic.command :
     ?group:Tezos_clic.group ->
     desc:string ->
     ('b, 'ctx) Tezos_clic.options ->
     ('a, 'ctx) Tezos_clic.params ->
     ('b -> 'a) -> 'ctx Tezos_clic.command

First, commands have a group and description that are used to generate
documentation. Groups are used to organize commands of related
functionality. This is convenient for applications such as "octez-
client" that defines a large number of commands which are grouped on
themes such as querying, testing, and address management. You can see
the documentation online.

In our example, using groups is not required, but we add a group to
demonstrate the feature. A group is just a name and a description of
the commands in that group.

   let wallet_group =
     {Tezos_clic.name = "wallet_group"; title = "Wallet-related commands"}


The third argument to "Tezos_clic.command" specifies the set of
options that commands take, which modulate its behavior (think "--
verbose" or "--output json"). The value of the options will be
collected as a value of type "'b".

The command is specified through a sequence of *params*, given as the
fourth argument to "Tezos_clic.command". Params can be *prefixes*:
fixed strings that must be given when calling the command. Above, we
mentioned the "get balance" command of "octez-client". The sequence
"get balance" is an example of such a prefix. A param can also define
a *hole* to be filled by the user on the command line. An example is
given by "octez-client get balance for <contract>". Here, the command
consists of a sequence of prefixes "get balance for" followed by the
hole "<contract>", filled by the user on the command-line. No matter
how the params specification is constructed, it is terminated by the
combinator "Tezos_clic.stop". The params specification will construct
a function type "'a", which together with the type "'b" from the
options is used to construct the signature "'b -> 'a" that the
*command handler* must adhere to.

Note also that by construction, the type "'a" will always be of the
form "... -> 'ctx -> unit tzresult Lwt.t", so that commands always
receive a context and must return "unit" in the "tzresult Lwt.t"
monad.

The fifth argument "Tezos_clic.command" is the command handler. This
function implements the actual command. It is passed any supplied
command-line options (as a value of type "'b") and the contents of any
holes in the params (which are, respectively, types of the arguments
of the function type "'a").

An example will be helpful to illustrate the signature of the command
handler. A command that takes no options will have "'b = unit". If,
furthermore, its params have no holes, then "'a = context -> unit
tzresult Lwt.t". The command is thus a function

   unit -> context -> unit tzresult Lwt.t

In other words, taking no arguments except for "()" and the context,
and returning "unit" wrapped in the "tzresult" and "Lwt.t" monads.

As a second illustration of the signature of the command handler,
consider a command "get balance for <contract>", where "<contract>" is
of type "Contract.t". Additionally, the command should consume a "--
output <format>" option (where "<format>" is a string such as
""json"", ""csv"", etc.). In this case, "'b = string option", "'a =
Contract.t -> context -> unit tzresult Lwt.t", and the command handler
a function with the signature:

   string option -> Contract.t -> context -> unit tzresult Lwt.t

We now have enough meat on our bones to define the "list known
contracts" command.

   module List_known_contracts = struct
     let options = Tezos_clic.no_options

     let params = Tezos_clic.(prefixes ["list"; "known"; "contracts"] stop)

     let list_known_contracts_handler :
         unit -> context -> unit Tezos_error_monad.Error_monad.tzresult Lwt.t =
      fun () ctxt ->
       let module C = (val ctxt) in
       C.list_known_contracts () ;
       Lwt_result_syntax.return_unit

     let command =
       Tezos_clic.command
         ~group:wallet_group
         ~desc:"Prints the list of known contracts"
         options
         params
         list_known_contracts_handler
   end

   let commands = [List_known_contracts.command]


We wrap the command and its related definitions in a module
"List_known_contracts". We specify that the command should have no
options through "Tezos_clic.no_options". We specify that the params is
a list of prefixes without holes. We then define the command handler
"list_known_contracts_handler". As the command has no options and its
params no holes, the signature of the handler becomes:

   unit -> context -> unit tzresult Lwt.t

This command handler does no more than unwrap and call the appropriate
function of the context. Finally we add the command to the full list
of commands "commands" that the application will provide.

Having thus defined the commands, we now define the entrypoint of our
application:

   let () =
     (* 1. Setup formatter with color *)
     ignore
       Tezos_clic.(
         setup_formatter
           Format.std_formatter
           ~isatty:(Unix.isatty Unix.stdout)
           Short) ;
     (* 2. Setup context and dispatch commands *)
     let ctxt = (module Dummy_context : CONTEXT) in
     let result =
       Lwt_main.run
         (Tezos_clic.dispatch commands ctxt (Array.to_list Sys.argv |> List.tl))
     in
     (* 3. Handle results *)
     match result with
     | Ok () -> ()
     | Error [Tezos_clic.Help _command] ->
         Format.printf "<display help>\n" ;
         exit 0
     | Error _ ->
         Format.printf "Could not parse command-line arguments.\n" ;
         exit 1

It consists of three sections. We first setup a formatter that
depending on whether the command is executed in a tty (as opposed to
e.g. being piped to a file) enables color in the output. Then, we pack
our context in a first-class module, that we pass to the
"Tezos_clic.dispatch". This function takes the full list of commands,
as defined by "commands", the context, and the list of raw command-
line arguments passed through the application. The list of command-
line arguments should not contain the first element (the name of the
program itself), so this is why the "List.tl" function is used. The
"dispatch" function will parse the arguments, and call the appropriate
command handler if a valid command was given. If this is the case, "Ok
()" is returned. If no arguments have been passed, or if "--help" is
given, then "Error [Tezos_clic.Help _command]" is returned. In this
case the application should print the appropriate usage instruction.
If some other unrecognized arguments are given we give a placeholder
error message, which we'll replace with something more helpful below.

We use "dune" to compile the example, with the following "dune" file:

   (executable
    (name clic_example)
    (libraries
     octez-libs.clic
     lwt.unix))

The dependencies of the example are "tezos-clic" and "lwt.unix" which
can be installed through "opam install tezos-clic lwt". Let's try it
out:

   $ dune exec ./clic_example.exe -- list known contracts
   <Print the list of known contracts>

which is as expected. Giving no arguments, or when passed the "--help"
flags, our placeholder help message is output:

   $ dune exec ./clic_example.exe -- --help
   <display help>
   $ dune exec ./clic_example.exe --
   <display help>

Similarly, if we attempt to call an unrecognized command:

   $ dune exec ./clic_example.exe -- foobar
   Could not parse command-line arguments.


Conclusion
==========

This example demonstrates how to define a simple "clic" application
with one simple command. This is far from a complete demonstration of
"clic". "clic" also includes facilities for generating interactive,
searchable documentation, with both command-line and HTML outputs.
"clic" also gives facilities for implementing shell auto-completion.
For more information, refer to "clic"'s API documentation .

The "data_encoding" library
***************************

Throughout the Tezos protocol, data is serialized so that it can be
used via RPC, written to disk, or placed in a block. This
serialization/de-serialization is handled via the data-encoding
library by providing a set primitive encodings and a variety of
combinators.


Examples/Tutorial
=================

The following is a very brief introduction to the data-encoding
library.

Note:

  A more complete documentation, including a full tutorial with many
  examples and a reference of all available combinators, is available
  in data-encoding's API documentation.


Encoding an integer
-------------------

Integers are defined as other concrete data types with a generic
encoding type "type 'a encoding". This means that it is an encoding
to/from type "int". There are a variety of ways to encode an integer,
depending on what binary serialization you want to achieve:

* "Data_encoding.int8"

* "Data_encoding.uint8"

* "Data_encoding.int16"

* "Data_encoding.uint16"

* "Data_encoding.int31"

* "Data_encoding.int32"

* "Data_encoding.int64"

For example, an encoding that represents a 31 bit integer has type
"Data_encoding.int31 = int Data_encoding.encoding".

   let int31_encoding = Data_encoding.int31


Encoding an object
------------------

Encoding a single integer is fairly uninteresting. The data-encoding
library provides a number of combinators that can be used to build
more complicated objects. Consider the type that represents an
interval from the first number to the second:

   type interval = int64 * int64

We can define an encoding for this type as:

   let interval_encoding =
     Data_encoding.(obj2 (req "min" int64) (req "max" int64))

In the example above we construct a new value "interval_encoding" by
combining two "int64" integers using the "obj2" (object with two
fields) constructor.

The library provides different constructors, i.e. for objects that
have no data ("Data_encoding.empty"), constructors for object up to 10
fields, constructors for tuples, list, etc.

These are serialized to binary by converting each internal object to
binary and placing them in the order of the original object and to
JSON as a JSON object with field names.


Lists, arrays, and options
--------------------------

List, arrays and options types can be built on top of ground data
types.

   type interval_list = interval list

   type interval_array = interval array

   type interval_option = interval option

And the encoders for these types as

   let interval_list_encoding = Data_encoding.list interval_encoding
   let interval_array_encoding = Data_encoding.array interval_encoding
   let interval_option_encoding = Data_encoding.option interval_encoding


Union types
-----------

The Octez codebase makes heavy use of variant types. Consider the
following variant type:

   type variant = B of bool
                | S of string

Encoding for this types can be expressed as:

   let variant_encoding =
     let open Data_encoding in
     union ~tag_size:`Uint8
       [ case ~title:"B" (Tag 0) bool
           (function B b -> Some b | _ -> None)
           (fun b -> B b) ;
         case ~title:"S" (Tag 1) string
           (function S s -> Some s | _ -> None)
           (fun s -> S s) ]

This variant encoding is a bit more complicated. Let’s look at the
parts of the encoding:

* We include an optimization hint to the binary encoding to inform it
  of the number of elements we expect in the tag. In most cases, we
  can use "`Uint8", which allows you to have up to 256 possible cases
  (default).

* We provide a function to wrap the datatype. The encoding works by
  repeatedly trying to decode the datatype using these functions until
  one returns "Some payload". This payload is then encoded using the
  dataencoding specified.

* We specify a function from the encoded type to the actual datatype.

Since the library does not provide an exhaustive check on these
constructors, the user must be careful when constructing union types
to avoid unfortunate runtime failures.


How the Dataencoding module works
=================================

This section is 100% optional. You do not need to understand this
section to use the library.

The library uses GADTs to provide type-safe serialization/de-
serialization. From there, a runtime representation of JSON objects is
parsed into the type-safe version.

First we define an untyped JSON AST:

   type json =
     [ `O of (string * json) list
     | `Bool of bool
     | `Float of float
     | `A of json list
     | `Null
     | `String of string ]

This is then parsed into a typed AST (we eliminate several cases for
clarity):

   type 'a desc =
     | Null : unit desc
     | Empty : unit desc
     | Bool : bool desc
     | Int64 : Int64.t desc
     | Float : float desc
     | Bytes : Kind.length -> Bytes.t desc
     | String : Kind.length -> string desc
     | String_enum : Kind.length * (string * 'a) list -> 'a desc
     | Array : 'a t -> 'a array desc
     | List : 'a t -> 'a list desc
     | Obj : 'a field -> 'a desc
     | Objs : Kind.t * 'a t * 'b t -> ('a * 'b) desc
     | Tup : 'a t -> 'a desc
     | Union : Kind.t * tag_size * 'a case list -> 'a desc
     | Mu : Kind.enum * string * ('a t -> 'a t) -> 'a desc
     | Conv :
         { proj : ('a -> 'b) ;
           inj : ('b -> 'a) ;
           encoding : 'b t ;
           schema : Json_schema.schema option } -> 'a desc
     | Describe :
         { title : string option ;
           description : string option ;
           encoding : 'a t } -> 'a desc
     | Def : { name : string ;
               encoding : 'a t } -> 'a desc

* The first few constructors define all ground types.

* The constructors for "Bytes", "String" and "String_enum" include a
  length field in order to provide safe binary serialization.

* The constructors for "Array" and "List" are used by the combinators
  we saw earlier.

* The "Obj" and "Objs" constructors create JSON objects. These are
  wrapped in the "Conv" constructor to remove nesting that results
  when these constructors are used naively.

* The "Mu" constructor is used to create self-referential definitions.

* The "Conv" constructor allows you to clean up a nested definition or
  compute another type from an existing one.

* The "Describe" and "Def" constructors are used to add documentation

The library also provides various wrappers and convenience functions
to make constructing these objects easier. Reading the documentation
in the mli file should orient you on how to use these functions.

Using The Event Logging Framework
*********************************

Developers of most modules should consider generating useful
information for the Logging subsystem.

In the Octez code base, logging is instrumented using an asynchronous
event system, where log events are emitted in various part of the code
and consumed by the so called sinks.  This is done by using the tezos-
event-logging library.

We use two levels of abstraction to define these events. Heavy events
are defined using the most generic API and have the full expressive
power and structure. Simple events are record-like structures which
are constructed on top of the generic event api, and are meant to be
less verbose and more developer friendly.


Adding Events
=============


Simple Events
-------------

Simple events are record-like structures which are constructed
directly (i.e. there is no high-level OCaml type, and parsing events
is not accessible programmatically). The API is less generic than the
“Heavy” Events'.

See for instance src/lib_node_config/config_file.ml (search for the
string "declare_"):

* one declares typed-record events with
  "Internal_event.Simple.declare_<number-of-fields>",

* and they are called with "val Simple.emit: 'a Simple.t -> 'a ->
  (unit, trace) result Lwt.t".

Please read the guidelines that are documented in the
Internal_event.Simple module. .


“Heavy” Events
--------------

The most generic API for defining events "Internal_event.Make" gives
full expressive power and structure.

See for instance: src/proto_alpha/lib_delegate/client_baking_blocks.ml
(module "Block_seen_event"):

* one needs to call the functor "module Event =
  Internal_event.Make(Definition)" where "Definition:
  Internal_event.EVENT_DEFINITION";

* this provides "val Event.emit: ?section:Section.t ->
  Block_seen_event.t -> (unit, trace) result Lwt.t".


Creating Sinks
==============

A sink is a module of type "Internal_event.SINK" which is registered
within the system (just like events or errors kinds) using: "let () =
Internal_event.All_sinks.register (module My_sink)". Its chosen URI-
scheme should be unique.

See for instance:

* The (simplest) module "Lwt_log_sink":
  src/lib_event_logging/internal_event.ml,

* or, the much more complex,
  src/lib_base/unix/file_descriptor_sink.ml.

Octez Architecture Details
**************************

The following pages offer different modeling perspectives of some key
components within Octez. They form an onboarding guide for anyone
looking to navigate and contribute to the codebase. This section
assumes the reader is familiar with both Octez and the current
protocol reference manuals.

* Baker application's architecture

  * Scheduling

  * States

  * Transitions

  * Actions

  * Operation worker

  * Forge worker

* RPC handling architecture

  * RPC server initialization

  * Connection management

  * The case of streams and chunks

  * Software stacks

  * Debugging

  * See also

Baker application's architecture
********************************

The Octez baker is an executable responsible for the creation of
blocks and consensus operations on the Tezos blockchain according to
the Tenderbake consensus algorithm.

The baker is designed as finite-state machine, with states,
transitions and specific consensus-related events .

The machine is orchestrated by a scheduler, which operates in a never-
ending loop to listen for ongoing events, trigger transitions
accordingly, and perform side-effects.

It relies on two workers running alongside the main scheduling loop to
manage auxiliary tasks:

* The operation worker maintains a pool of fresh operations by
  listening to the node's mempool. It is responsible for tracking
  consensus and reports "Prequorum_reached" and "Quorum_reached"
  events to the main scheduler.

* The forge worker is responsible for crafting blocks and operations.
  It receives crafting requests from the scheduler and notifies
  completion with "New_forge_event (Block_ready block)",
  "New_forge_event (Preattestation_ready op)", or "New_forge_event
  (Attestation_ready op)" events.


Scheduling
==========

Alongside the operation and forge workers, the scheduler tracks the
following events itself:

* new blocks via the "/monitor/heads" and "/monitor/validated_blocks"
  RPCs, where new inputs are converted into "New_head_proposal p" and
  "New_valid_proposal p" events, respectively.

* end of rounds and baking times by spawning timers that trigger
  "Timeout" events.

Upon receiving a new event, the scheduler applies the corresponding
transition based on the current state and the event. The transition
function returns a new state and an action. These actions are side-
effect requests, such as injecting operations, forging blocks, or
monitoring prequorums.

The scheduler then executes the current action and prepares for the
next loop iteration: it sets a timer to track the next timeout
according to the current state and then sleeps until the next event.


States
======

Documentation


Transitions
===========

Documentation


Actions
=======

Documentation


Operation worker
================

The operation worker maintains an operations pool by listening for new
operations incoming from the node via the
"/mempool/monitor_operations" streaming RPC. Every time the node
changes head, the worker flushes the operation pool, opens a new
monitoring stream and starts listening again for new incoming
operations.

It carries a unique local proposal for which it monitors consensus.
Said proposal is set/overridden using "monitor_preattestation_quorum"
and "monitor_attestation_quorum" functions. In practice, such
functions are called when the scheduler perfoms the actions
"Watch_prequorum" and "Watch_quorum" respectively.

Once the consensus operations within the operation pool accumulate
more voting power than the proposal threshold, the operation worker
informs the scheduler by escalating "Prequorum_reached" or
"Quorum_reached" events.


Forge worker
============

Documentation

RPC handling architecture
*************************

   [image: Components of Local and External RPC servers][image]

The Cohttp library is a vendored component responsible for handling
HTTP requests and responses, providing the core functionality for HTTP
communication in both the Local and External RPC servers.

Resto is a library for declaratively defining services, binding them
to given paths, and then either starting an RPC server to serve the
RPCs on these paths or making RPC calls to these services. For
monitoring requests, a stream is created and updates are sent out as
soon as new data is available. Resto is responsible for the following:

* Providing primitives to describe services.

* Assembling the services into directories which are essentially maps
  of paths and methods to services.

* Spinning up a Cohttp server that serves the chosen directory.

* Making requests to services as a client. The client automatically
  builds paths based on parameters to the service, assembles other
  HTTP details, and parses the response.

Additionally, Resto provides features for configuring ACL and for
serving a self-description service - a service that describes all
services of a directory.

In the rest of the document, we will use the term "Resto server" (as
it is called in the "resto" library), but it is worth clarifying that
the Cohttp server is the actual server which listens for events. Resto
server is sitting on top of Cohttp server, providing its handlers for
events. Resto server is not listening for events on its own but it
handles the events. So they split the functionality:

* Cohttp listens and involves event handlers

* Resto provides handlers for events

The RPC middleware module in the External RPC server receives accepted
connections from the Resto server. Depending on the RPC type, it
either handles the underlying RPC request locally or forwards it to
the Local RPC server next to the Tezos node (they share the same PID)
by initiating a connection to it. When forwarding, the RPC middleware
maintains a mapping between the accepted and the initiated
connections. If the client of the initial RPC request dies or closes a
connection, the RPC middleware is notified by Resto and then closes
the corresponding initiated connection to the Local RPC server.


RPC server initialization
=========================

Local RPC server and External RPC server are initialized at Tezos node
start in src/bin_node/node_run_command.ml. Each of them calls "launch"
in src/lib_rpc_http/RPC_server.ml for a Resto server.

An RPC server is declared in src/lib_rpc_http/RPC_server.ml by
"Resto_cohttp_server.Server.Make". At launch, Resto RPC server takes 2
callbacks:

* "callback" - the main callback for accepted connections

* "conn_closed" - the callback notifying about the connection closed
  externally, e.g. by EOF

Resto server launches Cohttp server which also takes "callback" and
"conn_closed" plus one more callback:

* "sleep_fn" - a callback implementing sleep with desired timeout
  which is used to listen for EOF while the main callback is
  unresponsive handling the request.

Resto provides a default "callback" implementation, "resto_callback"
in resto/src/server.ml, which serves Resto directory.

The forwarding logic is handled in file
src/lib_rpc_http/RPC_middleware.ml which defines
"proxy_server_query_forwarder".


Connection management
=====================

The Client, Baker or Indexer can initiate RPC requests. As a result, a
TCP connection is established to the Tezos Node or to the External RPC
server depending on the settings in the Client, Baker or Indexer.

The External RPC server decides based on the RPC endpoint type if the
request has to be handled locally or should be forwarded to the Tezos
node. For performance reasons External RPC server tries to offload
Tezos node.

If the request is forwarded, RPC middleware initiates a new request
that is sent to the Tezos node via an initialized Unix connection,
using Cohttp client. From now on, Cohttp client needs to monitor the
connection health to notify the Client/Baker/Indexer if the connection
died.

This becomes critical as some requests are endless (e.g.
"/monitor/heads/main" or "/chains/main/mempool/monitor_operations").
The Client/Baker/Indexer have no timeout for that and the
responsibility of keeping these connections healthy is on External RPC
server.

The Cohttp server monitors the health of the Unix connection. If EOF
is received, RPC middleware is notified. Then it closes the connection
towards the Client/Baker/Indexer. See Cohttp functions
"handle_request()" in cohttp/cohttp-lwt/src/server.ml and
"wait_eof_or_closed" in cohttp/cohttp-lwt-unix/src/io.ml.

Cohttp uses Conduit library for low-level operation with connections.
The Unix file descriptor of the connection is kept within Cohttp and
is hidden for higher levels of the stack which makes debugging quite
complicated. Cohttp server passes the closing function to the Resto
server allowing it to close the connection if the forwarding operation
failed.

The RPC server provides its callbacks to Cohttp which actually starts
a server, receives requests and involves the provided callbacks. RPC
server launches Cohttp as "RPC_server.launch" in
src/lib_rpc_process/main.ml. The callback "conn_closed" takes a
connection ID. This connection ID is provided by Cohttp when the
connection is created and then Resto stores it in forwarder resources.
Resto creates a new forwarded connection in "make_transform_callback"
and stores the connection ID and a closing function in
"forwarder_resources" in src/lib_rpc_http/RPC_middleware.ml.

The number of callbacks is confusing. So let’s take a closer look to
src/lib_rpc_http/RPC_middleware.ml:

* Cohttp runs a server. So a connection request received from
  Client/Baker/Indexer/ arrives at Cohttp and then passed to Resto via
  "make_transform_callback".

* Resto passes the callback to Cohttp at server start to deal with a
  connection which has to be closed. As it is provided at server
  start, it takes connections storage and a connection ID as an input.
  The provided callback is essentially "forwarding_conn_closed" as it
  handles dead forwarded Unix connection. Therefore, Resto keeps a
  mapping of connections from Client/Baker/Indexer to the shutdown
  function on the corresponding forwarded connections towards the
  Tezos node. So if a client dies, the connection towards Tezos Node
  is also closed.

So when a connection from Client/Baker/Indexer is received, Resto is
involved via  "make_transform_callback". And if that connection dies,
Cohttp invokes "forwarding_conn_closed". If the connection is handled
locally by External RPC server, Resto does nothing. If the request was
forwarded, Resto will call "shutdown" for the Unix connection towards
Tezos node.


The case of streams and chunks
==============================

In Resto, service handlers return an "'a Answer.t" (as defined in
resto/src/resto_directory.ml). The type Answer.t  models the different
possible HTTP status codes. Here’s an excerpt:

   type ('o, 'e) t =
     [ `Ok of 'o (* 200 *)
     | `Not_found of 'e option (* 404 *)
     … ]

The full type definition has more variants for more codes.
Interestingly, it actually has three different variants for the code
200 (OK).

   type ('o, 'e) t =
     [ `Ok of 'o (* 200 *)
     | `OkChunk of 'o (* 200 *)
     | `OkStream of 'o stream (* 200 *)
     …]


Ok
--

"`Ok" is for just returning a value. No big deal. The handler just
returns a value; the server serialises this value for the body of the
HTTP response.

[graph]


OkChunk
-------

"`OkChunk" is for returning values which are very large. When a
handler returns "`OkChunk", it is the handler’s way of instructing the
server to send the response over as multiple chunks. This reduces the
peak CPU and I/O usage of the server which is nicer for the health of
the octez-node process.

The server transmits the chunks via the chunked transfer encoding. In
this HTTP response format, the server sends a series of chunks with
size headers and separators (see link). The server closes the
connection only once all the chunks have been sent.

[graph]

The client recovers the data by deserialising the concatenation of all
the chunks received.

Note that the chunking only really impacts the "application/json"
media type. This is because of historical reasons: the JSON
serialisation is very costly and was blocking the node for noticeable
spans of times. The serialisation to binary could be improved to
benefit from chunking but this requires modifying the de/serialisation
backend.


OkStream
--------

"`OkStream" is for returning not one single value but a sequence of
different values that a variable can have in the "octez-node". E.g.,
the RPC entry point /monitor/heads/<chain_id> sends a sequence of
blocks, one for each time the node changes head on the chain passed as
parameter.

The server transmits each new value as a chunk using the chunk
transfer encoding (see above). Unlike with "`OkChunk" each of the
chunk transmitted for "`OkStream" is a fully formed element of a
stream. The client doesn’t concatenate the chunks together: it decodes
them one after the other.

[graph]

The server never closes the connection (unless the stream ends, which
is not the case for the values monitored in the Tezos stream RPCs).

The payload of the "`OkStream" constructor is a "stream" which is
essentially a function "next" returning a promise for the next
available value:

   type 'a stream = {next : unit -> 'a option Lwt.t; shutdown : unit -> unit}

The resto server (resto/src/server.ml) transforms this "stream" into
an "Lwt_stream" and passes it to cohttp which uses it to request new
values to be transmitted.


Software stacks
===============

There are a number of modules and libraries involved in the RPC system
of Tezos. This section presents them by theme.


Declaring
---------

In the "declaring" part of the stack, the services are merely
described: how many parameters and of what type, what type of value is
returned, etc.

The services are declared in multiple files peppered around the source
tree. E.g., the p2p-related services are declared in
src/lib_p2p_services/, module P2p_services . These declarations are
split from the registration so that both the serving and the querying
stacks can depend on it without introducing unneeded dependencies.

The files declare services by calling into src/lib_rpc/RPC_service.ml
the module Service  which:

* instantiates the functor MakeService  from resto/src/resto.ml with a
  de/serialisation process, and

* specialises the service type and constructors with the error-
  management type "tzresult"

[graph]


Serving
-------

In the "serving" part of the stack, the queries to the services are
answered.

Setting up the serving part is done in two phases.

First, the services are assembled into a directory, by declaring a
directory and registering services into it. In this step, the services
are associated to a handler: the *procedure* in *Remote Procedure
Call*. This happens in multiple files peppered around the source code,
generally in modules which have "directory" in their names. E.g.,
src/lib_shell/chain_directory.ml.

The registration into directory is done by calling into
src/lib_rpc/RPC_directory.ml which instantiates the functor "Make"
from resto/src/resto_directory.ml with a de/serialisation process. It
also adds a layer of error-management.

Second, the assembled directory is used to initialise and start a
server. This is done by passing the directory to the functions in
src/lib_rpc_http/RPC_server.ml.

src/lib_rpc_http/RPC_server.ml instantiates the functor in
resto/src/server.ml and shadows a few functions.

resto/src/server.ml translates the directory passed during
initialisation into a dispatch callback: a function which, given a
request’s method and path will find the corresponding service (by
calling the "lookup" function in resto/src/resto_directory.ml) and
call its handler. The callback also includes a significant amount of
legwork related to de/serialisation of service arguments and outputs,
error management, access-control checks ans so on. It then passes this
callback to "cohttp".

The "cohttp" library handles the parsing and printing of HTTP message,
and passes this information to the callback it was given.

The "cohttp" library delegates the network management (bind, accept,
close, etc.) to the "conduit" library.

[graph]


The lifetime of a request
~~~~~~~~~~~~~~~~~~~~~~~~~

The description of the stack above, can also be displayed as a
sequence diagram which shows the lifetime of a request.

[graph]


Querying
--------

In the "querying" part of the stack, queries to services are formed
and sent, and then the responses are parsed. This is used by the
different executables to communicate with other executables. Most
commonly this is used by "octez-client" to communicate with "octez-
node".

The "octez-client" (or another binary) obtains a description of the
some services from the "src/lib_*_services" files. This is one of the
reason service declaration and registration are separate steps: the
former can be used by clients were the handler wouldn’t necessarily
make sense.

The "octez-client" instantiates a *client context* object from
src/lib_client_base_unix/client_context_unix.ml. This object is passed
around the code under the variable name "cctxt" and it is responsible
for the side-effects (both making RPC calls through the network and
logging and displaying results and accessing local keys from storage
and all).

The aim of this abstraction is essentially dependency injection:
allowing the native client to run on unix using the unix context
client, allowing other clients to run on, say, JavaScript using a
specialised context for this. The ability to run the client on other
backends than native unix application is a discontinued project and
this abstraction has no purpose anymore.

The instantiation of the unix client context in
src/lib_client_base_unix/client_context_unix.ml instantantiates (via
the class inheritance mechanism) a unix rpc context defined in
src/lib_rpc_http/RPC_client_unix.ml.

The code of src/lib_rpc_http/RPC_client_unix.ml is a functor
application of the logic defined in src/lib_rpc_http/RPC_client.ml.
The functor application sets up some retry-on-failure mechanism around
the default cohttp client.

The code in src/lib_rpc_http/RPC_client.ml is a wrapper around
resto/src/client.ml. The wrapper provides:

* glue between the logging mechanism of resto and Octez

* error-management and error-related UI/UX (translating HTTP errors
  into more readable messages)

* media-types management and de/serialisation.

[graph]


Debugging
=========

If you want to learn more about the exchange of RPCs between node and
client you can pass the option "-l" and the client will print all the
calls with their input/output.

A useful tool to manipulate JSON is jq.

To enable the logs for RPC-related components, prepend Tezos scripts
with "TEZOS_LOG="*->debug"" and "COHTTP_DEBUG=true".


See also
========

* RPC How-To

Testing in Octez
****************

Testing is important to ensure the quality of the Octez codebase by
detecting bugs and avoiding regressions. Octez and its components use
a variety of tools and frameworks for testing.

This page groups the documentation helping developers to design,
write, and execute tests on the Octez codebase. It covers different
kinds of tests (Unit, Integration, etc.) and the associated tools and
frameworks. It also includes tutorials for testing more specific parts
such as protocol code.

* Overview of Testing in Octez

  * Testing frameworks

  * Michelson unit tests

  * Executing tests locally

  * Executing tests through the GitLab CI

  * Conventions

* Ppx_expect

  * How to run tests

  * How to add tests

  * Where to put the tests

  * Integration with Lwt

* Tezt: OCaml Tezos Test Framework

  * How to Run Tests

  * How to Write New Integration Tests

  * How to Write New Unit Tests

  * Regression Tests

* Tezt: Long Tests and Performance Regression Test Framework

  * Adding a Long Test

  * Performance Regression Test framework: Time Series, Alerts and
    Graphs

  * Example

  * Providing Large Data

  * Testing Your Benchmarks Locally

* Alcotezt: An Alcotest Compatibility Wrapper for Tezt

  * Running Alcotezts

  * Notable Differences Between Alcotest and Alcotezt

* Making legacy code unit-testable

  * Solutions

* How to Test a Protocol Proposal

  * The "master" Branch

  * Unit Tests and Sandboxed Mode

  * Testing Migration Code

  * Migration Testing

  * Wrap up the Migration Procedure

  * Tips and Tricks

  * Anatomy of Migration Code

Overview of Testing in Octez
****************************

The goal of this document is to give an overview on how testing is
done in Octez, and to help Octez contributors use the test suite and
write tests by pointing them towards the most appropriate testing
framework for their use case. Finally, this guide explains how tests
can be run automatically in the Octez CI and how to measure test
coverage.

The frameworks used in Octez can be categorized along two axes: the
type of component they test, and the type of testing they perform. We
distinguish the following components:

* Node

  * Protocol

    * Michelson interpreter

    * Stitching

* Networked nodes

* Client

* Ledger application

* Baker

Secondly, these components can be tested at different levels of
granularity. Additionally, tests can verify functionality, but also
non-functional properties such as performance (execution time, memory
and disk usage). We distinguish:

Unit testing
   Unit testing tests software units, typically functions, in
   isolation.

Integration testing
   Integration testing tests compositions of smaller units.

System testing
   System testing tests the final binaries directly.

Regression testing
   In general, regression testing aims to detect the re-introduction
   of previously identified bugs. It can also refer to a coarse-
   grained type of testing where the output of a test execution is
   compared to a pre-recorded log of expected output. We here use
   "regression testing" to refer to the second meaning.

Property testing / Fuzzing
   Both property testing and fuzzing test code with automatically
   generated inputs. Property testing is typically used to ensure
   functional correctness, and gives the user more control over
   generated input and the expected output. Fuzzing is typically used
   to search for security weaknesses and often guides input generation
   with the goal of increasing test coverage.

Performance testing
   Testing of non-functional aspects such as run-time, memory and disk
   usage.

Acceptance testing
   Testing of the software in real conditions. It is usually slower,
   more costly and less amenable to automation than integration or
   system testing. It is often the final step in the testing process,
   performed before a release. However, in Octez, acceptance testing
   is decoupled from releases, and currently consists in manually
   running a net of resilience tests on a regular base. These tests
   use various testing frameworks.

By combining the two axes, we obtain the following matrix. Each cell
contains the frameworks appropriate for the corresponding component
and testing type. The frameworks are linked to a sub-section of this
page where the framework is presented in more detail.


Testing frameworks and their applications in Octez. EXP: Ppx_expect, AT: Alcotezt, QC: QCheck, TZ: Tezt, LTF: Long Tests and Performance regression Test Framework, TZT: Michelson unit tests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Component      | Unit           | Property       | Integration    | System         | Regression     | Performance    |
|================|================|================|================|================|================|================|
| Node           | AT             | QC             | AT             | TZ             |                | LTF            |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| -- Protocol    | AT, EXP        | QC             |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| -- --          | AT, TZT        |                |                | TZ             | TZ             |                |
| Michelson      |                |                |                |                |                |                |
| interpreter    |                |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Client         | EXP            | QC             |                | TZ             |                | LTF            |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Networked      | --             |                | TZ             |                |                |                |
| nodes          |                |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Attester       |                |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Baker          |                |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+


Testing frameworks
==================


Ppx_expect
----------

Ppx_expect is a framework for writing tests for OCaml code generating
textual output, similar to Cram which is used for testing command line
applications.

Typical use cases:
   * Unit tests and integration tests leveraging existing printers
     instead of checking properties.

   * Tests that change on purpose over time. One can easily make tests
     pass again with a single dune invocation "dune runtest --auto-
     promote"

Example tests:
   * Unit tests for src/lib_micheline, in
     src/lib_micheline/test/test_parser.ml. To execute them locally,
     run "dune runtest src/lib_micheline/test" in the Octez root.

References:
   * Section in Octez Developer Documentation on Ppx_expect

   * Ppx_expect README.

   * Dune documentation about inline expectation tests.

   * Ppx_inline_test README.


QCheck
------

QCheck is a library for property-based testing in OCaml.

Typical use cases:
   * Verifying input-output invariants for functions with randomized
     inputs.

Example test:
   * QCheck is used in src/lib_base/test/test_time.ml to test the
     Tezos_base.Time  module. For instance, subtracting and then
     adding a random amount of seconds to a random time should give
     back the original time: this tests that "add" and "diff" are
     consistent (and the inverse of each other). To run this test, you
     need to run "dune exec src/lib_base/test/test_time.exe".

References:
   * QCheck README

   * QCheck module documentation


Tezt
----

Tezt is a system testing framework for Octez. Tezt is capable of
regression testing. Tezt focuses on tests that run in the CI, although
it is also used for some manual tests (see the tezt/manual_tests
folder). Its main strengths are summarized in its section in the Tezos
Developer Documentation. Conceptually Tezt consists of a generic
framework for writing tests interacting with external processes, and a
set of Octez-specific modules for interacting with the Octez binaries:
the client, baker, etc.

Typical use cases:
   * Testing the commands of "octez-client". This allows to test the
     full chain: from client, to node RPC to the implementation of the
     economic protocol.

   * Test networks of nodes, with daemons.

   * Detecting unintended changes in the output of a component, using
     regression tests.

Example tests:
   * Testing baking (in tezt/tests/basic.ml)

   * Testing double baking and double attestation scenarios (in
     tezt/tests/double_bake.ml).

   * Testing absence of regressions in encodings (in
     tezt/tests/encoding.ml)

References:
   * Section in Tezos Developer Documentation on Tezt

   * General API documentation

   * Octez-specific API documentation


Long Tests and Performance regression Test Framework
----------------------------------------------------

A library called "tezt-performance-regression", which is a wrapper
around Tezt, is used for tests that are too long to run in the CI and
for performance testing. Those tests are run on dedicated machines and
can send data points to an InfluxDB instance to produce graphs using
Grafana and/or detect performance regressions. See Tezt: Long Tests
and Performance Regression Test Framework.


Alcotezt
--------

Alcotezt is an Alcotest-compatible wrapper for Tezt. With it, unit
tests originally written for Alcotest (now deprecated) can be executed
using Tezt instead. We are currently in the progress of migrating all
Alcotests to Tezt, and we are using Alcotezt as a stop gap towards
this goal. For new unit and integration testing suites, prefer using
Tezt directly.

Typical use cases:
   * Conversion of pre-existing Alcotests to Tezt

Example tests:
   * Unit tests for tezos-clic . To execute them locally, run "dune
     build @src/lib_clic/runtest".

   * Unit tests for octez-version. To execute them locally, run "dune
     build @src/lib_version/runtest".

See Alcotezt: An Alcotest Compatibility Wrapper for Tezt for more
information on how to convert tests to Alcotezt, and how to execute
them.


Alcotest (usage deprecated)
---------------------------

Alcotest is a library for unit and integration testing in OCaml.
Alcotest was historically the primary tool in Octez for unit and
integration testing of OCaml code. However, we are currently migrating
to Tezt for unit, integration and system testing. To ease migration
from Alcotest to Tezt, the Alcotezt wrapper was introduced. Alcotezt
is briefly described above and in more detail in Alcotezt: An Alcotest
Compatibility Wrapper for Tezt. The below Alcotest description applies
to tests that have not yet been converted to Alcotezt.

Typical use cases:
   * Verifying simple input-output specifications for functions with a
     hard-coded set of input-output pairs.

   * OCaml integration tests.

References:
   * Alcotest README.


Michelson unit tests
====================

The TZT syntax extension defines a format for unit tests for Michelson
snippets. The tests written in it are executable:

* through "octez-client run unit test" for the Michelson interpreter
  written in OCaml;

* through "cargo run --bin tzt_runner" for the Michelson interpreter
  written in Rust.

Example use cases:

* Verifying the functional (input--output) behavior of snippets of
  Michelson instructions.

* Conformance testing for Michelson interpreters.

A conformance test suite for Michelson interpreters written using the
TZT extension is maintained and is used for non-regression testing.


Executing tests locally
=======================

Whereas executing the tests through the CI, as described below, is the
standard and most convenient way of running the full test suite, they
can also be executed locally.

All tests can be run with "make test" in the project root. However,
this can take some time, and some tests are resource-intensive or
require additional configuration. Alternatively, one can run subsets
of tests identified by a specialized target "test-*". For instance,
"make test-unit" runs the alcotest tests and should be quite fast. See
the project "Makefile" for the full list of testing targets.


Measuring test coverage
-----------------------

We measure test coverage with bisect_ppx. This tool is used to see
which lines in the code source are actually executed when running one
or several tests. Importantly, it tells us which parts of the code
aren't tested.

We describe here how "bisect_ppx" can be used locally. See below for
usage with CI.

To install "bisect_ppx", run the following command from the root of
the project directory:

   make build-dev-deps

The OCaml code should be instrumented in order to generate coverage
data. This is done by prepending

   ./scripts/with_coverage.sh

to build and test commands run from the root of the project directory.
For example,

   ./scripts/with_coverage.sh make
   ./scripts/with_coverage.sh make test-coverage

Generate the HTML report from the coverage files using

   make coverage-report

The generated report is available in "_coverage_report/index.html". It
shows for each file, which lines have been executed at least once, by
at least one of the tests.

Clean up coverage data (output and report) with:

   make coverage-clean

The helper "./scripts/with_coverage.sh" can also be used outside make
commands (e.g. with "dune", "poetry"). For example,

   ./scripts/with_coverage.sh dune runtest src/lib_shell/
   ./scripts/with_coverage.sh dune exec tezt/tests/main.exe -f basic.ml

However you launch the tests, the same commands are used to get the
report (e.g. "make coverage-report").


Enabling instrumentation for new libraries and executables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To ensure that all libraries and executables are included in the
coverage report, the following field should be added to all "library"
and "executable(s)" stanzas in all "dune" files, e.g.:

   (library
     (name ...)
     (instrumentation
       (backend bisect_ppx)))

The manifest will add this stanza automatically unless
"~bisect_ppx:false" is specified.

This enables the conditional instrumentation of the compilation unit
through the "./scripts/with_coverage.sh" helper as described above.

Exempted from this rule are the "dune" files that belong to tests,
developer utilities and old protocols. In particular:

* benchmarks, e.g. "src/lib_shell/bench/dune"

* bindings, e.g. "src/lib_sapling/bindings/dune"

* test frameworks, e.g. "tezt/lib/dune"

* test packages, e.g. "src/*/test/dune"

* old protocols, e.g. "src/proto_00*/*/*dune"

* helper utilities, e.g.:

  * "src/openapi/dune", (executable name "openapi")

  * "src/lib_client_base/gen/dune" (executable name "bip39_generator")

  * "src/lib_protocol_compiler/dune" (executable name "replace")

  * "src/proto_alpha/lib_parameters/dune" (executable name "gen")

  * "src/proto_011_PtHangz2/lib_parameters/dune" (executable name
    "gen")

  * "src/lib_protocol_environment/ppinclude/dune" (executable name
    "ppinclude")

  * "src/lib_store/legacy_store/dune" (executable name
    "legacy_store_builder")


Truncated coverage files
~~~~~~~~~~~~~~~~~~~~~~~~

Occasionally, tests write corrupted coverage data. If you run into the
issue, you will see a message like:

   $ make coverage-report
   Error: coverage file '_coverage_output/foobar.coverage' is truncated

   make: *** [Makefile:105: coverage-report] Error 1

or

   $ make coverage-report
   bisect-ppx-report: internal error, uncaught exception:
                      Bisect_common.Invalid_file("_coverage_output/foobar.coverage", "unexpected end of file while reading magic number")

   make: *** [Makefile:112: coverage-report] Error 125

Typically, this indicates that a instrumented binary that was launched
by the test was terminated abruptly before it had time to finish
writing coverage data. You can just rerun the test, and most likely,
it won't produce a corrupted trace on the second run. However, this is
not a long-term solution. Below, we present some hints on how to debug
this issue:

Binaries instrumented with "bisect_ppx" attach an "at_exit" handler
that writes collected coverage data at termination of the tested
process execution.

To ensure that this process is not disrupted, one should follow these
guidelines:

For system test frameworks
   System test frameworks like Tezt: OCaml Tezos Test Framework, run
   binaries e.g. "octez-client" and "octez-node". Typically, they do
   so with calls to "exec" so the resulting process does not inherit
   the signal handlers from the parent process (the test framework).
   When writing tests in these frameworks, the author must ensure that
   the processes launched are instrumented and that they do proper
   signal handling: they should catch "SIGTERM" and call exit in their
   "SIGTERM" handler. This should already be the case for the binaries
   in octez.  They should also ensure that the framework terminates
   the processes with "SIGTERM".

For integration test frameworks
   Some integration test frameworks, such as the "lib_p2p" test
   framework, spawn subprocesses through "fork". These subprocesses
   inherit the signal handler of the parent process (the test
   framework). Such frameworks should themselves be instrumented and
   themselves do proper signal handling as described above. Bisect
   provides a convenience for doing so, through the "--sigterm" flag:

      (preprocess (pps bisect_ppx --bisect-sigterm))

   When enabled, it ensures that the instrumented process writes
   coverage data successfully on receiving "SIGTERM". For an
   illustration of how to implement this, and the problem it resolves,
   see !3792.

General process handling
   If possible, do not leave processes "hanging" in tests. Instead,
   use e.g. "wait" or "Lwt.bind" to ensure that processes get a chance
   to terminate before the full test terminates. For an illustration
   of how to implement this, and the problem it resolves, see !3691.


Comparing reports
~~~~~~~~~~~~~~~~~

At times, it is convenient to compare two coverage reports. This can
be used to ensure that coverage does not regress when e.g. migrating a
test from one framework to another. We provide a fork of bisect_ppx
with this functionality. It adds the command "compare-html" to
"bisect-ppx-report".

Running:

   bisect-ppx-report compare-html -x x.coverage -y y.coverage

will create an HTML report comparing the coverage of in "x.coverage"
and "y.coverage". A limitation of this tool is that it assumes that
only coverage has changed -- not the underlying source files.


Executing tests through the GitLab CI
=====================================

To execute the tests on a merge request, the "trigger" job needs to be
manually executed. To trigger it, go to the GitLab merge request page.
Click the grey gear in the leftmost circle of the pipeline, then click
the "Play" button. If necessary, Marge-bot will trigger the merge
request pipeline before merging it. For instances, to see the latest
runs of the CI, visit this page. You can click the status of a
pipeline for more details.

The results of the test suite on terminated pipelines is presented on
the details of the merge request page corresponding to the pipeline's
branch (if any). For more information, see the GitLab documentation on
Unit test reports.

By default, the "test" of the CI runs the tests as a set of
independent jobs that cluster the tests with a varying grain. This
strikes a balance between exploiting GitLab runner parallelism while
limiting the number of jobs per pipeline. The grain used varies
slightly for different types of tests:

Tezt integration and regression tests
   By default, Tezt tests are grouped in several batch jobs named
   "tezt" and are executed in merge request pipelines. According to
   the tags attached to them, the tests can be handled differently.
   The description of these tags can be found in src/lib_test/tag.mli.

The OCaml package tests (Alcotest & QCheck)
   The OCaml package tests are regrouped in a set of jobs per protocol
   package, in addition to one job regrouping tests for remaining
   packages.


Adding tests to the CI
----------------------

When adding a new test that should be run in the CI (which should be
the case for most automatic tests), you need to make sure that it is
properly configured. The procedure for doing this depends on the type
of test you've added:

Tezt integration and regression tests
   New Tezt tests will be included automatically in the CI.

The OCaml package tests (Alcotest & QCheck)
   Any tests located in a folder named "src/**/test/" that are
   executed through "dune runtest" will be picked up automatically by
   the CI. No intervention is necessary.

Other
   For other types of tests, you need to manually modify the CI
   configuration. Please refer to the CI configuration's README
   (ci/README.md) for more information.


Test coverage in merge requests
-------------------------------

Build and tests are instrumented with "bisect_ppx" in the CI for each
merge request on Octez. To measure test coverage in the CI, it
launches the job "unified_job" in stage "test_coverage" which
generates the coverage report. They are stored as an HTML report that
can be downloaded or browsed from the CI page upon completion of the
job (see the Artifacts produced by the MR pipeline in the GitLab UI).

The summary report gives the merge request an overall test coverage
percentage (displayed just next to the MR pipeline in the GitLab UI).

Additionally, using "bisect-ppx-report cobertura", we produce and
upload a Cobertura artifact activating the test coverage visualization
in GitLab:

[image]


Known issues
~~~~~~~~~~~~

1. After termination of the "unified_coverage" job, test coverage
   visualization can take some time to load. Once the coverage report
   is processed by GitLab, you will have to refresh the "Changes" tab
   of the MR to see the results.

2. Instrumenting the code with both "ppx_inline_test" and "bisect_ppx"
   can produce misplaced locations. This is caused by a bug in
   "ppx_inline_test" version "0.14.1" that will be in their next
   release.

3. Occasionally, tests write corrupted coverage data. In this case,
   the job "unified_coverage" will fail. We've done our best to ensure
   this happens rarely. If it happens, you can either try:

   * Re-running the full pipeline.

   * Reading the log of the job "unified_coverage". It'll direct you
     to the test job that produced the corrupted coverage file.  You
     can then retry the test job, and once finished, retry the
     "unified_coverage" job.

   * Finally, if the problem persists, adding the label "ci--no-
     coverage" will disable the "unified_coverage" job. You can add
     this as a last resort to merge the MR.


Test coverage on master
-----------------------

In addition to computing test coverage on merge request, we also
associate coverage information to each merge commit on the master
branch. Instead of running the test suite on master, which would be
wasteful, we fetch it from the most recent merge request.

The job "unified_coverage" detects when it runs on "master". In this
case, it reads the history of the branch to find the latest pipeline
on the most recently merged branch. It then fetches the coverage
result from there, and also retrieves the artifacts which contains the
HTML coverage report. GitLab also produces a graph of the coverage
ratio over time.


Conventions
===========

Besides implementing tests, it is necessary to comment test files as
much as possible to keep a maintainable project for future
contributors. As part of this effort, we require that contributors
follow this convention:

1. For each unit test module, add a header that explains the overall
   goal of the tests in the file (i.e., tested component and nature of
   the tests). Such header must follow this template, and be added
   after license:

   (** Testing
       -------
       Component:    (component to test, e.g. Shell, Micheline)
       Invocation:   (command to invoke tests)
       Dependencies: (e.g., helper files, optional so this line can be removed)
       Subject:      (brief description of the test goals)
   *)

2. For each test in the unit test module, the function name shall
   start with "test_" and one must add a small doc comment that
   explains what the test actually asserts (2-4 lines are enough).
   These lines should appear at the beginning of each test unit
   function that is called by e.g. "Alcotest_lwt.test_case". For
   instance,

   (** Transfer to an unactivated account and then activate it. *)
   let test_transfer_to_unactivated_then_activate () =
   ...

3. Each file name must be prefixed by "test_" to preserve a uniform
   directory structure.

4. OCaml comments must be valid "ocamldoc" special comments.

5. If a test takes 2 minutes or more on the CI, it should be tagged as
   "slow" (see Tezt integration and regression tests). Note that tests
   with tag "slow" do not run automatically in the CI of merge
   requests.

Ppx_expect
**********

Ppx_expect is a framework for writing tests for OCaml code printing
textual output.

There is pretty comprehensive documentation about "Inline expectation
tests" and the "ppx_expect" tool for writing them:
   * Dune documentation about inline expectation tests.

   * Ppx_expect README.

Here, we will just cover enough to get started using them inside the
Octez codebase.


How to run tests
================

If you want to run tests and see whether they succeed, build the
standard "runtest" alias:

   dune runtest src/lib_stdlib
   # or dune build @@src/lib_stdlib/runtest

The result would look like this:

   File "src/lib_stdlib/bloomer.ml", line 97, characters 0-817: random_read_writes (0.006 sec)
   File "src/lib_stdlib/bloomer.ml", line 123, characters 0-1339: peek and poke work with bits = [1 .. Sys.int_size - 7] (0.012 sec)
   File "src/lib_stdlib/bloomer.ml", line 163, characters 0-739: sequential_read_writes (0.021 sec)
   File "src/lib_stdlib/bloomer.ml", line 182, characters 0-705: read_over_write (0.001 sec)
   File "src/lib_stdlib/bloomer.ml", line 282, characters 0-835: consistent_add_mem_countdown (0.275 sec)
   File "src/lib_stdlib/bloomer.ml", line 309, characters 0-1531: consistent_add_countdown_count (2.079 sec)
   File "src/lib_stdlib/bloomer.ml", line 456, characters 4-137 (11.966 sec)
   File "src/lib_stdlib/bloomer.ml", line 461, characters 4-1695: <<match Sys.getenv_opt "BLOOMER_TEST_GNUPLOT_PA[...]>> (0.000 sec)

A test can fail because it generates an output different from what is
expected or because it raises an exception. Failures are reported as a
diff in both cases.

Different output:

   File "src/proto_alpha/lib_client/limit.ml", line 1, characters 0-0:
   ------ src/proto_alpha/lib_client/limit.ml
   ++++++ src/proto_alpha/lib_client/limit.ml.corrected
   File "src/proto_alpha/lib_client/limit.ml", line 57, characters 0-1:
    |      if eq x y then Result.return_some x
    |      else error_with "Limit.join: error (%s)" where
    |
    |let%expect_test "join" =
    |  let pp_print_err fmt = function
    |    | Result.Error _ -> Format.pp_print_string fmt "error"
    |    | Ok x ->
    |        Format.(
    |          pp_print_option
    |            ~none:(fun fmt () -> pp_print_string fmt "None")
    |            pp_print_bool)
    |          fmt
    |          x
    |  in
    |  let print x = Format.fprintf Format.std_formatter "%a" pp_print_err x in
    |  print (join ~where:__LOC__ Bool.equal (Some true) (Some true)) ;
   -|  [%expect {| false |}] ;
   +|  [%expect {| true |}] ;
    |  print (join ~where:__LOC__ Bool.equal None None) ;
    |  [%expect {| None |}] ;
    |  print (join ~where:__LOC__ Bool.equal None (Some true)) ;
    |  [%expect {| true |}] ;
    |  print (join ~where:__LOC__ Bool.equal (Some true) None) ;
    |  [%expect {| true |}] ;
    |  print (join ~where:__LOC__ Bool.equal (Some true) (Some false)) ;
    |  [%expect {| error |}]
    |
    |let get ~when_unknown = function
    |  | None -> error_with "Limit.get: %s" when_unknown
    |  | Some x -> ok x
    |
    |let%expect_test "get" =
    |  let pp_print_err fmt = function
    |    | Result.Error _ -> Format.fprintf fmt "error"

Exception raised:

   File "src/lib_stdlib/bloomer.ml", line 1, characters 0-0:
   ------ src/lib_stdlib/bloomer.ml
   ++++++ src/lib_stdlib/bloomer.ml.corrected
   File "src/lib_stdlib/bloomer.ml", line 309, characters 0-1:
    |    let bloomer = create ~hash ~index_bits ~hashes ~countdown_bits in
    |    let rec init n acc =
    |      if n = 0 then acc
    |      else
    |        let x = Random.int (1 lsl 29) in
    |        add bloomer x ;
    |        assert (mem bloomer x) ;
    |        init (n - 1) (x :: acc)
    |    in
    |    let all = init 1000 [] in
    |    for _ = 0 to (1 lsl countdown_bits) - 2 do
    |      List.iter (fun x -> assert (mem bloomer x)) all ;
    |      countdown bloomer
    |    done ;
    |    List.iter (fun x -> assert (not (mem bloomer x))) all
    |  done
   +|[@@expect.uncaught_exn {|
   +|  (* CR expect_test_collector: This test expectation appears to contain a backtrace.
   +|     This is strongly discouraged as backtraces are fragile.
   +|     Please change this test to not include a backtrace. *)
   +|
   +|  "Assert_failure src/lib_stdlib/bloomer.ml:287:4"
   +|  Raised at Tezos_stdlib__Bloomer.(fun) in file "src/lib_stdlib/bloomer.ml", line 287, characters 4-18
   +|  Called from Expect_test_collector.Make.Instance_io.exec in file "collector/expect_test_collector.ml", line 262, characters 12-19 |}]


How to add tests
================

Ppx_expect is based on "ppx_inline_test" machinery. That is, it
collects tests automatically for you.

To add a new test, make sure you have the following in your library
stanza in the dune file:

   (inline_tests)
   (preprocess (pps ppx_expect))

In the manifest, just add the following argument to the library
containing expect tests:

   ~inline_tests:ppx_expect

Adding new tests is then just a matter of adding "let%expect_test" at
top level:

   let%expect_test "optional name" =
     print_endline "hello world";
     [%expect {||}]

Running test with the example above will fail and show you a diff
between your source and the corrected one:

    |let%expect_test "optional name" =
    |  print_endline "hello world";
   -|  [%expect {||}]
   +|  [%expect {| hello world |}]

If you agree with the diff, just ask dune to promote the source and
you're done:

   dune runtest --auto-promote


Where to put the tests
======================

Expect tests can live next to the implementation or in a different
library dedicated to tests (e.g. if you don't want to polute your
source/binary or if your want to only test the exposed API).


Integration with Lwt
====================

Ppx_expect can be used in combination with Lwt, see the package
description. This integration has not been tested on the Octez
codebase yet, hence some work will be needed to a have specific
support for the codebase.

Tezt: OCaml Tezos Test Framework
********************************

Tezt is a generic test framework written in OCaml. It focuses on
integration tests that launch external processes. In particular, Tezt
can be used to orchestrate tests involving Tezos nodes and clients.

To facilitate the use with Tezos, Tezt has been complemented by a
specific library called Tezt-Tezos.

Tezt is pronounced /tɛzti/ (think "tezty", as in *Tez* are tas*ty*).

The main benefits of using Tezt-Tezos are:

* tests are written in the same language as Octez itself (OCaml),
  which reduces context switch for developers;

* tests do not actively poll the node as they passively listen to node
  events instead, which results in faster and more reliable tests;

* in verbose mode, logs show the interleaved output of all external
  processes, while the tests are running;

* it should be easy to use and extend.

Therefore, Tezt and Tezt-Tezos have been leveraged to build a test
suite for Octez. See tezt/README.md for details on its implementation.

The rest of this page explains how to run the test suite and how to
add new tests.


How to Run Tests
================

It is recommended to set up an alias:

   alias tezt='dune exec tezt/tests/main.exe --'

Try it with "--help" to get the list of command-line options as
follows:

   tezt --help

Just run "tezt" with no options to run all tests. However, most often
you only want to run a subset of the tests. You can get the list of
tests with "--list":

   tezt --list

For instance, to run only tests from files "basic.ml" and
"RPC_test.ml" that have tag "alpha" but not tag "regression", with log
level "info", run:

   tezt -f basic.ml -f RPC_test.ml alpha /regression -i

You can also run tests in parallel, although in that case it is
recommended to use the default log level to avoid interleaving logs.
For instance, the following command runs tests declared in file
"encoding.ml" with up to 8 tests in parallel:

   tezt -f encoding.ml -j 8


How to Write New Integration Tests
==================================

The best way to get started is to have a look at existing tests in
directory "tezt/tests" of the Octez repository.

Most integration tests are part of the same executable
"tezt/tests/main.exe". The source of this module is
tezt/tests/main.ml. This executable runs all tests, but you can
restrict the set of tests to run by specifying tags on the command
line, or even the titles of the tests to run (with the "--test"
option).

All tests do not have to be implemented in tezt/tests/main.ml though.
You can of course add more modules and have them be linked into
"main.exe" together. The best way to do this is to write your tests as
functions and call them from the main module.

For instance, let's create a new basic test in a new file named
tezt/tests/basic.ml:

   let check_node_initialization (history_mode : Node.history_mode) :
       Protocol.t list -> unit =
     Protocol.register_test
       ~__FILE__
       ~title:
         (sf "node initialization (%s mode)" (Node.show_history_mode history_mode))
       ~tags:["basic"; "node"; Node.show_history_mode history_mode]
     @@ fun protocol ->
     let metrics_addr = Constant.default_host in
     let metrics_port = Port.fresh () in
     let* node =
       Node.init [History_mode history_mode] ~metrics_addr ~metrics_port
     in
     let* client = Client.init ~endpoint:(Node node) () in
     let* () = Client.activate_protocol ~protocol client in
     Log.info "Activated protocol." ;
     let* () = repeat 10 (fun () -> Client.bake_for_and_wait client) in
     Log.info "Baked 10 blocks." ;
     let* level = Node.wait_for_level node 11 in
     Log.info "Level is now %d." level ;
     let* identity = Node.wait_for_identity node in
     if identity = "" then Test.fail "identity is empty" ;
     Log.info "Identity is not empty." ;
     let addr = Format.sprintf "http://%s:%d/metrics" metrics_addr metrics_port in
     let* metrics =
       Process.spawn ~log_output:false "curl" ["-s"; addr]
       |> Process.check_and_read_stdout
     in
     if metrics = "" then Test.fail "Unable to read metrics" else return ()

   (* This test checks that tests with tag [ci_disabled] do not run in the CI.
      It happened once when [Test.run] was accidentally called twice.
      To make sure that the test is selected by Manifezt often enough,
      it is tagged with some [~uses] that are often selected. *)
   let () =
     Test.register
       ~__FILE__
       ~title:"meta: failing test with tag ci_disabled"
       ~tags:[Tag.ci_disabled]
       ~uses:
         [
           Constant.octez_node;
           Constant.octez_client;
           Constant.octez_dal_node;
           Constant.octez_dac_node;
           Constant.octez_smart_rollup_node;
           Constant.octez_evm_node;
           Constant.WASM.evm_kernel;
         ]
     @@ fun () ->
     Test.fail
       "This test is not supposed to be run because it has tag ci_disabled. If it \
        does run (in the CI), it means that the tag ci_disabled is not working \
        properly anymore."

   let register ~protocols =
     check_node_initialization Archive protocols ;
     check_node_initialization (Full None) protocols ;
     check_node_initialization (Rolling None) protocols

Then, let's launch the test from tezt/tests/main.ml by calling:

   Basic.register ~protocols:[Alpha] ;
   Test.run () (* This call should already be there. *)

Finally, let's try it with:

   dune exec tezt/tests/main.exe -- basic --info

The "--info" flag allows you to see the "Log.info" messages. It is the
same as "-i". Here is what you should see:

   $ dune exec tezt/tests/main.exe -- basic --info
   [13:45:36.666] Starting test: Alpha: node initialization (archive mode)
   [13:45:37.525] Activated protocol.
   [13:45:38.215] Baked 10 blocks.
   [13:45:38.215] Level is now 11.
   [13:45:38.215] Identity is not empty.
   [13:45:38.231] [SUCCESS] (1/3) Alpha: node initialization (archive mode)
   [13:45:38.231] Starting test: Alpha: node initialization (full mode)
   [13:45:39.113] Activated protocol.
   [13:45:39.813] Baked 10 blocks.
   [13:45:39.813] Level is now 11.
   [13:45:39.813] Identity is not empty.
   [13:45:39.828] [SUCCESS] (2/3) Alpha: node initialization (full mode)
   [13:45:39.828] Starting test: Alpha: node initialization (rolling mode)
   [13:45:40.708] Activated protocol.
   [13:45:41.407] Baked 10 blocks.
   [13:45:41.407] Level is now 11.
   [13:45:41.407] Identity is not empty.
   [13:45:41.422] [SUCCESS] (3/3) Alpha: node initialization (rolling mode)


Detailed Walk through the Basic Test
------------------------------------

Let's review what our basic test in the previous section does.

* First, note that the Tezt library and its Base module are opened
  automatically by Dune based on its configuration file. The Base
  module contains useful functions such as "let*" (which is
  "Lwt.bind") or "sf" (a short-hand for "Printf.sprintf").

* Then, we define a function "check_node_initialization" which
  registers one test. It is parameterized by the history mode.
  "Protocol.register_test" is partially applied here;
  "check_node_initialization" is also implicitly parameterized by the
  list of protocols to run the test on.

* The function "Protocol.register_test" registers a test. It is a
  wrapper over "Test.register". This wrapper is preferred when the
  test is parameterized by a list of protocols. The "~__FILE__"
  argument gives the source filename so that one can select this file
  with the "--file" argument, to only run tests declared in this file.
  Each test has a title which is used in logs and on the command-line
  with the "--test" option (which allows to run a particular test from
  its title). Each test also has a list of tags. We gave our test the
  tag "basic" in particular. No other test has this tag, so it is easy
  to run all of the tests of our new "Basic" module, and only them, by
  adding "basic" on the command-line.

* The function "Protocol.register_test" takes a function as an
  argument. This function contains the implementation of the test.

* First, we initialize a node with "Node.init". This creates a node
  and runs the node command "identity generate", then "config init"
  and finally "run". It then waits until the node is ready, and
  returns the node. Note that you do not have to call "Node.init". For
  instance, if you want to test the behavior of the node without an
  identity, you can call "Node.create", followed by "Node.config_init"
  and "Node.run".

* Then, we initialize a client with "Client.init". We give it a node,
  which is the node that the client will connect to by default. Note
  that we can still use this client to perform operations on other
  nodes if we want to, it's just convenient to specify it once and for
  all.

* Then, we activate the protocol with the "activate protocol" command
  of the client. By default, this activates the protocol given as
  argument, with some default parameters and using the default
  activator key (defined in the "Constant" module). This activator key
  was added to the client by "Client.init". You can override all of
  this. For instance, if you don't want the client to know the default
  activator key, use "Client.create" instead of "Client.init" (you can
  use "Client.import_secret_key" to import another activator key, for
  instance). Or, if you want to change the fitness or the parameter
  file, you can use the "?fitness" and "?parameter_file" optional
  arguments of "Client.activate_protocol".

* Then, we log a message using "Log.info". This message is not visible
  with the default verbosity, but you can see it by running "main.exe"
  with the "--info" option (or "--verbose").

* Then, we repeat "Client.bake_for" 10 times, to bake 10 blocks.

* Then, we wait for the level of the node to be at least 11 (the
  activation block plus the 10 blocks that we baked) using
  "Node.wait_for_level". If you call this function and the level is
  already 11 or greater, "Node.wait_for_level" returns immediately.
  (Note: "Node.wait_for_level" makes the test fail if the node stops
  before reaching level 11.)

* Finally, we read the identity of the node using
  "Node.wait_for_identity" which returns as soon as the node reads the
  identity file. In fact, this was probably done much sooner, but Tezt
  stores the identity in case you try to query it later, just like the
  level. (Note: "Node.wait_for_identity" makes the test fail if the
  node stops before reading the identity file.)

* We check that the identity is not empty, and if it is we call
  "Test.fail". This causes the test to terminate immediately with an
  error. Note that it is not the only cause of failure for this test:
  we already saw that "Node.wait_for_level" and
  "Node.wait_for_identity" can cause a test failure, and if anything
  goes wrong (failing to initialize the node or the client, failing to
  activate the protocol...) "Test.fail" is called automatically as
  well.

* After the test succeeds or fails, "Test.run" cleans up everything.
  It terminates all running processes by sending "SIGTERM". It waits
  for them with "waitpid" to avoid zombie processes. And it removes
  all temporary files, in particular the data directory of the node
  and the base directory of the client.

* We run this test three times, once per history mode: "archive",
  "full" and "rolling". Note that we added the history mode as a tag
  to "Test.register", so if we want to run only the test for history
  mode "full", for instance, we can simply run "dune exec
  tezt/tests/main.exe -- basic full". You can see our list of basic
  tests and their tags with "dune exec tezt/tests/main.exe -- basic
  --list".


How to Write New Unit Tests
===========================

For Tezt, the main difference between a unit test and an integration
test is that a unit test does not use the "Process" module. Indeed, an
integration test tests an executable by running it, while a unit test
links with a library to test its functions directly.

Since the scope of unit tests is much smaller, they are usually
registered in executables that are dedicated to the library being
tested. For instance, tests for "src/lib_base" could be put in an
executable named "src/lib_base/tezt/main.exe". Such executables are
usually faster to compile and run than "tezt/tests/main.exe" since
their dependency cone is limited.

To add unit tests to a library which does not yet have a Tezt
executable, create files such as "src/lib_base/tezt/example.ml" and
"src/lib_base/tezt/other.ml" and use "Test.register" to register tests
in those files, at toplevel. Here is a minimal example:

   let () =
     Test.register
       ~__FILE__
       ~title:"test title here"
       ~tags:["test"; "tags"; "here"]
     @@ fun () ->
     (* your test here *)
     unit

Then, declare those files in "manifest/", for instance in
manifest/product_octez.ml:

   let _octez_base_tezts =
     tezt
       ["example"; "other"]
       ~path:"src/lib_base/tezt"
       ~opam:"tezos-base"
       ~deps:[octez_base]

This causes the manifest to generate executable
"src/lib_base/tezt/main.exe" for you. This executable calls
"Test.run". It also declares a Dune alias "runtest" so that you can
run your tests with either of the following commands:

   dune build @src/lib_base/runtest
   dune exec src/lib_base/tezt/main.exe

Note that your tests will actually also be available in
"tezt/tests/main.exe". This executable gathers all tests so that the
CI can auto-balance them.


JavaScript
----------

If you want to be able to run your test with Node.js, declare them in
the manifest with "~js_compatible:true" and with "JS" in "~modes". For
instance:

   let _octez_base_tezts =
     tezt
       ["example"; "other"]
       ~path:"src/lib_base/tezt"
       ~opam:"tezos-base"
       ~js_compatible:true
       ~modes:[Native; JS]
       ~deps:[octez_base]

Running "dune build" will generate not only a native executable
("src/lib_base/tezt/main.exe") but also a JavaScript file
("src/lib_base/tezt/main_js.bc.js") that you can run with Node.js:

   nodejs _build/default/src/lib_base/tezt/main_js.bc.js

Note however that tests that use "Tezt.Process", "Tezt.Temp" or
"Tezt.Runner" cannot use the JavaScript backend. In other words,
integration tests cannot be run with Node.js, only unit tests.


Regression Tests
================

Regression tests are used to prevent unintended changes to existing
functionality by ensuring that the software behaves the same way as it
did before introduced changes.

Regression tests capture commands and output of commands executed
during a test. An output of some regression test is stored in the
repository and is expected to match exactly with the captured output
on subsequent runs. An added advantage of this is that when a change
in behaviour is intentional, its effect is made visible by the change
in test's output.

To run all the regression tests, use the "regression" tag:

   dune exec tezt/tests/main.exe regression

When the change in behaviour is intentional or when a new regression
test is introduced, the output of regression test must be
(re-)generated. This can be done with the "--reset-regressions"
option, e.g.:

   dune exec tezt/tests/main.exe regression -- --reset-regressions

Regression tests are registered with "Regression.register" instead of
"Test.register". Use "Regression.capture" or "Regression.hooks" to
capture output that you want to be stable. Regression tests can be
used both in unit tests and integration tests.

Tezt: Long Tests and Performance Regression Test Framework
**********************************************************

Tezt can also be used for long tests. Here are the differences with
regular Tezt tests:

* long tests are not run in the CI but on dedicated machines with
  stable, predictable performance, and with no global timeout like in
  the CI;

* long tests are declared in tezt/long_tests/main.ml instead of
  tezt/tests/main.ml;

* long tests are registered with "Long_test.register" instead of
  "Test.register";

* long tests have easy access to a Performance Regression Test
  framework which provides these features:

  * Persist measurement samples (in an InfluxDB database) such as how
    long it takes to do something. These samples will be used to
    prevent regressions of performance.

  * Provide easy access to a Grafana instance that can be updated to
    create graphs displaying the samples sent to InfluxDB;

  * Call Slack webhooks to send alerts when a performance regression
    has been detected.


Adding a Long Test
==================

To add a long test, you need to register it with "Long_test.register"
(respectively "Long_test.register_with_protocol") instead of
"Test.register" (respectively "Protocol.register_test"). The test must
be registered in tezt/long_tests/main.ml instead of
tezt/tests/main.ml. "Long_test.register" is very similar to
"Test.register". The main difference is that you have to declare a
timeout (that should be significantly overestimated, see the
documentation of the "timeout" type in
tezt/lib_performance_regression/long_test.mli) and that
"Long_test.register" will handle InfluxDB, Grafana and Slack alerts.
Just like regular Tezt tests, your test should be implemented in the
same file with other thematically-related tests, with only a single
line in tezt/long_tests/main.ml.

If you declare a test in tezt/long_tests/main.ml and merge it into
"master", it will be automatically run on dedicated machines that
regularly pull the latest version of "master" and run long tests. If
your test takes a significant time to run (days), you should however
ask that a new dedicated machine is created to run your test. Please
ask on the "#tests" Slack channel of "tezos-dev" before merging.


Performance Regression Test framework: Time Series, Alerts and Graphs
=====================================================================

Long tests can use functions from the "Long_test" module
(tezt/lib_performance_regression/long_test.mli) to send data points to
InfluxDB, which is a time-series database. A time-series database
stores values annotated with a timestamp. In the particular case of
InfluxDB, data points are composed of:

* a timestamp (usually the time the measure was taken);

* a measurement name (such as "client load time");

* optionally, some tags that can be used to refine the measurement
  name (such as a tag "mode" equal to "client", "mockup" or "proxy");

* fields, which are composed of field names and field values (such as
  a field "duration" equal to how long it took for the client to
  load).

The "Long_test" module allows you to send any data point using the
"add_data_point" function. They will be sent at the end of the test
execution. You can query statistics about data points from previous
tests using function "get_previous_stats" to compare them with your
new data points and send an alert on Slack with function "alert" if
you detect that something is wrong. Note that function
"check_regression", which is essentially a combination of
"get_previous_stats" and "alert", can be used in most cases.  And if
all you want to do is measure the time it takes for a chunk of code to
run though, you can use the "time" function instead, which does
everything for you: measure time, send data points, compare them with
previous runs, and send an alert if the difference is too large.

The "Long_test" module also provides function
"update_grafana_dashboard" called in tezt/long_tests/main.ml with a
specification to create/overwrite a dashboard in Nomadic Labs' Grafana
(restricted access). Default is named "Long Tezts" but you can add
additional dashboards using the "Long_test.update_grafana_dashboard"
function. To add a dashboard for your tests, define it next to your
test (in the same file), and declare it in the call to
"update_grafana_dashboard" in tezt/long_tests/main.ml.

As always in Tezt, the above functions try to provide flexibility. The
"time" function in particular is parameterized by settings like the
number of times the test should be repeated, how many previous data
points should be fetched from InfluxDB when comparing with new
measurements, how much of a difference to tolerate before alerting,
etc. "time" itself being a combination of other lower-level functions
that are also provided and which you can combine to fit your needs.
And of course you can contribute to improve them.

Automated long tezts logs are available in Nomadic Labs' public S3
bucket browser. In case of InfluxDB issues, you can inspect metrics in
the InfluxDB dashboard (restricted access). InfluxDB itself is private
and direct access via CLI is restricted to administrators.


Example
=======

See tezt/long_tests/prt_client.ml, which is a very simple test that
measures how long it takes for "octez-client" to load.  It uses
"Long_test.time_lwt" to measure how long it takes for "Client.version"
to run and to emit alerts if this time is significantly higher than
usual. It also defines a graph of this time. This test and its graph
are registered in tezt/long_tests/main.ml.


Providing Large Data
====================

Your test may require data that is too large to commit in
"tezos/tezos". For example, a benchmark in which measurement is
dependent on some block's context would need to load the same data
directory on each execution.

There is an Amazon S3 bucket where you can upload your data which will
be made available for your test. Data will be synchronized with the
server your tests will be running on.

For security reasons, this storage has its access limited to
authorized people. If you want to upload data, please contact Jérémie
Goldberg (@jgonlabs) or anyone with admin access on the Tezos AWS
account to allow you to do so.

Please note that the S3 storage root folder is mounted in "/s3data/".
E.g. if your file is under "/myfolder/myfile" in the Amazon bucket,
your tests will find it under "/s3data/myfolder/myfile".


Testing Your Benchmarks Locally
===============================

When developing a benchmark depending on the Performance Regression
Test framework, it can be useful to test it using development backends
so that your tests does not impact production ones.

The Performance Regression Test framework now contains a setup that
can automatically provision and configure InfluxDB and Grafana
instances using Docker Compose.


Provisioning InfluxDB and Grafana
---------------------------------

The following steps assume that you already installed "Docker" as well
as "docker-compose" and correctly configured it. For more information
on this subject, please refer to:

* https://docs.docker.com/engine/install/#desktop

* https://docs.docker.com/compose/

From the root folder of "tezos" run the following commands from a
terminal to start the Docker containers in background:

   docker-compose -f tezt/lib_performance_regression/local-sandbox/docker-compose.yml up -d

After containers have been started, you can test that InfluxDB is
properly started and that the "performance_regression" database has
been automatically created:

   curl --get http://localhost:8086/query\?pretty\=true --data-urlencode "q=show databases"

The command should display the following:

   {
       "results": [
           {
               "statement_id": 0,
               "series": [
                   {
                       "name": "databases",
                       "columns": [
                           "name"
                       ],
                       "values": [
                           [
                               "performance_regression"
                           ],
                           [
                               "_internal"
                           ]
                       ]
                   }
               ]
           }
       ]
   }

Also, you should be able to connect to the Grafana web UI by
connecting to http://localhost:3000 in your browser. By going to the
"Datasources" menu in the webapp configuration
(http://localhost:3000/datasources), you can see that an InfluxDB
datasource has been pre-configured and is connected to the
"performance_regression" database.

Note that as security does not really matter for tests, it has been
disable for ease. This is why you can connect to the Grafana web app
with full privileges or send requests to InfluxDB without having to
authenticate.

To stop the container, simply run:

   docker-compose -f tezt/lib_performance_regression/local-sandbox/docker-compose.yml down

The created containers use persistent Docker volumes, so that data
stored in the database and created dashboards will be preserved
between container runs. To permanently remove these docker volumes,
run the command "docker volume rm local-sandbox_influxdb local-
sandbox_grafana".


Configuring and Running Tezt Long Tests
---------------------------------------

For more information about how to use the configuration file, please
refer to the Long test module API .

A predefined configuration has already been shipped in
tezt/lib_performance_regression/local-sandbox/tezt_config.json. It
allows to use the InfluxDB and Grafana instances set up by the Docker
compose file presented in the previous section.

All content related to Grafana and InfluxDB has already been set and
can be used as is.

Other aspects of the configuration (for example the "test_data_path")
should be updated to match the needs of your local machine.

To run Tezt long tests, run the following command:

   TEZT_CONFIG=tezt/lib_performance_regression/local-sandbox/tezt_config.json dune exec tezt/long_tests/main.exe

Alcotezt: An Alcotest Compatibility Wrapper for Tezt
****************************************************

Alcotezt is a compatibility wrapper for tests originally written in
Alcotest (now deprecated), enabling running them with Tezt.

Alcotezts (i.e. tests wrapped with Alcotezt) are programmed against an
API that is compatible with Alcotest. But, under the hood, they are
registered with Tezt. This means that they are executed in the same
way as all Tezt tests are (see Running Alcotezts below for more
details).

Alcotezts are typically declared through the "tezt" manifest function.
This will:

* Create a *test library* that registers the tests in the modules
  given as argument to "tezt".

* Create a *test executable* "main.exe" in the test folder. The test
  executable links with the test library and consists of a single call
  to Tezt's Test.run.

* Finally, link the test library with Tezt's main entrypoint at
  "tezt/tests/main.exe" so that it registers all Alcotezts.


Running Alcotezts
=================

For a given folder "$TEST_DIR", the Alcotezts contained therein can be
invoked in three ways:

1. By executing the "main.exe" runner binary generated by Manifest.
   Through "dune", this is done with "dune exec $TEST_DIR/main.exe".
   This will execute the tests in "$TEST_DIR" (but not in its
   subdirectories).

2. Through Dune, by building the "runtest" alias in the test target
   folder. That is, executing "dune build @$TEST_DIR/runtest". This
   will execute the tests in "$TEST_DIR" and its subdirectories.

3. By executing the full Tezt test suite, via the main entrypoint at
   "tezt/tests/main.exe". Through Dune, this is done with "dune exec
   tezt/tests/main.exe".


Execution through the test executable
-------------------------------------

The advantage of passing through the Manifest-generated runner is that
it only depends on the tests themselves and the tested modules (and
their recursive dependencies), and so is faster to compile.
Furthermore, you can give arguments to Tezt to control execution, e.g.
passing "--list" to list the tests or "--verbose" to see debug output.

For example, to run the tests of src/lib_clic/test:

   dune exec src/lib_clic/test/main.exe

This will execute the Alcotezts in src/lib_clic/test. To pass
arguments to Tezt, append "-- <TEZT_ARGS>" to the above command:

   dune exec src/lib_clic/test/main.exe -- --list


Execution through the Dune "runtest" alias
------------------------------------------

The "runtest" alias can be used to execute all tests, including
Alcotezts, in a folder and its recursive sub-folders. For example, to
run all tests of protocol Alpha, run:

   dune build @src/proto_alpha/runtest

On the other hand, there is no convenient way to pass arguments to the
underlying tests with this method.

The Alcotezts can be dynamically disabled in the "runtest" alias by
setting the environment variable "RUNTEZTALIAS" to "false". For
instance, to run all the unit tests in protocol Alpha exception the
Alcotezts, you can run:

   RUNTEZTALIAS=false dune build @src/proto_alpha/runtest


Execution through the Tezt main entrypoint
------------------------------------------

Finally, Alcotezts can be executed through the Tezt main entrypoint at
"tezt/tests/main.exe". All Alcotezts are linked with this binary, so
that all system, integration and unit tests registered with either
Tezt or Alcotezt can be executed by invoking:

   dune exec tezt/tests/main.exe

This is used to run all tests in the CI in a unified and load-balanced
manner. This is less convenient for local use though, as there is
currently no way of executing the subset of tests that corresponds to
a specific package, file or folder. A forthcoming release of Tezt will
ameliorate this.


Notable Differences Between Alcotest and Alcotezt
=================================================


Test naming
-----------

First of all, some definitions:

* In Alcotest, a "test suite" is a sequence of "tests" that are each
  composed of a sequence of "test cases".

* In Tezt, there are only "tests", no "test suites".

With Alcotezt, each Alcotest "test case" becomes a Tezt "test". The
Alcotest suite, test and case name are used to form the title of the
Tezt test with following format: "SUITE_NAME: TEST_NAME
(TEST_CASE_NAME)".

Given an Alcotest consisting of a suite "Suite A" that contains a test
"Test a" that itself contains the test cases "Test case a1" and "Test
case a2". It also contains a test "Test b" with the test case "Test
case b1":

   let () =
     Alcotest.run
       "Suite A"
       [
         ( "Test a",
           [
             ("Test case a1", `Quick, fun () -> ...);
             ("Test case a2", `Quick, fun () -> ...);
           ] );
         ( "Test b",
           [
             ("Test case b1", `Quick, fun () -> ...);
           ] );
       ]

Running it in with Alcotest produces:

   Testing `Suite A'.
   This run has ID `3F91T9S2'.

     [OK]          Test a          0   Test case a1.
     [OK]          Test a          1   Test case a2.
     [OK]          Test b          0   Test case b1.

   Full test results in `/home/tezos/_build/_tests/Suite A'.
   Test Successful in 0.000s. 2 tests run.

And running it with Alcotezt produces:

   [17:07:42.289] [SUCCESS] (1/2) Suite A: Test a (Test case a1)
   [17:07:42.289] [SUCCESS] (2/3) Suite A: Test a (Test case a2)
   [17:07:42.290] [SUCCESS] (3/3) Suite A: Test b (Test case b1)


Test Output
-----------

Alcotezt redirects "Format"'s output to Tezt's Log.debug. To see the
debug output of an Alcotezt, pass the "--verbose" flag to Tezt. See
the section Running Alcotezts above for more information on how to
pass flags to Tezt when executing Alcotezts.

There is no way to redirect the output of "Printf". Consequently, the
output of Alcotezts that call this module directly cannot be hidden.


Integration with the "runtest" aliases
--------------------------------------

Alcotezts are registered as a dependency on the "runtest" alias.
However, they are not executed through this alias in the CI. Instead,
they run through the Tezt main runner to enable load balancing.

Making legacy code unit-testable
********************************

This guide describes how to make legacy code testable and how to test
it. The following techniques do not require any API change: they
concern testing each single module as a **unit** without touching
modules depending on it. These techniques are applicable to **legacy**
code: you do not need to have an exact knowledge of every
implementation detail of a module to test it.

The goals are:

* to gain confidence in a module as a developer (or even part of test-
  driven development)

* to document it: tests are examples

* to have a safety net as a maintainer (non-regression testing), e.g.,
  when: - adding/changing features - performing various forms of
  refactoring (changing the API or not)

In the rest of this page, "testable" means "unit-testable", for
conciseness.

Here "legacy" is a loose term, encompassing:

* existing code written by other developers, not necessarily
  containing a right amount of built-in unit tests

* code that is only tested by heavy/slow/high level tests, like Tezt -
  this is often a symptom of code that is hard to unit-test

* tightly-coupled code (e.g. module "A" directly depends on functions
  of modules "B", "C" and "D", thus unit-testing "A" requires setting
  everything up for "B", "C" and "D" too, which not only makes unit
  testing harder, but is also complex and brittle)

* code that calls system functions (e.g. "Unix.chmod", which requires
  modifying actual files on the filesystem, also making unit testing
  harder) or more generally any kind of side-effecting function


Solutions
=========


Testing a module and all its dependencies
-----------------------------------------

Sometimes, legacy modules depending on other modules may be "unit-
tested" without modifying them at all.

This approach is technically not "unit" testing, as the
module/function is not tested in isolation from its dependencies.
However as the intention is close to unit testing, we include it here.

Pros:

* No change at all in business code (not even an "Internal_for_tests"
  module)

Cons:

* Code not tested in isolation

* Tests are harder to write and maintain

Works best when:

* Code and its dependencies are pure, or "pure enough" (e.g. code may
  contain logging and other unharmful/unimportant side effects)

* Dependency tree is not too deep (e.g. if A depends on B which
  depends on C which depends on D, then it becomes hard to
  write/read/maintain unit tests)


Using function records
----------------------

This approach decouples the business code from its dependency
functions, either by storing dependency functions in the state - if
any - or by taking them as additional parameters in each module
function.

Note that this does not decouple types, only functions! If your module
depends on a dependency type that can only be built by having side-
effects, then you are forced to have those side effects too in your
tests.

As an example, consider the following "Counter" module (signature and
implementation):

   module Counter : sig
     type t
     val create_from_file : unit -> t
     val increment_and_save_to_file : t -> t
   end = struct
     type t = {
       counter : int;
     }

     let create_from_file () =
       let counter = int_of_string (Files.read "/tmp/counter.txt") in
       {counter}

     let increment_and_save_to_file t =
       let t = {counter = t.counter + 1} in
       Files.write "/tmp/counter.txt" (string_of_int t.counter);
       t
   end

"Counter" is hardly unit-testable because of the calls to "Files.read"
and "Files.write" which have the side effect of reading from/writing
into "/tmp/counter.txt" file. Thus we extract "Files.read" and
"Files.write" into an argument that can be changed with a dumb
("mock") function in tests.


Store dependencies in state
~~~~~~~~~~~~~~~~~~~~~~~~~~~

One way to achieve this - when your module has a state, usually a type
"t" - is to store all dependency functions in a field, here
"dependencies":

   module Counter : sig
     type t
     val create_from_file : unit -> t
     val increment_and_save_to_file : t -> t

     module Internal_for_tests : sig
       type dependencies = {
         files_read : string -> string;
         files_write : string -> string -> unit;
       }
       val create_from_file : dependencies -> unit -> t
     end
   end = struct
     type dependencies = {
       files_read : string -> string;
       files_write : string -> string -> unit;
     }

     type t = {
       counter : int;
       dependencies : dependencies;
     }

     let create_from_file_internal dependencies () =
       let counter = int_of_string (dependencies.files_read "/tmp/counter.txt") in
       {counter; dependencies}

     let create_from_file = create_from_file_internal {files_read = Files.read; files_write = Files.write}

     let increment_and_save_to_file t =
       let t = {t with counter = t.counter + 1} in
       t.dependencies.files_write "/tmp/counter.txt" (string_of_int t.counter);
       t

     module Internal_for_tests = struct
       type nonrec dependencies = dependencies = {
         files_read : string -> string;
         files_write : string -> string -> unit;
       }

       let create_from_file = create_from_file_internal
     end
   end

Note that the direct calls to "Files.read" and "Files.write" were
replaced with indirect calls to "dependencies.files_read" and field
"t.dependencies.files_write":

* They are set to "Files.[read|write]" in the business constructor
  "Counter.create_from_file"

* They are changed at will in the test constructor
  "Counter.Internal_for_tests.create_from_file"

Also note that while the API was extended with test artifacts under
the "Internal_for_tests" sub-module, the public API is otherwise
unchanged, thus keeping this refactoring local - you do not need to
change any call sites!

Now we can test this module without any side effect:

   let test () =
     let counter_value_written = ref "" in
     let fake_files_read file_name = "41" in
     let fake_files_write file_name text =
       counter_value_written := text
     in
     let counter = Counter.Internal_for_tests.create_from_file {files_read = fake_files_read; files_write = fake_files_write} () in
     let _ = Counter.increment_and_save_to_file counter in
     Alcotest.(check string) "counter value was incremented in file" !counter_value_written "42"


Taking dependencies in function argument
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An alternative solution, more verbose but not requiring any "state"
value available in each function, is to take the dependencies directly
as an additional function argument:

   module Counter : sig
     type t
     val create_from_file : unit -> t
     val increment_and_save_to_file : t -> t

     module Internal_for_tests : sig
       type dependencies = {
         files_read : string -> string;
         files_write : string -> string -> unit;
       }

       val create_from_file : dependencies -> unit -> t
       val increment_and_save_to_file : dependencies -> t -> t
     end
   end = struct
     type dependencies = {
       files_read : string -> string;
       files_write : string -> string -> unit;
     }

     type t = {
       counter : int;
     }

     let business_dependencies = {
       files_read = Files.read;
       files_write = Files.write;
     }

     let create_from_file_internal dependencies counter =
       let counter = int_of_string (dependencies.files_read "/tmp/counter.txt") in
       {counter}

     let create_from_file = create_from_file_internal business_dependencies

     let increment_and_save_to_file_internal dependencies t =
       let t = {counter = t.counter + 1} in
       dependencies.files_write "/tmp/counter.txt" (string_of_int t.counter);
       t

     let increment_and_save_to_file t = increment_and_save_to_file_internal business_dependencies t

     module Internal_for_tests = struct
       type nonrec dependencies = dependencies = {
         files_read : string -> string;
         files_write : string -> string -> unit;
       }

       let create_from_file = create_from_file_internal
       let increment_and_save_to_file = increment_and_save_to_file_internal
     end
   end

Note that the direct calls to "Files.read" and "Files.write" were
replaced with indirect calls to arguments "dependencies.files_read"
and "dependencies.files_write":

* They are set to "Files.[read|write]" in each business function
  ("create_from_file" and "increment_and_save_to_file")

* They are changed at will in the test function "Counter.Internal_for
  _tests.[create_from_file|increment_and_save_to_file]"

As in the previous solution, notice that the public API has not
changed - save for additional APIs in "Internal_for_tests".

Now we can test this module without any side effect:

   let test () =
     let counter_value_written = ref "" in
     let fake_files_read file_name = "41" in
     let fake_files_write file_name text =
       counter_value_written := text
     in
     let mock_dependencies = Counter.Internal_for_tests.{
       files_read = fake_files_read;
       files_write = fake_files_write;
     } in
     let counter = Counter.Internal_for_tests.create_from_file mock_dependencies () in
     let _ = Counter.Internal_for_tests.increment_and_save_to_file mock_dependencies counter in
     Alcotest.(check string) "counter value was incremented in file" !counter_value_written "42"


Pros and Cons for Function records
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Works best when:

* Dependency types are not too hard to build

Pros:

* No side-effecting function is called (they are replaced with mocks)

* Enables validating the arguments passed to mock functions (e.g.
  "counter_value_written") have the right value

* Independent of the dependency depth (for functions): if "A" calls
  "B.f" which calls "C.g", your mock of "B.f" will never call "C.g"

Cons:

* All dependency types remain, so if it is difficult/side-effectful to
  create those values, testing remains difficult/not so unitary

* Adds a bit of boilerplate in "Internal_for_tests" module

* Adds a bit of indirection, by introducing indirect calls to
  dependency functions. The associated performance overhead should be
  negligible in most practical cases. There also is a slight decrease
  in code readability, but documenting this unit-testability pattern
  should avoid many headaches.

To choose between the field and the argument:

* If your module already has a kind of "state" (usually a type "t"),
  then add a "dependencies" field

* Else add a "dependencies" argument - but this requires duplicating
  each function, which ends up being very verbose if you have several
  functions

* If your "state" value (usually a value of type "t") is passed to a
  polymorphic function like "=" or "compare" (which throw on function
  fields, and are famous for being an anti-pattern), and it is not
  possible for you to fix this anti-pattern, then either switch to
  function arguments, or wrap in an object.


Using functors
--------------

This approach decouples the business code from its dependency modules.
Note that unlike the Function records solution, this decouples both
dependency functions **and abstract types**!

Consider the following code: it is similar to the previous "Counter"
example but this time, the "Files" dependency module (which could be
another module, a third party library, or even the "Stdlib") also has
an abstract type "t":

   (* The dependency *)
   module Files : sig
     type t
     val openf : string -> t
     val write : t -> string -> unit
     val close : t -> unit
     (* Many other functions and types *)
   end = struct (* omitted implementation *) end

   module Counter : sig
     type t
     val create : int -> t
     val increment_and_save_to_file : t -> t
   end = struct
     type t = {
       counter : int;
     }

     let create counter = {counter}

     let increment_and_save_to_file t =
       let t = {counter = t.counter + 1} in
       let file = Files.openf "/tmp/counter.txt" in
       Files.write file (string_of_int t.counter);
       Files.close file;
       t
   end

The technique is to transform "Counter" into a functor that takes a
module looking like "Files" in argument - but which can now be changed
in tests!

   module Counter : sig
     module type S = sig
       type t
       val create : int -> t
       val increment_and_save_to_file : t -> t
     end

     include S

     module Internal_for_tests : sig
       module type FILES = sig
         type t
         val openf : string -> t
         val write : t -> string -> unit
         val close : t -> unit
       end
       module Make (Files : FILES) : S
     end
   end = struct
     module type S = sig
       type t
       val create : int -> t
       val increment_and_save_to_file : t -> t
     end

     module type FILES = sig
       type t
       val openf : string -> t
       val write : t -> string -> unit
       val close : t -> unit
     end

     module Make (Files : FILES) = struct
       type t = {
         counter : int;
       }

       let create counter = {counter}

       let increment_and_save_to_file t =
         let t = {counter = t.counter + 1} in
         let file = Files.openf "/tmp/counter.txt" in
         Files.write file (string_of_int t.counter);
         Files.close file;
         t
     end

     (* Do not be mistaken: here [Files] refers to the real, business [Files] module! *)
     include Make (Files)

     module Internal_for_tests = struct
       module type FILES = FILES

       module Make = Make
     end
   end

As you can see, this is significantly more verbose!

However, now we can freely change/mock not only the dependency
functions "Files.[openf|close|write]", but also the implementation of
type "Files.t"!

   let test () =
     let written_content = ref "" in
     let module Counter = Counter.Internal_for_tests.Make (struct
       type t = unit
       let openf _ = ()
       let close _ = ()
       let write _ content = written_content := content
     end) in
     let counter = Counter.create 41 in
     let _ = Counter.increment_and_save_to_file counter in
     Alcotest.(check string) "counter value was incremented in file" !written_content "42"

While the real "Files.t" type probably contained a file descriptor,
our mock module has no side effect outside of the test!

Note on verbosity: some things are duplicated because of OCaml MLI
syntax, e.g. module type declaration. This can be partially mitigated
by using the _intf trick but this in turn induces a bit more
complexity, use with caution.

Works best when:

* You need to decouple (abstract) types, not only functions. For
  example, because building values of those types adds too much
  complexity, or requires side-effects.

* There are linear dependencies in modules ("A" depends on "B" which
  depends on "C", but "A" does not depend on "C")

Pros:

* Everything but exposed and private dependency types are mocked

* Enables validating the arguments passed to mock functions (e.g.
  "counter_value_written") have the right value

* Independent of the dependency depth (for functions): if "A" calls
  "B" which calls "C", your mock of "B" will never call "C" nor refer
  to its abstract types

Cons:

* Verbosity

* Additional complexity (functors, module types)

* Does not scale well in more complex dependencies ("A" depends on "B"
  and "C" types, and "B" also depends on "C" types) as it induces a
  lot of destructive substitutions and module noise to convince the
  typechecker that "C.t" in "A" is the same as "C.t" in "B"

* Does not work for exposed and private (exposed in read-only)
  dependency types

How to Test a Protocol Proposal
*******************************

In this tutorial we show how to test a newly developed protocol and,
in particular, how to test the migration process from its predecessor.
We also provide a short guide on how to read the migration code.

We start by describing the branch that contains the protocol proposal
that is under development.


The "master" Branch
===================

The current proposal is developed in the branch "master", which
contains both the new protocol and its migration code from the current
active protocol. The current protocol proposal under development is
referred to as the *Alpha protocol*.

Since the migration code is only used once, it is a good practice to
keep them clearly separated by marking them with the tag
"Proto/Migration:". Note that a first step when developing a new
protocol is to revert the migration code from the previous protocol.

We next describe how to run unit tests and how to activate the Alpha
protocol in sandboxed node.


Unit Tests and Sandboxed Mode
=============================

The first step for tweaking and testing the current proposal is to
checkout the latest code and run unit tests:

   $ git checkout master
   $ git pull
   $ make
   $ make test

We can run a node and a client in sandboxed mode by invoking:

   $ ./src/bin_node/octez-sandboxed-node.sh 1 --connections 0 &
   $ eval $(./src/bin_client/octez-init-sandboxed-client.sh 1)

By default, the sandbox starts from the "genesis" block at level 0,
and the sandbox's active protocol is the "Genesis protocol". Once the
sandbox is started, the Alpha protocol can be activated by invoking
the command:

   $ octez-activate-alpha

This command inserts a new block after which the active protocol is
the Alpha protocol. From this point on, making the chain progress is
straightforward because the sandbox contains accounts "bootstrap1" to
"bootstrap5" with implicit credentials that allow them to bake blocks
by using the usual RPCs in the shell (see Sandboxed mode):

   $ octez-client bake for --minimal-timestamp


Adding New Protocol Tests in OCaml
----------------------------------

Be sure you first read the introduction on the testing ecosystem for
Tezos. In addition to system tests written with Tezt, unit tests and
integration tests for the protocol can be found in
src/proto_alpha/lib_protocol/test. It is strongly recommended to write
unit tests and integration tests in addition to the system tests. To
test a new component, create a new file in this directory and add the
module in "main.ml". To print the errors of the "Error" monad
correctly, alcotests must be wrapped into the function "tztest"
defined in the module "Test" defined at the same level.

Some helpers are available in the module "Tezos_alpha_test_helpers"
defined in the subdirectory "helpers". For instance, it contains
context, operation and block fixtures that can be used in tests
requiring these components.


Testing Migration Code
======================

The remainder of the tutorial is organized as follows. Section
Migration testing provides detailed indications on how to test a
migration, and Section Wrap up the migration procedure summarizes
these indications by collecting all the steps needed to test the
migration. To conclude, Section Tips and tricks indicates how to use
the shell to inspect the context, and Section Anatomy of migration
code contains a primer on how to read and write migration code.


Migration Testing
=================

The most delicate part of migrating to a new protocol is to produce a
new context from the context of its predecessor. The migration code
takes care of producing the new context, which often involves
converting large data structures. Therefore, it is important to bench
the migration running time and the size of the context produced. For
these reasons it is imperative to test the migration on a real context
imported from Mainnet, bench it, and manually inspect the content of
the storage. We refer to this procedure as "migration on a context
imported from Mainnet".

However, sometimes we may want to perform some preliminary tests and
run the migration on an empty context that we populate manually. We
can do so by running a node in sandboxed mode, and by activating the
predecessor of the Alpha protocol on the genesis block. We refer to
this procedure as "migration on the sandbox".

This section describes a *migration procedure* in which the developer
is in charge of setting up the migration environment and of manually
baking the blocks that would eventually trigger the migration. For
convenience, we have batched parts of this manual migration procedure
by providing scripts that encompass some of its steps.

We will illustrate the migration procedure through an example where
the version of the Alpha protocol to which we migrate is "012".


Checkout Latest Code and Tweak Migration
----------------------------------------

We start by checking out the latest code for the Alpha protocol:

   $ git checkout master
   $ git pull

Now we could tweak our migration by adding any desired feature. For
instance, we could log the point at which migration takes place by
editing the file "src/proto_alpha/lib_protocol/init_storage.ml". This
can be done by modifying the match expression of the function
"prepare_first_block" in the said file to include the following lines:

   | Hangzhou_011 ->
       Logging.(log Notice "STITCHING!") ;

After making sure that our "master" branch contains all the migration
code that we want to test, we need to commit the changes locally:

   $ git commit -am 'My awesome feature'

The next section summarizes how to prepare the migration once we have
tweaked the Alpha protocol.


Prepare the Migration
---------------------

Preparing the migration comprises the following steps:

1. snapshot the Alpha protocol, if so wished,

2. link the snapshot Alpha protocol in the build system, if we wished
   to snapshot the Alpha protocol,

3. set "user-activated upgrade" that will trigger the migration at a
   given level,

4. compile the project,

5. import a context from Mainnet, if so wished, and

6. create a "yes-wallet" that stores fake baker signatures, if we wish
   to import the context from Mainnet.

Steps 1--7 can be batched by invoking the script
"scripts/prepare_migration_test.sh" in the way we explain below.
Alternatively, each of the steps above can be performed individually
by invoking the corresponding commands/scripts that we detail in the
rest of the section.

Before preparing the migration, we need to choose on which context the
migration will run. When on the sandbox, the steps 4, 5 and 7 above
are omitted because the sandbox starts on an empty context, and the
sandbox automatically contains accounts with implicit credentials that
will allow us to bake blocks and make the chain progress.

When on a context imported from Mainnet, we will use a *snapshot file*
(do not mistake "snapshot a protocol", like in step 1 above, with
"snapshot a node", which results in a snapshot file like in here) that
contains the real status of a Mainnet's node at a particular moment in
time. Such a snapshot file can be downloaded from several sites on the
internet (see Snapshots). Such websites store daily snapshot files
from both Mainnet and Testnet, in both "full" and "rolling" mode (see
History modes). For the purposes of testing the migration, a snapshot
file in "rolling" mode is enough. It is important to use a snapshot
file that is recent enough as to contain the predecessor of the Alpha
protocol. It is also important to note down the level at which the
snapshot file was taken, which determines at which level we want to
trigger the migration. The snapshots websites conveniently indicate
the date and the level (the block) at which each snapshot file was
taken.

In our example we will use a snapshot file "~/snapshot-
mainnet.rolling" which was taken at level "1617344".

The next subsections explain each of the individual steps 1--6.


1. Snapshot the Alpha Protocol
------------------------------

Snapshotting the Alpha protocol is an optional procedure whose
objective is to convert the Alpha protocol to a format that could be
injected into Mainnet, which is done by performing the following three
steps:

* specify the version and name of the current protocol in
  "raw_context.ml",

* compute the protocol's hash in "TEZOS_PROTOCOL", and

* replace names and protocol hashes in various places in the code
  base.

If so wished, these three steps can be performed by the script
"scripts/snapshot_alpha.sh", which receives a parameter with the name
of the Alpha protocol. This name parameter follows the convention
"<tag_starting_with_version_letter>_<version_number>". A valid name
for the Alpha protocol in our example could be "d_012" (we might also
have used "dummy_12"). We can snapshot the protocol by invoking the
following:

   $ ./scripts/snapshot_alpha.sh d_012

The script creates a new directory "src/proto_012_<short_hash>" where
"<short_hash>" is a short hash that coincides with the first eight
characters of the hash computed by the script and written in the file
"TEZOS_PROTOCOL".

If the Alpha protocol has been snapshot, proceed to Section 2. Link
the snapshot Alpha protocol in the build system below, which details
how to link the snapshot code in the build system. Otherwise proceed
directly to Section 3. Set user-activated upgrade.


2. Link the Snapshot Alpha Protocol in the Build System
-------------------------------------------------------

If the Alpha protocol was snapshot into
"src/proto_<version_number>_<short_hash>", this protocol can now be
linked in the build system. Note that linking the protocol is not
mandatory; we can always inject a protocol that compiles in a node and
link it dynamically on the fly. However, linking the protocol in the
client enables the use of the commands that may be present in the
folder "src/proto_<version_number>_<short_hash>/lib_client", if any.
Otherwise, only the commands accessible through the RPCs would be
available. Except for some specific scenarios in which the commands
accessible through the RPCs are enough, it is always convenient to
link the snapshot protocol in the build system. In our example, this
can be done by invoking:

   $ ./scripts/link_protocol.sh src/proto_012_*

Alternatively, you can snapshot Alpha and link it with one single
script: "snapshot_alpha_and_link.sh". This replaces steps 1 and 2.
This script effectively runs "snapshot_alpha.sh" and
"link_protocol.sh" for you. In particular it means you do not have to
find the short hash of the protocol yourself to pass it to
"link_protocol.sh". To run it, pass the protocol version number and
name as follows:

   $ ./scripts/snapshot_alpha_and_link.sh 012 d


3. Set User-Activated Upgrade
-----------------------------

The currently active protocol supports self-amending through the
voting procedure of Tezos. However, such procedure needs to go through
several voting periods that involve several quorums of bakers, and we
would rather test our migration in a less involved way. Besides the
amendments driven by the protocol, Tezos also supports *user-
activated* upgrades, which are triggered by the shell. The user-
activated upgrades allow the user to specify the level at which the
next protocol will be adopted, which can be used to perform emergency
bug fixes, but which is also useful to test migrations.

Depending on whether we test the migration on the sandbox or on a
realistic context imported from Mainnet, we would like to set the
user-activated upgrades respectively at a small level (some blocks
after the genesis block at level "0") or at a high level (some blocks
after the status imported from Mainnet, which contains several
hundreds of thousands of blocks). By convention, when setting a user-
activated upgrade the scripts would consider that the migration is on
the sandbox if the level is less or equal than "28082", and on a real
context imported from Mainnet otherwise, and the scripts would behave
differently.

If we are testing the migration on the sandbox, the user-activated
upgrade allows us to activate the predecessor of the Alpha protocol by
using an activation command after the sandbox starts, and to
automatically trigger the activation of the Alpha protocol when the
sandbox reaches a given level. Using this mechanism, we can start the
sandbox, activate the predecessor of the Alpha protocol, populate the
empty context at will by using the shell of the predecessor protocol,
and then have the migration triggered automatically at the desired
level. The script "scripts/user_activated_upgrade.sh" receives the
path of the protocol to which we would like to upgrade, and the
desired level.

In our example above, where the Alpha protocol was snapshot into
"src/proto_012_<short_hash>", we can set the user-activated upgrade
such that the migration is triggered at level three by invoking:

   $ ./scripts/user_activated_upgrade.sh src/proto_012_* 3

If we had opted for not snapshotting the Alpha protocol, we could pass
the path "src/proto_alpha" as the parameter of the command above.

Now we consider the case when testing the migration on a context
imported from the snapshot file. In that case, we should recall the
level at which the snapshot file was taken from the beginning of
Section Prepare the migration. In our example, this level is
"1617344". The user-activated upgrade allows us to start the node
imported from Mainnet, which would have the predecessor of the Alpha
protocol already active if the snapshot is recent enough, and then
have the migration triggered automatically at the desired level, which
has to be strictly bigger than the level at which the snapshot file
was taken.

In our example, where we the Alpha protocol was snapshot into
"src/proto_012_<short_hash>", we can set the user-activated upgrade
such that the migration is triggered three levels after the level
"1617344" at which the snapshot was taken by invoking:

   $ ./scripts/user_activated_upgrade.sh src/proto_012_* 1617347

As before, if we had opted for not snapshotting the Alpha protocol, we
could pass the path "src/proto_alpha" as the parameter of the command
above.

If we are testing the migration on an empty context on the sandbox,
then we should proceed directly to Section 4. Compile the project.
Otherwise, the next two subsections detail how to produce credentials
that will allow us to make the chain that we imported from Mainnet
progress.


4. Compile the Project
----------------------

At this point we have to compile the Alpha protocol (or the snapshot
Alpha protocol, in case we opted for it) that we will activate when
running the migration, as well as the shell if we patched it. We can
compile the whole project under the "src" folder by invoking:

   $ make


5. Import a Context From Mainnet
--------------------------------

If we wish to test the migration in a realistic scenario, we need to
import a context from a Mainnet's snapshot file. As explained in the
beginning of Section Prepare the migration, in our example we will use
a snapshot file "~/snapshot-mainnet.rolling" which was taken at level
"1617344".

We also need to generate a node identity, which we will keep in the
folder that contains the history of the node. Since importing a node
from a snapshot file is very time consuming, once the node is imported
and the identity is generated we will keep the original folder
unchanged, and we will copy its contents to a fresh test folder every
time we want to perform the migration.

For instance, the following commands import a context from the
snapshot file "~/snapshot-mainnet.rolling" into the folder "/tmp
/octez-node-mainnet", and generate an identity in the same folder:

   $ ./octez-node snapshot import ~/snapshot-mainnet.rolling --data-dir /tmp/octez-node-mainnet
   $ ./octez-node identity generate --data-dir /tmp/octez-node-mainnet

The "./octez-node snapshot import" command accepts an option "--block
<block_hash>" that instructs the command to check that the hash of the
last block in the imported chain is "<block_hash>". This mechanism
helps the developer to check that the imported chain contains blocks
that are part of the current main chain of the Tezos network. The
snapshots websites normally provide the hash of the last block in a
given snapshot file. Although we will not be using the "--block"
option in this tutorial, the developer is encouraged to check that
this prefix corresponds to the hash of a real block in Mainnet.

Importing the context from a snapshot file is optional and should be
performed only if we want to test the migration on a realistic context
from Mainnet. Otherwise the migration will run on the sandbox.


6. Create a Yes-Wallet
----------------------

We also need to create a yes-wallet, which is a special wallet where
secret keys actually encode the same bytes as their corresponding
public keys. By adding to the yes-wallet the existing accounts of
Mainnet bakers, we would have enough rights to bake blocks at will. We
can do so by running:

   $ dune exec devtools/yes_wallet/yes_wallet.exe -- create from context /tmp/octez-node-mainnet in /tmp/yes-wallet --active-bakers-only

This command creates a yes-wallet and places its folder in the
system's temp directory (in our example, "/tmp") as given by the path
argument "/tmp/yes-wallet". If no path argument was given, the command
would create the yes-wallet folder in the default path "./yes-wallet".

The command above will generate a wallet containing approximately 400
keys. If you wish to restrict to a given percentage of the attesting
power by retrieving the first bakers (with the biggest staking power),
you can also use the "--staking-share" option to provide a limit. For
instance, the first largest bakers with an accumulated stake of at
least 75 percent can be kept with:

   $ dune exec devtools/yes_wallet/yes_wallet.exe -- create from context /tmp/octez-node-mainnet in /tmp/yes-wallet --active-bakers-only --staking-share 75

Note:

  Prior to switching to the Tenderbake consensus algorithm it was
  sufficient to create a minimal yes-wallet with 8 Foundation keys.
  Starting from Protocol I this is no longer the case, because a
  number of bakers holding at least 2/3rds of the total attesting
  power have to attest a block for it to be considered valid.

By restricting the accumulated stake to 75% as in the command above,
the wallet is both "lighter" (it may contain around 30-40 keys and
therefore some commands like "octez-client bake for" will execute
faster) and its keys will represent more than the 2/3rds of the
attesting power for any given level.


Batch Steps 1--6 Above
----------------------

The script "scripts/prepare_migration_test.sh" batches steps 1--7
above.

The first parameter is optional and contains a name in the format
"<tag_starting_with_version_letter>_<version_number>". If some name is
passed, then the Alpha protocol is snapshot into
"src/proto_<version_number>_<short_hash>". If the name is omitted,
then the Alpha protocol in "src/proto_alpha" will be used for the
migration testing.

Now the script takes the level at which we want to set the user-
activated upgrade. The script distinguishes whether the migration is
on the sandbox or on an imported context based on this level. (Recall
that a level less or equal than "28082" corresponds to the sandbox,
and a level greater than "28082" corresponds to an imported context.)
In our example, if we want to test the migration on the sandbox and
want to trigger it at level three, we can use:

   $ ./scripts/prepare_migration_test.sh d_012 3

If on the contrary we have imported a realistic context from the
snapshot file "~/snapshot-mainnet.rolling" taken at level "1617344",
and we want to trigger the migration three levels after the level at
which the snapshot file was taken, we can use:

   $ ./scripts/prepare_migration_test.sh d_012 1617347 ~/snapshot-mainnet.rolling

In the latter case both the context and the yes-wallet folder will be
placed in the system's temp directory. In our example the temp
directory is "/tmp", and the context and yes-wallet would be placed in
paths "/tmp/octez-node-mainnet" and "/tmp/yes-wallet" respectively.

If the script detects that the yes-wallet folder already exists int
"/tmp", then it will clean it by removing spurious files "/tmp/yes-
wallet/blocks" and "/tmp/yes-wallet/wallet_locks", and it will not
create a new yes-wallet folder. If the script detects that the folder
"/tmp/octez-node-mainnet" already exists, or if the developer passes
the path of a folder instead of the path of a snapshot file, then the
script will use the corresponding folder as the original folder, and
will not import a new context.

In case we opted for not snapshotting the Alpha protocol, we could
batch steps 1--7 by respectively using the commands above, but
omitting the name parameter "d_012".

The script "scripts/prepare_migration_test.sh" receives an optional
"<block_hash>" as the last argument which, if passed, will be used for
the option "--block <block_hash>" to the "./octez-node snapshot
import" command when importing the context form Mainnet.

After performing the steps 1--7, the migration will be ready to be
tested. The next two subsections respectively detail how to run the
migration on the sandbox and on a context imported from Mainnet.


Run the Migration on the Sandbox
--------------------------------

If we run the migration on an empty context, then we would start a
sandboxed node as usual after having prepared the migration test (see
previous section). In our example we can run the following:

   $ ./src/bin_node/octez-sandboxed-node.sh 1 --connections 0 &

We can also start the client:

   $ eval $(./src/bin_client/octez-init-sandboxed-client.sh 1)

Instead of command "octez-activate-alpha", the sandboxed client script
"src/bin_client/octez-init-sandboxed-client.sh" now accepts a command
"octez-activate-XXX-<short_hash>" that activates the predecessor
protocol with version number "XXX" and short hash "<short_hash>". In
our example, the predecessor protocol is "011" with short hash
"PtHangz2". (Check the folder "src" for the version number and short
hash of the predecessor protocol for migrations to versions different
from "012".) We can activate this protocol by invoking:

   $ octez-activate-011-PtHangz2

Activation of the predecessor protocol produces one block and
increases the level by one. This unavoidable increase of the level has
to be taken into account when setting the desired level for the user-
activated upgrade.

Now we can use the client commands to bake blocks until we reach the
level at which migration will be triggered, which in our example is
"3". Since activating the predecessor protocol increases the level by
one, we need to bake two more blocks:

   $ octez-client bake for --minimal-timestamp
   $ octez-client bake for --minimal-timestamp

Note:

  Prior to Tenderbake activation (i.e. to the Protocol I) the command
  above requires a specific account to bake for. Any of
  "bootstrap[0-9]" accounts can be used to do it:"$ octez-client bake
  for bootstrap1 --minimal-timestamp"

At this moment migration will be triggered and the protocol
"proto_012_<short_hash>" will become active, and we will see the log
message "STITCHING!".

The migration can be tested again by restarting the sandboxed node and
client, by activating the predecessor of the Alpha protocol, and by
baking two blocks.


Run the Migration on a Context Imported From Mainnet
----------------------------------------------------

If we run the migration on a context imported from Mainnet, then we
would start the node using the context imported from the snapshot
file. Since importing a snapshot file is very time consuming, we will
leave the original folder unchanged, and every time we want to run the
test, we will copy its contents to a fresh test folder. In our
example, we can do this by taking advantage of an environment variable
"test_directory" and the tool "mktemp" as follows:

   $ test_directory=$(mktemp -d -t "octez-node-mainnet-XXXX") && cp -r "/tmp/octez-node-mainnet/." "$test_directory"

This command creates a fresh test folder in the system's temp
directory (in our example "/tmp") whose name is "octez-node-mainnet-
XXXX", where the "XXXX" are four random alphanumerical characters, and
sets the environment variable "test_directory" to the path of the test
folder, such that we can run the node in the test folder later. Then
it copies the contents of the original context folder into the test
folder.

In order to produce credentials to make the chain imported from
Mainnet progress, we need our node to forge and verify fake
signatures. This can be achieved with the use of the
"TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING" evironment variable that
replaces each signature with a concatenation of a public key and a
message, such that this fake signature is still unique for each key
and message.

Now, we can run the "octez-node" command by specifying the test folder
"$test_directory" as the data directory. We will also specify the RPC
address "localhost", such that the RPCs will be available at the url
"localhost:8732". In our example, by invoking the following:

   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-node run --synchronisation-threshold 0 --connections 0 --data-dir "$test_directory" --rpc-addr localhost &

We will now trigger the migration by baking blocks until the level
reaches the one specified when setting the user-activated upgrades.
The blocks can be baked with the yes-wallet created in step 5 above,
and with any of the accounts "foundation1" to "foundation8". In our
example, we can bake one block by running the following command:

   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-client -d /tmp/yes-wallet bake for --minimal-timestamp

Note:

  Prior to Tenderbake activation (i.e. to the Protocol I) this command
  requires a specific account to bake for. Any of "foundation[1-8]"
  accounts can be used to do it."$
  TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y  octez-client bake for
  foundation1 --minimal-timestamp"If the chosen account "foundation1"
  ceases to have the priority to bake, we can switch to any of the
  remaining accounts "foundation2" to "foundation8". We will always be
  able to make the chain progress since it is virtually impossible
  that at some moment all the eight accounts cease to have the
  priority to bake.

After baking three blocks the migration will be triggered and the
protocol "proto_012_<short_hash>" will become active.  We will see the
log message "STITCHING!".

The migration can be tested again by removing the test folder and the
spurious files "blocks" and "wallet_lock" in the yes-wallet folder. In
our example we can do this with the following command:

   $ rm -rf "$test_directory" && rm -f /tmp/yes-wallet/{blocks,wallet_lock}

Then we repeat the commands above in order to create a fresh test
folder, and to copy the context of the original folder into the test
folder. In our example:

   $ test_directory=$(mktemp -d -t "octez-node-mainnet-XXXX") && cp -r "/tmp/octez-node-mainnet/." "$test_directory"

Now we run the node in the test folder by invoking:

   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-node run --synchronisation-threshold 0 --connections 0 --data-dir "$test_directory" --rpc-addr localhost &

And finally, we bake the numbers of blocks specified by the user-
activated upgrade, with the command:

   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-client -d /tmp/yes-wallet bake for --minimal-timestamp


Wrap up the Migration Procedure
===============================

For convenience, this section collects all the steps needed to test
the migration, both on the sandbox and on a context imported from
Mainnet.


Migration on the Sandbox
------------------------

Check out latest code:

   $ git checkout master
   $ git pull

Tweak migration by checking that
"src/proto_alpha/lib_protocol/init_storage.ml" includes the following
lines:

   | Hangzhou_011 ->
       Logging.log_notice "\nSTITCHING!\n" ;

Commit the feature:

   $ git commit -am 'My awesome feature'

Prepare migration by snapshotting the Alpha protocol, linking it to
the build system, setting user-activate upgrades, and compiling the
project:

   $ ./scripts/prepare_migration_test.sh d_012 3

(Alternatively, each of these steps could be performed individually by
invoking the following fur commands):

   $ ./scripts/snapshot_alpha.sh d_012
   $ ./scripts/link_protocol.sh src/proto_012_*
   $ ./scripts/user_activated_upgrade.sh src/proto_012_* 3
   $ make

Run sandboxed node and client:

   $ ./src/bin_node/octez-sandboxed-node.sh 1 --connections 0 &
   $ eval $(./src/bin_client/octez-init-sandboxed-client.sh 1)

Activate predecessor of the Alpha protocol and move chain one level
forward:

   $ octez-activate-011-PtHangz2

Bake two more blocks:

   $ octez-client bake for --minimal-timestamp
   $ octez-client bake for --minimal-timestamp

You should see the "STITCHING!" message!

To test again, restart the sandboxed node and client:

   $ fg
   ./src/bin_node/octez-sandboxed-node.sh 1 --connections 0
   ^C
   $ ./src/bin_node/octez-sandboxed-node.sh 1 --connections 0 &
   $ eval $(./src/bin_client/octez-init-sandboxed-client.sh 1)

Activate predecessor of the Alpha protocol:

   $ octez-activate-011-PtHangz2

Bake two blocks:

   $ octez-client bake for --minimal-timestamp
   $ octez-client bake for --minimal-timestamp

You should see the "STITCHING!" message again!


Migration on a Context Imported From Mainnet
--------------------------------------------

Check out latest code:

   $ git checkout master
   $ git pull

Tweak migration by checking that
"src/proto_alpha/lib_protocol/init_storage.ml" includes the following
lines:

   | Hangzhou_011 ->
       Logging.log_notice "\nSTITCHING!\n" ;

Commit the feature:

   $ git commit -am 'My awesome feature'

Prepare migration by snapshotting the Alpha protocol, linking it to
the build system, patching the shell in order to obtain yes-node,
creating a yes-wallet, setting user-activated upgrades, importing a
context from Mainnet into the original context folder, generating an
identity in the same folder, and compiling the project:

   $ ./scripts/prepare_migration_test.sh d_012 1617344 ~/mainnet.rolling

(Alternatively, each of these steps could be performed individually by
invoking the following eight commands):

   $ ./scripts/snapshot_alpha.sh d_012
   $ ./scripts/link_protocol.sh src/proto_012_*
   $ ./scripts/user_activated_upgrade.sh src/proto_012_* 1617344
   $ make
   $ ./octez-node snapshot import ~/mainnet.rolling --data-dir /tmp/octez-node-mainnet
   $ ./octez-node identity generate --data-dir /tmp/octez-node-mainnet
   $ dune exec devtools/yes_wallet/yes_wallet.exe -- create from context /tmp/octez-node-mainnet in /tmp/yes-wallet --active-bakers-only

Copy original folder into test folder:

   $ test_directory=$(mktemp -d -t "octez-node-mainnet-XXXX") && cp -r "/tmp/octez-node-mainnet/." "$test_directory"

Run the node`:

   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-node run --synchronisation-threshold 0 --connections 0 --data-dir "$test_directory" --rpc-addr localhost &

Bake three blocks:

   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-client -d /tmp/yes-wallet bake for --minimal-timestamp
   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-client -d /tmp/yes-wallet bake for --minimal-timestamp
   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-client -d /tmp/yes-wallet bake for --minimal-timestamp

Note:

  Prior to Tenderbake activation (i.e. to the Protocol I) this command
  requires a specific account to bake for. Any of "foundation[0-9]"
  accounts can be used to do it.

You should see the "STITCHING!" message!

To test again, kill the node:

   $ fg
   TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-node run --synchronisation-threshold 0 --connections 0 --data-dir "$test_directory" --rpc-addr localhost
   ^C

Clean up by removing test folder and copying original folder into
fresh test folder, and by removing files "/tmp/yes-wallet/wallet_lock"
and "/tmp/yes-wallet/blocks":

   $ rm -rf "$test_directory" && rm -f /tmp/yes-wallet/{blocks,wallet_lock};
   $ test_directory=$(mktemp -d -t "octez-node-mainnet-XXXX") && cp -r "/tmp/octez-node-mainnet/." "$test_directory"

Run the node:

   TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-node run --synchronisation-threshold 0 --connections 0 --data-dir "$test_directory" --rpc-addr localhost &

And bake three blocks:

   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-client -d /tmp/yes-wallet bake for --minimal-timestamp
   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-client -d /tmp/yes-wallet bake for --minimal-timestamp
   $ TEZOS_USE_YES_CRYPTO_I_KNOW_WHAT_I_AM_DOING=Y ./octez-client -d /tmp/yes-wallet bake for --minimal-timestamp

You should see the "STITCHING!" message again!


Tips and Tricks
===============

Migrating a context mostly concerns editing existing data structures.
For this reason it is important to inspect the resulting context with
the RPCs "context/raw/json" and "context/raw/bytes". The former RPC
displays the json value relative to a key of the context, using its
json format. This is possible thanks to the storage functors of Tezos,
which are used to register every piece of storage in a node and are
aware of the json structure of the data. The latter RPC is more low
level and simply returns the bytes corresponding to a key. Both RPCs
support the option "depth" to control how much of the subtree of the
key should be displayed.

For example, if we use "context/raw/json" to inspect the size of the
current listings, which informs of how many rolls are allowed to vote
in the current period, we get:

   $ curl -s localhost:8732/chains/main/blocks/head/context/raw/json/votes/listings_size
   56639

On the other hand, if instead we use "context/raw/bytes" to inspect
the data corresponding to the same key, we obtain a string of bytes in
hexadecimal format:

   $ curl -s localhost:8732/chains/main/blocks/head/context/raw/bytes/votes/listings_size
   "0000dd3f"

This string of bytes can be converted using the OCaml toplevel to
obtain the same value retrieved before:

   utop # let h = 0x0000dd3f ;;
   val h : int = 56639

In our migration example above, we can inspect the json output of a
specific contract:

   $ curl -s localhost:8732/chains/main/blocks/head/context/raw/json/contracts/index/tz3bvNMQ95vfAYtG8193ymshqjSvmxiCUuR5 | jq .
   {
     "balance": "2913645407940",
     "big_map": [],
     "change": "2705745048",
     "counter": "0",
     "delegate": "tz3bvNMQ95vfAYtG8193ymshqjSvmxiCUuR5",
     "delegate_desactivation": 125,
     "delegated": [],
     "frozen_balance": [],
     "manager": "p2pk66n1NmhPDEkcf9sXEKe9kBoTwBoTYxke1hx16aTRVq8MoXuwNqo",
     "roll_list": 50696,
     "spendable": true
   }

The "raw/json" interface conveniently hides the disk representation of
data and keys. Notice how the hashes of public keys are not stored as
is, but instead they are encrypted using the more efficient base58
format.

In this case, in order to inspect the low level representation in
bytes, which we would often need to, we have to convert hashes of
public keys using "tztop" (utop adapted for protocol development) and
the functions "of_b58check" and "to_b58check" of module
"Contract_repr":

   # let's borrow some code from the protocol tests
   $ dune exec -- tztop src/proto_alpha/lib_protocol/test/

   # open Tezos_protocol_alpha.Protocol ;;

   # let b58check_to_path c =
   Contract_repr.of_b58check c |> fun (Ok c) ->
   Contract_repr.Index.to_path c [] |>
   String.concat "/"
   ;;
   # b58check_to_path "tz3bvNMQ95vfAYtG8193ymshqjSvmxiCUuR5" ;;
   ff/18/cc/02/32/fc/0002ab07ab920a19a555c8b8d93070d5a21dd1ff33fe

   # let path_to_b58check p =
   String.split_on_char '/' p |>
   Contract_repr.Index.of_path |> fun (Some c) ->
   Contract_repr.to_b58check c
   ;;
   # path_to_b58check "ff/18/cc/02/32/fc/0002ab07ab920a19a555c8b8d93070d5a21dd1ff33fe"  ;;
   "tz3bvNMQ95vfAYtG8193ymshqjSvmxiCUuR5"

On the other hand, we could have inspected the data corresponding to
the same key above with "raw/bytes", as we do below:

   $ curl -s localhost:8732/chains/main/blocks/head/context/raw/bytes/contracts/index/ff/18/cc/02/32/fc/0002ab07ab920a19a555c8b8d93070d5a21dd1ff33fe | jq .
   {
     "balance": "c4ddb296e654",
     "change": "98c9998a0a",
     "counter": "00",
     "delegate": "02ab07ab920a19a555c8b8d93070d5a21dd1ff33fe",
     "delegate_desactivation": "0000007d",
     "delegated": {
       "15": {
         "bb": {
           "9a": {
             "84": {
               "b5": {
                 "e3501428362c63adb5a4d12960e7ce": "696e69746564"
               }
             }
           }
         }
       },
       ...
     },
     "frozen_balance": {
       "114": {
         "deposits": "80e0f09f9b0a",
         "fees": "93bb48",
         "rewards": "809ee9b228"
       },
       ...
     },
     "manager": "0102032249732e424adfaf6c6efa34593c714720c15490cdb332f2ac84ef463784ff4e",
     "roll_list": "0000c608",
     "spendable": "696e69746564"
   }

Observe that while the value in json format above shows a "big_map"
field that is empty (i.e. ""big_map": [],"), the low-level
representation of the same value reveals that the field containing
such an empty "big_map" is not stored at all.


Anatomy of Migration Code
=========================

The migration code is triggered in
"init_storage.ml:prepare_first_block", so that function is the entry
point to start reading it. Notice that constants are migrated in
"raw_context.ml:prepare_first_block", which takes a "Context.t" and
returns a "Raw_context.t" containing the new constants. Migrating
other data can usually be done by manipulating the "Raw_context.t",
and such code should be placed in the match case "Alpha_previous" of
"init_storage.ml:prepare_first_block".

Conversions of data structures from the previous protocol are
typically found in "storage.ml,i", which may involve the functors in
"storage_functors.ml,i". Each migration is very custom, but there are
two recurring schemas that emerged over time.

For high-level changes, the interface offered by the
"storage_functors" is usually expressive enough. The migration would
copy the code to read the data structures in the previous version and
simply rename it by adding a suffix with the previous version number
(in our example above where we are migrating to version "012", the
identifiers in the old code would be renamed by appending the suffix
"_011" to them). The values are then written using the code for the
data structures of the current protocol, thus performing the
migration. The last step in the migration would be to manually remove
any remaining code with a suffix corresponding to the previous version
("_011" in our example).

Some migrations may require breaking the interface offered by the
"storage_functors", and to modify the file "raw_context.mli" directly.
In this case we usually *copy* the data to a temporary path, perform
the conversion, and then *recursively remove* the temporary path.

Maintaining
***********

As with any smart-contract blockchain platform, Octez is a quite
complex software system, at the crossroad of several domains including
distributed systems, cryptography, programming languages, and applied
formal methods. Additionally, the specific evolution features of
Tezos, enabling powerful online governance, imply an extra
infrastructure meta-layer and an orthogonal versioning scheme for
protocols, and hence, more complex maintenance procedures.

The following howtos are intended to facilitate different maintenance
tasks by providing detailed guides to platform developers, but are
also potentially interesting for other actors in the ecosystem
(communication, management, etc.).

* How to start reading protocol Alpha

  * The "TEZOS_PROTOCOL" structure

  * The big abstraction barrier: "Alpha_context"

  * Below the "Alpha_context"

  * Above the "Alpha_context"

* Adding a new instruction to Michelson language

  * Michelson primitives

  * Michelson types

  * Internal representation (IR)

  * The translator

  * The interpreter

  * The Gas model

  * Documentation

* Protocol Proposals

  * Developing a Protocol Proposal

  * Releasing a Protocol Proposal

  * Validation via on-chain governance

  * Activation

  * Issues and mitigations

* How to Freeze Protocols

  * Update Manifest

  * Remove Accuser, Baker

  * Remove Protocol Tests

  * Remove "lib_delegate"

  * Remove Protocol Tests From Tezt

  * Remove Testnet From "bin_node"

  * Remove RW Commands From "lib_client_commands"

  * Remove Mempool Protocol Plugins

  * Remove Old Docker-Compose Files

  * Add an Entry in "CHANGES.rst"

* Adding a new protocol environment

  * Bootstrap

  * Activating the environment

  * Making changes in the environment

* RPC How-To

  * How to declare a service

  * How to register services by declaring a directory

  * How to serve a directory

  * How to call RPCs

  * How to register an RPC in the protocol

How to start reading protocol Alpha
***********************************

Protocol Alpha is the name of the economic protocol under development
in the "master" branch. Alpha is a placeholder name and is not related
to the old Alphanet test network.

Before reading that document, you may want to:

* read the whitepaper,

* read about the protocol environment.

As all protocols, Alpha is made of a series of OCaml interface and
implementation files, accompanied by a "TEZOS_PROTOCOL" file.


The "TEZOS_PROTOCOL" structure
==============================

If you look at file "TEZOS_PROTOCOL" in the repository, you will see
that it is composed of the hash of the sources, and the list of its
modules, in linking order.

Protocol Alpha is structured as a tower of abstraction layers, a
coding discipline that we designed to have OCaml check as many
invariants as possible at typing time. You will also see empty lines
in "TEZOS_PROTOCOL" that denote these layers of abstraction.

These layers follow the linking order: the first modules are the
tower’s foundation that talk to the raw key-value store, and going
forward in the module list means climbing up the abstraction tower.


The big abstraction barrier: "Alpha_context"
============================================

The proof-of-stake algorithm, as described in the white paper, relies
on an abstract state of the ledger, that is read and transformed
during validation of a block.

Due to the polymorphic nature of Tezos, the ledger’s state (that we
call **context** in the code), cannot be specific to protocol Alpha’s
need. The proof-of-stake is thus implemented over a generic key-value
store whose keys and associated binary data must implement the
abstract structure.

The "Alpha_context" module enforces the separation of concerns
between, on one hand, mapping the abstract state of the ledger to the
concrete structure of the key-value store, and, on the other hand,
implementing the proof-of-stake algorithm over this state.

In more practical terms, "Alpha_context" defines a type "t" that
represents a state of the ledger. This state is an abstracted out
version of the key-value store that can only be manipulated through
the use of the few selected manipulations reexported by
"Alpha_context", that always preserve the well-typed aspect and
internal consistency invariants of the state.

When validating a block, the low-level state that result from the
predecessor block is read from the disk, then abstracted out to a
"Alpha_context.t", which is then only updated by high level operations
that preserve consistency, and finally, the low level state is
extracted to be committed on disk.

This way, we have two well separated parts in the code. The code below
"Alpha_context" implements the ledger’s state storage, while the code
on top of it is the proof-of-stake algorithm. Thanks to this barrier,
the latter can remain nice, readable OCaml that only manipulates plain
OCaml values.


Below the "Alpha_context"
=========================

For this part, in a first discovery of the source code, you can start
by relying mostly on this coarse grained description, with a little
bit of cherry-picking when you’re curious about how a specific
invariant is enforced.


Hashes and storage description layer
------------------------------------

This layer is implemented by module "storage_description.ml" and
modules named "*_hash". It contains mainly Blake2b hash
implementations specialized for various basic types of the protocol.


The representation layer
------------------------

This layer is implemented by modules named "*_repr". These modules
abstract the values of the raw key-value context by using
Data_encoding.

These modules define the data types used by the protocol that need to
be serialized (amounts, contract handles, script expressions, etc.).
For each type, it also defines its serialization format using
Data_encoding.

Above this layer, the code should never see the byte sequences in the
database, the ones of transmitted blocks and operations, or the raw
JSON of data transmitted via RPCs. It only manipulates OCaml values.


The "Storage" module and storage functors
-----------------------------------------

Even with the concrete formats of values in the context abstracted
out, type (or consistency) errors can still occur if the code accesses
a value with a wrong key, or a key bound to another value. The next
abstraction barrier is a remedy to that.

The storage module is the single place in the protocol where key
literals are defined. Hence, it is the only module necessary to audit,
to know that the keys are not colliding.

It also abstracts the keys, so that each kind of key get its own
accessors. For instance, module "Storage.Contract.Balance" contains
accessors specific to contracts’ balances.

Moreover, the keys bear the type of the values they point to. For
instance, only values of type "Tez_repr.t" can by stored at keys
"Storage.Contract.Balance". And in case a key is not a global key, but
a parametric one, this key is parameterized by an OCaml value, and not
the raw key part.

So in the end, the only way to be used when accessing a contract
balance is "Storage.Contract.Balance.get", which takes a
"Contract_repr.t" and gives a "Tez_repr.t".

All these well-typed operations are generated by a set of functors,
that come just before "Storage" in "TEZOS_PROTOCOL".


The "*_storage" modules
-----------------------

The two previous steps ensure that the ledger’s state is always
accessed and updated in a well-typed way.

However, it does not enforce that, for instance, when a contract is
deleted, all of the keys that store its state in the context are
indeed deleted.

This last series of modules named "*_storage" is there to enforce just
that kind of invariants: ensuring the internal consistency of the
context structure.

These transaction do not go as far as checking that, for instance,
when the destination of a transaction is credited, the source is also
debited, as in some cases, it might not be the case.


Above the "Alpha_context"
=========================

The three next sections describe the main entrypoints to the protocol:
validation of blocks by the shell (that we often also call
application), smart contracts, and RPC services.

The "Main" module is the entrypoint that’s used by the shell. It
respects the module type that all protocol must follow. For that, its
code is mostly plumbing,


Starting from "Apply"
---------------------

This is were you want to start on your first read. Even if some
plumbing code is woven in, such as error cases declaration and
registration, most of the proof-of-stake code has been written in a
verbose style, to be understood with minimum OCaml knowledge.

You want to start from the shell entry points (validation of the block
header, validation of an operation, finalization of a block
validation), and follow the control flow until you hit the
"Alpha_context" abstraction barrier. This will lead you to reading
modules "Baking" and "Amendment".


Smart contracts
---------------

From "Apply", you will also end up in modules "Script_ir_translator"
and "Script_interpreter". The former is the typechecker of Michelson
that is called when creating a new smart contract, and the latter is
the interpreter that is called when transferring tokens to a new smart
contract.


Protocol RPC API
----------------

Finally, the RPCs specific to Alpha are also defined above the
"Alpha_context" barrier.

Services are defined in a few modules, divided by theme. Each module
defines the RPC API: URL schemes with the types of parameters, and
input and output JSON schemas. This interface serves three purposes.
As it is thoroughly typed, it makes sure that the handlers (that are
registered in the same file) have the right input and output types. It
is also used by the client to perform RPC calls, to make sure that the
URL schemes and JSON formats and consistent between the two parties.
These two features are extremely useful when refactoring, as the OCaml
typechecker will help us track the effects of an RPC API change on the
whole codebase. The third purpose is of course, to make automatic
documentation generation possible (as in "octez-client rpc
list/format"). Each service is also accompanied by a caller function,
that can be used from the client to perform the calls, and by the
tests to simulate calls in a fake in-memory context.

It can be useful if you are a third party developer who wants to read
the OCaml definition of the service hierarchy directly, instead of the
automatically generated JSON hierarchy.

Adding a new instruction to Michelson language
**********************************************

From time to time a need arises to enrich Michelson language with an
additional instruction. It might be an intimidating task, especially
to someone who is not very familiar with the interpreter code. Thus,
here is a quick tutorial outlining the most important steps required
to add a new Michelson instruction. outlining the most important steps
required to add a new Michelson instruction.

Adding a new instruction requires providing the following elements
(detailed in the rest of this document):

1. a primitive

2. an internal representation (or IR for short)

3. a translation rule

4. an execution rule

5. a gas cost model

6. some documentation

Once all of the above are provided, we may try compiling the protocol.
We will most likely encounter some non-exhaustive pattern matching
errors, but after the following steps are complete, these will usually
be trivial to fix.

There are a lot of comments in the code base explaining further
details about the functions and data types described here. They are
attached either to their definitions in ".ml" files or in
corresponding ".mli" files if definitions are public.

For each function and data type mentioned on this page there is a
pointer to the source code file where it can be found.


Michelson primitives
====================

The first thing we need to provide for creating a new instruction is a
Michelson primitive. A primitive is a textual representation of our
instruction in the Michelson code. Michelson primitives are organized
into 5 namespaces:

1. keywords (lowercase words such as "parameter", "storage", "code")

2. types (lowercase words)

3. constructors (capitalised words)

4. constants (lowercase word "constant" followed by a string)

5. instructions (all uppercase words)

Of these only the last group is of interest to us here. "DUP", "CONS"
and "ADD" are some examples of primitives representing instructions.
By convention constructor names for primitives representing
instructions in the interpreter implementation (as opposed to: in
Michelson code) should start with capital letter "I", for instance
"I_ADD" for "ADD" instruction.

Primitives should be added with great care, though, as in binary
representation they are currently encoded as single bytes, so there
can be no more than 256 of them. Exceeding that number will force us
to reserve more bytes for some primitives, which will incur some space
and performance overheads for storing and executing contracts.
Therefore, whenever possible we should prefer reusing existing
primitives to adding new ones. Fortunately, we can take advantage of
the fact that Michelson has a strong type system, ensuring that a same
primitives can be used differently in several contexts. Thus, by
allowing an existing primitive in a new context we can effectively add
instructions without adding new primitives.

For instance suppose we have a "MAP" instruction and primitive, which
expects a list at the top of the stack, and which recursively applies
some given code block to elements of the list, thus creating a new
list, which is then put back at the top of the stack. Suppose we now
want a similar instruction for options, instead of lists of elements.
So far "MAP" is only considered valid in a very particular state of
the stack. By adding a new translation rule (see Script translator),
we can allow the same primitive to be used when there is an option
instead of the list at the top of the stack, thus effectively creating
a new instruction. This may be seen as "overloading" the primitive for
different argument types.

When adding a new primitive, it should be mapped to the appropriate
namespace in "namespace" function (in
src/proto_alpha/lib_protocol/michelson_v1_primitives.ml). In case of
an instruction, the appropriate namespace is "Instr_namespace". It is
also necessary to map the primitive constructor to the string
representing it in Michelson code in "prim_of_string" and
"string_of_prim" in the same module.

See src/proto_alpha/lib_protocol/michelson_v1_primitives.ml for a
complete list of primitives known to Michelson and their respective
textual representations.

See also Micheline to learn more about the concrete syntax of
Michelson language.


Michelson types
===============

Before we talk about internal representations, we need to have a brief
look at how the interpreter handles types of Michelson expressions
internally. Types "ty" and "stack_ty" are both defined in
src/proto_alpha/lib_protocol/script_typed_ir.ml. Type "ty" enumerates
all types known to the Michelson interpreter and is parametrised by
the underlying OCaml type in which Michelson values are actually
stored in memory. Thus, when values of the ty type are pattern-matched
on, their type parameters help convince the OCaml compiler that the
underlying Michelson expressions or instructions are compatible (i.e.
may be unified), and so that the execution model for Michelson is
type-safe.

"stack_ty" is a collection of "ty" values occurring in a particular
order, which represents the type of the whole stack. The outermost
value corresponds to the top of the stack, while the innermost one is
always "Bot_t", which corresponds to the bottom of the stack. Note
that "stack_ty" is parametrised by two type parameters. The first one
is the OCaml counterpart of the type of the value at the top of the
stack; the other combines the OCaml counterparts of types of the
remainder of the stack in the form of a *cons list*, i.e. a pair,
whose first element is a type and the other – another pair containing
the remainder of the list.

This is a recurring pattern within the interpreter: type parameters
corresponding to types of Michelson stacks usually come in pairs. As
we will shortly see, the first parameter in these pairs always
corresponds to the type of element at the top of the stack; the other
– to the remainder of the stack.


Internal representation (IR)
============================

A primitive is what represents our instruction in the Michelson
script. Now we need an IR, which is a piece of data containing all the
information necessary to actually execute the instruction. It should
belong to the type:

   type ('before_top, 'before, 'result_top, 'result) kinstr = (* ... *)

which is defined in "lib_protocol/script_typed_ir.ml". As the module's
name suggests, this representation is already guaranteed to be well-
typed. The type "kinstr" is the type of well-typed instruction
sequences. Adding a new instruction consists in defining a new
constructor of the "kinstr" type representing sequences starting with
the new instruction, providing a rule which constructs them from the
primitive corresponding to the new instruction and a rule which
interprets them when the script in executed.

Instructions are parametrised by 4 type parameters:

1. The type of the top element of the stack prior to execution

2. The type of the remainder of the stack prior to execution

3. The type of the top element of the stack following execution

4. The type of the remainder of the stack following execution

For instance "ICar" instruction is defined as:

   | ICar :
       ('a * 'b, 's) kinfo * ('a, 's, 'r, 'f) kinstr
       -> ('a * 'b, 's, 'r, 'f) kinstr

The reason why the third parameter of the resulting "kinstr" is "'r"
and not simply "'a" (which is the type of the first element of the
pair at the top of the initial stack) is because this constructor also
contains the next instruction, which produces a value of some
arbitrary type "'r". However, note that this next instruction should
expect "'a" at the top of its initial stack.

New instructions are added by extending the "kinstr" type with
additional variants. The variant should contain all the information
necessary to execute the instruction and its type parameters must
ensure that the shape of the stack prior to execution will provide
arguments for the instruction and that the stack following the
execution will contain its result.

Each IR should also contain an instance of type "kinfo", which holds
the information about the type of the stack prior to the instruction's
execution. This information is mostly used for error reporting and
logging. Another thing each instruction's IR should contain is a
continuation, i.e., another instruction that is going to be executed
next. This way the whole script can be represented as a single
instruction constructed by sequencing many instructions together. See
the next section to learn how such a sequence is constructed in the
process of translation.

Instructions taking all their arguments from the stack will require
nothing more than the above data, as the constructor itself will
inform the interpreter, what to do. In more complex cases, especially
when some control transfer is involved, some additional information
may be needed. For example a "MAP" instruction (regardless of the type
it operates on) requires some more instructions (a sub-program) to be
executed to map one value into another. A "PUSH" instruction needs to
know the type and the value it should push on the stack, and so on.


The translator
==============

Now that we have chosen a primitive to represent our instruction in
the code and an internal representation (IR), we need to provide a
rule that translates the former into the latter. "parse_instr"
function in src/proto_alpha/lib_protocol/script_ir_translator.ml is
responsible for this. Notice that the function "parse_instr", despite
what its name suggests, matches on pre-parsed Micheline AST. Micheline
parser is not a part of the protocol and therefore must be run by the
client before the script is submitted to the node in binary-encoded
form.

The translator iterates over the AST instruction after instruction,
maintaining the type of the stack after each operation. This way it
can also type check the script in a single run. Additionally this
makes information about the "current" type of the stack available when
parsing each consecutive instruction. Notice that this function not
only matches on Michelson primitives themselves, but also on the type
of the stack implied by translating previous instructions. This is
what lets us distinguish between different contexts in which the same
primitive may have different meaning. For instance the "MAP"
instruction that we mentioned before may perform mapping either on
lists or on options, depending on what the current type of the stack
might be. Of course, the internal representation will be different in
each case. If the primitive and the current stack type does not match
any possibility, it's a typing error and the whole script is
considered ill-typed.

Because the IR is well-typed by construction, it is necessary for
"parse_instr" translates the script and also type check it at the same
time. Motivations for this are twofold. For one thing type checking
gives us strong static guarantees about the behaviour of the
interpreter. In particular it's guaranteed to only produce well-typed
Michelson values, which means each instruction receives input that it
expects. Secondly, the interpreter does not have to check for types of
data it finds on the stack (they're guaranteed to be correct), which
makes for faster execution. For this reason it is essential that each
IR contains a value of type "kinfo" (or an equivalent thereof), from
which the translator can obtain the type the stack should have after
this instruction is executed. Function "kinfo_of_kinstr" in
src/proto_alpha/lib_protocol/script_typed_ir.ml is responsible for
this extraction.

An interesting situation occurs with instructions regulating control
flow. These usually receive one or more pieces of code (sub-programs)
to execute depending on some runtime conditions. "MAP", "IF_LEFT",
"IF_NONE" are all examples of such instructions. These sub-programs
must also be well-typed. Moreover, if there's more than one like in
case of "IF_LEFT" or "IF_NONE" instructions, a certain relationship
between types of these sub-programs must hold. In particular, they
must ensure, that the entire instruction will always render a
resulting stack of the same type.

To ensure this, these sub-programs must be type-checked (and
translated) recursively. The aforementioned relationships between sub-
programs' types should be guaranteed by the constructor of the
"kinstr" variant. However, in case of multiple execution branches
(sub-programs) their types must be unified before proceeding. This is
what "merge_stacks" function is for. It accepts two stack types and
verifies if they're equal. If so, the unified stack type is returned,
otherwise it results in a type error.

The precise return type of "parse_instr" is "judgement" defined in
src/proto_alpha/lib_protocol/script_ir_translator.ml:

   type ('a, 's, 'b, 'u) cinstr = {
     apply :
       'r 'f. ('a, 's) kinfo -> ('b, 'u, 'r, 'f) kinstr -> ('a, 's, 'r, 'f) kinstr;
   }

   type ('a, 's, 'b, 'u) descr = {
     loc : Script.location;
     bef : ('a, 's) stack_ty;
     aft : ('b, 'u) stack_ty;
     instr : ('a, 's, 'b, 'u) cinstr;
   }

   (* ... *)

   type ('a, 's) judgement =
     | Typed : ('a, 's, 'b, 'u) descr -> ('a, 's) judgement
     | Failed : {
       descr : 'b 'u. ('b, 'u) stack_ty -> ('a, 's, 'b, 'u) descr;
     }
       -> ('a, 's) judgement

This is quite a bit involved, so we'll explain these types and their
role one at a time. "cinstr" is an intermediate representation of an
instruction whose continuation is not yet known. As we can see, it's a
function actually, which given another instruction (the "kinstr"
argument), sequences it after the current one, which is held in the
function's closure. A precise stack description (a "kinfo") is also
required, but note the initial type of the stack is already
predetermined by the type of "cinstr". The resulting type of the
instruction passed into the function, on the other hand, can be
anything, as witnessed by the **forall** annotation "'r 'f.". The
construction of "cinstr" is such that we can already create it without
knowing what the next instruction will be (as opposed to a "kinstr",
which should already contain instruction's continuation).

A "descr" is a complete description of an instruction. It combines a
"cinstr" with precise descriptions of the stack type before and after
the operation. These stack types are typically assembled by inspecting
the initial stack type passed to "parse_instr" (which we pattern match
on in the said function). Location is typically copied from the
Micheline pattern as is and it only serves the error-reporting
purposes in case type-checking failed at a later step.

Finally a successful typing judgement contains a full description of
an instruction, but is only parametrised by its input type. The return
type is existential and can only be retrieved by inspecting the "aft"
field of the contained description. This is vital, because when
calling the type-checking procedure, we don't yet know the return type
of the program. In fact the whole point of it is (in a sense) to learn
that type. Typically when adding typing rules for new instructions, we
only need to be concerned with the successful result ("Typed"
constructor). "Failed" is returned when the instruction does not
produce an output stack. This is true for instance of "FAILWITH"
instruction, which immediately terminates the execution. Hence, it
cannot be given any sound type and therefore must be treated
specially.

As already mentioned above, some instructions also receive pieces of
code (sub-programs) as their arguments. As these sub-programs must
themselves be well-typed, typically we will recursively call
"non_terminal_recursion" function on them. If it succeeds in type-
checking the sub-program, we use its description to convert it into
the type of the whole instruction (which will usually be slightly
different than the type of the sub-program). If it fails, however, the
error will be transparently propagated up the call stack.

"Failed" judgements are treated differently by different instructions.
Some, like "MAP" convert them to typing errors, other unify them with
any type the other program branch might have, effectively treating
them as if they had type "forall 'a. 'a". In fact, this is precisely
the type that Michelson specification assigns to the "FAILWITH"
instruction.


The interpreter
===============

The interpreter is the heart of the Michelson implementation. This
specific implementation follows the small-step approach: at each
execution step, the interpreter rewrites a machine configuration made
of a value stack and a continuation stack. Therefore, the interpreter
takes a script's IR, a storage and an input to the script as
arguments, generates the initial stack containing the storage content
and the input, and then executes the script, returning the final
content of the stack. It's defined in
src/proto_alpha/lib_protocol/script_interpreter.ml by the "execute"
function.

The "execute" function does some preliminary preparations and then
passes control to the "step" function, which encodes the
interpretation loop and where execution rules for all instructions are
given. The "step" function accepts:

1. Context and step constants (see below)

2. Remaining gas

3. Instruction

4. Continuation

5. The top value on the stack (called *accu*)

6. The remainder of the stack

Typically this function computes the new value of the stack and then
calls itself recursively with a new instruction (already available in
the "kinstr" value). In some cases, however, some additional action
may be required either before or after the instruction is processed.
These additional actions are usually related to the control flow.

For instance, in the case of our "MAP" instruction for options, if it
finds "None" at the top of the stack, will leave it without a change
and simply proceed to the next instruction. If, however, it finds
"Some x", it should pass control over to the sub-program given as
parameter. This sub-program does not expect an option, though, it
expects our "x" unwrapped. Similarly it does not return an option, but
an arbitrary result which should be wrapped in a "Some" constructor
after control returns to the main program. Without this additional
action, the types of stacks produced by the two branches would differ
and the program would be ill-typed. To remedy this and similar
problems, the interpreter also defines the "continuation" type
(defined in src/proto_alpha/lib_protocol/script_typed_ir.ml). Whenever
the control is passed over to a sub-program, the "next" function can
be called to manage the flow of control around the sub-program (for
instance executing it multiple times in case of a loop). Also, each
Michelson program ends with a special instruction "IHalt", which calls
the same "next" function.

The "next" function occupies itself with continuations stack (argument
no. 4 on the list above). Typically before transferring control to a
sub-program, an appropriate continuation is pushed on the continuation
stack to manage its outcome and resume execution of the main program.
Only if there's no continuation to perform, the program is actually
terminated. In the "MAP" instruction example it would be wrapping the
value at the top of the stack in the "Some" constructor to match the
type of the other branch. We simply put an appropriate continuation in
the continuation stack before executing the sub-program with the rest
of the main program appended as continuation.

Sometimes adding an instruction may involve adding a new continuation
as well. However, continuations are completely internal to the
interpreter. They neither have a representation in the Michelson code
nor are they ever involved in translation. A continuation is a value
of type "('a, 'b, 'c, 'd) continuation" defined in
src/proto_alpha/lib_protocol/script_typed_ir.ml. Similarly to an
instruction, a new constructor of this type should contain all the
information required to execute the continuation. For instance "KCons"
continuation contains an instruction and a continuation which should
be executed after it. The special "KNil" continuation marks the end of
execution. After the continuation is defined, it can be used freely in
the interpreter.

The step constants passed to the function along with the context
contain some important information about the transaction itself, like
the sender and the target, the amount transferred and so on. See
"step_constants" type definition in
src/proto_alpha/lib_protocol/script_interpreter.ml for more details.


The Gas model
=============

Each Michelson instruction also has a corresponding *gas model*, which
estimates how much gas should be consumed by the interpreter when
executing the instruction. The main goal of this is to prevent scripts
from falling into infinite loops and to protect bakers from abuse of
their computing power. Of course, it's impossible to make sure
statically that a Michelson program always terminates, but its
computing time can be checked dynamically and that's the main purpose
of gas.

The gas model is a function approximating the time and computational
resources consumed by executing an instruction depending on the size
of its argument. Typically the role of the programmer adding an
instruction is to define the function for the instruction, but leaving
any constant values in it abstract. Additionally, a benchmark needs to
be defined in order to find the appropriate values for these
constants. At a later point benchmarks are run on a dedicated machine
and constant values are filled in based on the results. It is
essential that these benchmarks are always run on the same hardware so
that they return comparable results.

More information on the gas model and on benchmarking the interpreter
can be found on the pages dedicated to the "snoop" library: Snoop and
the Michelson Interpreter.


Documentation
=============

Last but not least, the new instruction needs to be documented. After
all, nobody will ever use it otherwise!

Documenting the instruction may involve:

* Editing the Michelson language page (only the page corresponding to
  the Alpha protocol needs be changed, because injected protocols
  cannot be extended with new instructions). The source for this page
  is docs/alpha/michelson.rst.

* Updating the Michelson reference website, by modifying the
  associated repository.

Protocol Proposals
******************

This document explains in detail the process followed by a team
crosscutting Nomadic Labs, Trilitech and Functori for developing and
submitting Protocol Proposals (hereafter called "the protocol team"),
which are proposals for amending the Tezos economic protocol. This
playbook covers more generally their whole lifecycle, including
discovering and managing related issues.

Note:

  The processes documented here present the best practices followed by
  the protocol team. They are documented for improving predictability
  and reproducibility between protocol proposals submitted by this
  team, and for easing the onboarding of new members in this team, and
  possibly more largely in the Tezos ecosystem, by demystifying the
  protocol amendment process. These best practices may also be useful
  as inspiration for other proposers of protocol amendments, but they
  do not aim to provide any guidelines or standards.

Protocol Proposals are regularly submitted (non-exclusively) by the
protocol team, typically every 3-4 months, to the on-chain governance
– the bakers – who decide whether one of these Proposals will be
activated on the mainnet and become the effective Protocol.

Activation of a Protocol Proposal involves a self-amendment of the
mainnet Protocol through hot swapping. This requires baker
infrastructure to be ready for a specific period – the Protocol
programmatically decides the activation block once voted.

The life cycle of a successful Protocol Proposal unfolds as follows:

1. Developing a Protocol Proposal

2. Releasing the Protocol Proposal

3. Validation via on-chain governance

4. Activation on mainnet

During any of these phases, issues may possibly be discovered.
Handling such issues is covered by the different procedures described
in section Issues and mitigations.


Developing a Protocol Proposal
==============================

The development of a protocol proposal proceeds in different phases,
and covers not only its code, but also other related artefacts such as
documentation or testnets.


Proposed protocol code and public documentation
-----------------------------------------------

The Protocol Proposal code consists of feature implementations that
can be either ready to be activated or under a feature flag. The
features that are ready for activation will be considered as part of
the Protocol Proposal. It’s important to note that the code under
feature flags will also be included in the Protocol Proposal code,
even if it is unreachable under mainnet.

A feature is **complete** once fully implemented, documented, and
validated. A feature is validated when the corresponding feature team
has enough confidence in its implementation thanks to a validation
process, usually designed with safety and security analysis, feature
tests, and regression tests (and potentially external audits).

The Protocol Proposal must not introduce unbearable and unpredicted
performance regressions during its activation or lifetime: it has to
be *validated against* *global and performance tests*.

A Protocol Proposal is **feature-complete** when all its features are
complete, and its whole code is validated by global performance
testing.

A feature-complete Protocol Proposal may transit to the
**stabilization period**, during which it undergoes additional testing
cycles, including performance and manual testing, QA testing,
integration testing by tool providers. During stabilization, the
protocol documentation is improved and completed to cover all new
features and changes.

During the development of a protocol proposal, feedback requests
involve bakers, tooling providers, and other stakeholders to gather
early input on the design and implementation of new features. These
aim to reach a consensus on feature design as early in the development
process as possible, avoiding delays in downstream support when
features are eventually released.

It is especially important to bring onboard the developers from
hardware wallets, web wallets, indexers to inform them of the new
features and identify possible changes (specially breaking changes),
arising from changes in the new proposal.

It is imperative that these tools are ready before the potential
activation date of the new protocol proposal.

Under exceptional circumstances, particular inputs from the feedback
loop might be taken into account during the protocol stabilization
period. In this case:

* The stabilization period is extended accordingly to allow gathering
  feedback and providing recommendations.

* Changes originating from this process may be incorporated in the
  protocol proposal code, or may translate in a new *variant* to be
  submitted along the original proposal.

* Modified proposals and/or new variants will also be subject to
  stablization.

During stabilization, the feature-complete Protocol Proposal lives in
"proto-beta/", and changes can only be triggered by a bug fix, a QA or
performance improvement, or a decided change from a feedback loop.

Once stabilized, a feature-complete Protocol Proposal can be declared
**ready for release.**

The code lives within the following directories under "src/":

* Before being feature-complete, the Protocol Proposal code is in
  "proto-alpha".

* A feature-complete Protocol Proposal code is in "proto-beta".

* A ready-for-release Protocol Proposal code is in "proto-<proposal
  name>".


Protocol Proposal testnet and hash
----------------------------------

The following testnets are used for protocol proposals:

* Before being feature-complete, the Protocol Proposal (protocol
  Alpha) can be tested on "weeklynet".

* A feature-complete Protocol Proposal stabilization requires a
  dedicated testnet: "betanet" is used for testing campaigns happening
  during stabilization. This testnet is shut down when the Proposal
  testnet starts.

* A ready-for-release Protocol Proposal testnet is "<proposal
  name>net" and its start is announced within the Protocol Proposal
  release.

A testnet needs an Octez suite and a testnet configuration for
running. An Octez suite needs a Protocol to activate hence a hash of
the code of this Protocol.

Consequently:

* A first hash of "proto-beta" is computed before entering into the
  stabilization

* At each upgrade of "proto-beta", it has to be rehashed so that
  "betanet" enables testing on the accurate Proposal

* For each version of "proto-beta", a version of Octez, compatible
  with "proto-beta", is released as "octez-beta"

The stabilization period continues until the feature-complete Protocol
Proposal code is ready for release.


Feature-completeness
--------------------

The objectives of this phase are:

* assessing the set of features from "proto-alpha" that are complete
  (with feature flag activated) to be part of a Protocol Proposal

* producing all the artifacts required for entering the stabilization
  period:

  1. a feature-complete Protocol Proposal in a dedicated directory
     called "proto-beta" (using proto-betaN in case we need to
     stabilize several versions of proto-alpha with a different set of
     activated features) from proto-alpha

  2. Its initial hash and the initial "octez-beta" suite

  3. Testnet configuration for "betanet"

Artifacts 1. and 2. are produced using the script
scripts/proto_manager.sh by the protocol team (this script supersedes
the older snapshot script  by automating much more steps, for instance
related to testing and CI). In case of breaking changes in the
protocol, the script should be updated by its maintainers. If
necessary, manual commits may be introduced in the Merge Request
introducing the stabilized protocol. In any case, such changes should
be taken into account and the script should be updated by its
maintainers afterwards. Artifact 3 should be produced by the protocol
team.

Two weeks before the feature-complete expected date, open a milestone
(along the lines of another recent milestone titled "<X> Protocol
Proposal is feature-complete") and open a dedicated channel
<x>-protocol with the tech leads and managers for each feature team
and for release management, the Protocol shepherd, the Head of Core
Engineering and the testnets manager. Make sure the "proto_manager"
script is usable with the current state of "proto_alpha".

While the productions of both the "proto-beta" snapshot and "octez-
beta" are under the responsibility of the protocol team, Release
Management must be involved in their review and actual releases.


Stabilization
-------------

The objectives of this phase are:

* Stabilizing the feature-complete Protocol Proposal on "proto-beta"

* Potentially adding decided changes triggered by feedback loops

* Producing artifacts for the Protocol Proposal release:

  * Final hash

  * Compatible Octez suite

  * Testnet configuration for "proto-<proposal name>"

  * Technical description of the Protocol Proposal features

To plan the end of the stabilization period, open a milestone (along
the lines of another recent milestone titled "<X> Protocol Proposal is
release-ready").

As with "proto-beta" and "octez-beta", producing the final protocol
snapshot and its associated release candidate are under the
responsibility of the protocol team. Obviously, Release Management
must be involved in their review and actual releases.


Releasing a Protocol Proposal
=============================

A Protocol Proposal consists of:

1. A hash of the proposed protocol code.

2. The proposed protocol public documentation.

3. An Octez suite release candidate that allows nodes, bakers, and
   accusers to support this Proposal.

4. A testnet.

Releasing a Protocol Proposal involves publishing these 4 elements in
a blog post, on Agora and on company blogs. Make sure of the technical
consistency of the blogpost content and social media package.

It is recommended to release a protocol proposal as early as possible
during the first half of a Proposal period of the on-chain governance
process, so as to allow bakers sufficient time to evaluate proposals
and cast their votes.

For more details on the release phase, see:

* Protocol Release Checklist

  * Summary

  * Timeline and Checklist


Validation via on-chain governance
==================================


Protocol Proposal proposition
-----------------------------

The on-chain governance of Tezos L1 consists of 5 consecutive periods
of 5 cycles (~ 2 weeks):

* Proposal period:

  * This is when the released Protocol Proposal is injected by a baker
    and submitted to the on-chain governance.

  * From potentially multiple proposals, bakers decide which is the
    only one to remain

  * Quorum: 5% of the stake

  * If no proposal reaches the quorum, we go back to a Proposal period

* Exploration period:

  * Are the bakers happy with the one Protocol Proposal that wins the
    Proposal period

  * Or, it fails and we go back to a Proposal period

  * Quorum: For a proposal to be approved, the participation (yea, nay
    and pass votes) should be higher than the expected quorum
    (currently around 50%) **and** 80% of the casted votes (either yea
    or nay) should be yea votes

* Cooldown (also called testing on some block explorers) period:

  * Supposedly readiness of the ecosystems, bakers’ discussions

  * no vote

* Promotion period:

  * After whatever happens during the cooldown period, are bakers
    still happy with the Protocol Proposal under vote

  * Or, it fails and we go back to a Proposal period

  * Quorum: For a proposal to be approved, the participation (yea, nay
    and pass votes) should be higher than the expected quorum
    (currently around 50%) **and** 80% of the casted votes (either yea
    or nay) should be yea votes

* Adoption period:

  * The Protocol Proposal will be activated in 5 cycles, the ecosystem
    is getting ready. This constitutes a hard deadline for critical
    tools (such as Tezos apps on hardware wallets, or major indexers
    and block explorers) to support the test protocol and be deployed,
    if not already so.


Activation
==========


Before Activation
-----------------

**One Month Before Potential Activation**

* **Test Network End of Life**: announce the end of the previous test
  network. More precisely, announce that the test network for protocol
  "n-1" will be stopped when (if) protocol "n+1" activates. This gives
  one month for users to migrate to the test network for protocol "n"
  or "n+1".

**A Few Days Before Activation**

* Reminder for bakers to upgrade.

  The goal is to ensure that 3 first block proposers for the
  activation, the first new protocol block, and <P> second blocks are
  informed and have upgraded their infrastructure following the
  operation procedure for bakers.

**Before Activation, the following are prepared**

* Communication launch: blog article, social media, news/press release
  preparation for succesful activation

* Doc Update MR: A documentation update reflecting the new active
  protocol and droping the documentation of the previous protocol has
  to be ready. Technically, an MR instantiating meta-issue #2170
  (Section "Protocol activation") for the new protocol has to be ready
  and must have sufficient approvals to be triggered right after
  activation.


Activation time
---------------

The activation of the Protocol Proposal happens at the end of the
Adoption period.

Switching to the new protocol happens at the same time on the nodes in
the network and requires a coordinated distributed deployment on every
baker infrastructure (node, baker, accuser) – as we must have ⅔ + 1 of
each block’s consensus committee up to date, to have a smooth
activation we target the whole stake, and tolerate at least 85% of the
stake – and nodes on the network need also to deploy Octez binaries
that "understand" the new Protocol.

All the preparation before activation, even before release should have
ensured that the activation is smooth. Hence pre-activation readiness
data, monitoring, war room, snapshot to restore <O> states, and – most
importantly — readiness to celebrate the success of the activation in
the activation room.


After Activation
----------------

A few hours after activation, when it is certain that everything has
happened successfully, a blog post goes out to Agora and company
blogs, and then a news release is sent out.

On the main tezos/tezos repository, on the master branch, the
"proto_alpha" directory is reset to the newly activated protocol and
its associated daemons, with the exception of vanity nonce and
migration that should be reverted.

Soon after the activation (preferably on the same or next day), the MR
updating the documentation to reflect the new active protocol (see
above) has to be merged. Also, part of the code related to the old
protocol can now be dropped, see How to Freeze Protocols.

One month after the activation of protocol N, we deactivate the N-1
test network. (For example, the Babylon net was deactivated one month
after Carthage went live on the main network.) This deactivation has
been already announced one month before activation (see above).


Issues and mitigations
======================


Decision Tree in case of issues
-------------------------------

The following decision tree applies in case an issue has been found in
an active protocol or a protocol proposal already submitted to on-
chain governance, during any of the voting periods.

This decision tree is extended to the Tezos Layer 2 by a **Solidarity
Principle**. This Solidarity Principle ensures that Layer 1 of the
Tezos protocol extends its responsibility to enshrined Layer 2
components such as Etherlink. In practice, this means that if an issue
arises in these components that can be resolved or mitigated by
actions at the Layer 1 level, the protocol team will help the Layer 2
team with the same level of commitment as if the issue were within
Layer 1 itself. This principle aims to strengthen trust in Layer 2 and
other enshrined components by guaranteeing not only their security
through Layer 1 but also the commitment of Layer 1 to support and
safeguard their operation when necessary.

[image: Decision Tree in case of issues][image]

Attention:

  To enable a user activated action, more than 2/3rd of the stake has
  to trigger it. Thus one must ensure a large majority of the stake
  has been upgraded.


User-Activated Actions
----------------------


UAPO: *User-Activated Protocol Override*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The process to amend a Protocol Proposal under the governance voting
process.

In the following, we denoted <P> the Protocol Proposal that needs to
be UAPOed.

1. The patched "<P>" Protocol Proposal should be done and validated

2. If it does not create an attack vector, a public MR that is
   targeting alpha and has no relation to <P> nor mentions any issue
   should be ready to be merged ASAP

3. A snapshot and a hash for the patched <P> Protocol are prepared
   with the release manager

4. Migration tests, monitoring, etc should be ready

5. The comm must be ready

6. Then a public merge of the patched <P> Protocol on "master" of
   "tezos/tezos" (this will be public and demand comms to be ready),
   this MR should include the necessary code for the UAPO in the Octez
   shell (which would have already been produced to test the migration
   anyway)

7. Form then a (quick) release of Octez with the UAPO and everything
   can go back to the usual business

Note:

  * "<P>"net must be upgraded and the tool providers might need some
    time to update their tools.

  * Ensure with bakers that a large majority of the stake has been
    upgraded.

  * In terms of communication, we favor transparency.


UAU: *User-Activated Upgrade*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Process to amend the active protocol, to be avoided as much as
possible (particularly on Mainnet), and only occur in an extreme
emergency.

1. The patched "<P>" Protocol Proposal should be done and validated.

2. A beta-like protocol is prepared using the same tools used to
   produce a stabilized protocol.

3. This protocol is patched to include the necessary fixes

4. It is tested accordingly

5. It is snapshotted into a new "proto-<proto name>" protocol

6. The shell is updated to produce the automatic upgrade at a decided
   block (preferably **the last block of a cycle**).

7. Communication must be ready

8. Then, a public merge of the patched <P> protocol on master
   including the Octez shell changes

9. A (quick) release of Octez with the UAU

Note:

  * "<P>"net must be upgraded and the tool providers might need some
    time to update their tools.

  * Ensure with bakers that a large majority of the stake has been
    upgraded.

  * In terms of communication, we favor transparency.


UAAC: *User-Activated Activation Cancelation*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The process to cancel a Protocol Activation at the end of the
governance voting process.

* Prepare a configuration with the UAAC

* Optionally: prepare an Octez release hardcoding the UAAC


KO: Cancel organically
----------------------

This process consists in transparently provide information on the
issue and leave the decision in bakers' hands to reject (or not) the
proposal using the governance process.

Protocol Release Checklist
**************************

This page documents the steps needed for releasing and injecting a
protocol proposal for Tezos. These include:

* technical steps such as releasing code and launching a test network;

* meta-technical steps such as extending the developer documentation;

* non-technical steps such as managing public relations (PR) via
  appropriate communication.

Protocol release is one step of the process of developing a protocol
proposal, see Protocol Proposals.


Summary
=======


Documentation & Public Relations Steps
--------------------------------------

Before protocol proposal time, there must be both a complete technical
description of the changes in the protocol (including all necessary
documentation), and a complete set of blog posts; one blog post must
introduce the proposal to a non-technical audience and describe who
created it, while a second blog post points to the technical
documentation and gives a high level overview of the technical
changes.


Technical Steps
---------------

The release manager, in coordination with both protocol and shell
teams, must ask for the protocol to be released, a test network must
be created, and whatever tests are currently demanded for the protocol
must pass as green.

The new shell should be released before injection occurs (see Release
System), to make it simpler for third parties to run tests before
voting. This is especially necessary if a new environment is required
for the new protocol.


Timeline and Checklist
======================

**Protocol Name:**

The protocol name is chosen by active members of the development team
at least one week before the planned release. *Selecting the name is
the privilege of the whole set of developers who were involved in the
release.*

* Traditionally, the name must be that of an ancient city, and its
  first letter must be the successor of the first letter of the
  previous protocol.

**Communication Preparation:**

During this phase, blog and social network posts are drafted and
reviewed.

**Protocol CI Checklist:**

The release of a new protocol requires some manual configuration in
the CI:

* The new "n+1" protocol must be added to those being built and tested

* The old "n-1" protocol must be removed from those being built and
  tested

This is done by adding and removing the respective protocols to
script-inputs/active_protocol_versions and script-inputs/slim-mode-
dune, which is done by defining the protocols with the right functions
(typically "active" or "frozen") in manifest/product_octez.ml and
running "make -C manifest".

**Final Tests and sign-off:**

In the below, “Protocol Shepherd” is abbreviated “Shepherd”.

* **Shepherd Initiates Final QA Check**:

  * declares feature freeze;

  * checks final gas rescaling;

  * checks final migration code.

* **Tests Are Conducted**

  * **Unit Tests:** All unit tests must pass.

  * **Integration Tests:** All integration tests must pass.

  * **Protocol Migration Test**: the Mainnet migration test must be
    run and must pass.

* **Developer Meeting Agrees to Freeze the Code** (up to bug fixes)

* **Shepherd compiles the doc page with the changelog** (e.g. Protocol
  Carthage), and resets the Alpha changelog, as the diff between
  protocol Alpha and the proposed protocol restarts at zero (e.g.
  !3123)

* **Shepherd Declares We Are Ready For Proposal Creation**

**At this point there should be at least one week left** before
injection to give time for the test network and the release to be
ready and tested before the injection. If some features need to be
dropped to meet this deadline (including because they were not
reviewed thoroughly enough), they should be dropped.

* **Protocol Hash**: a vanity hash for the new protocol is computed
  and chosen by its developers

  * Several developers may create hashes which are meant to capture as
    much of the protocol name as possible; the specific hash that is
    chosen is picked by consensus of the developers involved in the
    release.

* **Release Artifacts are Created**: an MR on "master" is made with
  the result of the protocol snapshot (mostly automated by script
  scripts/snapshot_alpha_and_link.sh) linked into the node and client.

  The final snapshot is produced from the latest version of "proto-
  beta". It is rehashed and renamed. Release material (including
  blogposts and other communication artifacts) should be made ready
  using the final protocol hash and name.

  In the merge request introducing "proto-<proposal name>", **``proto-
  beta`` is deleted from the repository**. In order to avoid a period
  of time where no testnet is easily reachable to test the new
  proposed protocol, this change should be made (as in ‘merged into
  the master branch of tezos/tezos repository’) only once the
  communication and testnets are ready. **Release and** **Injection of
  this protocol amendment proposal should follow minutes/hours
  after.**

  The few additional manual steps to be done on the documentation part
  are listed in meta-issue #2170 (Section "Protocol snapshot"). The
  delegates must also be built in this branch. CI must be run on this
  branch and pass.

  **NB: Make sure NOT to merge this MR about the same day when a
  previous protocol gets activated.** It is recommended to avoid
  merging a snapshot for a new protocol within 1-2 days of another
  protocol's activation, due to significant interactions and
  interferences between the snapshotting process and that for
  upgrading the documentation site to reflect a protocol activation.
  Note that the activation date cannot be modified, but is known at
  least two weeks in advance.

* **Test Network Preparation**: everything before the Spawn Test
  Network section of:
  https://gitlab.com/romain.nl/howtos/-/blob/master/HOWTO-launch-a
  -test-network.md

* **New Shell Release Preparation**: see the Preparation section of:
  https://gitlab.com/romain.nl/howtos/-/blob/master/HOWTO-release-
  tezos.org

* **Release Manager Declares Release Artifacts Final**: If previous
  steps all work, the artifacts are ready for public release.

* Blogpost and Agora post are prepared to announce <X> Protocol
  Proposal release.

**At the time of release: Past here, preparation work is done: we are
at the point of no return.**

* **Agora Posts, Tweets**: Blog posts are published on Tezos Agora
  announcing the release; tweets linking to the Agora posts may go out
  at this time. (The Agora posts go out first to encourage people to
  view Agora as the place to go first and earliest for Tezos
  announcements.)

* **Company Blog Posts, Tweets**: Blog posts are made on company web
  sites 15 minutes to an hour later than the Agora posts (so that the
  Agora post is the first announcement) and tweets pointing to company
  blogs may go out at this time.

* **Shell Release(s)**: new version with the protocol and the
  delegates (and possibly adding a new protocol environment), see the
  Release section of:
  https://gitlab.com/romain.nl/howtos/-/blob/master/HOWTO-release-
  tezos.org

* **Test Network**: the test network for the new protocol is started,
  see https://gitlab.com/romain.nl/howtos/-/blob/master/HOWTO-launch-a
  -test-network.md

* **Injection**: a baker injects the protocol using a Proposal
  operation. This must happen ideally during the first half of a
  Proposal period.

How to Freeze Protocols
***********************

After each protocol activation, old protocols are kept in the
codebase. However large parts of the code developed for the previous
protocol can now be removed to avoid accumulating dead code in the
repository. All daemons, tests associated to old protocols, and RW
(read-write) commands available to the client can be removed.

In this document, "protocol" refers to the protocol that we want to
freeze, N, and "current protocol" refers to its successor, N+1. For
instance, if "protocol" refers to 008, "current protocol" refers to
009.

The various pieces of code to be removed are within directory
"src/proto_<N>_<Hash>/", unless indicated otherwise.


Update Manifest
===============

In manifest/product_octez.ml, look for "let alpha = active
Name.alpha". In one of the lines above it is the declaration of the
protocol you are freezing. Replace "active" by "frozen" for this
protocol. Run the manifest with "make -C manifest".


Remove Accuser, Baker
=====================

These daemons are no longer needed. Thus, the code in
"bin_{accuser,baker}/" can be safely removed.


Remove Protocol Tests
=====================

All tests developed for the protocol can be removed.

These are defined in

* "lib_client/test"

* "lib_protocol/test"


Remove "lib_delegate"
=====================

This code is no longer needed and can be safely removed.


Remove Protocol Tests From Tezt
===============================

The code of Tezt must be updated manually removing the protocol from
the type "Protocol.t" and adapting the tests accordingly.


Remove Testnet From "bin_node"
==============================

Mentions of the protocol's testnets should be removed from the node
executable. In particular the file src/lib_node_config/config_file.ml
should be amended.

The protocol plugin registration module should be removed from
src/bin_node/dune and opam/octez-node.opam.


Remove RW Commands From "lib_client_commands"
=============================================

The client commands at
"proto_XXX/lib_client_commands/client_proto_context_commands.ml"
define both read-only and read-write commands, where reads and writes
refer to accesses to and modifications of the state of the chain. The
RW commands can now be safely removed as they are no longer needed.


Remove Mempool Protocol Plugins
===============================

The Mempool protocol plugin located in
"proto_XXX/lib_plugin/plugin.ml" can be removed.  This implies
removing the plugin code, and remove the registration of the plugin in
the file "proto_XXX/lib_plugin/plugin_registration.ml".

Other plugins should be evaluated case-by-case. At the moment of
writing, the "Mempool" plugin is the only one that can be safely
removed.


Remove Old Docker-Compose Files
===============================

The docker-compose file with the corresponding protocol name can be
removed from the "scripts/docker" directory.


Add an Entry in "CHANGES.rst"
=============================

Add an entry in CHANGES.rst to summarize all changes for the user.

In particular it should mention:

* that the protocol's daemons were removed;

* that the corresponding "--network" alias was removed.

Adding a new protocol environment
*********************************

The economic protocols of Tezos are compiled against a restricted set
of libraries. This is for security reasons (so that, e.g., the
protocol can never be leveraged for accessing files of the host
machine) and safety reasons (so that, e.g., the protocol is immune to
some categories of errors). See the general overview in Protocol
environment.

The set of libraries a protocol is compiled against is called the
*protocol environment*, or simply, the *environment*.

Each protocol is compiled against a given environment, as declared in
the protocol's "TEZOS_PROTOCOL" manifest file. In order to ensure that
old protocols can still be compiled in the future, all the
environments are set in stone: their interface cannot ever be
modified. Consequently, when new features are needed for new
protocols, a whole new environment must be created (otherwise, if no
new feature is needed, a protocol can use the same environment as its
predecessor's).

This page details the process of creating a new environment by copying
the latest environment and building upon it. In the rest of this page,
"<N>" is the version number for the new environment you are creating
and "<N-1>" is the version number for the existing environment that
you are copying from.


Bootstrap
=========

The following steps are roughly the steps taken in the V6 bootstrap MR

1. Copy the existing environment files:

   * Copy the directory "src/lib_protocol_environment/sigs/v<N-1>"
     into "src/lib_protocol_environment/sigs/v<N>".

   * Update the "ocamlformat" configuration file
     src/lib_protocol_environment/sigs/.ocamlformat-ignore.

   * Copy the file "src/lib_protocol_environment/sigs/v<N-1>.in.ml"
     into "src/lib_protocol_environment/sigs/v<N>.in.ml" and change
     any reference from "v<N-1>" to "v<N>" in the copied file.

2. Make the new environment buildable by updating
   manifest/product_octez.ml:

   * Bump the "latest_environment_number" in
     manifest/product_octez.ml.

   * Run "make -C manifest".

3. Copy the existing compatibility layer if any (see details in Struct
   compatibility layer).

   * Update  "src/lib_protocol_environment/structs/tezos_protocol_env
     ironment_structs.ml" to add a new submodule "V<N>" by copying the
     submodule "V<N-1>".

4. Copy and adapt the environment functor:

   * Copy "src/lib_protocol_environment/environment_V<N-1>.ml[i]" to
     "src/lib_protocol_environment/environment_V<N>.ml[i]".

   * Change any reference from "V<N-1>" to "V<N>" in all those copied
     files.

   * Update
     "src/lib_protocol_environment/environment_context_intf.ml".

5. If the protocol signature is expected to change then copy and adapt
   it otherwise leave it as is:

   "Environment_protocol_T_V<X>" is the current protocol signature and
   "<X>" is equal to the environment version that introduces it.

   * Copy
     "src/lib_protocol_environment/environment_protocol_T_V<X>.ml" to
     "src/lib_protocol_environment/environment_protocol_T_V<N>.ml".

   * Change "Environment_protocol_T_V<X>" to
     "Environment_protocol_T_V<N>" in
     "src/lib_protocol_environment/environment_V<N>.ml".

6. Add references to the new environment version number in the rest of
   the code:

   * "src/lib_base/protocol.ml[i]"

   * "src/lib_validation/block_validation.ml"

7. Adapt demo protocols to the new environment:

   * Modify the required environment in
     src/proto_demo_noops/lib_protocol/TEZOS_PROTOCOL and
     src/proto_demo_counter/lib_protocol/TEZOS_PROTOCOL.

   * Run "make -C manifest" to regenerate
     src/proto_demo_noops/lib_protocol/dune and
     src/proto_demo_counter/lib_protocol/dune.

   * Verify they both compile with "dune build
     @src/proto_demo_noops/runtest_compile_protocol" and "dune build
     @src/proto_demo_counter/runtest_compile_protocol".

8. Change the following files, adding references to "V<N>" to match
   the references to "V<N-1>":

   * src/lib_protocol_environment/tezos_protocol_environment.ml

   * "src/lib_protocol_updater/registered_protocol.ml[i]"

   * "src/lib_protocol_compiler/registerer/tezos_protocol_registerer.
     ml[i]"

9. Commit all those changes and open an MR with your changes.

It is recommended that you test your work more comprehensively
offline. To that end, follow the instructions below on how to activate
the environment, and then run the protocol tests locally. Do not
commit the changes or at least, do not push the changes.


Struct compatibility layer
--------------------------

The struct compatibility layer is for providing compatibility between
a signature of the protocol environment (which is set in stone) and
the interface of an external library that provides it (which might
change from version to version). E.g., at the time of the V0
environment the OCaml Stdlib did not include an "Option" module and so
a custom one was provided in the whole of the Tezos project including
the protocol environment; later, when the Tezos project switched to
the now available and standard "Stdlib.Option" module, the struct
compatibility module
"src/lib_protocol_environment/structs/v0_option.ml" was added.

More recent protocol environments generally need fewer struct
compatibility modules. Occasionally, the most recent environment needs
no compatibility layer at all. You can know if this is the case by
checking the file "src/lib_protocol_environment/structs/tezos_protoco
l_environment_structs.ml": if the submodule "V<N>" exists and is not
empty then there is a compatibility layer, otherwise there isn't.

Either way, the instructions in the list above are sufficient for
creating the new environment.


Activating the environment
==========================

The new environment as it stands now is not activated. More precisely,
it cannot be used by any protocol. A few more changes are needed
before it can be used.


When to activate
----------------

This is on purpose: we do not want to release an unfinished
environment because it interferes with the distributed nature of Tezos
protocol development. Specifically, if an unfinished protocol was made
available in a release of the Octez suite, then anyone could propose a
protocol built upon this version. But then further work on the
protocol (to finish it) would create multiple different environments
that have the same name. To avoid this, we only activate the
environment once it is safe.

The new environment should only be activated after the last release
that precedes the injection of the protocol that uses it. Don't worry
too much about this, simply reach out to a release manager and work
with them on the schedule.


How to activate
---------------

To activate the environment, you will need to bump environment version
in:

* "src/bin_client/test/proto_test_injection/TEZOS_PROTOCOL"

* "tezt/tests/voting.ml" (in the embedded "TEZOS_PROTOCOL")

* "src/lib_store/unix/test/test_consistency.ml"

And finally, bump environment version in
"src/proto_alpha/lib_protocol/TEZOS_PROTOCOL", and run "make -C
manifest".

For an example, check the MR in which the environment V6 was
activated.

Additionally, you have to update the documentation of protocol Alpha
to reflect the fact that it now uses environment "V<N>". For that, see
meta-issue #4155, which explains all the necessary changes (don't
worry, the changes are very limited).


Making changes in the environment
=================================

You can make changes to the newly created environment until it is
released. For this purpose release candidates do not count. Below are
examples of changes from previous work on the environment.

* Adding the Result module in environment V3

  * Add the interface file
    "src/lib_protocol_environment/sigs/v3/result.mli"

  * Add a reference to the file in
    "src/lib_protocol_environment/sigs/v3.in.ml"

  * Declare the "Result" module in the functor in
    "src/lib_protocol_environment/environment_V3.ml"

* Updating the data-encoding dependency

  * Provide backwards compatibility layers for older environments

  * Modify existing
    "src/lib_protocol_environment/sigs/v3/data_encoding.mli"

* Rehauling the List module in the environment V3

  * Replace some of the environment modules with a new one (remove old
    files)

  * Remove struct compatibility module (the new interface is identical
    to the one in the most recent library)

RPC How-To
**********

This document explains how to declare, register, and serve RPC entry-
points in the Octez codebase. A lot of the complexity comes from the
following facts:

* Resto was developped in a separate repository

* the code of Octez was split by OS usage (Unix vs others)

* Resto and the RPC stack in general have accumulated significant
  technical debt

For background knowledge about the RPC architecture, see RPC handling
architecture.

The main operations related to the RPC lifetime, explained below, are:
- declaring an RPC (service): describing the service, including a
textual description and the types and de/serialization of arguments
and return value - declaring an RPC directory that bundles multiple
services - registering RPCs: associating a handler to each service -
serving an RPC directory: launching a server listening and serving RPC
requests - calling an RPC: using externals tools to invoke Tezos RPCs


How to declare a service
========================

Declaring a service is the same as declaring a module signature. This
includes a textual description and the types and de/serialization of
its inputs/outputs. The short answer is: call
"{get,post,put,…}_service" in Tezos_rpc.Service . Usage examples
below.

These functions are defined in src/lib_rpc/RPC_service.ml as a wrapper
(for de/serialisation and error management) around "Resto.MakeService"
(from resto/src/resto.ml).

The call returns a "('method,'prefix,'params,'query,'input,'output)
service" where the parameters have the following meaning:

* "'method" (a.k.a. "'m"): one of "[`GET]", "[`POST]", "[`PUT]", etc.
  matching the service’s method. (See
  https://en.wikipedia.org/wiki/HTTP#Request_methods.)

* "'prefix" (a.k.a. "'pr"): the type of arguments passed to the
  service handler, which the client passes as part of the path. Unlike
  "'params" (see next), the "'prefix" parameters are set when
  declaring the directory of services (see later) rather than in this
  here declaration. (See example below.)

* "'params" (a.k.a. "'p"): the type of arguments passed to the service
  handler, which the client passes as part of the path of the URL.
  Think "/blocks/<id>/operations/<index>".

* "'query" (a.k.a. "'q"): the type of arguments passed to the service
  handler, which the client passes as part of the query parameters of
  the URL. Think "?order=ascending&sort=author". (See
  https://en.wikipedia.org/wiki/Query_string.)

* "'input" (a.k.a. "'i"): the type of arguments passed to the service
  handler, which the client passes as the request’s body. (See
  https://en.wikipedia.org/wiki/HTTP_message_body.)

* "'output" (a.k.a. "'o"): the type of values the service handler
  returns. The handler is actually expected to return values of the
  type "'output tzresult" which is what the
  :src`src/lib_rpc/RPC_service.ml` wrapper provides: the services in
  "resto/src/resto.ml" have an additional type parameter for "'error"
  which is hard-coded in src/lib_rpc/RPC_service.ml. (See
  https://en.wikipedia.org/wiki/HTTP_message_body.)

These "service" values are *description* of services: they list the
number and type of arguments, their de/serialisation format, the type
of the return value, and some documentation. They do not have an
associated handler: this is registered separately (see How to register
services by declaring a directory). Think of these values as the items
in a ".mli" file: a description of inputs and outputs without code.


"'prefix" vs "'params"
----------------------

Both the "'prefix" and "'params" type parameters describe arguments
passed to the service handler. They differ in that "'prefix" is
provided by the directory that multiple services are bundled under
whereas "'params" is provided by the service itself.

Thus the prefix can be used for common arguments that a set of
services use. Just a way to avoid repeating some common argument.
E.g., if we wanted to have a version number in all the RPC entry-
points ("/<version>/chain/<chain-id>/heads") that would be the way to
do it.

The prefix can also be used when an argument cannot be described at
the point where the service is declared but it is available when the
service is registered (see How to register services by declaring a
directory) as described in the documentation of "subst" in
resto/src/resto.mli.

The "'prefix" is always a subset of the "'params". Meaning that if
"'prefix" parameters exist, they also appear (in duplicate) in the
"'params". This is enforced by construction of paths (see the module
"Path" in resto/src/resto.ml). The "'prefix" is used to enforce that
the directory does define the necessary parameters (thus you cannot
register a service in a directory without providing the descriptions
for the prefix parameters, see How to register services by declaring a
directory); the "'params" is used to enforce that the service handler
has the matching number of parameters.


Usage example
-------------

src/lib_p2p_services/p2p_services.ml declares all the p2p-related RPC
services. Excerpt:

   let list =
     Tezos_rpc.Service.get_service  (* a GET service *)
       ~description:"List the running P2P connection."
       ~query:Tezos_rpc.Query.empty  (* no qeury parameters *)
       ~output:(Data_encoding.list connection_info_encoding)  (* what the service returns
                                                                 and how it's de/serialised *)
       Tezos_rpc.Path.(root / "network" / "connections")  (* plain path without parameters *)

   let kick =
     Tezos_rpc.Service.delete_service  (* a DELETE service *)
       ~query:wait_query  (* a query parameter defined earlier in the file as a flag (yes/no) *)
       ~output:Data_encoding.empty  (* no value returned *)
       ~description:
         "Forced close of the current P2P connection to the given peer."
       Tezos_rpc.Path.(root / "network" / "connections" /: P2p_peer.Id.rpc_arg)
           (* a path with an argument identifying a peer *)

And the matching ".mli" excerpt:

   val list :
     ( [`GET]  (* method: get *)
     , unit  (* prefix: none *)
     , unit  (* params: none *)
     , unit  (* query: none *)
     , unit  (* input: none *)
     , connection_info list  (* output *)
     ) Tezos_rpc.Service.t

   val kick :
    ( [`DELETE]  (* method: delete *)
    , unit  (* prefix: none *)
    , unit * P2p_peer.Id.t  (* params: one parameter *)
    , < wait : bool >  (* query: one parameter *)
    , unit  (* input: none *)
    , unit  (* output: none *)
    ) Tezos_rpc.Service.t

Note that params (and prefix) parameters are represented as nested
tuples of parameters. Zero parameters is represented as "unit", a
single "x" parameter is represented as "unit * x", two parameters "x"
and "y" are represented as "(unit * x) * y", etc.

Note that query parameters are represented as objects. This helps
naming the different components of the query (it could also have been
a record). In our case, it’s only a single query argument.


Another usage example
---------------------

src/lib_shell_services/chain_services.ml declares all the chain-data-
query RPC services. Excerpt:

   module Levels = struct
     (* define a path for that part of the services, as a kind of hierarchy of
        paths matching a hierarchy of concepts *)
     let path = Tezos_rpc.Path.(path / "levels")

     let checkpoint =
       Tezos_rpc.Service.get_service  (* GET *)
         ~description:"The current checkpoint for this chain."
         ~query:Tezos_rpc.Query.empty  (* no query *)
         ~output:block_descriptor_encoding  (* output de/serialisation *)
         Tezos_rpc.Path.(path / "checkpoint")  (* sub-path *)

     let savepoint = …

     let caboose = …
   end

And the matching ".mli" excerpt:

   type prefix = unit * chain
   …
   module Levels : sig
     val checkpoint :
       ( [`GET]  (* method: get *)
       , prefix  (* prefix: one parameter (defined above) *)
       , prefix  (* params: same as the prefix, no additional service-specific parameters *)
       , unit  (* query: none *)
       , unit  (* input: none *)
       , Block_hash.t * int32  (* output *)
       ) Tezos_rpc.Service.t

     val savepoint : …

     val caboose : …
   end


How to register services by declaring a directory
=================================================

Directories are sets of services, each with a handler. Therefore,
declaring the directory and registering its services is done at the
same time.

More specifically, you:

1. start with the empty directory "Tezos_rpc.Directory.empty"

2. populate the directory by calling the registration functions in
   Tezos_rpc.Directory

   There are a variety of registration functions in
   "Tezos_rpc.Directory" depending on the number of path parameter the
   service has, whether the service can fail with an error or not,
   etc. E.g., "register2" registers a 2-parameter service which may
   fail. E.g., "lwt_register1" registers a 1-parameter service which
   cannot fail (its handler uses lwt as its monad, hence the prefix).

3. may combine multiple directories together by calling
   "Tezos_rpc.Directory.merge".

   Note that the merging of directories may fail by raising an
   exception if there are services registered for conflicting paths.
   You can use the other directory combinator "prefix" to put a whole
   directory under a given namespace. (You can also use "map" to
   provide prefix parameters (see above).)

"Tezos_rpc.Directory" is the module for the file
src/lib_rpc/RPC_directory.ml and it is a thin wrapper around
Resto_directory.Make  which is in resto/src/resto_directory.ml. The
wrapper instantiates the functor with de/serialisation functions and
shadows the "*register*" functions with some error-handling features.


Usage example
-------------

src/lib_p2p/p2p_directory.ml assembles the p2p-related services
defined in src/lib_p2p_services/. More specifically, it provides a
"build_rpc_directory" function which returns a directory of the p2p-
related services.

   let build_rpc_directory net =
     let dir = Tezos_rpc.Directory.empty in
     … (* some registrations *)
     let dir =
       Tezos_rpc.Directory.lwt_register1
         dir  (* the dir being populated *)
         P2p_services.Connections.S.kick  (* the pre-declared service being registered *)
         (fun peer_id q () ->  (* the handler for the service, with the different
                                  parameters: the path parameter, the query parameter *)
           match P2p.pool net with
           | None -> Lwt.return_unit
           | Some pool -> …)
     in
     let dir =
       Tezos_rpc.Directory.register0
         dir
         P2p_services.Connections.S.list
         (fun () () ->
           match P2p.pool net with
           | None -> tzfail P2p_errors.P2p_layer_disabled
           | Some pool -> …)
     in
     … (* more registrations *)
     dir

Note the differences between the two registrations. The "kick"
registration uses "lwt_register1" because it cannot fail (it’s “lwt
only”) and it takes one (1) path parameter. The "list" registration
uses "register0" because it can fail (general case, no prefix) and it
takes zero (0) path parameters.

Also note that the helper functions for registration convert between
the nested-tuples representation of the "'params" parameters and the
curried representation of parameters for the handler function. E.g.,
the handler for "kick" takes a "peer_id" parameter instead of "((),
peer_id)".


Additional usage example
------------------------

src/lib_shell/node.ml brings in multiple directories from different
parts of the code.

   let build_rpc_directory ~node_version ~commit_info node =
     let dir : unit Tezos_rpc.Directory.t ref = ref Tezos_rpc.Directory.empty in
     let merge d = dir := Tezos_rpc.Directory.merge !dir d in
     merge (Chain_directory.build_rpc_directory node.validator) ;
     merge (P2p_directory.build_rpc_directory node.p2p) ;
     …  (* more directories being merged *)
     !dir


How to serve a directory
========================

Serving a directory involves configuring and launching a server that
listens to a port and handles RPC requests.

First, get a "server" value by calling
"Tezos_rpc_http.RPC_server.init_server". This function takes a
directory (see How to register services by declaring a directory).

Then, call "Tezos_rpc_http.RPC_server.launch". This function takes the
"server" value initialised above as well as some server-configuration
parameters (think port number and such). The executable which calls
this function now serves the RPC services registered in the directory.

The responsibilities are handled as follows:

* Tezos_rpc_http.RPC_server  provides a thin wrapper (for
  de/serialisation and logging) around "Resto_server".

* "Resto_server" translates the directory into a callback: it takes an
  HTTP request and finds the matching handler to call.

* "Resto_server" provides a thin wrapper (error management, startup
  and teardown, some de/serialisation glue, etc.) around Cohttp.

* Cohttp does the low-level HTTP management (parsing HTTP requests,
  printing HTTP responses, populating headers, etc.) and delegates the
  actual network management (sockets, connections, etc.) to Conduit.

* Conduit does the bind/accept/etc. dance.

For details about RPC handling and RPC server initialization, see RPC
handling architecture.


Usage example
-------------

src/bin_node/node_run_command.ml spins up the RPC server which is part
of the "octez-node" executable.

   let launch_rpc_server (config : Config_file.t) dir rpc_server_kind addr =
     … (* some things ommitted for scope *)
     let server =
       RPC_server.init_server
         ~cors
         ?acl
         ~media_types:(Media_type.Command_line.of_command_line media_types)
         dir  (* this is the directory of services, passed to the server *)
     in
     … (* some things ommitted for scope *)
     RPC_server.launch
       ~host
       server  (* this is the server (with its directory) being started *)
       ~callback
       ~max_active_connections:config.rpc.max_active_rpc_connections
       mode

Where "dir" is initialised in another part of the code as

   let dir = Node.build_rpc_directory ~node_version ~commit_info node in
   let dir = Node_directory.build_node_directory config dir in
   let dir =
     Tezos_rpc.Directory.register_describe_directory_service
       dir
       Tezos_rpc.Service.description_service
   in


How to call RPCs
================

In principle, you can call RPCs using curl or whichever HTTP client,
but it can be difficult to de/serialise arguments and responses. It is
even more difficult when using the "application/octet-stream" media
type. Although octez-codec, the executable from "src/bin_codec", can
help, it is still difficult and further explanations are beyond the
scope of this document.

Instead, you can use the octez-client executable. This provides some
safety checks, some UI/UX niceties, and some built-in
de/serialisation. The "octez-client" executable uses abstractions
similar to the node’s RPC server in order to make RPC calls.

The stack is:

* "src/bin_client/*" defines the actual UI of the "octez-client"
  binary: the commands, the parameters, etc. These commands use a
  client-context (variable name: "cctxt") (see details) to actually
  make the call.

* src/lib_rpc_http/RPC_client_unix.ml instantiates an "RPC_client"
  (see next item) with the actual underlying calling method. The
  actual underlying calling method is a thin wrapper around the one
  provided by cohttp-client.

* src/lib_rpc_http/RPC_client.ml provides a thin wrapper (error
  management, de/serialisation, some media-type dispatch) around
  "resto_client" to package it into a client-context.

* src/lib_rpc/RPC_context.ml defines a client-context: a simple class
  (in the OOP sense) with methods to perform RPC calls.

  * The main idea behind this abstraction is roughly dependency
    injection: code can handle all the logic of calling RPC entry-
    points and using the returned value but the actual backend is
    passed dynamically as a client-context parameter. This is a
    leftover from a time we were trying to separate the code into
    native-vs-javascript parts and it could be greatly simplified.

  * The methods for performing the calls take a service as an argument
    as well as all the arguments (path, query, body) that the service
    expects. It computes the correct path based on the service
    declaration.


How to register an RPC in the protocol
======================================

Shortly: declaring and registering an RPC in the protocol is the same
as for other services but is done inside the plugin part of the
protocol so the protocol services can be patched without having to
inject a new protocol.

Documenting
***********

This documentation is available online at
https://octez.tezos.com/docs/, and always up to date with branch
"master" on GitLab.


Building the documentation
==========================

You can build the documentation locally or in the CI.


Building the documentation in the CI
------------------------------------

When reviewing a merge request (MR) in the Gitlab interface, you may
build the documentation in the CI without checking out the source
branch of the MR, and without installing Python locally. Proceed as
follows:

* trigger the CI if needed in the home page of the MR, and make sure
  that job "documentation:build_all" under the CI stage "build" is
  being executed

* once the whole CI is finished, check the built documentation in the
  exposed artifacts on the home page of the MR

If you cannot wait for the whole CI to finish, the artifacts are not
yet exposed on the home page of the MR; but click on the CI job
"documentation:build_all", and in the job's page (once finished),
click on "Browse" to retrieve only the doc artifacts.

In both cases, visualize file "docs/_build/index.html".


Building the documentation locally
----------------------------------

To build the documentation locally, you need to install the Python
package manager Poetry. For instructions on how to obtain Python and
Poetry, see the installation instructions for the Python environment.

Another pre-requisite for building the documentation is making sure
that the Octez sources on your branch are compiled, because part of
the documentation is generated by Octez executables. This involves
executing "make" in the parent directory (the repository root). If
this step results in errors, you usually have to restart the compiling
procedure from "make clean" onwards.

Once this is done, you can do:

   make -C docs

The output is generated and available in "docs/_build". It is built by
Sphinx, and uses the Read The Docs theme.


OCaml documentation
-------------------

As part of the above procedure, Odoc is used for OCaml API generation.
You can install Odoc with:

   opam install odoc

Octez generates the API documentation for all libraries in HTML
format. The generated HTML pages are put in "_build/<context>/_doc".
It creates one sub-directory per public library and generates an
"index.html" file in each sub-directory.

The documentation is published online at OCaml APIs - Reference.


Writing documentation
=====================

Online documentation is written in reStructuredText format, also known
as RST. reStructuredText is the default plaintext markup language used
by Sphinx, which is the tool used to compile this format into plain
web pages in HTML format.

For the RST syntax, see the Sphinx RST primer and also the Sphinx
extensions below.


Sphinx extensions
-----------------

Some ad-hoc reference kinds are supported.

* ":package:`name`" or ":package:`text<name>`" points to the "odoc"
  page of the package, checking that the page exists

* ":package-name:`name`" or ":package-name:`text<name>`" just displays
  the package name (no link), checking that the package exists

* ":package-api:`path/to/api-page.html`" or ":package-
  api:`text<path/to/api-page.html>`" points to an API page generated
  by odoc, checking that the page exists. The path is relative to the
  root of the odoc-generated pages (normally,
  "_build/api/odoc/_html"). It must start with a package name,
  optionally followed by a library name, then by a series of nested
  module names, and ended by a page name, usually "index.html". It may
  optionally be suffixed by a section name, using the standard HTML
  "#section" suffix. This role is meant to point to APIs that do not
  correspond to a whole package (for that case, prefer to use the
  ":package:" role).

* ":src:`/path/to/file/or/dir`" or ":src:`text</path/to/file/or/dir>`"
  points to the gitlab source tree viewer. It is not possible to refer
  to a particular line in a file using a line number suffix of the
  form "#Lnnn", because such links are usually too fragile to be used
  in documentation.

* ":opam:`package`" or ":opam:`text<package>`" points to the package
  page on "opam.ocaml.org", version number is supported
  ("package.version")

* ":gl:`[special gitlab reference]`" or ":gl:`text <[special gitlab
  reference]>`" expands and links GitLab special references, like for
  merge requests tezos/tezos!123 (":gl:`tezos/tezos!123`"), issues
  tezos/tezos#999 (":gl:`tezos/tezos#999`") and commits 28309c81
  (":gl:`28309c81`"). The default project and namespace is
  "tezos/tezos". In other words, "tezos/tezos#999", "tezos#999" and
  "#999" all refer to the same thing. Currently supports usernames,
  projects, issues, merge requests, snippets, milestone ids, commits
  and commit ranges. The implementation of this role is in
  docs/_extensions/gitlab_custom_role.py.


Style guidelines
----------------

Currently there are no enforced guidelines about the style in writing
documentation. In particular, the choice of American, British,
Canadian, ... English (alphabetical, non-exhaustive list!) is up to
each contributor. So is the capitalization convention of section
names, and other typesetting aspects. The focus should be on the
contents: on logical structure of documents, on uniform use of terms,
on avoiding incoherencies between pages, and so on.

However, when adding a new page or modifying an existing one, you
should check that your text displays correctly and introduces no new
problems. For that, you should build the documentation (by running
"make" in the "docs" directory), address any new error message, and
check the generated pages ("docs/_build/index.html") in a browser.


Links
~~~~~

When introducing cross-references between documentation pages as well
as references to external resources, please consider using the most
appropriate kind of link:

* When referring to a whole documentation page, you should use a
  ":doc:" role rather than introducing a label at the start of the
  page. Indeed, labels incur an overhead, especially when pages get
  duplicated for different protocol versions. In particular, when
  referring to a page of the currently active protocol, consider using
  "active/" as the directory of that page, instead of a hardcoded
  protocol name.

* When referring to an artifact in the code repository (source file,
  commit, etc.), you may use an appropriate custom or GitLab role (see
  Sphinx extensions) instead of a plain HTML link. Indeed, specific
  roles are checked for correctness more effectively and more
  efficiently than HTML links.


Line breaking
~~~~~~~~~~~~~

When writing documentation in text formats such as RST, it is not
required to respect a maximal line width, such as 80 columns.
Therefore, you may choose between the different line breaking policies
your text editor proposes. However, you should be aware that file
comparison tools such as "diff" tend to output large differences for a
paragraph that has been reformatted after only a small change in one
phrase. Also, reviewing tools such as the one in the GitLab user
interface associate comments and change suggestions to lines, while
these comments and suggestions are usually logically associated with
whole phrases.

For such reasons:

* Some contributors use one line per complete phrase, which allows to
  make rephrasing suggestions more easily in "gitlab", associated to
  this (possibly long) line; and which allows "diff" to isolate
  modified phrases, instead of showing the whole container paragraph
  as modified.

* Other contributors, whose editor breaks lines at a fixed width,
  introduce an extra line break at the end of each phrase. This also
  allows "diff" to isolate modified phrases.

Thus, you may choose your own formatting style, while tolerating
different styles from other contributors.


Images
------

When including images in the documentation:

* commit under Git an image format which can be displayed by Gitlab's
  "View file" feature, such as "png" or "jpeg", to ease reviewing;

* preferably, also commit a source form of the image that can be
  maintained, explaining which tool to use if necessary (in a local
  Readme file, see docs/images/README.md as an example);

* image files are usually placed next to the pages where they are
  used; as an exception, images to be used in several pages from
  different directories, or in toplevel page such as "index.rst",
  should be placed under docs/images/.


Writing executable documentation
--------------------------------

When you are writing documentation containing executable parts, such
as sequences of instructions to install, configure, or launch some
tool, there is sometimes a better way than copying those instructions
from a terminal (where you supposedly tried them before!) to a
documentation page. This better way is to write "executable
documentation". The idea is to write such executable scripts separated
from the documentation, and to automatically copy them in the
documentation whenever it is (re)generated. Executable documentation
allows one to test those scripts, e.g. in CI (continuous integration),
ensuring they work and are up to date with the code and with its
environment.

Typically, Octez installation scripts not only have to evolve with the
Octez codebase, but also with various other evolving resources, such
as OPAM packages, package managers, Linux distributions, and so on. By
continuously testing such installation scripts, executable
documentation allows one to detect problems and fix obsolete
instructions as early as possible, avoiding headaches and frustration,
for new end users and experienced developers alike.

Technically, executable documentation can be created by using the
Sphinx directive literalinclude, which may include whole scripts or
parts of them. For example, the following directive includes a script
fragment detailing a step in compiling the Octez sources:

   .. literalinclude:: compile-sources.sh
     :language: shell
     :start-after: [install packages]
     :end-before: [test executables]

Whenever appropriate, in addition to including the script (fragment)
in the documentation as above, make sure it is regularly tested,
manually and/or within a CI job.


Writing protocol documentation
------------------------------

Writing protocol documentation is a special case because protocol-
related documentation pages are duplicated for several protocol
versions (under directories named as the protocols, e.g.,  "alpha/"),
and possibly also in a protocol-independent part (typically under
directory "shell/").

Besides the need of maintaining several versions of these pages, this
duplication introduces the need to carefully handle documentation
cross-references, in particular to avoid duplicate labels (i.e.,
multiple labels with the same name in different pages) and wrong
references (i.e., escaping from one protocol version into another).

The following rules promote a systematic way of handling documentation
cross-references that avoids introducing such errors.


Definitions
~~~~~~~~~~~

First let us introduce the following definitions:

* A *label* is an identifier defining a specific position in a
  documentation page (typically, before a section name). A *reference*
  is a link to a label, in the same or another page. In Sphinx, labels
  are written ".. _label:" and references are written ":ref: `textual
  description <label>`", or ":ref: `label`". Labels and references are
  case-insensitive.

* A *versioned* label is suffixed by  protocol name (e.g.
  "label_alpha"); an  *unversioned* label doesn't (i.e. just "label")

* A *local* reference is a link from a protocol-specific page to the
  same page or to another protocol-specific page. An *external*
  reference is a reference from a protocol-independent page to a label
  in a protocol-specific page.


Rules
~~~~~

The following simple rules are proposed for safely managing cross-
references:

1. In all but the **current** protocol, any defined label must be
   versioned:

      .. _<label>_<proto>:

2. In the **current** protocol, labels may be versioned (as targets of
   local references), unversioned (as targets of external references),
   or both. The last case is done by defining *two* labels for such
   location:

      ..  _<label>:
      ..  _<label>_<proto>:

3. Any local reference in protocol "<proto>" must be versioned
   "<proto>". This includes references appearing in the currently
   active protocol.

4. External references must be unversioned.

The rationale of the above rules:

* Any label defined in a protocol-specific page must be versioned to
  avoid name conflicts (as by definition the containing page is
  duplicated).

* External references must be unversioned to avoid modifying protocol-
  independent pages when the current protocol is changed.

* Local references in the current protocol could also work if
  unversioned, but when the protocol is changed, they should be
  rewritten as versioned. It is much simpler to enforce the rule that
  all local references in a page for any protocol "<proto>" must be
  versioned "<proto>".


Protocol changes
~~~~~~~~~~~~~~~~

When a new protocol is adopted, its pages must be "linked" with the
protocol-independent pages:

* remove in the old protocol all the unversioned labels (this
  operation is unnecessary if the pages of the old protocol are
  removed altogether)

* add in the new protocol an unversioned label before each versioned
  label

**NB** no rewriting of any reference is needed on protocol changes.

On creating a new protocol proposal version "<proto>" out of alpha:

* rename all versioned labels AND references _alpha in its pages to
  version _<proto>


Rules automation
~~~~~~~~~~~~~~~~

To help enforcing the above cross-referencing rules in protocol-
specific pages, the following scripts are provided under
"docs/scripts":

* "check_proto_xrefs.py": checks the references, and optionally the
  labels, in all pages of a given protocol version

  * can be used at any time, e.g. when changing a protocol-specific
    page

* "add_labels_without_proto.py": adds unversioned labels before each
  versioned label in a protocol-specific page

  * can be used when a new protocol is adopted, to "link" its
    documentation into protocol-independent pages

* "remove_labels_without_proto.py": removes unversioned labels in a
  protocol-specific page

  * can be used when a new protocol is adopted for "unlinking" the
    pages of the old protocol, only if those pages are not removed
    altogether

Moreover, the script "scripts/snapshot_alpha.sh", used to create a new
protocol proposal version "<proto>" out of alpha integrates renaming
of labels and references.


Documenting protocols
~~~~~~~~~~~~~~~~~~~~~

Due to the duplication of the documentation for multiple protocol
versions, the following extra guidelines should be observed.

* In principle, protocol-independent pages should only refer to the
  currently active protocol. Indeed, until newer protocols are
  adopted, there is no guarantee that their features will be part of
  Tezos someday. Note that there is a symbolic link called "active"
  within the documentation folder pointing to the currently active
  protocol directory. Use it whenever appropriate to avoid introducing
  hardcoded protocol numbers.

* When modifying the pages of a given protocol version, you might have
  to also modify it for later versions. Otherwise, when newer
  protocols are adopted, your changes will vanish! In particular, when
  fixing a problem in the documentation of the current protocol (e.g.
  adding a term in the glossary), you might have to fix it also for
  the candidate protocol (if there is one under the voting procedure)
  and for the Alpha protocol under development (assuming that the
  features of the candidate protocol will be inherited by or proposed
  in another form in Alpha).

* As there is a considerable overhead for maintaining protocol-
  specific pages, think twice before duplicating a page as protocol-
  specific. Does this page really refer to the protocol? If yes, does
  *all* the page refer to the protocol? If the answer to the last
  question is "no", consider splitting the page in two parts,
  respectively protocol-specific and protocol-independent. This kind
  of splitting is however unadvised when there are many local cross-
  references between the parts; in this case, keeping everything in a
  same page may avoid introducing many labels (this is why the
  glossary pages are not split into shell and protocol pages).

Platform Development tools
**************************

The development and maintenance process of the Tezos platform is
facilitated by some specialized tools. Some of these tools are
included in the Octez repository, because of a close coupling with the
code itself (see Scope of the Octez repository for the policy of
selecting such tools). They provide, for example, support for
profiling or for benchmarking different subsystems of Octez.

On the other hand, contributing to the development of the Octez
repository requires installing some additional infrastructure, which
is not needed by regular Octez users. For instance, developers need
Python for building the documentation.

The tools for platform developers, as well as the configuration of the
additional infrastructure, are documented in the following pages.

* Profiling the Octez node

  * Memory profiling the OCaml heap

  * Memory profiling the C heap

  * Performance profiling

  * The Profiler module

* Benchmarking with Snoop

  * Architecture of "octez-snoop"

  * Snoop usage tutorial

  * In-depth usage example: more control over your benchmark

  * Rewriting Micheline with "tezos-micheline-rewriting"

  * Snoop and the Michelson Interpreter

* Time measurement PPX

  * Getting started

  * Compatible OCaml Attributes

  * Instrumenting the octez-node executable

* Python Environment

  * Installation

  * Adding new dependencies

* Pre-Commit Hook

Profiling the Octez node
************************

If you want to profile Octez, several solutions are available.


Memory profiling the OCaml heap
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* Memory profiling the OCaml heap

  * Using Memtrace

  * Using Statmemprof-emacs


Memory profiling the C heap
^^^^^^^^^^^^^^^^^^^^^^^^^^^

* Memory profiling the C heap


Performance profiling
^^^^^^^^^^^^^^^^^^^^^

* Performance profiling


Profiler module
^^^^^^^^^^^^^^^

* The Profiler module

  * Example of use

  * Example file

  * Create a new profiler

  * Use the profiler

Memory profiling the OCaml heap
*******************************

The profiler offers specific support for displaying the memory
footprint of the OCaml heap. This is how you can use it


Using Memtrace
==============

* Install an OCaml switch that support "statmemprof": until the
  "4.14.2" or after the "5.3"

* Install "memtrace" and "memtrace_viewer".

* Add the following line at the beginning of the program you want to
  profile "Memtrace.trace_if_requested ();"

* Recompile your program

* Add the following environment variable to your executable command:
  "MEMTRACE=trace.ctf ./my_executable"

* "trace.ctf" will be filled with data from the execution of the
  program and can be analysed with "memtrace-viewer trace.ctf".

* Open your favorite web browser and go to "localhost:8080"


Using Statmemprof-emacs
=======================

This section is legacy and might not work any more with recent OCaml
versions.

* Install an OCaml switch with the "statmemprof" patch: "4.04.2
  +statistical-memprof" or "4.06.0+statistical-memprof"

* Install "statmemprof-emacs".

* Enable loading "statmemprof" into the node.

  Add the "statmemprof-emacs" package as a dependency to the main
  package, and add "let () = Statmemprof_emacs.start 1E-4 30 5" to the
  "node_main.ml" file.

  Arguments:

  * "sampling_rate" is the sampling rate of the profiler. Good value:
    "1e-4".

  * "callstack_size" is the size of the fragment of the call stack
    which is captured for each sampled allocation.

  * "min_sample_print" is the minimum number of samples under which
    the location of an allocation is not displayed.

* Load sturgeon into emacs, by adding this to your ".emacs":

   (let ((opam-share (ignore-errors (car (process-lines "opam" "config" "var" "share")))))
     (when (and opam-share (file-directory-p opam-share))
     (add-to-list 'load-path (expand-file-name "emacs/site-lisp" opam-share))))

   (require 'sturgeon)

* Launch the node then connect to it with sturgeon.

  If the process is launched with pid "1234" then

     M-x sturgeon-connect octez-nodememprof.1234.sturgeon

  (tab-completion works for finding the socket name)

Memory profiling the C heap
***************************

Several language-independent tools are available for displaying the
memory usage of the heap in any compiled program.

* Install "valgrind" and "massif-visualizer"

   valgrind --tool=massif octez-node run ...

* Stop with "Ctrl-C" then display with

   massif-visualizer massif.out.pid

Performance profiling
*********************

If you are interested to know how much time is spent in different
functions in your program, this is how to proceed.

* Install "perf" (the "linux-perf" package for debian).

  If the package does not exist for your current kernel, a previous
  version can be used. Substitute the "perf" command to "perf_4.9" if
  your kernel is 4.9).

* Either:

  * Run the node, find the pid.

    Attach "perf" with "perf record -p pid -F 99 --call-stack dwarf".

    Then stop capturing with "Ctrl-C". This can represent a lot of
    data. Don't do that for too long. If this is too much you can
    remove the "--call-stack dwarf" to get something more manageable,
    but interpreting the information can be harder.

  * Let "perf" run "octez-node": "perf record -g -F 99 --call-
    graph=dwarf -- ./octez-node run ..."

    This will write the output in file "perf.data" after having
    stopped the node with "Ctrl-C".

  In both cases, the "-F" argument specifies the frequency of sampling
  of data (in hertz).

  If too much data is generated, use a smaller value. If data is not
  precise enough, try using a higher value.

* display the result with "perf report", or use a more advanced
  visualizer (recommended). Such visualizers include:

  * flamegraph: command-line tool for generating flamegraphs (example
    for octez-node)

  * gprof2dot: command-line tool for generating callgraphs (example
    for octez-node)

  * hotspot: a GUI for the "perf" tool

The Profiler module
*******************

Octez offers a profiler module that is better suited than external
tools like "perf" for the monadic programming model of Lwt and for
generating traces, as it offers more control and is able to handle the
elusive nature of Lwt.


Example of use
==============

This step-by-step guide shows how a profiler is created, plugged and
used in Octez (based on the "lib_shell" profiling):


Example file
============

We'll start with this simple file:

   let read_int ic =
     let rec aux acc =
       match input_char ic with
       | ' ' | '\n' -> acc
       | c -> aux ((10 * acc) + (Char.code c - 48))
     in
     aux 0

   let read_test_int () =
     let ic = open_in sample in
     let max = ref 0 in
     try
       while true do
         let e = read_int ic in
         if e > !max then max := e
       done
     with End_of_file ->
       close_in ic ;
       Format.eprintf "%d@." !max

   let read_test_line () =
     let ic = open_in sample in
     let max = ref 0 in
     try
       while true do
         input_line ic |> String.split_on_char ' '
         |> List.iter (fun e ->
                let e = int_of_string e in
                if e > !max then max := e)
       done
     with End_of_file ->
       close_in ic ;
       Format.eprintf "%d@." !max

   let () =
     read_test_int () ;
     read_test_line () ;
     read_test_scanf ()

We have three functions that we would want to profile to see which one
is faster and to see what's taking the longest time in each.


Create a new profiler
=====================

Start by creating a unique profiler:

   let read_profiler = unplugged ()

You can see it as an API to the profiling machinery that isn't able to
do anything useful for now. Why is that? Because you need to attach it
to a "backend".

A "backend" is defined in two steps:

* Select a "Driver" (like "this driver writes text files in an unix
  filesystem")

* Define a specific "instance" of a "Driver" (like "this driver will
  write in this file with this level of detail")

Octez already provides two "Drivers":

   val auto_write_as_txt_to_file : (string * Profiler.verbosity) Profiler.driver

   val auto_write_as_json_to_file : (string * Profiler.verbosity) Profiler.driver

These "Drivers" are specifically crafted to write text or JSON files
in an Unix filesystem. As you can see, they expect two 'arguments', a
"string" (where to write) and a "Profiler.verbosity" (the maximum
level of detail/verbosity expected from the profiler).

   type verbosity = Notice | Info | Debug

For this tutorial, we will use "verbosity = Info", but developers are
encouraged to use their preferred option.

To choose the verbosity of the profiler at runtime, the "PROFILING"
environment variable is used. It follows the same pattern as the
"TEZOS_LOG" environment variable (see Logging features) except that it
doesn't execute profilers which verbosity is not set.

Starting a node with
"PROFILING='shell_profiling->Notice;mempool_profiling->Debug'" will
set the maximum verbosity for these two profilers and not execute the
rest of the profilers.

We can now easily create an instance for a "Driver":

   let read_instance =
     Tezos_profiler.Profiler.instance
       Tezos_profiler_backends.Simple_profiler.auto_write_as_txt_to_file
       ("read_profiling.txt", Profiler.Info)

We just need one last thing. We have a "read_profiler" and a
"read_instance" that writes in "read_profiling.txt" but they are not
connected. That's where the following function needs to be used:

   val plug : profiler -> instance -> unit

So we just need to

   Profiler.plug read_profiler read_instance

And voilà!, when we'll call functions attached to "read_profiler" the
reports will be properly written in "read_profiling.txt" (It should be
noted here that a profiler can be plugged to multiple instances
allowing to write infos in different files or with different format).

Since it would be a little bit annoying to call each functions by
giving it "read_profiler" as a parameter, the "Profiler" module offers
a convenient function that creates a module allowing to call all the
"read_profiler" functions without providing it:

   val wrap : profiler -> (module GLOBAL_PROFILER)

This will give access to the functions in the Profiler.GLOBAL_PROFILER
module .


Use the profiler
================

We can now wrap our profiler to create a module that we will use to
profile our code.

   module Read_profiler = (val Profiler.wrap read_profiler)

Since "Read_profiler" is already plugged to "read_instance", calling
"Read_profiler" functions will work as expected.

We can now start monitoring our code.

First, we need to plug our profilers to backends to gather results.
Plugging a profiler to one or more backends is done with
"PROFILING_BACKENDS=<list of ; separated backends>". For example, you
could run a node with

   PROFILING="*->Debug" PROFILING_BACKENDS="txt; json" octez-node run

We can start with a simple change:

   let () =
     Read_profiler.plug instance ;
     (Read_profiler.record_f ~verbosity:Info ("read_test_line", []) @@ fun () -> read_test_line ()) ;
     (Read_profiler.record_f ~verbosity:Info ("read_test_int", []) @@ fun () -> read_test_int ()) ;
     Read_profiler.record_f ~verbosity:Info ("read_test_scanf", []) @@ fun () -> read_test_scanf ()

Note: the "Read_profiler" functions take a "(id, metadata) : string *
metadata" as their first parameter: - "id" is used to create sections
and represent the part being profiled by the function - "metadata" is
used by backends that need specific informations (like Prometheus)

Looking at the result gives us:

   2024-09-18T09:46:46.376-00:00 read_test_line .... 1           42.707ms 100% +0.002ms
   2024-09-18T09:46:46.419-00:00 read_test_int ..... 1          106.481ms 100% +42.865ms
   2024-09-18T09:46:46.525-00:00 read_test_scanf ... 1          122.623ms 100% +149.439ms

Now that we know that the profiler outputs correctly to our chosen
file, let's monitor our functions more precisely:

   let profiler = Read_profiler.unplugged ()

   module Read_profiler = (val Profiler.wrap profiler)

   let instance =
     Profiler.instance Tezos_profiler_backends.Simple_profiler.auto_write_as_txt_to_file
       ("/tmp/test_profiler.txt", Profiler.Info)

   let read_int ic =
     let rec aux acc =
       match input_char ic with
       | ' ' | '\n' -> acc
       | c -> aux ((10 * acc) + (Char.code c - 48))
     in
     aux 0

   let read_test_int () =
     Read_profiler.record_f  ~verbosity:Info ("read_test_int", []) @@ fun () ->
     let ic = Read_profiler.aggregate_f ~verbosity:Info ("open_in", []) @@ fun () -> open_in sample in
     let max = ref 0 in
     try
       while true do
         Read_profiler.aggregate_f ~verbosity:Info ("read_int", []) @@ fun () ->
         read_int ic |> fun e -> if e > !max then max := e
       done
     with End_of_file ->
       Read_profiler.aggregate_f ~verbosity:Info ("close_in", []) @@ fun () ->
       close_in ic ;
       Format.eprintf "%d@." !max

   let read_test_line () =
     Read_profiler.record_f ~verbosity:Info ("read_test_line", []) @@ fun () ->
     let ic = Read_profiler.aggregate_f ~verbosity:Info ("open_in", []) @@ fun () -> open_in sample in
     let max = ref 0 in
     try
       while true do
         Read_profiler.span_f ~verbosity:Info (["input_line"], []) @@ fun () ->
         input_line ic |> String.split_on_char ' '
         |> List.iter (fun e ->
                let e = int_of_string e in
                if e > !max then max := e)
       done
     with End_of_file ->
       Read_profiler.aggregate_f ~verbosity:Info ("close_in", []) @@ fun () ->
       close_in ic ;
       Format.eprintf "%d@." !max

   let read_test_scanf () =
     Read_profiler.record_f ~verbosity:Info ("read_test_scanf", []) @@ fun () ->
     let ic =
       Read_profiler.aggregate_f ~verbosity:Info ("open_in", []) @@ fun () -> Scanf.Scanning.open_in sample
     in
     let max = ref 0 in
     try
       while true do
         Read_profiler.mark ~verbosity:Info (["Scanf.bscanf"], []) ;
         Scanf.bscanf ic "%d " (fun i -> i) |> fun e -> if e > !max then max := e
       done
     with End_of_file ->
       Read_profiler.aggregate_f ~verbosity:Info ("close_in", []) @@ fun () ->
       Scanf.Scanning.close_in ic ;
       Format.eprintf "%d@." !max

   let () =
     Read_profiler.plug instance ;
     read_test_line () ;
     read_test_int () ;
     read_test_scanf ()

You should obtain something like this:

   2024-09-18T09:19:13.555-00:00
   read_test_line ... 1           44.079ms 101% +0.002ms
     close_in ....... 1            0.049ms 101%
     input_line ..... 1002        42.992ms 100%
     open_in ........ 1            0.013ms 109%
   2024-09-18T09:19:13.599-00:00
   read_test_int .... 1         1660.119ms 100% +44.247ms
     close_in ....... 1            0.048ms  99%
     open_in ........ 1            0.035ms 100%
     read_int ....... 1003003    807.536ms 101%
   2024-09-18T09:19:15.259-00:00
   read_test_scanf .. 1          300.168ms  99% +1s704.432ms
     Scanf.bscanf ... 1003003
     close_in ....... 1            0.063ms 102%
     open_in ........ 1            0.036ms 100%

The execution time of "read_int" seems off. Replacing the following
lines:

   Read_profiler.aggregate_f ~verbosity:Info ("read_int", []) @@ fun () ->
   read_int ic |> fun e -> if e > !max then max := e

By:

   Read_profiler.mark ~verbosity:Info (["read_int"], []) ;
   read_int ic |> fun e -> if e > !max then max := e

Gives a completely different result:

   2024-09-18T09:25:23.516-00:00
   read_test_line ... 1           44.440ms 100% +0.001ms
     close_in ....... 1            0.081ms 100%
     input_line ..... 1002        43.287ms 100%
     open_in ........ 1            0.014ms 102%
   2024-09-18T09:25:23.560-00:00
   read_test_int .... 1          267.466ms 100% +44.609ms
     close_in ....... 1            0.046ms 103%
     open_in ........ 1            0.008ms 102%
     read_int ....... 1003003
   2024-09-18T09:25:23.828-00:00
   read_test_scanf .. 1          289.068ms 100% +312.139ms
     Scanf.bscanf ... 1003003
     close_in ....... 1            0.055ms 103%
     open_in ........ 1            0.037ms  98%

This is expected because "aggregate"-like and "record"-like functions
will call "Unix.gettimeofday" for each occurrence. Here we're calling
it "1003003" times and losing a lot of time. Out of the 1660ms spent
in "read_int", almost 900ms were spent computing "Unix.gettimeofday".
You can either choose to keep these slowdowns while making sure you
know where they happen and why they happen or you can choose simpler
functions like "mark" that just count a number of occurrences.

As you can see, though, monitoring your code with the "Profiler" can
lead to extreme slowdowns. The first solution is to call
"Profiler.plug" only when needed. Since your "profiler" is just an
API, calling its functions has little to no impact. The other solution
is to use the "PPX" specially crafted for the "Profiler".


PPX Profiler
^^^^^^^^^^^^

* Profiler PPX

  * Why a Profiler PPX?

  * How to use this PPX?

  * Structure of an attribute

  * Adding functionalities

Profiler PPX
************

The profiler PPX is an OCaml preprocessing tool allowing to use the
"Profiler" functions in any part of the code. The PPX allows to choose
at compile time if we want to have the profiler enabled or not.

This guide is aimed at explaining how to use the PPX, not the
"Profiler". For explanations about the profiler, please look at this
page.

*Both the PPX rewriter (our specific instance defined in* the
Ppx_profiler module ) *and the PPX, the mechanism used to preprocess
files in OCaml, are called PPX in this document.*


Why a Profiler PPX?
===================

After having created a profiler, you can create a wrapper around it in
any file with

   module Profiler = (val Profiler.wrap my_profiler)

That can be used like this

   Profiler.aggregate_f ~verbosity:Info ("advertise mempool", []) @@ fun () ->
   advertise pv_shell advertisable_mempool;

   let* _res =
     Profiler.aggregate_s ~verbosity:Info ("set mempool", []) @@ fun () ->
     set_mempool pv_shell our_mempool
   in ...

The issue with this direct approach is that it creates wrapper around
functions that may hinder their functioning. This is not wanted since
the profiler is only used by devs hence the use of a PPX that is
controlled by an environment variable.

   TEZOS_PPX_PROFILER=<value> make

Will preprocess the code before compiling. The "value" field is
described Enabled drivers.

This will allow to preprocess

   () [@profiler.record {verbosity = Info} "merge store"] ;

into

   Profiler.record ~verbosity:Info ("merge store", []) ;
   () ;

It should be noted that a "Profiler" module has to be available and
has to have the signature of the Profiler.GLOBAL_PROFILER module  that
can be obtained with "module Profiler = (val Profiler.wrap
my_profiler)".

Of course you can create any module with this signature but in case
you didn't name it "Profiler" (let's say you name it "My_profiler")
you'll have to declare your PPX attributes with a "profiler_module"
field:

   () [@profiler.record {verbosity = Info; profiler_module = My_profiler} "merge store"] ;

This will be preprocessed into

   My_profiler.record ~verbosity:Info ("merge store", []) ;
   () ;


How to use this PPX?
====================

There are three types of functions in the Profiler library.


1. Inline functions
-------------------

These functions are (for details about them, look at the The Profiler
module document)

* "aggregate : verbosity:verbosity -> string * metadata -> unit"

* "mark : verbosity:verbosity -> string list * metadata -> unit"

* "record : verbosity:verbosity -> string * metadata -> unit"

* "stamp : verbosity:verbosity -> string * metadata -> unit"

* "stop : unit -> unit"

The PPX allows to replace

   Profiler.stop ();
   Profiler.record ~verbosity:Info ("merge store", []);
   ...

with

   ()
   [@profiler.stop]
   [@profiler.record {verbosity = Info} "merge store"] ;
   ...

You can also decompose it to be sure of the evaluation order:

   () [@profiler.stop] ;
   () [@profiler.record {verbosity = Info} "merge store"] ;
   ...


2. Wrapping functions
---------------------

These functions are:

* "aggregate_f : verbosity:verbosity -> string * metadata -> (unit ->
  'a) -> 'a"

* "aggregate_s : verbosity:verbosity -> string * metadata -> (unit ->
  'a Lwt.t) -> 'a Lwt.t"

* "record_f : verbosity:verbosity -> string * metadata -> (unit -> 'a)
  -> 'a"

* "record_s : verbosity:verbosity -> string * metadata -> (unit -> 'a
  Lwt.t) -> 'a Lwt.t"

* "span_f : verbosity:verbosity -> string list * metadata -> (unit ->
  'a) -> 'a"

* "span_s : verbosity:verbosity -> string list * metadata -> (unit ->
  'a Lwt.t) -> 'a Lwt.t"

The PPX allows to replace

   (Profiler.record_f ~verbosity:Info ("read_test_line", []) @@ fun () -> read_test_line ())
   ...

with

   (read_test_line () [@profiler.record_f {verbosity = Info} "read_test_line"])
   ...


3. Custom values
----------------

This PPX library provides a special construct, which basically acts as
a "#ifndef TEZOS_PPX_PROFILER" / "#else":

* "expr_if_ppx_not_used [@profiler.overwrite expr_if_ppx_used]"

This construct will be preprocessed as "expr_if_ppx_not_used" if you
are not using the PPX, or "expr_if_ppx_used" if you are.

If you want to write a custom function that will use the intial value
instead of simply removing it from the code, there are two functions
that allow you to use a wrapper.

Both "profiler.wrap_f" and "profiler.wrap_s" (the "Lwt.t" variant)
allow you to wrap a delayed version of the initial value (using "fun
() -> ...") with a custom function.

This will rewrite

   let wrapper expr = do_something (); expr ()
   let _ = expr [@profiler.wrap_f wrapper]
   ...

into

   let wrapper expr = do_something (); expr ()
   let _ = wrapper (fun () -> expr)
   ...


Structure of an attribute
=========================

An attribute is a decoration attached to the syntax tree that allow
the PPX to preprocess some part of the AST when reading them. It is
composed of two parts:

   [@attribute_id payload]

An attribute is attached to:

* "@": the closest node (expression, patterns, etc.),

  "let a = "preprocess this" [@attr_id payload]", the attribute is
  attached to ""preprocess this""

* "@@": the closest block (type declaration, class fields, etc.),

  "let preprocess this = "and this" [@@attr_id payload]", the
  attribute is attached to the whole value binding

* "@@@": *floating attributes are not used here*

The grammar for attributes can be found in this page.

In the case of our PPX, the expected values are the following.


"attribute_id"
--------------

Allows to know the kind of functions we want to use (like
"@profiler.mark" or "@profiler.record_s") and to link our PPX to all
the "attribute_ids" it can handle. *The use of* "profiler." *allows to
make sure we don't have any conflict with another PPX.*


"payload"
---------

The payload is made of two parts, the first one being optional:

   payload ::= record? args

   record ::= { fields }

   fields ::= field ; fields | empty

   field ::=
     | verbosity = (Notice | Info | Debug)
     | profiler_module = module_ident
     | metadata = <(string * string) list>
     | driver_ids = <(Prometheus | OpenTelemetry | Text | Json) list>

   args ::= <string> | <string list> | <function application> | ident | empty

As an example:

   f x [@profiler.aggregate_s {verbosity = Info} g y z] ;
   g x [@profiler.span_f {verbosity = Debug; profiler_module = Prof} "label"]
   ...

will be preprocessed as

   Profiler.aggregate_s ~verbosity:Info (g y z) @@ f x ;
   Prof.span_f ~verbosity:Debug ("label", []) @@ g x
   ...


Enabled drivers
---------------

When enabling the ppx with "TEZOS_PPX_PROFILER=<value>", "value" can
have two possible types:

* A dummy one, all attributes will be preprocessed except the ones
  with a non-empty "driver_ids" field

* A list of driver ids like "prometheus; opentelemetry" that will
  allow to preprocess attributes:

  * with an empty "driver_ids" field

  * with a "driver_ids" field where one of the driver ids is also
    present in "value"


Adding functionalities
======================

To add a function that needs to be accepted by our PPX (let's say we
want to add "my_new_function" that was recently added to the
"Profiler" module) the following files need to edited:

* "src/lib_ppx_profiler/rewriter.ml":

  * Add a "my_new_function_constant" to "Constants"

  * Add this constant to "Constants.constants"

  * Add "My_new_function of content" to "Rewriter.t"

  * Add a "my_new_function key location" constructor with its accepted
    payloads (usually "Key.Apply", "Key.Ident" and "Key.List" or
    "Key.String")

* If this function needs to accept a new kind of payload (like an
  integer) you'll need to edit "src/lib_ppx_profiler/key.ml" and the
  "extract_key_from_payload" function in "Rewriter" (you can look at
  the ppxlib documentation)

* "src/lib_ppx_profiler/expression.ml" where you'll just need to add
  "Rewriter.my_new_function" to the "rewrite" function

Benchmarking with Snoop
***********************

If you have a piece of code for which you'd like to construct a model
predictive of its performance, "octez-snoop" is the tool to help you
do that. This tool allows to benchmark any given piece of OCaml code
and use these measures to fit cost models predictive of execution
time.

It is in particular used to derive the functions in the Michelson gas
cost API , computing the gas costs in the Tezos protocol.


Architecture of octez-snoop
^^^^^^^^^^^^^^^^^^^^^^^^^^^

* Architecture of "octez-snoop"

  * High-level description

  * Code organization

  * Defining benchmarks: the "Generator" module

  * Defining a predictive model: the "Model" module

  * The "Measure" module

  * Computing parameter fits: the "Inference" module


Using octez-snoop by example
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* Snoop usage tutorial

  * Step 1: Defining the benchmark

  * Step 2: Running the benchmark, inferring parameters, and
    generating code

  * Step 3: checking the generated files

  * Step 4: options

  * Using Snoop in the development cycle


octez-snoop: going further for more control
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* In-depth usage example: more control over your benchmark

  * Step 1: Checking the timer

  * Step 2: Benchmarking

  * Step 2.5: (optional) Removing outliers

  * Step 3: Fitting the model

  * Step 4: Generating code


Rewriting Micheline terms
^^^^^^^^^^^^^^^^^^^^^^^^^

* Rewriting Micheline with "tezos-micheline-rewriting"

  * Constructing Micheline terms

  * Manipulating paths in Micheline terms

  * Pattern-matching Micheline

  * Performing substitutions

  * An example?


Writing your very own benchmarks and models for the Michelson interpreter
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* Snoop and the Michelson Interpreter

  * Interpreter Workload

  * Interpreter Models

  * Interpreter benchmarks

  * Testing

Architecture of "octez-snoop"
*****************************

The following figure describes the main functionalities and data
processed by "octez-snoop", to be read from top to bottom. The boxed
nodes represents the various kinds of data processed by the tool,
while the unboxed items represent computational steps.

[image]

The code architecture of "octez-snoop" is itself divided in the
following main packages:

* "bin_snoop" is the main binary (you can have a look at the manual).

* "tezos-benchmark" is a library for performing measurements, writing
  models and infering parameters for these models.

* "tezos-micheline-rewriting" is used to perform rewriting of
  Micheline terms (documentation available here). It is mainly used
  when writing protocol-specific benchmarks but is independent from
  the protocol.

There are other packages containing shell-specific and protocol-
specific benchmarks, these are not documented here.

Here, we will focus on the "tezos-benchmark" library, which is the
core of the tool.


High-level description
======================

"octez-snoop" is a tool for benchmarking and fitting statistical
models which predict the performance of any piece of code of interest.

More concretely, let us consider a piece of code for which we wish to
predict its performance. To understand the performance profile of this
piece of code, we must execute it with different arguments, varying
the size of the problem to be solved. As "the size of the problem to
be solved" is a long expression, we will use the shorter term
*workload* for that.

The notion of workload is abstract here, and indeed, it is not
necessarily a scalar. Here are a few examples of workloads:

* Timer benchmarks measure the latency of the timer itself and the
  associated workloads record nothing.

* IO benchmarks measure the execution time of performing specific read
  and write patterns to the underlying key-value store and the
  associated workloads record the size of the storage as well as the
  parameters (bytes read/written, length of keys, etc) of these
  accesses.

* Translator benchmarks measure the execution time of various pieces
  of "Script_ir_translator" (the *translator* for short, which handles
  typechecking/unparsing of code and data) as well as Micheline
  serialization, and corresponding workloads record the size of the
  typechecked/unparsed/(de)serialized terms.

* Michelson benchmarks measure the execution time of the interpreter
  on specific programs and the associated workloads record the list of
  all executed instructions together with, for each instruction, the
  sizes of the operands as encountered on the stack.

Once this notion of workload is clear, we can describe Snoop's user
interface.

Using "tezos-benchmark" requires to provide, for each benchmark, the
following main items:

* a type of execution "workload";

* a statistical model, corresponding to a function which to each
  "workload" associates an expression (possibly with free variables)
  denoting the predicted execution time for that workload. In simple
  cases, the model consists in *a single* expression computing a
  predicted execution time for any given workload.

* A family of pieces of code (i.e. closures) to be benchmarked, each
  associated to its "workload". Thus, each closure contains the
  application of a piece of a code to arguments instantiating a
  specific workload. We assume that the execution time of each closure
  has a well-defined distribution. In most cases, these closures
  correspond to executing *a single* piece of code of interest with
  different inputs.

From this input, "tezos-benchmark" can perform for you the following
tasks:

* perform the timing measurements;

* infer the free parameters of the statistical model;

* display the results of benchmarking and inference;

* generate code from the model.


Code organization
=================

The data type that wraps everything up is the module type
"Benchmark.S". The main items required by this type are:

* "create_benchmarks", a function that must generate closures and
  their associated workloads (packed together in type "'workload
  Generator.benchmark").

* "models", a list of statistical models that we'd like to fit to
  predict the execution time of the piece of code of interest.

The library is meant to be used as follows:

* define a "Benchmark.S", which requires

  * constructing benchmarks

  * defining models, either pre-built (via the "Model" module) or from
    scratch (using the "Costlang" DSL);

* generate empirical timing measurements using
  "Measure.perform_benchmark";

* given the data generated, infer parameters of the models using
  "Inference.make_problem" and "Inference.solve_problem";

* exploit the results:

  * input back the result of inference in the model to make it
    predictive

  * plot the data ("tezos-benchmark" can generate CSV)

  * generate code from the model ("Codegen" module)

Modules implementing the "Benchmark.S" signature can also be
registered via the "Registration.register" function which makes them
available to "octez-snoop", a binary that wraps these features under a
nice CLI.


Defining benchmarks: the "Generator" module
===========================================

The "Generator.benchmark" type defines the interface that each
benchmark must implement. At the time of writing, this type specifies
three ways to provide a benchmark (but more could easily be added):

   type 'workload benchmark =
     | Plain : {workload : 'workload; closure : unit -> unit} -> 'workload benchmark
     | With_context : {
         workload : 'workload;
         closure : 'context -> unit;
         with_context : 'a. ('context -> 'a) -> 'a;
       } -> 'workload benchmark
     | With_probe : {
         workload : 'aspect -> 'workload;
         probe : 'aspect probe;
         closure : 'aspect probe -> unit;
       }
         -> 'workload benchmark


Plain benchmarks
----------------

The "Plain" constructor simply packs a workload and a closure
together. The implied semantics of this benchmark is that the
"closure" is a stateless piece of code, ready to be executed thousands
of times by the measure infrastructure.


With_context benchmarks
-----------------------

The "With_context" constructor allows to define benchmarks we require
to set up and cleanup a *context*, shared by all executions of the
closure. An example (which prompted the addition of this feature) is
the case of storage benchmarks, where we need to create a directory
and set up some files before executing a closure containing e.g. a
read or write access, after which the directory must be removed.


With_probe benchmarks
---------------------

The "With_probe" constructor allows fine-grained benchmarking by
inverting control: the user is in charge of calling the pieces of code
to be benchmarked using the provided "probe". The definition of a
probe consists in a small object with three methods:

   type 'aspect probe = {
     apply : 'a. 'aspect -> (unit -> 'a) -> 'a;
     aspects : unit -> 'aspect list;
     get : 'aspect -> float list;
   }

The intended semantics of each method is as follows:

* calling "probe.apply aspect f" executes "f", performing e.g. a
  timing measurement of "f"'s execution time and returns the result of
  the evaluation. The measurement is associated to the specified
  "aspect" in a side-effecting way.

* "probe.aspects" returns the list of all aspects.

* Finally, "probe.get aspect" returns all the measurements associated
  to "aspect".

Note that "With_probe" benchmarks do not come with a fixed workload,
but rather with an aspect-indexed family of workloads. This reflects
the fact that this kind of benchmark can measure several different
pieces of code in the same run, each potentially associated to its own
cost model.

The function "Measure.make_timing_probe" provides a basic probe
implementation. The unit test in
"src/lib_benchmark/test/test_probe.ml" contains an example.


Defining a predictive model: the "Model" module
===============================================

As written above, the "Benchmark.S" signature also requires a list of
*models* (note that users only interested in measures of execution
time can leave this list empty). At the time of writing, "tezos-
benchmark" only handles *linear models*.


Linear models: a primer
-----------------------

We aim at predicting the cost (typically, execution time) for various
parts of the codebase. To do this, we must first come up with a
*model*. These cost models take as input some notion of "size"
(typically a vector of integers) and output a prediction of execution
time (or, up to unit conversion, a quantity of gas). If S is the
abstract set of sizes, we're trying to infer a function of type S
\rightarrow \mathbb{R}_{\ge 0} from a finite list of examples (s_n,
t_n)_n \in (S \times \mathbb{R}_{\ge 0})^\ast which minimizes some
error criterion. This is an example of a **regression** problem.

Note that since S is typically not finite, S \rightarrow
\mathbb{R}_{\ge 0} is an infinite-dimensional vector space. We will
restrict our search to a n-dimensional subset of functions f_\theta,
with \theta \in \mathbb{R}^n, of the form

   f_\theta = \sum_{i=1}^n \theta_i g_i

where the (g_i)_{i=1}^n is a **fixed** family of functions g_i : S
\rightarrow \mathbb{R}_{\ge 0}. An n-dimensional linear cost model is
entirely determined by the g_i.

Enumerating the currying isomorphisms, a linear model can be
considered as:

1. a **linear** function \mathbb{R}^n \multimap (S \rightarrow
   \mathbb{R}_{\ge 0}) from "meta" parameters to cost functions;

2. a function S \rightarrow (\mathbb{R}^n \rightarrow \mathbb{R}_{\ge
   0}) from sizes to linear forms over "meta" parameters;

3. a function S \times \mathbb{R}^n \rightarrow \mathbb{R}_{\ge 0}.

The two first forms are the useful ones. The first form is useful in
stating the inference problem: we seek \theta that minimizes some
empirical error measure over the benchmark results. The second form is
useful as it allows to transform the linear model in vector form, when
applying the size.


The "Costlang" DSL
------------------

The module "Costlang" defines a small language in which to define
terms having both free and bound variables. The intended semantics for
free variables is to stand in for variables to be inferred during the
inference process (corresponding to \theta_i in the previous section).
The language is defined in tagless final style. If this does not ring
a bell, we **strongly** recommend you take a look at
https://okmij.org/ftp/tagless-final/index.html in order to make sense
of the rest of this section. The syntax is specified by the
"Costlang.S" module type:

   module type S = sig
     type 'a repr
     type size
     val true_ : bool repr
     val false_ : bool repr
     val int : int -> size repr
     val float : float -> size repr
     val ( + ) : size repr -> size repr -> size repr
     val ( - ) : size repr -> size repr -> size repr
     val ( * ) : size repr -> size repr -> size repr
     val ( / ) : size repr -> size repr -> size repr
     val max : size repr -> size repr -> size repr
     val min : size repr -> size repr -> size repr
     val log2 : size repr -> size repr
     val free : name:Free_variable.t -> size repr
     val lt : size repr -> size repr -> bool repr
     val eq : size repr -> size repr -> bool repr
     val shift_left : size repr -> int -> size repr
     val shift_right : size repr -> int -> size repr
     val lam : name:string -> ('a repr -> 'b repr) -> ('a -> 'b) repr
     val app : ('a -> 'b) repr -> 'a repr -> 'b repr
     val let_ : name:string -> 'a repr -> ('a repr -> 'b repr) -> 'b repr
     val if_ : bool repr -> 'a repr -> 'a repr -> 'a repr
   end

In a nutshell, the type of terms is "type 'a term = \pi (X : S). 'a
X.repr", i.e. terms must be thought of as parametric in their
implementation, provided by a module of type "S".

It must be noted that this language does not enforce that built terms
are linear (in the usual, not type-theoretic sense) in their free
variables: this invariant must be currently enforced dynamically. The
"Costlang" module defines some useful functions for manipulating terms
and printing terms:

* "Costlang.Pp_impl" is a simple pretty printer,

* "Costang.Eval_impl" is an evaluator (which fails on terms having
  free variables),

* "Costlang.Eval_linear_combination_impl" evaluates terms which are
  linear combinations in their free variables to vectors
  (corresponding to applying a size parameter to the second curried
  form in the previous section),

* "Costlang.Subst" allows to perform substitution of free variables,

* "Costlang.Hash_cons" allows to manipulate hash-consed terms,

* "Costlang.Beta_normalize" allows to beta-normalize...

Other implementations are provided elsewhere, e.g. for code or report
generation.


Definition of cost models: the "Model" module
---------------------------------------------

The "Model" module provides a higher-level interface over "Costlang",
and pre-defines widely used models. These pre-defined models are
independent of any specific workload: they need to be packaged
together with a conversion function from the workload of the benchmark
of interest to the domain of the model. The "Model.make ~conv ~model"
function does just this.


The "Measure" module
====================

The "Measure" module is dedicated to measuring the execution time of
closures held in "Generator.benchmark" values and turn these into
timed workloads (i.e. pairs of workload and execution time). It also
contains routines to remove outliers and to save and load workload
data together with extra metadata.


Measuring execution time of "Generator.benchmark" values
--------------------------------------------------------

The core of the functionality is provided by the
"Measure.perform_benchmark" function.

   val perform_benchmark :
     Measure.options -> ('c, 't) Tezos_benchmark.Benchmark.poly -> 't workload_data

Before delving into its implementation, let's examine its type. A
value of type "('c, 't) Tezos_benchmark.Benchmark.poly" is a first
class module where "'c" is a type variable corresponding to the
configuration of the benchmark and "'t" is a variable corresponding to
the type of workloads of the benchmark. Hence "perform_benchmark" is
parametric in these types.

Under the hood, this functions calls to the "create_benchmarks"
function provided by the first class module to create a list of
"Generator.benchmark" values. This might involve loading from
benchmark-specific parameters from a JSON file if the benchmark so
requires. After setting up some benchmark parameters (random seed, GC
parameters, CPU affinity), the function iterates over the list of
"Generator.benchmark" and calls
"Measure.compute_empirical_timing_distribution" on the closure
contained in the "Generator.benchmark" value.  This yields an
empirical distribution of timings which must be determinized: the user
can pick either a percentile or the mean of this distribution. The
function then records the execution time together with the workload
(contained in the "Generator.benchmark") in its list of results.


Loading and saving benchmark results
------------------------------------

The "Measure" module provides functions "save" and "load" for
benchmark results. Concretely, this is implemented by providing an
encoding for the type "Measure.measurement" which corresponds to a
"workload_data" together with some meta-data (CLI options used,
benchmark name, benchmark date).


Removing outliers from benchmark data
-------------------------------------

It can happen that some timing measurement is polluted by e.g. another
process running in the same machine, or an unlucky scheduling. In this
case, it is legitimate to remove the tainted data point from the data
set in order to make fitting cost models easier. The function
"Measure.cull_outliers" is dedicated to that:

   val cull_outliers : nsigmas:float -> 'workload workload_data -> 'workload workload_data

As its signature suggests, this function removes the workloads whose
associated execution time is below or above "nsigmas" standard
deviations of the mean. **NB** make a considerate use of this
function, do not remove data just because it doesn't fit your model.


Computing parameter fits: the "Inference" module
================================================

The inference subsystem takes as input benchmark results and
statistical models and fits the models to the benchmark results.
Abstractly, the benchmark results consist of a list of pairs  "(input,
outputs)" for an unknown function while the statistical model
corresponds to a parameterised family of functions. The goal of the
inference subsystem is to find the parameter corresponding to the
function that best fits the relation between inputs and outputs.

In our case, the "inputs" correspond to workloads and the "outputs" to
execution times, as described in some length in previous sections.

The goal of the "Inference" module is to solve the regression problem
described in the primer on linear models. As inputs, it takes a cost
model and some empirical data under the form of a list of workloads as
produced by the "Measure" module (see the related section).
Informally, the inference process can be decomposed in the two
following steps:

* transform the cost model and the empirical data into a matrix
  equation A x = T where the input dimensions of A (i.e. the columns)
  are indexed by free variables (corresponding to cost coefficients to
  be inferred), the output dimensions of A are indexed by workloads
  and where T is the column vector containing execution times for each
  workload;

* solve this problem using an off-the-shelf optimization package,
  yielding the solution vector x assigning execution times to the free
  variables.

Before looking at the code of the "Inference" module, we consider for
illustrative purposes a simpler case study.


Case study: constructing the matrices
-------------------------------------

We'd like to model the execution time of an hypothetical piece of code
sorting an array using merge sort. We *know* that the time complexity
of merge sort is O(n \log{n}) where n is the size of the array: we're
interested in predicting the actual execution time as a function of n
for practical values of n.

We pick the following cost model:

   \text{cost}(n) = \theta_0 + \theta_1 \times n \log{n}

Our goal is to determine the parameters \theta_0 and \theta_1. Using
the Costlang DSL, this model can be written as follows:

   module Cost_term = functor (X : Costlang.S) ->
   struct
     open X
     let cost_term =
       lam ~name:"n"
       @@ fun n ->
       free ~name:"theta0" + (free ~name:"theta1" * n * log2 n)
   end

Assuming we performed a set of benchmarks, we have a set of timing
measurements corresponding to pairs (n_i, t_i)_i where n_i and t_i
correspond respectively to the size of the array and the measured
sorting time for the i th benchmark.

By evaluating the model cost on each n_i, we get a family of linear
combinations \theta_0 + \theta_1 \times n_i \log{n_i}. Each such
linear combination is isomorphic to the vector (1, n_i \log{n_i}).
These vectors correspond to the row vectors of the matrix A and the
durations t_i form the components of the column vector T.

In terms of code, this corresponds to applying n_i to "cost_term" and
beta-reducing. The "Inference" module defines a hash-consing partial
evaluator "Eval_to_vector":

   module Eval_to_vector = Beta_normalize (Hash_cons (Eval_linear_combination_impl))

All these operations (implemented in tagless final style) are defined
in the "Costlang" module. "Beta_normalize" beta-normalizes terms,
"Hash_cons" shares identical subterms and
"Eval_linear_combination_impl" transforms an evaluated term of the
form "free ~name:"theta0" + (free ~name:"theta1" * n_i * log2 n_i)"
into a vector mapping ""theta0"" to "1" and "theta1" to "n_i * log2
n_i".

Applying "cost_term" to a constant "n_i" in tagless final form
corresponds to the following term:

   module Applied_cost_term = functor (X : Costlang.S) ->
   struct
     let result = X.app Cost_term(X).cost_term (X.int n_i)
   end

and performing the partial evaluation is done by applying
"Eval_to_vector":

   module Evaluated_cost_term = Applied_cost_term (Eval_to_vector)

The value "Evaluated_cost_term.result" corresponds to the row vector i
of the matrix A.


Structure of the inference module
---------------------------------

We now describe in details the two main functionalities of the
"Inference" module:

* making regression problems given a cost model and workload data;

* solving regression problems.


Making regression problems
~~~~~~~~~~~~~~~~~~~~~~~~~~

As explained in the previous section, a regression problem corresponds
to a pair of matrices A and T. This information is packed in the
"Inference.problem" type.

   type problem =
     | Non_degenerate of {
         lines : constrnt list;
         input : Scikit.Matrix.t;
         output : Scikit.Matrix.t;
         nmap : NMap.t;
       }
     | Degenerate of {predicted : Scikit.Matrix.t; measured : Scikit.Matrix.t}

Let's look at the non-degenerate case. The "input" field  corresponds
to the "A" matrix while  the "output" field corresponds to the "T"
matrix. The "nmap" field is a bijective mapping between the dimensions
of the matrices and the variables of the original problem. The "lines"
field is an intermediate representation of the problem, each value of
type "constrnt" corresponding to a linear equation in the variables:

   type constrnt = Full of (Costlang.affine * quantity)

The function "make_problem" converts a model and workload data (as
obtained from the Measure module) into an "Inference.problem". Let's
look at the signature of this function:

   val make_problem :
     data:'workload Measure.workload_data ->
     model:'workload Model.t ->
     overrides:(string -> float option) ->
     problem

The "data" and "model" arguments are self-explanatory. The "overrides"
argument allows to manually set the value of a variable of the "model"
to some fixed value. This is especially useful when the value of a
variable can be determined from a separate set of experiments. The
prototypical example is how the timer latency is set (see the snoop
usage example).

The job performed by "make_problem" essentially involves applying the
cost model to the workloads, as described in the previous section.


Solving the matrix equation
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once we have a "problem" at hand, we can solve it using the
"solve_problem" function:

   val solve_problem : problem -> solver -> solution

Here, "solver" describes the available optimization algorithms:

   type solver =
     | Ridge of {alpha : float; normalize : bool}
     | Lasso of {alpha : float; normalize : bool; positive : bool}
     | NNLS

The "Lasso" algorithm works well in practice. Setting the positivity
constraint to "true" forces the variables to lie in the positive
reals. At the time of writing, these are implemented as calls to the
Python "Scikit-learn" library. The "solution" type is defined as
follows:

   type solution = {
     mapping : (Free_variable.t * float) list;
     weights : Scikit.Matrix.t;
   }

The "weights" field correspond to the raw solution vector to the
matrix problem outlined earlier. The "mapping" associates the original
variables to their fit.


Parameter inference for sets of benchmarks
------------------------------------------

As hinted before, benchmarks are not independent from one another: one
sometimes needs to perform a benchmark for a given piece of code,
estimate the cost of this piece of code using the inference module and
then inject the result into another inference problem. For short
chains of dependencies this is doable by hand, however when dealing
with e.g. more than one hundred Michelson instructions it nice to have
an automated tool figuring out the dependencies and scheduling the
inference automatically.

"octez-snoop" features this. The "infer parameters" command is
launched in "full auto" mode when a *directory* is passed to it
instead of a simple workload file. The tool then automatically scans
this directory for all workload files, compute a dependency graph from
the free variables and performs a topological run over this dependency
graph, computing at each step the parameter fit and injecting the
results in the subsequent inference problems. The dependency graph
computation can be found in the "Dep_graph" module of "bin_snoop".

Snoop usage tutorial
********************

We present a typical benchmark workflow using "octez-snoop". We'll
consider the case of the "blake2b" hashing function, which is used
among other things to hash blocks, operations and contexts:

   Tezos_crypto.Blake2B.hash_bytes : ?key:bytes -> bytes list -> Tezos_crypto.Blake2B.t

At the time of writing, this function is a thin wrapper which
concatenates the list of bytes and passes it to the "blake2b"
implementation provided by HACL*.


Step 1: Defining the benchmark
==============================

Benchmarks correspond to OCaml modules implementing the "Benchmark.S"
signature. These must then be registered via the
"Registration.register" function. Of course, for this registration to
happen, the file containing the benchmark and the call to
"Registration.register" should be linked with "octez-snoop". See
Architecture of octez-snoop for complementary details.

We'll define the benchmark module chunk by chunk and describe each
part. For a starter, names are associated to various entities, and we
use namespaces to help organize them.

   open Tezos_benchmark

   let ns = Namespace.(make root "example")

Benchmarks are referenced by "name". The "info" field is a brief
description of the benchmark. "module_filename" will help users
finding where the benchmark is implemented simply using the command
line interface. If we need to generate OCaml code to a specific
location, we can set it with the "generated_code_destination" field.
Finally, there's also a system of "tags" that allows listing
benchmarks by kind.

   module Blake2b_bench : Benchmark.S = struct
     let name = ns "Blake2b_example"
     let info = "Illustrating tezos-benchmark by benchmarking blake2b"
     let module_filename = __FILE__
     let generated_code_destination = None
     let tags = ["example"]

Typically, a benchmark will depend on a set of parameters for the
samplers which generate input data to the function being benchmarked.
These parameters correspond to the type "config". A "default_config"
is provided, which can be overridden by specifying a well-formatted
JSON file. Reading and writing this file is made possible by defining
a "config_encoding" using the data-encoding library.

   type config = {max_bytes : int}
   let default_config = {max_bytes = 1 lsl 16}
   let config_encoding =
     let open Data_encoding in
     conv
       (fun {max_bytes} -> max_bytes)
       (fun max_bytes -> {max_bytes})
       (obj1 (req "max_bytes" int31))

Benchmarking involves measuring the execution time of some piece of
code and using the recorded execution time to fit a model. As
explained in Architecture of octez-snoop, a model is in fact a
function of three parameters: a "workload", the vector of free
parameters to be fitted, and a name for future reference. The
"workload" corresponds to the information on the input of the function
being benchmarked required to predict its execution time. Typically,
it corresponds to some notion of "size" of the input. In order to be
saved to disk, we must define a "workload_encoding" as well. The
"workload" type is abstract seen from the outside of the module,
however, for plotting purposes, it is necessary to exhibit a vector-
like structure on these workloads. The "workload_to_vector" function
maps workloads to sparse vectors. If one is not interested in
plotting, this function can be made to always return
"Sparse_vec.String.zero".

   type workload = {nbytes : int}
   let workload_encoding =
     let open Data_encoding in
     conv
       (fun {nbytes} -> nbytes)
       (fun nbytes -> {nbytes})
       (obj1 (req "nbytes" int31))
   let workload_to_vector {nbytes} =
     Sparse_vec.String.of_list [("nbytes", float_of_int nbytes)]

We expect the execution time of "Blake2b.hash_bytes" to be
proportional to the number of bytes being hashed, with possibly a
small constant-time overhead. Hence, we pick an "affine" model. The
"affine" model is generic, of the form \text{affine}(n) = \theta_0 +
\theta_1 \times n with \theta_i the free parameters. One must explain
how to convert the "workload" to the argument "n". This is the purpose
of the "conv" parameter.

   let models =
     [ ( "blake2b",
         Model.make
           ~conv:(fun {nbytes} -> (nbytes, ()))
           ~model:
             (Model.affine
                ~name
                ~intercept:(Free_variable.of_namespace (ns "blake2b_const"))
                ~coeff:(Free_variable.of_namespace (ns "blake2b_ns_p_byte"))) ) ]

Finally, we can define the actual benchmark. The function to be
defined is "create_benchmarks", which expects to be given an
"rng_state", a "bench_num" and a "config" and returns a list of
suspensions, each suspension yielding a benchmark when evaluated.

One might wonder why this particular signature has been chosen,
instead of returning directly a list of benchmarks, or simply
requiring a benchmark generator to be defined. The reasons for this
are:

* the current signature allows for setup code to be shared by all
  benchmarks being generated (not the case here);

* returning a list of suspensions allows to delay the sampling process
  and the memory allocation associated to benchmark generation until
  actually needed, hence preventing memory leaks.

The auxiliary function "blake2b_benchmark" is in charge of preparing a
"closure", corresponding to a call to "Blake2b.hash_bytes" applied to
a random "bytes", and the associated "workload", containing the size
of the random "bytes". We want benchmarks to be easily replayable
given a seed, hence the closure-generation function is parameterized
with an explicit "rng_state" of type "Random.State.t".

     let blake2b_benchmark rng_state config () =
       let nbytes =
         Base_samplers.sample_in_interval
           rng_state
           ~range:{min = 1; max = config.max_bytes}
       in
       let bytes = Base_samplers.uniform_bytes rng_state ~nbytes in
       let workload = {nbytes} in
       (* The closure here is the piece of code to be benchmarked. *)
       let closure () = ignore (Tezos_crypto.Blake2B.hash_bytes [bytes]) in
       Generator.Plain {workload; closure}
     let create_benchmarks ~rng_state ~bench_num config =
       List.repeat bench_num (blake2b_benchmark rng_state config)
   end (* module Blake2b_bench *)

This concludes the definition of the benchmark. Let's register it:

   let () = Registration.register (module Blake2b_bench)


Step 2: Running the benchmark, inferring parameters, and generating code
========================================================================

Now, we are ready to run our benchmark. All we need is the full name
of the benchmark, which involves namespaces. Fortunately, "octez-
snoop" enjoys a few commands to help us if we are not sure what the
final name of the benchmark is.

   octez-snoop list all benchmarks | grep blake2b

Which prints:

   example/Blake2b_example: Illustrating tezos-benchmark by benchmarking blake2b

We can also query more information about the benchmark, resulting from
the registration process.

   octez-snoop display info for benchmark example/Blake2b_example

And here is what we get.

   Name:
       example/Blake2b_example
   Filename:
       src/lib_benchmark/example/blake2b.ml
   Generated code destination:
       Destination not specified
   Info:
       Illustrating tezos-benchmark by benchmarking blake2b
   Tags:
       example
   Models:
       blake2b:
           Aggregated model containing the following abstract models:
               example/Blake2b_example
               builtin/timer_model

Models and other entities that Snoop manipulates can be displayed, and
the Snoop manual has a dedicated section that we can explore.

Also, we need to install some Python libraries before going further.

   pip install scikit-learn statsmodels

Now we can proceed with our initial goal, and ask Snoop to perform
some measurements and generate OCaml code that reflects the gas cost
of running "blake2b" depending on its input.

   octez-snoop generate code for benchmarks example/Blake2b_example --out-dir /tmp/snoop_results

The tool is quite verbose, but we will not detail what it is telling
us here; this is the purpose of the Snoop in-depth example section of
the documentation. Two lines are worth noticing though.

   Adding solution example/blake2b_ns_p_byte := 0.976187
   Adding solution example/blake2b_const := 295.080202

These are the values that "octez-snoop" has inferred for the
parameters of the model we declared, and based on the measurements it
performed. The obtained values are highly dependent on the
architecture of the computer and the processes running in parallel
while measuring the execution time.

What Snoop did was to:

* run the benchmark on a certain number of random inputs, and a
  certain number of times for each input;

* infer values for the model parameters so that the model fits as
  closely as possible the measures obtained at the previous step;

* generate an OCaml function representing the model, where rational
  values are approximated using integers manipulation;

* plot the model and the measurements together in a report.


Step 3: checking the generated files
====================================

"octez-snoop" created several files in "/tmp/snoop_results" (the
argument of the "out-dir" option). Let's have a look at two of them.

"blake2b.ml" contains the gas cost function: it estimates the
computational time of running "blake2b" on an input of the given size.

   (* Do not edit this file manually.
      This file was automatically generated from the models registered at
      src/<protocol>/lib_benchmarks_proto/interpreter_model.ml.
      If you wish to update a function in this file,
      a. update the corresponding model, or
      b. define a new function in michelson_v1_gas_costs.ml, potentially referencing this one. *)

   [@@@warning "-33"]

   module S = Saturation_repr

   (* Inference model name: blake2b *)

   (* model example/Blake2b_example *)
   (* fun size -> (295.080202248 + (0.976187336999 * size)) *)
   let cost_Blake2b_example size =
     let open S.Syntax in
       let size = S.safe_int size in
       let v0 = size in
       (S.safe_int 300) +
         ((((v0 lsr 1) + (v0 lsr 2)) + (v0 lsr 3)) + (v0 lsr 4))

The rational values of the gas parameters have been rounded using
integer arithmetics. The file can be copied and pasted in the
protocol, and the cost function can be called at a specific location
so that we can account for gas when running "blake2b".

Another file of interest is "blake2b.tex" that we can convert to PDF
format.

   pdflatex blake2b.tex

The document shows some plots, like this one:

[image]

In abscissa, we find the size of inputs randomly chosen during the
measurements, and in ordinate is the computation time. The blue
circles are the values measured by Snoop when running "blake2b", and
the red squares are the values obtained for the declared model with
the parameters inferred by Snoop. This allows to evaluate how good the
model is: the more the two overlap, the more precise we are.


Step 4: options
===============

Some of our benchmarks can take a long time to run. "octez-snoop"
offers options to vary the number of random values or the number of
times they are measured, using options "--nsamples" and "--bench-num"
(see the manual). The default values are 300 random values and 100
measurements for each. We can specify lower numbers to reduce
benchmark time, but it will also degrade the precision of the results.

Another option of particular interest is "--plot". It can be used to
automatically open windows at the end of the whole process. They will
show the same plots as in the report generated at the previous step,
but we can interact with the plots, which is very convenient when they
are multi-dimensional (i.e. when the complexity model relies on more
parameters).


Using Snoop in the development cycle
====================================

So when are we supposed to use Snoop? There are two cases.


Developing a new feature in the protocol
----------------------------------------

When a feature is added in the protocol, it must account for the gas
that bakers and attesters will spend by running the feature. Here is a
typical workflow:

* developers implement the feature;

* they also implement the corresponding benchmarks (usually in
  "src/proto_alpha/lib_benchmarks_proto");

* they generate the cost function from the benchmarks using Snoop;

* they copy the resulting, generated code to the appropriate location
  in the protocol (potentially a new file);

* they create a merge request;

* when the request is merged, the gas team will automatically see new
  benchmark results. Then, they will update the cost function with the
  results obtained from running the benchmarks on the reference
  machine for gas.


Updating dependencies or code
-----------------------------

When developers update a library that Tezos depends on, or modify a
protocol feature in such a way that gas may be impacted, they should
check so by running Snoop before and after the modifications.

The usual steps would be to:

* use the "list all benchmarks" and "display info for benchmark"
  commands to find the concerned benchmarks;

* run the benchmark and update the resulting cost function with the
  code generated by Snoop;

* open a merge request and let the gas team know that gas has changed;

* when the request is merged, the gas team will update the cost
  function with the results obtained from running the benchmarks on
  the reference machine for gas.

In-depth usage example: more control over your benchmark
********************************************************

This section borrows the example from the Snoop tutorial on the
"blake2b" hashing function, but relies on Snoop sub-commands for
better control over the benchmark process (corresponding to Step 2 in
the tutorial). The objectives and the output files are the same as in
the tutorial: benchmarking, inferring gas parameter values and
generating the corresponding OCaml code, but sub-commands help to
understand each sub-step, especially in case of unexpected results.


Step 1: Checking the timer
==========================

Before we perform the benchmarks, we need to ensure that the system
timer is sufficiently precise. This data is also useful to subtract
the latency of the timer for benchmarks of very small duration (which
is not required here). We invoke the tool on the built-in benchmark
"TIMER_LATENCY" and specify (through "--bench-num") that we want only
one closure to benchmark (since all closures are identical for this
benchmark) but to execute this closure "100000" times (through "--
nsamples").

   octez-snoop benchmark TIMER_LATENCY and save to timer.workload --bench-num 1 --nsamples 100000

The tool returns the following on standard output:

   Benchmarking with the following options:
   { options = { flush_cache=false;
                 stabilize_gc=false;
                 seed=self-init;
                 bench #=1;
                 nsamples/bench=100000;
                 determinizer=percentile 50;
                 cpu_affinity=none;
                 minor_heap_size=262144 words;
                 config directory=None };
      save_file = timer.workload;
      storage = Mem }
   Using default configuration for benchmark TIMER_LATENCY
   {}
   benchmarking 1/1
   stats over all benchmarks: { max_time = 25.000000 ; min_time = 25.000000 ; mean_time = 25.000000 ; sigma = 0.000000 }

This commands measures "100000" times the latency of the timer, that
is the minimum time between two timing measurements. This yields an
empirical distribution on timings. The tool takes the 50th percentile
(i.e. the median) of the empirical distribution and returns the
result: 25ns latency. This is reasonable. Since there's only one
benchmark (with many samples), the standard deviation is by definition
zero. One could also run many benchmarks with fewer samples per
benchmark:

   octez-snoop benchmark TIMER_LATENCY and save to timer.workload --bench-num 1000 --nsamples 100

This yields on standard output:

   Benchmarking with the following options:
   { options = { flush_cache=false;
                 stabilize_gc=false;
                 seed=self-init;
                 bench #=1000;
                 nsamples/bench=100;
                 determinizer=percentile 50;
                 cpu_affinity=none;
                 minor_heap_size=262144 words;
                 config directory=None };
      save_file = timer.workload;
      storage = Mem }
   Using default configuration for benchmark TIMER_LATENCY
   {}
   benchmarking 1000/1000
   stats over all benchmarks: { max_time = 40.000000 ; min_time = 23.000000 ; mean_time = 24.130000 ; sigma = 0.653529 }

This is consistent with the previous results.

A reliable timer should have a latency of the order of 20 to 30
nanoseconds, with a very small standard deviation. It can happen on
some hardware or software configurations that the timer latency is of
the order of *microseconds* or worse: this makes benchmarking short-
lived computations impossible.


Step 2: Benchmarking
====================

If the results obtained in the previous section are reasonable, we can
proceed to the generation of raw timing data. We want to invoke the
"Blake2b_example" benchmark and save the resulting data to
"./blake2b.workload". We want "500" distinct random inputs, and for
each input we will perform the timing measurement "3000" times. The "
--determinizer" option specifies how the empirical timing distribution
corresponding to the per-input "3000" samples will be converted to a
fixed value: here we pick the 50th percentile, i.e. the median (which
happens to also be the default, so this option could have been
omitted). We also use an explicit random seed in case we want to
reproduce the exact same benchmarks. If not specified, the PRNG will
self-initialize using an unknown seed.

   octez-snoop benchmark Blake2b_example and save to blake2b.workload --bench-num 500 --nsamples 3000 --determinizer percentile@50 --seed 12897

Here's the output:

   Benchmarking with the following options:
   { options = { flush_cache=false;
                 stabilize_gc=false;
                 seed=12897;
                 bench #=500;
                 nsamples/bench=3000;
                 determinizer=percentile 50;
                 cpu_affinity=none;
                 minor_heap_size=262144 words;
                 config directory=None };
      save_file = blake2b.workload;
      storage = Mem }
   Using default configuration for benchmark Blake2b_example
   { "max_bytes": 65536 }
   benchmarking 500/500
   stats over all benchmarks: { max_time = 71957.000000 ; min_time = 284.000000 ; mean_time = 34750.532000 ; sigma = 20155.604394 }

Since the size of inputs varies a lot, the statistics over all
benchmarks are less useful.


Step 2.5: (optional) Removing outliers
======================================

It is possible to remove outliers from the raw benchmark data. The
command is the following:

   octez-snoop remove outliers from data ./blake2b.workload above 3 sigmas and save to blake2b-cleaned.workload

In this particular example, the data seems clean though:

   Measure.load: loaded ./blake2b.workload
   Removing outliers.
   Stats: { max_time = 71925.000000 ; min_time = 289.000000 ; mean_time = 34988.436000 ; sigma = 20766.341788 }
   Validity interval: [-27310.589365, 97287.461365].
   Removed 0 outliers out of 500 elements.

The best defense against outliers is to have clean data in the first
place: use a stable environment for benchmarking.


Step 3: Fitting the model
=========================

We can now proceed to inferring the free parameters from the model
using the data. At the time of writing, the tool offloads the
regression problem to the scikit-learn (aka sklearn) and the
statmodels Python libraries: install them before proceeding.

   pip install scikit-learn statsmodels

Let's execute the following command:

   octez-snoop infer parameters for model blake2b on data blake2b.workload using lasso --lasso-positive --dump-csv blake2b.csv --save-solution blake2b.sol --plot

   Initializing python... Done.
   Measure.load: loaded blake2b.workload
   Applying model to workload data 500/500
   Initializing matrices 500/500
   Importing blake2b.csv
   Exporting to blake2b.csv
   Saved solution to blake2b.sol

The command performed the following tasks:

* load the workload data from "blake2b.workload";

* construct a linear regression problem using the chosen model: here,
  the "Blake2b_example" benchmark only provides the "blake2b" model;

* solve this problem using the specified "lasso" algorithm, with the
  constraint that the inferred coefficients must be positive;

* dump the result of inference to a csv file named "blake2b.csv";

* save a structured solution (useful for code generation) to
  "blake2b.sol";

* plot the result of inference.

Let's first have a look at the contents of the CSV solution
"blake2b.csv".


Inference results
^^^^^^^^^^^^^^^^^

+----------------------------------------------------+----------------------------------------------------+
| blake2b_const                                      | blake2b_ns_p_byte                                  |
|====================================================|====================================================|
| 129.279086813                                      | 1.09627036127                                      |
+----------------------------------------------------+----------------------------------------------------+

The columns correspond to the inferred values for the free variables
of the "blake2b" cost model. The units are respectively ns/bytes for
"blake2b_ns_p_byte" and ns for "blake2b_const".

The tool also produces a plot:

[image]

The leftmost figure plots the empirical data, i.e. the raw execution
time (in nanoseconds) as a function of the input size (here, in bytes
-- other data structures might use different notions of sizes). The
rightmost figure plots the empirical data along the predicted
execution time. If the model is good and the parameters were correctly
fitted, these should match. The central plot is useful when using
complex nonlinearities to model the execution time of some piece of
code: the tool will project back the raw data in the linear space
spanned by the chosen nonlinearities and if the model is good, one
should observe that the empirical data lies along a linear subspace.
Here, the model is trivial so the central plot is less interesting.


Step 4: Generating code
=======================

As a final step, we demonstrate how to generate code corresponding to
the model. This is typically used to generate gas consumption
functions for Michelson instructions and not for raw functions like
"blake2b" but the principle is similar.

   octez-snoop generate code using solution blake2b.sol and model blake2b_codegen

By default, the tool produces integer code by casting floating point
constant to integers. The tool produces the following code on
"stdout":

   let model_blake2b_codegen size =
       (int_of_float 144.753899773) + (int_of_float 1.17988921492) * size

It is also possible to generate code implementing the cost function
using fixed-point arithmetic. This requires specifying some codegen
parameters in a JSON file. For instance, we can require to consider 5
bits of precision and use rounding to nearest to convert constants,
failing if we make more than 10% relative error when casting. The
"inverse_scaling" and "resolution" parameters respectively specify the
fraction of digits considered to be not significant, and the
resolution of the grid used when prettifying constants (in
nanoseconds).

   { "precision": 6, "max_relative_error": 0.1, "cast_mode": "Round", "inverse_scaling": 10, "resolution": 5 }

Calling the tool:

   octez-snoop generate code using solution blake2b.sol and model blake2b_codegen --fixed-point codegen_params.json

We get:

   let model_blake2b_codegen size =
       let v0 = size in
       150 + ((v0 + (v0 lsr 3)) + (v0 lsr 5))

Rewriting Micheline with "tezos-micheline-rewriting"
****************************************************

"tezos-micheline-rewriting" allows to construct rewriting rules on
Micheline, enumerate matches and perform rewritings. It is fairly
generic and deserves a documentation of its own. This library
manipulates the following basic elements:

* Micheline terms

* Paths

* Patterns

Informally, the library allows to find patterns in terms: the result
is a list of paths, corresponding to the location of these patterns.
Rewriting is then performed by substituting a subterm at a given path
by another one. The library also provides facilities for manipulating
hash-consed terms and patterns, which is crucial in all non-trivial
applications.


Constructing Micheline terms
============================

The type of Micheline terms has two parameters: the type of locations
and the type of primitives. We abstract away from the underlying
representation of terms via the  "Micheline_sig.S" module type:

   module type S = sig
     type label
     type head
     type node = (label, head) Micheline.node
     val default_label : label
     val int : Z.t -> node
     val string : string -> node
     val bytes : Bytes.t -> node
     val prim : head -> node list -> string list -> node
     val seq : node list -> node
     val label : node -> label
   end

We provide two functors for manipulating Micheline terms:

* "Micheline_without_hash_consing.Make"

* "Micheline_with_hash_consing.Make"

Both functors provide implementations of the "Micheline_sig.S"
signature (with distinct type constraints however). They both take as
input some additional structure on the type of primitives, as
specified in the "Signature.S" module type:

   module type S = sig
     type t
     val compare : t -> t -> int
     val hash : t -> int
     val pp : Format.formatter -> t -> unit
   end

ie primitives must be totally ordered, hashable and pretty-printable.

"Micheline_without_hash_consing.Make" has type

   functor (X : Signature.S) (Label : sig type t val default : t end) ->  Micheline_sig.S with type label = Label.t and type head = X.t

and provides a simple overlay over the Micheline constructors, setting
the label to its default value.

"Micheline_with_hash_consing.Make" has type

   functor (X : Signature.S) (P : sig val initial_size : int option end) -> Micheline_sig.S with type label = hcons_info and type head = X.t

We use the label slot to store hash-consing information on terms: the
type "hcons_info" contains a unique id for the term and its hash. This
implementation of "Micheline_sig.S" performs maximal sharing: it is
important that terms are only constructed through the interface
provided by the module for the required invariants to hold true. A
weak hash table is used to store terms, its initial size is fixed by
"P.initial_size". One can use the unique id contained in "hcons_info"
to construct a separate map from the term to some other annotation.


Manipulating paths in Micheline terms
=====================================

A Micheline term is either a primitive application, a sequence, or one
of three atoms (int, string or byte). The primitive application and
sequence constructors have variable arity (the take lists of subterms
as arguments). It follows that one can designate a subterm of a given
term as a list of integers, each integer denoting an index in the list
of subterms of either a primitive or sequence application.

We provide two implementations of path-manipulating modules: one
without hash-consing and one with. Both implement the following
signature:

   module type S = sig
     type desc = private Root | At_index of int * t
     and t = private {tag : int; hash : int; rev_path_desc : desc}
     val compare : t -> t -> int
     val root : t
     val at_index : int -> t -> t
     val concat : above:t -> under:t -> t
     val to_string : t -> string
   end

We observe that paths, contrary to our intuition, are not directed
from the root to the subterm but rather from the subterm to the root.
This allows easy hash-consing and follows the usual way paths are
constructed during the pattern matching process. The two fundamental
operations are "root", corresponding to an empty path, and "at_index i
p", corresponding to the ith subterm of the term at path "p".


Pattern-matching Micheline
==========================

The "Pattern" module provides two functors implementing a small
pattern description language, as well as functions for enumerating
matches of a pattern in a given term. The signature is the following:

   module type S = sig
     type head
     type path
     type t
     type plist
     type node
     val pattern_matches : t -> node -> bool
     val all_matches : t -> node -> path list
     val focus_matches : t -> path list -> path list
     val int : (Z.t -> bool) option -> t
     val string : (string -> bool) option -> t
     val bytes : (Bytes.t -> bool) option -> t
     val prim : head -> plist -> t
     val prim_pred : (head -> bool) -> plist -> t
     val seq : plist -> t
     val any : t
     val focus : t -> t
     val list_any : plist
     val list_empty : plist
     val list_cons : t -> plist -> plist
     val ( @. ) : t -> plist -> plist
     val pp : Format.formatter -> t -> unit
     val uid : t -> int
   end

The comments describing all these constructs can be found in
src/lib_benchmark/lib_micheline_rewriting/pattern.mli. It is worth
describing a subset of these functions here:

* "pattern_matches patt node" returns true if and only if "patt"
  matches "node".

* "all_matches patt node" returns the list of all paths in of subterms
  of "node" matching "patt".

* "focus patt" constructs a focused subpattern. There can be several
  focused subpatterns but the cannot be nested.

* "focus_matches patt matches" converts a list of matches for "patt"
  into a list of matches for the focused subpatterns of "patt".

The focusing mechanism allows patterns to have a contextual part,
corresponding to the subterm matched by the whole pattern, and a
"point of interest" in the context, corresponding to a subterm of the
subterm matched by the whole pattern. For instance, we can match on
integers that are directly under a particular primitive, etc.

The signatures of the non-hash-consing functor is as follows:

   Make : functor (X : Signature.S) (Micheline : Micheline_sig.S with type head = X.t) (Path : Path.S) -> S with type head = X.t and type path = Path.t and type node = Micheline.node

While the hash-consing implementation has the following slightly more
complicated type:

   module Make_with_hash_consing : functor
       (X : Signature.S)
       (Micheline : Micheline_sig.S
                      with type head = X.t
                       and type label = Micheline_with_hash_consing.hcons_info)
       (Path : Path.S) -> sig
     include
       S
         with type head = X.t
          and type path = Path.t
          and type node = Micheline.node

     val all_matches_with_hash_consing : t -> node -> path list
   end

I.e the default implementation of match enumeration does not use hash-
consing; one has to use "all_matches_with_hash_consing" to do so.


Performing substitutions
========================

The "Rewrite" module provides facilities for performing substitutions.
There is only one implementation here (as all hash-consing is taken
care of in previously described modules). The module provides a
functor taking implementations for terms, paths and patterns and
provides the following functions:

   module type S = sig
     type label
     type head
     type path
     type patt
     type node = (label, head) Micheline.node
     exception Rewrite_error of string * node option
     val get_subterm : term:node -> path:path -> node
     val subst : term:node -> path:path -> replacement:node -> node
     val pattern_matches : patt -> node -> bool
     val all_matches : patt -> node -> path list
   end

The key function here is "subst" which performs the substitution. The
implementation proceeds as one might expect, by recursive descent on
the term together with the specified path.


An example?
===========

An example can be found in the "test" subdirectory. It consists in a
reimplementation of the migration of addresses towards pairs of
addresses and chain ids in multisignature contracts.

Snoop and the Michelson Interpreter
***********************************

In order to evaluate gas costs for the Michelson interpreter, we want
to have a collection of benchmarks and models that cover every
instruction of the language. This document explains how to modify the
files in src/proto_alpha/lib_benchmarks_proto/ in order to fully
integrate a new instruction for "octez-snoop".


Interpreter Workload
====================

We first need to update the "interpreter_workload.ml" file. It
specifies the data that is collected and saved as a "workload" for
each benchmark. First, we add an entry to "instruction_name" with the
name of the new instruction prefixed with "N_", it will be the name
"octez-snoop" will refer to this instruction as. Then add it to the
list "all_instructions", and update "string_of_instruction_name". Now,
we can add a function to the "Instructions" module. This function
should take a number, possibly zero, of integer arguments that defines
our workload. We want these arguments to be the different parameters
that may influence the benchmarking time. For instance, the time "DIG"
takes to be executed varies with the depth of digging, so the "depth"
should be in the workload:

   module Instructions = struct
     (...)
     let dig depth = ir_sized_step N_IDig (unary "depth" depth)

We make these functions by calling "ir_sized_step" with the
instruction name we defined above, and a list of arguments, usually
formed with the helper functions "nullary", "unary", "binary", etc...
Each argument must be given a descriptive name, like "length", or
"int1" and "int2". Note that the order of the elements matters, as it
will be the order in which they will be passed to the model later.

The only step left here is to update the "extract_ir_sized_step"
function: update the pattern matching to include the new instruction,
extract the relevant data from the instruction and the stack, and call
the newly built function.

   let extract_ir_sized_step :
    fun ctxt instr stack ->
     let open Script_typed_cps_ir in
     match (instr, stack) with
     (...)
     | (INil (_, _), _) ->
         Instructions.nil
     | (IDig (_, n, _, _), _) ->
         Instructions.dig n
     | (IConcat_bytes (_, _), (ss, _)) ->
         let list_size = Size.list ss in
         let total_bytes =
           List.fold_left
             (fun x s -> Size.(add x (bytes s)))
             Size.zero
             ss.elements
         in
         Instructions.concat_bytes list_size total_bytes

Here are three examples: the first one is "Nil", its execution time is
constant, so its function is called as is, and its workload will be
empty. For "Dig", the depth needs to be passed to the
"Instructions.dig" function, and it will simply be saved in its
workload, to be used later for the model. The last example requires
some computation: the execution time of "Concat_bytes" depends on the
size of the list on top of the stack, as well as the total number of
bytes to concatenate. We calculate these values here, so we can
retrieve the appropriate workload.

In that last example, we use the module "Size" to convert the bytes
into their size as an integer. It is oftentimes more appropriate to
consider the size of the arguments, instead of their values, as the
actual workload on the instructions.

   match (instr, stack) with
     (...)
     | (IMul_teznat (_, _), (x, (y, _))) ->
         Instructions.mul_teznat (Size.mutez x) (Size.integer y)

In this example, and for most mathematical operations in Michelson,
the computation time actually depends on the size in memory of the
arguments, not their values. Here, the size of an integer is the
number of bytes needed to represent them, likewise for mutez. In this
kind of scenario, we will make use of the "Size" module.


Interpreter Models
==================

We now need to specify a cost model for the instruction in
"interpreter_model.ml". Said models are in the module "Models", which
already contains many generic ones.

   module Models = struct
     let const1_model name =
       Model.unknown_const1 ~const:(fv (sf "%s_const" name))

     let affine_model name =
       Model.affine
         ~intercept:(fv (sf "%s_const" name))
         ~coeff:(fv (sf "%s_coeff" name))

     (...)

They directly derive from generic models available in "model.ml".
"const1_model" is used for constant-time instructions, "affine_model"
is for instructions with cost function \lambda size. const + coeff *
size, where "size" is a value that appears in the workload, while
"const" and "coeff" are free variables, etc... Note that the arity of
the model must match the number of elements in the workload. Some
instructions may require a specific model that does not yet exist. In
this case, we can add it to the "Models" module, like
"join_tickets_model", which has arity 4, and is exclusively used for
the instruction "IJoin_tickets".

The affine model expects an argument for the constant value, called
"intercept". It is the execution time of the instruction when the
workload is 0. Some models assume that the intercept is 0, like the
linear models. Otherwise, we may want to make two benchmarks for the
instruction, one of which made exclusively for the computation of that
"intercept".

When a model has been chosen, we can update the function "ir_model".
Add your instruction to the pattern matching, and return the following

   model_[i] instr_or_cont ([your_model] name)

where "your_model" is your choosen model, and "i" its arrity. For
instance:

   | N_INil ->
        model_0 instr_or_cont (const1_model name)
   | N_IDig | N_IDug | N_IDipN | N_IDropN | N_IDupN ->
        model_1 instr_or_cont (affine_model name)
   | N_IConcat_bytes ->
        model_2 instr_or_cont (concat_model name)


Interpreter benchmarks
======================

The interpreter benchmarks are located in "interpreter_benchmarks.ml",
in the module "Registration_section", which is itself divided in
multiple sub-modules for each family of instructions. Most benchmarks
are created and registered using the following helper function
"benchmark", or one of its derivatives.

   val benchmark:
       ?amplification:int ->
       ?intercept:bool ->
       name:Interpreter_worload.instruction_name ->
       kinstr_and_stack_sampler:(Default_config.config ->
                                 Random.State.t ->
                                 unit ->
                                 ex_stack_and_kinstr) ->
       unit

This function builds the "Benchmark.S" module and registers it, doing
most of the work defined in Step 1 of the usage example. Its arguments
are as follows:

* "amplification", if provided, is the number of times an operation
  must be run in a single execution of the benchmark. If not
  specified, the instruction will be run only once. It is useful for
  instance when the operation itself takes very little time, and the
  calls to the timer take the most of the benchmark runtime.

* "intercept" (default "false") is "true" if and only if the benchmark
  is covering the intercept case. It can be used when the expected
  intercept for the chosen model is not 0. The user should then make
  sure that the provided sampler generates zero workload for the
  instruction.

* "name" is simply the name of the benchmarked instruction. It will
  also appear in the name of the registered benchmark. When searching
  for a benchmark in snoop, the names are structured as follows:
  "<name>{_intercept}_<protocol_version>", where "<protocol_version>"
  is either the version number of the protocol, or "alpha".
  "_intercept" only appears if the previous argument was set to
  "true". The function cannot register two benchmarks with the same
  "name", unless they have different "intercept" values.

* "kinstr_and_stack_sampler" is a function that, given a configuration
  and a random state, returns a stack and a "kinstr" on which the
  benchmark will be performed. If "intercept" is "ŧrue", this sampler
  should be so that zero workload is generated for the instruction.

Some other functions are provided to cover some simpler cases, they
exist mostly to avoid creating a full sampler for both the stack and
the instruction.

* "simple_benchmark" can be used when the stack can be whatever (as
  long as it is compatible with the benchmarked instruction). Its
  optional argument "intercept_stack" is the stack on which the
  intercept case will be run. If this argument is provided, two
  benchmarks will be registered for the instruction.

* "benchmark_with_stack_sampler" requires a sampler only for the
  stack, useful when the "kinstr" is fixed, which happens most of the
  time. "benchmark_with_fixed_stack" requires a fixed stack instead of
  a sampler. Both variations also exist for "simple_benchmark".

* Finally, "continuation_benchmark" works similarly to "benchmark",
  but for continuations instead of instructions.


Examples
--------


A simple benchmark
~~~~~~~~~~~~~~~~~~

   let () =
      simple_benchmark
        ~amplification:100
        ~name:Interpreter_workload.N_ISwap
        ~kinstr:(ISwap (kinfo_unitunit, halt_unitunit))
        ()

Benchmark for "ISwap". Here the execution time is small enough that we
want to use an amplification factor to obtain reasonable timings. The
definition of "kinstr" uses some predefined values for "kinfo" and
"kinstr", available at the begining of the registration section.

   let kinfo kstack_ty = {iloc = 0; kstack_ty}
   let halt stack_ty = IHalt (kinfo stack_ty)
   let halt_unit = halt (unit @$ bot)
   let halt_unitunit = halt (unit @$ unit @$ bot)
   let kinfo_unit = kinfo (unit @$ bot)
   let kinfo_unitunit = kinfo (unit @$ unit @$ bot)

In our example, "ISwap" is preformed for two unit types on a stack
with nothing else. There is no need to specify a stack sampler, the
function "simple_benchmark" infers it automatically from the given
"kinstr".


A simple benchmark with an intercept stack
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   let () =
      simple_benchmark
        ~name:Interpreter_workload.N_IConcat_bytes
        ~intercept_stack:(Script_ir_translator.list_empty, ((), ()))
        ~kinstr:
          (IConcat_bytes (kinfo (list bytes @$ bot), halt (bytes @$ bot)))
        ()

For "IConcat_bytes", the intercept case would be an empty list: we are
concatenating zero bytes on a list with zero elements, so both values
of the workload are zero, as expected. In the non intercept case, the
stack is randomly generated in accordance to the specification, which
in this case is simply a random list of bytes. The sampling
parameters, such as the bounds for the size of the list and its
elements, can be controlled by a config file at runtime.


A less simple benchmark
~~~~~~~~~~~~~~~~~~~~~~~

   let () =
      let dig = Micheline.(Prim (0, I_DIG, [Int (0, Z.of_int 0)], [])) in
      benchmark
        ~amplification:100
        ~intercept:true
        ~name:Interpreter_workload.N_IDig
        ~kinstr_and_stack_sampler:(fun _cfg rng_state () ->
          let node = dig in
          parse_instr rng_state node long_stack)
        ()

    let () =
      let dig n = Micheline.(Prim (0, I_DIG, [Int (0, Z.of_int n)], [])) in
      benchmark
        ~name:Interpreter_workload.N_IDig
        ~kinstr_and_stack_sampler:(fun _cfg rng_state () ->
          let node = dig (sample_depth rng_state) in
          parse_instr rng_state node long_stack)
        ()

Here we define two benchmarks, the first being the interception case,
which is when "DIG" is called with the integer "0", which is enough
since the depth is exactly the workload. Here the sampler uses
"parse_instr", which builds the correct "kinstr" and "stack", given a
Micheline node and a generated stack (here we use "long_stack", a big
stack of unit types). The "kinstr" here must also be sampled, because
we want to benchmark the instruction for different workloads, i.e
different depths, which are sampled with "sample_depth".


Testing
=======

Assuming the workload is correctly defined for our benchmarking and
modeling needs, we just need to check if the chosen model fits the
data. For that, we can refer to the usage example, and follow the
given steps. If the resulting plot shows that the predicted execution
time fits the empirical data, then it should be good. Otherwise, it
should provide some insight to choose a more fitting model for the
instruction.

Time measurement PPX
********************

The time measurement PPX is an OCaml preprocessing tool that intends
to embed generated benchmark tooling into specific pieces of OCaml
code.

It is able to measure the time spent in the execution of annotated
OCaml expressions and to log these measurements when desired. Since it
uses "Tezos_event_logging" for the logging part, this PPX can easily
be used together with "Tezt" framework to perform the benchmarking of
specific parts of Octez node.

**This PPX is only intended to be used for tests. As the current
runtime implementation performs memory allocation, an unwise usage
could mess with the garbage collector or blow up your memory.**

**PLEASE, MAKE SURE THAT IT IS NOT ACTIVATED WHEN COMPILING CODE FOR
PRODUCTION.**


Getting started
===============

Suppose we want to measure the performance of some specific parts of
the following OCaml function inside the module "lib_my_module":

   let my_function () =
     let open Lwt_syntax in
     let a = f () in
     let b = g () in
     let* c = h () in
     foo a b c

Suppose also that module "lib_my_module" contains the following dune
file:

   (library
     (name tezos_my_module)
     (libraries lwt)
     (flags (:standard -open Lwt)))

We can measure the execution time of "f ()" and "g ()" and log them by
adding the following OCaml attributes:

   let my_function () =
     let a = f () [@time.duration f_time] in
     let b = g () [@time.duration g_time] in
     let* c = h () in
     foo a b c [@time.flush]

"[@time.duration]" will be used to measure the time of "f ()" and "g
()" expressions execution and to name them respectively "f_time" and
"g_time". "[@time.flush]" will then be used to log these measurements.

When the preprocessing will occur, the code will be transformed as
follows:

   let my_function () =
     let a = Tezos_time_measurement_runtime.Default.Time_measurement.duration
       ("f_time", [])
       (fun () -> f ())
     in
     let b = Tezos_time_measurement_runtime.Default.Time_measurement.duration
       ("g_time", [])
       (fun () -> g ())
     in
     let* c = h () in
     Lwt.bind
       (foo a b c)
       (fun __flush__id__0 ->
         Lwt.map
           (fun () -> __flush__id__0)
           (Tezos_time_measurement_runtime.Default.Time_measurement.flush ()))

Woah! What a mess... Let's see what this means.

The first thing that can be noted is that our annotated expression "f
()" has been wrapped in a closure "fun () -> f ()" to delay the
execution of "f ()". The resulting thunk is passed to the function
"Time_measurement.duration" from the module
"Tezos_time_measurement_runtime.Default" along with the argument
"("f_time", [])".

"Time_measurement.duration" measures the current time before and after
executing the given thunk in order to compute the span between the two
timestamps. The resulting measurement is then bufferized in memory
and, at last, the function evaluates in the result of the thunk so
that we can respect the invariants of the initial program. Hence, the
result of the expression "f ()" is well bounded to the identifier "a"
as expected.

The couple "("f_time", [])" given as argument represents a key that is
bound to the time measurement in order to discriminate it later. This
key is computed from the payload that was given inside the attribute
"[@time.duration f_time]". ""f_time"" is the label of the key, an
identifier that represents the name of the measurement. "[]" is a list
of strings that contains some additional metadata. It can be useful to
distinguish several measurements registered with the same label, for
example if the expression is evaluated in a loop. In our case, the
list is empty because no metadata was provided in the attribute's
payload.

The same logic applies for "g ()". We measure the time at the limits
of its execution, we bufferize the resulting span along with the key
"("g_time", [])", and we then bind the result of the expression to the
identifier "b".

At the end of the treatment, the "Lwt.t" promise "foo a b c" is bound
to a fresh identifier "__flush__id__0". This permit to memoize its
result while calling the "Time_measurement.flush" function from the
module "Tezos_time_measurement_runtime.Default". This function logs
all the time measurements that where previously bufferized and also
removes them from memory. The flushing promise is then bounded again
to return "__flush__id__0" value to preserve the program invariants as
well.

That's great, but since OCaml attributes are ignored by default, we
still need to update the dune stanza of "lib_my_module" so that it can
take effect:

   (library
     (name tezos_my_module)
     (libraries lwt)
     (flags (:standard -open Lwt))
     (instrumentation (backend tezos-time-measurement)))

This update adds the "tezos-time-measurement" instrumentation backend,
which, if set using "--instrument-with tezos-time-measurement" on
"dune build" command line, will preprocess our OCaml code using the
PPX.

This is useful to prevent our code from embedding benchmarking tooling
in production by mistake: If no backend is specified for the
compilation, added attributes will just be ignored by the OCaml
compiler and that's it!

We can now compile our ready-to-benchmark code:

   dune build --instrument-with tezos-time-measurement

We can then run the executable:

   ./my_program.exe

When "my_function" is executed, an event named
""time_measurements.v0"" is displayed on standard output. For example:

   Aug 23 17:52:58.593 - benchmarking: time measurements:
   Aug 23 17:52:58.593 - benchmarking:   [(f_time, 0.000177); (g_time, 0.005658)]


Compatible OCaml Attributes
===========================

The PPX provides the handling of three attributes:

* "[@time.duration <label> (<metadata>)]" is used to measure the time
  of OCaml expressions execution. The "<label>" inside the payload
  will be used to tag the measured time. The "<metadata>" is an OCaml
  expression that can be added optionally and should evaluate to a
  list of "string"s. It can be given to add additional contextual
  information to the measurement and it can permit to discriminate it
  from other measurements registered with the same label.

  Be careful, annotating "Lwt.t" values with this attribute may not
  give consistent time measurements since it will only measure the
  time spent to return the corresponding promise.

* "[@time.duration_lwt <label> (<metadata>)]" does the same as
  "[@time.duration]" except that it must annotate an expression
  evaluating in a "Lwt.t" value. The measured time will then be the
  time spent by the promise to be fulfilled.

* "[@time.timestamp_pre <label> (<metadata>)]" is used to measure the
  current timestamp before the annotated expression is evaluated. The
  measurement will be tagged with the given "<label>" and optional
  "<metadata>" like with "[@time.duration]".

* "[@time.duration]" is used to log each time measurements that were
  registered using "[@time.duration]", "[@time.duration_lwt]" or
  "[@time.timestamp_pre]". Since logging will be done using "tezos-
  event-log" library, it must be done inside a "Lwt.t" monad. So, this
  attribute must be placed on an expression evaluating in a "Lwt.t"
  value in order to compile.

Some of these attributes are used, for instance, in the implementation
of the performance regression test framework.


Instrumenting the octez-node executable
=======================================

A helper has been added in the "Makefile", so you just need to run the
following command to instrument the node during the compilation:

   ./make enable-time-measurement

Python Environment
******************

Octez uses Python to build this documentation website
(https://octez.tezos.com/docs/) and for a limited set of utility
scripts (although this latter usage of Python is deprecated). This
page contains installation instructions for the Python environment.


Installation
============

Prerequisites:

* "python 3.12.3". It is recommended to use pyenv to manage the python
  versions. If you want to use "pyenv":

  * Follow the installation instructions. In particular, this ensures
    that "eval "$(pyenv init -)"" has been executed first during the
    shell session, by adding this line to an environment script
    sourced automatically.

  * You can use then "pyenv install 3.12.3", followed by:

    * "pyenv local 3.12.3" to use "python 3.12.3" only in the current
      directory (and its subdirectories, unless redefined)

    * "pyenv global 3.12.3" to set the python version to "3.12.3"
      globally

    * "pyenv shell 3.12.3" to use "python 3.12.3" only in the current
      shell

* poetry 1.8.3 to manage the python dependencies and run the tests in
  a sandboxed python environment. Follow the installation
  instructions.

  Before using the python environment for the first time, the
  dependencies must be installed. To achieve this, run "poetry
  install" in the root of the project.

A typical installation of the above prerequisites (including their own
prerequisites) proceeds as follows, see below for full details:

   # 1. install pyenv
   # 2. restart shell, to ensure "pyenv init -" has been evaluated
   # 3. then install python using pyenv:
   pyenv install 3.12.3
   pyenv global 3.12.3
   # 4. install poetry
   # 5. restart shell, to activate the poetry setup
   # 6. then install dependencies for Octez using poetry:
   cd tezos/
   poetry install


Installation details for Ubuntu 20.04
-------------------------------------

First, make sure "curl" and "git" are installed, as they are required
by the "pyenv" and "poetry" installers:

   sudo apt-get install curl git --yes


For "pyenv" to compile Python, you'll have to install python's build
dependencies:

   sudo apt-get install make build-essential libssl-dev zlib1g-dev \
     libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm \
     libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev --yes


Now, install "pyenv" using pyenv-installer:

   curl https://pyenv.run | bash


To make "pyenv" available in your shell session, add the following to
your shell configuration file, e.g. "~/.bashrc" or "~/.zshrc":

   export PATH="$HOME/.pyenv/bin:$PATH" # add pyenv to path
   eval "$(pyenv init --path)"          # adds pyenv plugins to path
   eval "$(pyenv init -)"               # adds pyenv setup to environment
   eval "$(pyenv virtualenv-init -)"    # adds virtualenv setup to environment


To verify the "pyenv" installation, restart your terminal and try
executing "pyenv":

   pyenv --version


Now we can use "pyenv" to install Python 3.12.3 and set it as the
default version:

   pyenv install 3.12.3
   pyenv global 3.12.3


Now verify that the correct version is called when running "python":

   python --version # should output 3.12.3


Now, on to installing "poetry". We'll use poetry's official installer:

   curl -sSL https://install.python-poetry.org -o install-poetry.py
   python install-poetry.py --version 1.8.3 --yes


As for "pyenv", we need to do some shell setup to put "poetry" in the
shells path. Add the following to your shell configuration file, e.g.
"~/.bashrc" or "~/.zshrc":

   export PATH=$PATH:$HOME/.local/bin


Restart the terminal and verify that the correct version is called
when running "poetry":

   poetry --version # should output 1.8.3


Finally, let's use "poetry" to install the python dependencies of
Octez. This command needs to run in the root of the Octez checkout:

   cd tezos
   poetry install


If the installation went well, you should now have the correct version
when executing poetry in the Octez checkout for our main dependency
("sphinx-build" for documentation):

   poetry run sphinx-build --version # should output 7.3.7



Adding new dependencies
=======================

Dependencies are managed by poetry in the file pyproject.toml. See the
reference for the pyproject.toml files. The file poetry.lock is
generated by running "poetry lock", and must never be changed
manually.

Pre-Commit Hook
***************

Pre-commit is a framework for managing and maintaining multi-language
pre-commit Git hooks.

Using this framework you can specify a list of hooks you want and pre-
commit manages the installation and execution of any hooks before
every commit. pre-commit is specifically designed to not require root
access.

Before you can run hooks, you need to have the pre-commit package
manager installed. There are multiple ways of installing "pre-commit",
either via "apt" in Debian or using other installation options on the
pre-commit website.

Pre-commit is configured via a simple configuration file .pre-commit-
config.yaml that is kept in sync by the Tezos developers with the best
practice used on the Tezos repository. Installing "pre-commit" will
minimize CI failure, by avoiding failures that can be detected by
sanity checks before pushing upstream.

We can now install the git hooks using the command "pre-commit
install".

From now on before every commit, the git hook will be run by "pre-
commit" and make sure all checks are green.

You can find more information and advanced uses on https://pre-
commit.com/.

Encodings
*********

Throughout the Tezos protocol, data is serialized so that it can be
used via RPC, written to disk, or placed in a block. Conversely, data
is deserialized to in-memory data structures, for performing
operations on it, such as checking the validity of a transaction.
These serialization and deserialization operations rely on
*encodings*, that is, data encoding rules for the various Tezos data
structures, towards/from several forms. Thus, there exists binary
encodings and JSON encodings.


Tools
=====

For performing serialization/deserialization of data structures into
binary form within OCaml code, refer to the documentation of the data
encoding library.

For studying and understanding all the different encodings, the
definitive source of truth is the "octez-codec" tool. It is a command-
line tool allowing users and developers to:

* describe the binary and JSON encoding schemas for all the supported
  data structures (see How to read the output of octez-codec)

* encode and decode data into/from binary or JSON form, using a
  specific encoding (see How to encode/decode values)

You may refer to the "octez-codec" online manual for more details. The
rest of this page gives a gentle introduction to this tool by showing
on some examples how to perform the two tasks above.

Note that for the particular case of Micheline expressions, the
"octez-client" tool can also be used to convert between several data
representations, covering not only the binary and JSON
representations, but also OCaml and Michelson notations (see How to
convert Micheline with octez-client).


How to read the output of "octez-codec"
=======================================

The list of data structures that can be encoded/decoded may be
obtained as follows:

   $ octez-codec list encodings

The binary encoding of any supported data structure can be described
using the following command:

   $ octez-codec describe <id> binary schema

Similarly, the JSON encoding of any supported data structure can be
described using the following command:

   $ octez-codec describe <id> json schema

However, the output of the above commands is rather verbose, so here
is a short introduction for how to read the schemas produced by
"octez-codec".


JSON schemas
------------

The descriptions of JSON schemas are rather self-explaining. For
instance, the encoding of a Micheline expression is done according to
a few simple JSON encoding principles, that can be retrieved by doing:

   $ octez-codec describe alpha.script.expr json schema

The schema output by this command is as follows (slightly abbreviated
and reformatted for better readability):

   $alpha.script.expr
   $alpha.michelson.v1.primitives:
     "ADD"
     | "IF_LEFT"
     | "SELF_ADDRESS"
     ...
     | "code"
   $alpha.script.expr:
     { "int": $bignum } /* Int */
     || { "string": $unistring } /* String */
     || { "bytes": /^[a-zA-Z0-9]+$/ } /* Bytes */
     || [ $micheline.alpha.michelson_v1.expression ... ] /* Sequence */
     || { /* Generic prim (any number of args with or without annot) */
          "prim": $alpha.michelson.v1.primitives,
          "args"?: [ $micheline.alpha.michelson_v1.expression ... ],
          "annots"?: [ string ... ] }
   $bignum:
     /* Big number: Decimal representation of a big number */
     string
   $micheline.alpha.michelson_v1.expression:
     ...
   $unistring:
     /* Universal string representation:
        Either a plain UTF8 string, or a sequence of bytes for strings that
        contain invalid byte sequences. */
     string || { "invalid_utf8_string": [ integer ∈ [0, 255] ... ] }

The schema starts with the main non-terminal (here, a Micheline
expression, denoted by "$alpha.script.expr"), whose definition appears
lower in the schema. The schema also defines all the non-terminals
which are used, directly or indirectly, in this definition. Note that
we omitted in the listing above the definition of non-terminal
"$micheline.alpha.michelson_v1.expression", because it is identical to
that of the main non-terminal "$alpha.script.expr".

As can be seen, non-terminals are defined as disjunctions of JSON
elements such as constants, objects, and arrays. Some attached
comments further clarify the meaning of most alternatives or fields.


Binary schemas
--------------

The descriptions of binary schemas are more complex to some extent,
mainly for two reasons:

* Binary schemas are lower level than the JSON schemas. Thus, the
  encoding of elementary types has to be precisely defined: strings
  must include a field containing their length; discriminated unions
  must include a field containing a tag, whose possible values must be
  enumerated; the precise binary layout of various integer types must
  be made explicit, and so on.

* The binary encodings are optimized for certain common cases, in
  order to save space. For instance, Micheline primitive applications
  with one or two arguments uses specialized encodings that are more
  compact (see the binary encoding principles for Micheline).

To illustrate these differences, let us consider the same example as
above, that of a Micheline expression:

   $ octez-codec describe alpha.script.expr binary schema

The binary schema produced by this command is as follows (abbreviated
and reformatted for better readability):

   +-----------------+----------------------+----------+
   | Name            | Size                 | Contents |
   +=================+======================+==========+
   | Unnamed field 0 | Determined from data | $X_8     |
   +-----------------+----------------------+----------+


   Z.t
   ***
   A variable length sequence of bytes, encoding a Zarith number. ...
   +------+----------------------+----------+
   | Name | Size                 | Contents |
   +======+======================+==========+
   | Z.t  | Determined from data | bytes    |
   +------+----------------------+----------+

   micheline.alpha.michelson_v1.expression (Determined from data, 8-bit tag)
   *************************************************************************

   Int (tag 0)
   ===========
   +------+----------------------+------------------------+
   | Name | Size                 | Contents               |
   +======+======================+========================+
   | Tag  | 1 byte               | unsigned 8-bit integer |
   +------+----------------------+------------------------+
   | int  | Determined from data | $Z.t                   |
   +------+----------------------+------------------------+

   String (tag 1)
   ==============
   +-----------------------+----------+-------------------------+
   | Name                  | Size     | Contents                |
   +=======================+==========+=========================+
   | Tag                   | 1 byte   | unsigned 8-bit integer  |
   +-----------------------+----------+-------------------------+
   | # bytes in next field | 4 bytes  | unsigned 30-bit integer |
   +-----------------------+----------+-------------------------+
   | string                | Variable | bytes                   |
   +-----------------------+----------+-------------------------+

   Sequence (tag 2)
   ================
   ...

   Prim (no args, annot) (tag 3)
   =============================
   ...

   Prim (no args + annot) (tag 4)
   ==============================
   ...

   Generic prim (any number of args with or without annot) (tag 9)
   ===============================================================
   ...

   Bytes (tag 10)
   ==============
   ...

   alpha.michelson.v1.primitives (Enumeration: unsigned 8-bit integer):
   ********************************************************************
   +-------------+-----------------------+
   | Case number | Encoded string        |
   +=============+=======================+
   | 0           | parameter             |
   +-------------+-----------------------+
   | 1           | storage               |
   +-------------+-----------------------+
   | 2           | code                  |
   +-------------+-----------------------+
   ...
   +-------------+-----------------------+
   | 140         | GET_AND_UPDATE        |
   +-------------+-----------------------+

The binary schema starts with the binary layout of the main non-
terminal (here, "alpha.script.expr"), and also defines the other non-
terminals that are used, directly or indirectly in this definition.
Each definition forms a section (whose heading is underlined by
all-"*" lines). Sections corresponding to disjunctions are further
structured in subsections (whose headings are underlined by all-"="
lines), one for each possible value of the discriminating tag.

For instance:

* The layout of an "Int" as a "bignum" is explicitly defined as a
  Zarith number (non-terminal "Z.t").

* The layout of a "String" starts with a field containing the number
  of bytes in the string.

* The values of the discriminating tag are 0 for "Int" expressions, 1
  for "String" expressions, and so on.

* The encoding of expressions involving a primitive operator
  application defines both the generic case of an arbitrary number of
  operators (the same as in the JSON schema above), and a number of
  specialized common cases (zero operator with or without annotations,
  etc.).

* The operators themselves are encoded as an enumeration of values
  (non-terminal "alpha.michelson.v1.primitives").


How to encode/decode values
===========================

Beyond examining the various available encodings, the "octez-codec"
tool can also be used to encode and decode data. This can be useful
for developers when debugging, but also for end users when trying to
understand the contents of a block or transaction, for instance.

Let us consider a few examples of encoding and decoding some commonly
used types.


Strings
-------

To encode a string as a Micheline expression, proceed as follows:

   $ octez-codec encode alpha.script.expr from '{"string":"Hello world!"}'
   010000000c48656c6c6f20776f726c6421

As can be seen, strings are serialized as follows:

* a leading "01" tag to indicate type string

* four bytes (eight hex chars) to indicate the length of the string:
  "0000000c = 0x0c = 12" in our case

* the string represented by its ASCII values:
  "48656c6c6f20576f726c6421" in our case

The same tool can be used in the other direction, to decode a byte
sequence representing a serialized string expression:

   $ octez-codec decode alpha.script.expr from '010000000c48656c6c6f20776f726c6421'
   { "string": "Hello world!" }


Integers
--------

There are various encoding for integers, including:

* "ground.int16": Signed 16 bit integers

* "ground.uint16": Unsigned 16 bit integers

* "ground.Z": Arbitrary precision integers

which can be detailed by describing their schemas, e.g.:

   $ octez-codec describe ground.Z binary schema
   ...
   A variable length sequence of bytes, encoding a Zarith number. Each byte has
   a running unary size bit: the most significant bit of each byte tells is this
   is the last byte in the sequence (0) or if there is more to read (1). The
   second most significant bit of the first byte is reserved for the sign
   (positive if zero). Size and sign bits ignored, data is then the binary
   representation of the absolute value of the number in little endian order.

To illustrate the Zarith representation, let us encode the Micheline
representation of the number "1,000,000" (one million):

   $ octez-codec encode alpha.script.expr from '{"int":"1000000"}'
   0080897a

Here:

* The first byte "00" indicates that the type is integer.

* The number is represented by the bytes "80897a" for 1000000 (1
  million).

Reading each byte from left to right, in binary form:

   0x80897a = 0b10000000, 0b10001001, 0b01111010

* The first bit in each byte indicates whether it is the last byte (0)
  in the sequence or if there is more to read (1).

* The second bit in the first byte indicates that this is a positive
  number.

* The remaining bits are then "0b000000", "0b0001001", "0b1111010".
  Reversing the byte order (because little-endian) we get:
  "0b11110100001001000000" = "0xf4240" = "1000000".


Pairs
-----

Let us see how an OCaml pair is encoded:

   $ octez-codec encode alpha.script.expr from '{"prim":"Pair","args":[{"int":"1"},{"int":"2"}]}'
   070700010002

Here:

* "07": the first tag denotes the micheline constructor. "Pair 1 2" is
  a primitive application with 2 arguments and no annotation. The
  corresponding tag is "0x07".

* "07": the next tag denotes the Michelson primitive "Pair". It so
  happens that the corresponding tag is also "0x07".

* "0001": encoding of the integer 1

* "0002": encoding of the integer 2

Let's try another example, the encoding of the value "Left 1" of type
"or nat bool":

   $ octez-codec encode alpha.script.expr from '{"prim":"Left","args":[{"int":"1"}]}'
   05050001

Here:

* "05": the expression "Left 1" is a primitive application with one
  argument and no annotations. The corresponding tag is "0x05".

* "05": the michelson primitive is "Left", for which the corresponding
  tag is also "0x05".

* "0001": encoding of the integer 1.


Operations
----------

Finally, let us consider a more complex example. Assume that we try to
understand an operation included in a block. We can decode the binary
string as follows:

   $ octez-codec decode alpha.operation from '008f1d96e2783258ff663f03dacfe946c026a5d194c73d1987b3da73fadea7d46c008cb5baedee4dc3ec261dfcf57a9600bb0a8e26c0f00bdd85a0018452ac02e0a712000153957451d3cc83a71e26b65ea2391a1b16713d2d009595facf847a72b4c3fe231c0e4185e68e9b2875aa3c639382c86bcf0af23699f47fe66a6550ade936a5b59d5919ad20703885750314e0c368b277de39e7d10a'
   { "branch": "BKiXcfN1ZTXnNNbTWSRArSWzVFc6om7radWq5mTqGX6rY4P2Uhe",
     "contents":
       [ { "kind": "transaction",
           "source": "tz1YU2zoyCkXPKEA4jknSpCpMs7yUndVNe3S", "fee": "1520",
           "counter": "2622173", "gas_limit": "10500", "storage_limit": "300",
           "amount": "300000",
           "destination": "tz2FwBnXhuXvPAUcr1aF3uX84Z6JELxrdYxD" } ],
     "signature":
       "sighZMqWz5G8drK1VTsmTnQBFEQ9kxQQxL88NFh8UaqDEJ3R3mzgR3g81azadZ9saPwsWga3kEPsyfbzrXm6ueuDvx3pQ5Q9" }

In order to understand how the transaction has been decoded from the
binary sequence, we have to examine the encoding schema of a block
operation:

   $ octez-codec describe alpha.operation binary schema
   +-----------+----------+---------------------------------------------+
   | Name      | Size     | Contents                                    |
   +===========+==========+=============================================+
   | branch    | 32 bytes | bytes                                       |
   +-----------+----------+---------------------------------------------+
   | contents  | Variable | sequence of $alpha.operation.alpha.contents |
   +-----------+----------+---------------------------------------------+
   | signature | 64 bytes | bytes                                       |
   +-----------+----------+---------------------------------------------+
   ...
   alpha.operation.alpha.contents (Determined from data, 8-bit tag)
   ****************************************************************
   ...
   Transaction (tag 108)
   =====================
   +----------------------------------+----------------------+-------------------------------------+
   | Name                             | Size                 | Contents                            |
   +==================================+======================+=====================================+
   | Tag                              | 1 byte               | unsigned 8-bit integer              |
   +----------------------------------+----------------------+-------------------------------------+
   | source                           | 21 bytes             | $public_key_hash                    |
   +----------------------------------+----------------------+-------------------------------------+
   | fee                              | Determined from data | $N.t                                |
   +----------------------------------+----------------------+-------------------------------------+
   | counter                          | Determined from data | $N.t                                |
   +----------------------------------+----------------------+-------------------------------------+
   | gas_limit                        | Determined from data | $N.t                                |
   +----------------------------------+----------------------+-------------------------------------+
   | storage_limit                    | Determined from data | $N.t                                |
   +----------------------------------+----------------------+-------------------------------------+
   | amount                           | Determined from data | $N.t                                |
   +----------------------------------+----------------------+-------------------------------------+
   | destination                      | 22 bytes             | $alpha.contract_id                  |
   +----------------------------------+----------------------+-------------------------------------+
   | ? presence of field "parameters" | 1 byte               | boolean (0 for false, 255 for true) |
   +----------------------------------+----------------------+-------------------------------------+
   | parameters                       | Determined from data | $X_0                                |
   +----------------------------------+----------------------+-------------------------------------+

Using the above information, the sample binary sequence can be broken
down as follows:

   branch
   = 0x008f1d96e2783258ff663f03dacfe946c026a5d194c73d1987b3da73fadea7d4
   = BKiXcfN1ZTXnNNbTWSRArSWzVFc6om7radWq5mTqGX6rY4P2Uhe

   tag = 0x6c = 108 (transaction)

   source
   = 0x008cb5baedee4dc3ec261dfcf57a9600bb0a8e26c0
   = tz1YU2zoyCkXPKEA4jknSpCpMs7yUndVNe3S

   fee = 0xf00b = 1520
   counter = 0xdd85a001 = 2622173
   gas_limit = 0x8452 = 10500
   storage_limit = 0xac02 = 300
   amount = 0xe0a712 = 300000

   destination
   = 0x000153957451d3cc83a71e26b65ea2391a1b16713d2d
   = tz2FwBnXhuXvPAUcr1aF3uX84Z6JELxrdYxD

   has_parameters = 0x00 = false

   signature
   = 0x9595facf847a72b4c3fe231c0e4185e68e9b2875aa3c639382c86bcf0af23699f47fe66a6550ade936a5b59d5919ad20703885750314e0c368b277de39e7d10a
   = sighZMqWz5G8drK1VTsmTnQBFEQ9kxQQxL88NFh8UaqDEJ3R3mzgR3g81azadZ9saPwsWga3kEPsyfbzrXm6ueuDvx3pQ5Q9

As usual, "octez-codec" can be used the other way around, to encode
the same transaction:

   $ octez-codec encode alpha.operation from '{ "branch": "BKiXcfN1ZTXnNNbTWSRArSWzVFc6om7radWq5mTqGX6rY4P2Uhe", "contents": [ { "kind": "transaction", "source": "tz1YU2zoyCkXPKEA4jknSpCpMs7yUndVNe3S", "fee": "1520", "counter": "2622173", "gas_limit": "10500", "storage_limit": "300", "amount": "300000", "destination": "tz2FwBnXhuXvPAUcr1aF3uX84Z6JELxrdYxD" } ], "signature": "sighZMqWz5G8drK1VTsmTnQBFEQ9kxQQxL88NFh8UaqDEJ3R3mzgR3g81azadZ9saPwsWga3kEPsyfbzrXm6ueuDvx3pQ5Q9" }'


How to convert Micheline with "octez-client"
============================================

The "octez-client" can be used to convert Micheline expressions
between the following forms: binary, JSON, Michelson, and OCaml.

Note that the client has to be run in conjunction to a running node
for the following commands to work (unless option "--protocol" is
specified):

   $ octez-client convert data '(Pair 1 2)' from michelson to binary
   0x070700010002
   $ octez-client convert data 0x070700010002 from binary to michelson
   (Pair 1 2)
   $ octez-client convert data 0x070700010002 from binary to json
   { "prim": "Pair", "args": [ { "int": "1" }, { "int": "2" } ] }
   $ octez-client convert data 0x070700010002 from binary to ocaml
   Prim (0, D_Pair, [Int (1, Z.one); Int (2, Z.of_int 2)], [])

Merkle Proof Encoding Formats
*****************************

A Merkle proof is a datum which demonstrates that a Merkle tree has a
given value. Typically a Merkle root and a subtree of a Merkle tree
are used as a Merkle proof. Verification is done by computing the
Merkle root and checking it is the same as the given hash. In Octez,
Merkle proofs are used for Optimistic Rollups (see Smart Optimistic
Rollups) in the event an invalid hash is submitted from a layer 2 node
to layer 1. An honest layer 2 node can then present a Merkle proof to
demonstrate that the previously submitted hash is in fact fraudulent.

This document shows the encoding format of the Merkle proof
implemented in
src/lib_context/merkle_proof_encoding/merkle_proof_encoding.ml. There
are 2 versions of encodings (defined as V1 and V2), each generating 2
types of Merkle proofs (named tree_proof and stream_proof), for 2
types of Irmin Trees (32-tree and binary tree). The data structure is
defined in src/lib_context/sigs/context.ml (API ) as below. The
internal structure of Irmin, which is used to manage contexts in
Octez, appears in it. Encoding formats give the conversion between a
"tree_proof" and a byte sequence, and between a "stream_proof" and a
byte sequence.

   type index = int
   type hash = Context_hash.t (* 32-byte Blake2b hash *)

   type kinded_hash =[
     | `Value of hash
     | `Node of hash
   ]

   type 'a inode = {length : int; proofs : (index * 'a) list}

   type 'a inode_extender = {length : int; segment : index list; proof : 'a}

   type tree
     | Value of value
     | Blinded_value of hash
     | Node of (step * tree) list
     | Blinded_node of hash
     | Inode of inode_tree inode
     | Extender of inode_tree inode_extender

   type inode_tree =
     | Blinded_inode of hash
     | Inode_values of (step * tree) list
     | Inode_tree of inode_tree inode
     | Inode_extender of inode_tree inode_extender

   type elt =
     | Value of value
     | Node of (step * kinded_hash) list
     | Inode of hash inode
     | Inode_extender of hash inode_extender

   type stream = elt Seq.t

   type tree_proof = { version : int; before : kinded_hash; after : kinded_hash; state : tree }

   type stream_proof = { version : int; before : kinded_hash; after : kinded_hash; state : stream }

"before" and "after" in both proofs are Merkle roots and denote that
the Merkle root of a tree changed from "before" to "after" by applying
the get/set operations for the tree. "tree_proof" is a partial tree
containing the hashes of nodes needed for verification and the values
the operations refer to. It omits extra nodes and hashes that are
recoverable by re-computing. "stream proof" can produce on demand a
partial Merkle tree corresponding to a "tree_proof".


Merkle proof formats
^^^^^^^^^^^^^^^^^^^^

* V1 for Tree32

* V1 for Tree2

* V2 for Tree32

* V2 for Tree2

Note: all encoding versions use big endian.


Notation Example
================

In order to explain the notation used for describing the encodings,
let us consider the "Bytes" format as an example (this format is not
actually used). In our notation, the format is defined as in the table
below.

+-------------------------+-------------------------+-------------------------+
| Name                    | Size                    | Contents                |
|=========================|=========================|=========================|
| tag                     | 1 byte                  | 0b000000yy where yy is  |
|                         |                         | tag for length (0b00    |
|                         |                         | for p = 1, 0b01 for p = |
|                         |                         | 2, 0b10 for p = 4, 0b11 |
|                         |                         | for p = 8)              |
+-------------------------+-------------------------+-------------------------+
| length                  | p bytes                 | int                     |
+-------------------------+-------------------------+-------------------------+
| content                 | (length) bytes          | bytes                   |
+-------------------------+-------------------------+-------------------------+

This format means that given a byte sequence "b", the first 1 byte of
"b" represents the tag which determines the number of bytes "p" of the
next record "length"; the next "p" bytes of "b" represent the length
of the content as an integer, and the following (length) bytes
represent the content.

"0b000000yy" in the description of "tag" means that the first 6 bits
of the tag must be "0", and we represent “yy” as the last 2 bits of
the tag.

For example, when "b" is "0x000c48656c6c6f20576f726c6421", "tag" is
"0x00" (so the byte length of "length" is 1), "length" is "0x0c" (so
the byte length of "content" is 12) and "content" is
"0x48656c6c6f20576f726c6421"


See also
========

Another implementation of the encoder for the Merkle proofs, written
in the Rust language, is available here.

Grafazos: Grafana dashboards for Octez node monitoring.
*******************************************************

Grafazos allows to generate dashboards from jsonnet programs.

It relies on the metrics exposed by the Octez node, see Supported Open
Metrics, to build grafana dashboards.


Resources
=========


Jsonnet
-------

To build the dashboards, you need to install "jsonnet" with version
"0.18.0" minimum. We recommend to use https://github.com/google/go-
jsonnet.


Grafonnet
---------

"Grafonnet" is the jsonnet library used to generate grafana
dashboards.

"Grafonnet" should be installed using jsonnet-bundler.

To do so, run "go install -a github.com/jsonnet-bundler/jsonnet-
bundler/cmd/jb@latest" and update your environment variables such as
"export GOPATH=$HOME/go" and "export PATH=$PATH:$GOPATH/bin".

Once "jsonnet-bundler" installed you can import dependencies (here
"grafonnet") by running the following:

   jb install


Tools
=====

* Emacs mode: https://github.com/tminor/jsonnet-mode


Build
=====

To create the dashboards:

   make

The dashboards will be availbale in the "grafazos/output/" folder.

You can also create a specific dashboard with

   make dashboard_name

with "dashboard_name" one of:

   * "compact": a single page compact dashboard which aims to give a
     brief overview of the node's health,

   * "basic": a simple dashboard displaying detailed node's metrics,

   * "logs": same as "basic" but also displaying node's logs (thanks
     to Loki and promtail)

   * "full": same as "logs" but also displaying hardware metrics
     (thanks to netdata)

   * "dal-basic": a simple dashboard displaying some DAL node's
     metrics,

To create the dashboards for a different branch

   BRANCH=foo make

By default, the instance names label is set to "instance". If you are
using a particular label for the instance names, you can use

   NODE_INSTANCE_LABEL=my_instance_label

If you need hardware metrics, note that, by default, the storage stat
considered is the used space of the whole disk. Optionnally you can
enable storage monitoring with "filecheck":

   STORAGE_MODE=filecheck

By default, all mountpoints are monitored. If you want to selectively
change mountpoints to "/" and another specific mountpoint, you can
enable and add it with the "MOUNTPOINT" variable : .. code-block::
shell

   MOUNTPOINT="/opt"

By default, the logs are fetched from Loki. If you want to use GCP
logs, you can enable it with the "LOGSRC" variable, the GCP project id
with the "GCP" variable and the "GCP_DATASOURCE_UID" for it's unique
id in grafana. .. code-block:: shell

   LOGSRC="gcp" GCP_PROJECT_ID="project_id"
   GCP_DATASOURCE_UID="datasource_uid"

By default, dashboards are created without uid, resulting in a random
one assigned by grafana. You can set a specific uid with the "UID"
variable, which will be concatenated to a default uid set in each
dasboard's definition. If "UID" is set to an empty string, then the
uid is set to the default uid value set for each dashboard. .. code-
block:: shell

   UID="some_string"