Mainnet May 2019#
Activation of Athens#
In anticipation of the activation of Athens on May 29th, we strongly encourage bakers to run a node in full mode and rebuild their storage from a snapshot.
From our latest benchmark the time to validate the migration block on the old storage can be as long as 15 minutes on a hdd, <4 minutes on a ssd, while a node from a full snapshot takes less than 5 minutes on hdd, <40 seconds on ssd. This is due to our current storage backend that will be improved in the near future.
Storage Migration#
The new node requires an upgrade of the storage format that is performed at the launch of new Tezos node. For this reason, once the storage migrated, it will not be possible to use the old Tezos node.
Here is an example migration from Mainnet April 2019 to a node in mode
full. Assuming a running node from Mainnet April 2019 with default
storage directory ~/.tezos-node, begin by gracefully stopping the
node.
Back up your storage and upgrade it, it will only take a few seconds:
cp -r ~/.tezos-node tezos-node-$(date +%Y%m%d)
tezos-node upgrade storage
We now have the option to restart the node and it will automatically
run in archive mode.
This mode is useful only in specific cases and most users should run a
full node.
We can proceed to export a snapshot:
tezos-node snapshot export mainnet-$(date +%Y%m%d).full
The default is to snapshot the chain at the position of our last checkpoint, which is usually 5 cycles in the past. This is the safest option as the Tezos node guarantees that no reorganization of the chain is possible earlier than a checkpoint. However it requires to synchronize from the network the last 5 cycles.
Alternatively we can also specify the block to snapshot:
tezos-node snapshot export mainnet-$HASH.full --block $HASH
We can now delete our upgraded archive storage and import the snapshot
(without forgetting, if you need, to save {identity,peers,config}.json):
rm -r ~/.tezos-node
tezos-node snapshot import ./mainnet-*.full
There should be a rather smaller ~/.tezos-node directory populated
from which we can start our node:
tezos-node run
It is good practice to snapshot regularly our own node, for example with a cron job, once compressed the files are rather small.
Docker#
If you are running Docker with the helper script mainnet.sh there
are two new commands.
The node storage can be upgraded, before running the new node in
archive mode, using:
./mainnet.sh node upgrade
Otherwise you can delete your storage altogether and import a fresh snapshot using:
./mainnet.sh clear
./mainnet.sh snapshot import mainnet-*.full
Known issues#
In principle it is possible to export a snapshot while the node is running but again because of our current storage it doesn’t work for now. It will be fixed soon. For the time being, you should stop your node, export a snapshot and restart it.
Changelog#
RPC#
Note: new RPCs providing access to the shell are listed with SHELL changes
- Commits:
c866f8e1e:Client RPC : adding support for PUT and DELETE on the command line.7513853e8:client RPC: Rebuild errors from json and print them
P2P#
Fixes
Better handling of private and trusted nodes
Change version negotiation system
new
network/versionRPCnetwork/versionsRPC is deprecated, usenetwork/versioninstead- Commits:
10c37f153:Make 2-min sleep configurable (P2p_maintenance)9abcd0e55:P2p: change configuration of P2p_maintenance7e4a70fac:Remove peer from active_peers when Peer_validator.create has failed52b196a72:P2p: Fix error message from improve cancellation983bdee01:P2p: don't kick node which are private and trusted on too_many_connections01d0c880c:p2p: trusted_node is trusted by peer or by point134e1d7a7:P2p: change version selection0b4d7fa75:P2p, Shell: write Time.System types instead of equal types83f3f16f0:Shell_services: return P2P version as is59b0624df:P2p: earlier detection of "public" nodes.e88888707:P2p: Don't announce banned points84cdbcea5:P2p : random and parallel kill strategy on too_many_connection
CRYPTO#
- Commits:
c393f7f27:Crypto: add Signature.pp_watermark
SHELL#
Fixes
Support for test-chains
new RPC:
active_chainsRPCs return new and more specific errors (chain expired, block parsing error)
new parameter
--disable-testchainto node executablenew
stats/gcRPC to get info about memory consumptionnew
stats/memoryRPC to get info about process memory consumptionsupport for history modes
new
checkpointRPC to get the current checkpoint for a chain- Commits:
a3379065e:Distributed_db: fix concurrency issue on Peer_validator creation3e2172b2d:Distributed_db: don't fail when receiving a notification for a closing worker6aec720b0:Shell: propagate prevalidation error (#432)6a7c05118:Shell: ensure that the chain exists when calling the valid_blocks RPC186c96484:Shell: fix typo in error title3ac0ca16c:Shell: reorganize mutually recursive datatype85dea244b:Shell_services: fix error string typoada9581b5:Shell: changes the genesis hypothesis => it is genesis if its predecessor is itself4edc3df14:Shell/Testchain: correctly register the test chain into the global store and initialize the test protocol44e00f3ea:Shell/RPCs: add unhandled cases0c12ca0d5:Shell/RPCs: fix typo448c54089:Shell: improved faked genesis block for testnetdbb0d5782:Shell/testnet: preserve the hash of the forked context in the main contexta87212941:Shell/Testchain: fetch the missing protocol when forking a test chain188f94a0b:Shell: revert breaking hash computation change in test chain blocksa2a428a26:Shell: cleanup the test chain header temporary table09596d5c9:Shell/Testchain: unset chain's child on testchain destructiona7e374492:Shell: add a parameter to disable the testchain12881f1ce:Shell/Test: change chain_id's testchaind33923d0e:Shell: make the testchain start on the forking block189344bce:Shell: replace the temporary table contents with the forking block's hash and move the test protocol initialisation to Chain_validator2fe41b04a:Shell: switch testchain to not_running on current block's timestamp to ensure mainnet compatibility578b1610c:Shell: set the testchain to the mainchain when the testchain state is already in storee47acdf0d:Fix race condition in Distributed_functor.Memory_table54edb7748:shell: adds static link protocols in stored8efc14c3:Shell: fixes lwt logs for protocol store77db3f69e:Shell: refactor outputs and don't store protocols with fake hashes9f69abe82:Shell: refactor protocol fetching0617232cb:Shell: Replace read_exn with read_opt28e489077:Shell/test: replace read_exn with read_optbe4cb3494:Shell: cosmetic patchc8cd7b058:Shell: remove unused read_exna6ee34981:Shell: add pipeline librarya27728c41:Shell: tests for the pipeline931a61ca0:Shell: drop some dead read_exn functions829d06785:Shell/Worker: remove zombiesbfa9cba01:Shell/Workers: refactoring84bbf11a1:Shell/Worker: add find_opt function611d2ca60:Shell/Worker: add information function and encoding (for future RPCs)6e07e5ab8:Shell: minor renaming38add10ad:Shell/worker: expand wildcard patternsf842bf1c5:Shell/worker: minor documentation fix1fdbe9f94:Shell: add RPC to query GC stats4c99b0c78:Shell: add RPC to query proc_statm infoca439ecf3:Shell/Stat_services: add page_size in proc_statme3d34f72e:Shell/Stat_services: better handling of stats when running on a darwin (OSX) os6329d47e9:Shell_services/Stats: small renaminga51e7ad5d:RPCs: refactor memory services (split into three parts in stdlib/stdlib_unix and lib_shell)71ed49e88:Shell: don't use float for time, use Ptime0b4d7fa75:P2p, Shell: write Time.System types instead of equal types83f3f16f0:Shell_services: return P2P version as isecc358771:Shell/RPC: export a Base58Check representation of block headersaf1f9a7ae:Shell: use private type for State.Block.Header.tc02aa1052:Shell: store the full block header of the checkpointd187be487:Shell: remove test made incoherent by new checkpoint reprc54bfbec0:Shell: use "atomic" upgrade of disk storage4f10f7872:Shell: extend atomic storage operationsf0ef5d1d6:Shell_services: introduce history modesfacff17fd:Shell: add header field in contents typeebd67f2f8:Shell: remove contents field in block type0bec5d75e:Shell: remove type hashed_header9292450e7:Shell: replace block header abstractione040cb7cd:Shell: remove Store.Block.Headera26faf595:Shell: a block is known and known_valid if we know its header7fd92971e:Shell: add Save_point and Caboose in store6d7653f3b:Shell: init node with caboose and save_point valuesbdc448711:Shell: update block_locator so it handles partial chain709e71c3a:Shell: known_ancestor, unknown_prefix handle partial chain36da9092e:Shell: unopt genesis accessord138d4594:Shell: Block_validator avoid validation under save_point8887080b9:Shell: dismiss locators from rolling peers whose caboose is too far aheade8e6197da:Shell: use tzresult and chain_traversal handles partial chain65e21bb55:Shell: fail with error when reading contents and header in stateb255226fd:Shell: Chain_validator, move init_head in create38446f867:Shell: remove Operation_path from store08dd93c7c:Shell: add Protocol_hash in store6d3873f00:Shell: populate Protocol_hashcc65da9ce:RPCs: add rpc_header_directory7d0c1e5b0:Shell: relax with_atomic_rw type signatureed7ddf1d0:Shell: remove invalid heads operation metadatae1d40755d:Shell: add Chain.store in state9c32743c6:Shell: full/rolling purge when setting a new checkpoint485cfa6bc:Shell: improve loggingaa02c8580:Shell: delete block when cutting alternative headsef3d7c84d:Shell: check and save history mode9025412e7:Shell: disallow switching from an history-mode to another47c36a01f:Shell: use contents_not_found error when reading block contents0bea9ed83:Shell/RPC: replace Level 0 with Genesis780388c1a:Shell: remove useless include
PROTO#
add
fitnessargument toactivate protocolcommandadd
parametersargument toactivate protocolcommandadd
delayargument to thefork test protocolcommandFixes
- Commits:
53b7c9d21:Proto/Genesis: add fitness to Activate_testchain operationcc7b39ea7:Proto/Genesis: add a delay argument to the fork testchain command015f402b1:Michelson: Fix macro unexpansionafbcb71df:ProtoDemo: set the last_allowed_fork_level to 0 to make sure the previous blocks are not pruned
BINS#
Fixes
Improved logs
New
chainargument for the clientsupport for test-chains
remove conflicting CLI options in client:
-bmanual upgrade of old file-formats with
upgrade baking statecommandbetter support for user interruption (CTRL-C), especially when forking test chain
add
--forceargument to client’ssubmit proposalsadd
--dry-runargument to client’ssubmit proposalsandsubmit ballotadd
list orphan noncesandfilter orphan noncescommands for the bakeradd
fetch protocolcommand in clientclient’s
list protocolsdoes not contain duplicates anymoreadd
--verbose-signingargument to all client commands that require signinghistory modes
add
show current checkpointcommand to admin clientnew data encoding version for the node (version 0.0.3)
add snapshot command
- Commits:
b7c2546e5:Daemon: await_bootstrapped_node now retries on Connection_failed2268a1164:Client: break line after "No invalid blocks." messagefbf4c8492:Functorize Client_main_run and use it for the signer234bf416b:src/lib_client_base_unix/client_main_run.ml: apply global_options once for alld99603af8:remove useless maps7966e9006:simplifications2188d3c34:add some commentsea997b541:rebase on masterd5bf06fff:add some code documentation3967e43b7:sandboxed-node: preserve context if DATA_DIR variable is defined09d1822a8:sandboxed-node: document DATA_DIR capability9f0a9b621:Alpha/Accuser: handle different chain ids for double baking/endorsing580b7aea6:Alpha/Accuser: increase the number of considered levels to a cycle (i.e. 4096)a1849ab69:Alpha/Endorser: allow endorsements on different chain idsef202674b:Alpha/Baker: allow baking on different chain idsed22cae07:Alpha/Daemons: lift the chain id selection to the Delegate_commands12f6b9838:Client/Base: add a chain argument to the client333782792:Alpha/Daemons: use the tezos-client context to select the chain98bdbbedb:Baking/Alpha: fix block listing rpc call46bd72176:Baker: handle test chain forking casese15250809:Daemons/State: replace the daemon state functor with a highwatermark system9ff061976:Baker/Nonces: add chain_id to nonces file and functionalize the representation18a96d1f1:Baker/Nonces: change logging priority for nonce injection53f16df32:Baker/Nonces: clean and refactor nonce revelations7b5e73729:Endorser/Alpha: improve the endorser's logging and improve double-endorsing security444024497:Baker: letting the shell handle a protocol upgrade2e6887997:Alpha/Daemon: monitor the active chains to start and stop the daemons on the test chain4ccf071e8:Alpha/Daemons: refactor some code6139762f7:Client/CLI: improve the documentation and remove conflicting shortname optionba4c8ef7d:Baker: index nonces by chain_id and not chain alias3da7836a2:Baker: clean up testchain's nonces on testchain terminatione7cfd698c:Alpha/Daemons: handle migration from legacy files to new highwatermarks mechanism6c933dba4:Delegates: enforce manual upgrade of the baking state when detecting older file formats552edf7c7:Client: remove intrusive error messagee107fb92d:Daemon: remove the files upgrade from the daemons2ed2078bb:Daemon: refactor highwatermarks5245cb1b8:Baker: replace metadata RPC calls to header calls to handle purged blocks metadata8ef52cb8d:Accuser: remove dead-codeb2748ed4c:Baker: update baker's state structure659d1fe65:Accuser: ensure the test chain is launched before starting2c1fd0270:Baker: improve logging info for endorsement waiting3ecda1d96:Daemon: allow to ^C baker/endorser while waiting for the testchainff78a6119:Client/Alpha: fix the client's RPCs to use the correct chain id2d8e8b17d:Daemons: restore nonces file backward compatibility57f17e50c:Daemons: restore highwatermarks file backward compatibility52b00f650:Baker: restore the --force option for the 'bake for' commande3b604c03:Baker: factor out some code858afda30:Node: escape doclang in doc304bb75d8:Client: add --force to submit proposalsbed959b23:Client: improve Too many proposals error messagea17b42214:Client: check for duplicate proposalsbe46b7394:Client: improve display of proposal errors6728bbb1f:Client: improve display of proposals (show voting)58ee0418a:Client: fix punctuation of submit proposalscd1898a14:Client: add --dry-run to submit proposalsca5b13057:Client: add --dry-run to submit ballot187d24e64:Client: improve voting error display1aecd5ff0:Client: mention “yea” and add Oxford comma in helpc24b7e927:Baker/Nonces: filter out nonces associated to blocks that cannot be found in the chain66638cac6:Baker: enforce chain arguments through the API92f00dc93:Baker/Nonces: do not remove orphan nonces anymore but warn users insteadf4d4628ca:Baker/Commands: add a command to filter orphans from the nonce file6bcf7213e:Baker: add a command to display filtered orphan nonces8c62ad12d:Baker: prevent spurious error message while filtering orphan noncesc6f5fb4c7:Baker: saving orphan nonces file before overwriting the nonces filef4da04129:Client: add protocol fetch commandf22148117:Client: test string prefix directly rather than with splitc2ddac4be:Admin client: no duplicate in list protocols output6f653d703:Client: fix missing chain argument to rpc05ce3229e:Client: add option --verbose-signingdc569b812:Client: add code comments (verbose-signing !994)227799e4f:Baker: improve the notification in case of a new block while waiting for endorsement473ee0cbe:Client: reduce the dependency of lib_client_base over Unix by parameterizing Unix function calls in the wallet objects090a1073a:Client: simpler password retrievalb2658722f:Endorser: remove duplicate addresses for which an endorser endorsese98447e9f:Bin_node: use a module for the main chain genesis6c473044b:Bin_node: add bare option to ensure_data_dir02ce2047b:Node: history mode node initializationace8497a1:Bin_node: typo5fe11e227:Bin_node: use a module for patch context7fd70e063:Shell: add checkpoint client admin command249bbd86e:Node: properly display an error message when trying to switching history-modedb85a7e38:Bin_node: add read_data_dir config file7f4f2e2e7:Bin_node: update node data version handlingdb85a7e38:Bin_node: add read_data_dir config file7f4f2e2e7:Bin_node: update node data version handling3155bc7e7:Shell/Snapshots: Add snapshot export/import featurescb18ba9f0:Bin_node: Upgrade data_version of storage (0.0.2 -> 0.0.3)142680262:Bin_node: change rolling history-mode to experimental-rolling
LEDGER#
Drop support for Wallet and Baking apps older than 1.4.0.
Drop support for firmware older that 1.4.0
CLI and performance improvements.
Baking: support test-chain high-water mark, improve configuration (
setup ledger ...).APDU support
- Commits:
6ce10791b:Ledger: add deauth APDU support, better get-authorized-key support, and friendlier CLIdc87b9a7c:Ledger: clean-up bip32 module4714daaed:Ledger: create Signer_implementation module391126533:Ledger: switch to new APId5b08fd98:Ledger: improve output of --test-sign2a640e1d0:Ledger: explicitly forbid versions < 1.4.02bc9b7497:Ledger: fix display of get authorized path
MISC#
Fixes
- Commits:
d7ba8bb07:Resto: Fix #480: Keep the host: set by the user with TEZOS_SIGNER_HTTP_HEADERS35727d4ac:Clic: stop being colorless.0034c8df9:Resto: log exception and continue when catching an unexpected exception in restoc2b2f2002:Resto: client: Remove parameters (as 'charset') after 'type/subtype' in Content-Type header before matching against expected media typef9b5ad7a6:StdLib/Ring: add add_and_return_erased functiona54855c20:StdLib/WeakRingTable: hide internal key handling from usersaa987cc35:StdLib/WeakRingTable: add WeakRingTable to available iterators/iterable5fcf5a337:StdLib/hashPtree: typo in commente8ac45375:Stdlib: improve WeakRingTable fold/iterf3e85a9a6:Clic: add args173df9d715d:lmdb: specify the correct env var in error message1ae97b84f:StdLib: Typo323d1e12d:Base/Time: rewrite Timeab7f0c6cc:Base/Time: documentation of new interface8e81069f8:Separate unix dependencyc921903c0:client_base: move unix dependent code into client_base_unix6d8145acd:Base/time: tweak pretty-printing for backwards compatibility9ba57b815:Everywhere: use built-in return operators0b8575b46:Crypto: make Base58.encode tail recursive32e04d814:Store: add Pruned_contentsbc4c485bc:Storage: add comments63d11dd1e:Stdlib: add progress displaya63daf798:Stdlib_unix: add read/write_string6037d0edc:Lib_storage: add context export/import featureba968354c:Storage: Add export/import tests
TOOLING/DOC/TESTS/BUILD/CI/ETC.#
Fixes
More tests (including two new test frameworks)
Contributing guidelines
OCaml 4.07.1 and other dependencies update
- Commits:
b4b97dada:Fix identity.json preservation in docker images after a resete4a5d3c45:test for check point32a1d2647:Opam: remove unused dependencies from alpha-{baker,endorser,accuser}f9cbdf911:docker: improve argument passingc3dd98dc0:Dune: do not use the infamous "OCaml syntax" anymore28f680ab9:Dune: remove src in proto_*/lib_protocol0796b0355:CI: test that the various dune.inc are up-to-dateeba7f5ac8:Dune: fix dune.inc for added file in proto democ842ef6a2:Opam: fix protocol packaging16ae0ced7:Opam: add synopsis in all packages0883ee314:Mostly cosmetic changes in Michelson docf5c50c8ba:Update example commands + add extra docf45067cf3:doc: typo fixe6710ef87:fix documentation CSSd503ebb1b:proto_compiler: remove useless jbuild_templates6805d11af:emacs: michelson-mode, changing tmp file mode to access it from vagrant9eb46932c:Scripts: change default parameters used by sandbox to include nonces01665c65b:Docker: add test baker commands920a7f7bd:Docker: update docker file removal3c301d68d:add install in opam files for lib_resto-directory & lib_restoa7e37439c:Docker: fix docker scripts70d171b86:Docker: update alphanet.sh script3ddbc0e5e:Test: adapt voting test to new 'list protocol' semanticsf1f0f8ce2:Typo in "Howtouse"cffd27e80:Add all autogenerated files to the gitignore file39d1f229c:Only publish docker/docs on tezos/tezos82190b65c:CI: publish docker images for *net-snapshots branches5a14c1f4a:Michelson:Test: Add two simple unexpansion tests181a70235:Doc: add page for Mainnet April release1577b71cb:Stdlib-unix: Fix concurrency bug in create_dir19b925db8:Data-encoding: Add the With_version moduledb0bbfa5f:Clic: Add Scriptable (for --for-script)30816ed09:Build,CI: Add the tezos-event-logging library3c53dbc7d:Event-logging: Add module Internal_event_unixb425a3e9d:Stdlib-unix: Add module File_event_sink622b7312b:Baker: Use structured logging in Client_baking_blocks2939dac72:Admin-client: Add event-logging commands25ff4e037:Replace legacy logging with Internal_eventf117848d4:Event-logging: Make section a private string listb2b3f04ba:Event-logging: Forbid '.' in event namesa1ac99c1a:Event-logging: Make check_name_exn safer8f07970f8:Data-encoding: Fix type signature1f27dbf2b:Event-logging: Rename Error_event.to_lwt9db735b12:Data-encoding: Add implementation commented5e135df:CI: Fix build (dune package version)5de3487f0:Lib_delegate: make logging event names unique74e85db30:gitlab-ci: run ./scripts/update_opam_test.shaf3c5222e:Event logging: improve mli doc7f7c12796:Event logging: use only ascii characters in comments1ee47493b:Use POSIX compliant flags with sed49055d897:Delete CREATE_CONTRACT without literal from docs1f4c074c3:Doc: add contributing.md, improve contributing doce5c396997:[bin client]: remove wrong dependency to tezos-shellaf5ed9cd9:Opam: update dependencies1491675ea:Account for ocaml.4.07.1 changes in Tezos38a3e629a:Opam: force external dependencies on react and reactiveData9c3ea6f78:Docs: clean index to fix warninge7940dae2:Vendor/ledger-wallet: depend on hidapi 1.0c3f0fbfb1:Opam repository: put git final git hashesb80790013:CI: Add "opam lint" on all opam files in CIb1afd8078:[Opam]: add synopsis to tezos-event-loggingbe5ce3cc4:[opam]: fix opam-version in event-logging11e6c05e0:[dune files] Various changes to compile packages via opamb1cc7b331:Fix tezos-client-demo.opam0365eb175:CI: quotes in lint commandb5782b875:vendors/*.opam: use [[ ]] in run-test9535096ff:Docs: fix ASSERT macros in Michelson specificatione860a7bb1:build-deps: really install and use opam-depext708286ab5:add more build dependencies1479bb40d:expand2b1d15340:add 'which' build dep7a5fca120:Scripts: fix snapshot_alpha and activate_protocolddbe199ea:Doc: fix many odoc errors69d174f5d:Makefile: make clean removes the dune-project filescf678dd40:Fix LSR documentationb58af9893:Documentation: minor fixes22c3e7255:P2P versioning: adapt reset_chain.sh4a6f85740:Docs: don't linkcheck reddit to avoid rate limit errorsf3b064e2b:Tests: rename demo to proto_test_injectionaf1613a0b:Dune: use the same set of warnings everywhereea8470542:Various: bugfixes after (re)activating warning 32.7d5ff87da:Various: remove dead code after activating warning 32e0a9d1365:Various: activate the infamous warning 950da261f9:Protocol compiler: remove safe-string from dune_protocol.template9aebc957c:Tests: add python framework54195fe81:Doc: fix undefined label and linkcheck82e06dab9:Tests: add OCaml framework flextesa236c779fa:Tests: basic RPC test936e06273:Tests: porting test_tls.sh to test_tls.py05e863295:Tests: test tls added to CIe16ba18f5:Fix logging in tests3d511057b:Doc: add glossaryf03711b5d:Tests: add test for stats rpcs98d553db9:version/CI: update opam hash8c7fa7499:Test/OCaml: adapt framework with the new time system7eab940d1:Test/Python: change time between blocksa99612569:build: remove merge artifact in src/lib_shell/dunea711ac1fc:CI: update opam test18eeec0b4:Alpha/Client: update opamd6a3b46f1:Opam packaging: install TEZOS_PROTOCOL as part of tezos-embedded-protocol-%.raw4511710d4:Flextesa: voting command takes the TEZOS_PROTOCOL file as argumentb6619de3a:lib_network_sandbox: Make it 32bits arch compatible6e91d5ec2:Opam packaging: fix tezos-client-alpha-commands deps38e4ae8ed:Opam packaging: fix flextesa build772907364:Tests/Flextesa: remove .opame648fc42e:Test/Python: correct time between blocks - multiple bakers3f6c6c2f7:Tests: ported test_cors.sh to test_cors.pyc6998296f:Tests: test_cors.py added to CI7eaea17ea:Tests/python: remove numbering in CI21bb2dbd6:Doc: the Michelson type address is actually comparablee9e816d0d:Flextesa accusation test: fix44f655783:Shell: add testsuite for prune and delete6ffabdd8b:CI: update opam