# Testing Toolkit 🏗 A set of CLIs, tools and tests to set up, manage and operate Polygon devnets. The **Testing Toolkit** is built on top of `express-cli`, an extension of `matic-cli` which uses `terraform` to deploy, test and monitor any devnet on AWS stacks from any local system. It currently supports **only** devnets running `v0.3.x` stacks. The `express-cli` interacts with `terraform` to create a fully working setup on AWS. This setup is composed by a set of `EC2 VM` instances running a specific `ubuntu 22.04 ami`, mounted with `gp3 disks` , and a `public-subnet` with its `VPC`. In case the infrastructure already exists, `matic-cli` can be used as a standalone tool to deploy Polygon stacks on pre-configured VMs. Please, refer to the section of this file you are more interested in (`express-cli` or `matic-cli`) ## `express-cli` ### Requirements To use the `express-cli` you have to execute the following steps. - [install aws cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - [install terraform](https://learn.hashicorp.com/tutorials/terraform/install-cli) on your local machine - use [nvm](https://github.com/nvm-sh/nvm#installing-and-updating) to switch to the proper `node` version, `v16.17.1`, by running `nvm use` from the root folder - install `express-cli` and `matic-cli` locally with command `npm i` - generate a keypair on AWS EC2 and download its certificate locally (`.pem` file) - copy `secret.tfvars.example` to `secret.tfvar` with command `cp secret.tfvars.example secret.tfvars` and check the commented file for details - **If you are a Polygon employee**, connect to the company VPN - modify `secret.tfvar` with addresses of the allowed IPs (as specified in `secret.tfvars.example` file) - copy `.env.example` to `.env` with command `cp .env.example .env` and check the heavily commented file for details - make sure `PEM_FILE_PATH` points to a correct AWS key certificate, the one you downloaded in the previous steps - define the number of nodes (`TF_VAR_VALIDATOR_COUNT` and `TF_VAR_SENTRY_COUNT`) and adjust the `DEVNET_BOR_USERS` accordingly - use `TF_VAR_DOCKERZIED=no` to have one VM per node, otherwise the stack will run on one VM only in a dockerized environment - (optional) replace `TF_VAR_VM_NAME` with your own identifier (it can be any string, default is "polygon-user") - (optional) replace `TF_VAR_DISK_SIZE_GB` with your preferred disk size in GB (default is 100 GB) - `VERBOSE=true` prints logs from the remote machines. If set to `false`, only `express-cli` and `matic-cli` logs will be shown - **If you are a Polygon employee**, please refer to [this page](https://www.notion.so/polygontechnology/Testing-Toolkit-d47e098641d14c80b2e9a90b3b1b88d9) for more info ### Auth Configuration As a prerequisite, you need to configure authentication on `aws` This will create the folder `~/.aws` in your system To do so, please run ```bash aws configure sso ``` This command will interactively ask for some configs **If you are a Polygon employee**, please use the following - SSO session name: leave empty - SSO start URL: https://polygon-technology.awsapps.com/start#/ - SSO region: us-east-1 The browser will open and authorize your request. Please allow it. In case there are multiple accounts available to you, please select > posv1-devnet Then, the command will ask for other configs, please use - CLI default client Region: us-west-2 - CLI default output format: json - CLI profile name: default Note that it's **mandatory** to use `CLI profile name: default`, as used by `terraform` in `express-cli` (for more context see [this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs)) Here an output example ```bash SSO session name (Recommended): WARNING: Configuring using legacy format (e.g. without an SSO session). Consider re-running "configure sso" command and providing a session name. SSO start URL [None]: https://polygon-technology.awsapps.com/start#/ SSO region [None]: us-east-1 Attempting to automatically open the SSO authorization page in your default browser. If the browser does not open or you wish to use a different device to authorize this request, open the following URL: https://device.sso.us-east-1.amazonaws.com/ Then enter the code: There are 2 AWS accounts available to you. Using the account ID The only role available to you is: () Using the role name "" CLI default client Region [None]: us-west-2 CLI default output format [None]: json CLI profile name []: default To use this profile, specify the profile name using --profile, as shown: aws s3 ls --profile default ``` Now you can log into aws by running the following command. It needs to be executed every time the token expires. ```bash aws sso login ``` Congrats! You're all set to use `express-cli` commands. ### Commands Instructions to run `express-cli`. For the list of commands, please run `express-cli --help` First off, you need to `--init` terraform on your local machine, by executing the following command. - `./bin/express-cli --init` - Initializes a new devnet folder with terraform and creates some git-ignored files locally. This step is mandatory before running any other command. The new devnet folder created will be `devnet-` where `id` is a monotonically increasing count for the devnets. Once created, you can `cd deployments/devnet-` and run the other commands. This allows you to work with multiple devnets at once. Then, a remote devnet can be created with the `--start` command, as follows. - `../../bin/express-cli --start` - Creates the desired remote setup, based on the preferences defined in the `.env.devnet` file - `--start` command can be used also to target an existing AWS setup. If changes to `.env.devnet` file are detected, the previous devnet will be destroyed and a new one created, reusing the same AWS VMs To destroy the remote devnet, you can execute the `--destroy` command. - `../../bin/express-cli --destroy` - Destroys the remote setup and delete the dedicated VMs The `express-cli` also comes with additional utility commands, listed below. Some of them are only available for non-dockerized devnets. - `../../bin/express-cli --update-all [index]` - Fetches `heimdall` and `bor` branches defined as `HEIMDALL_BRANCH` and `BOR_BRANCH` in `.env.devnet` file, pulls relative changes and restarts those services on the remote machines. If an integer `index` is used, the job will be performed only on the VM corresponding to that index. - `../../bin/express-cli --update-bor [index]` - Fetches `bor` branch defined as `BOR_BRANCH` in `.env.devnet` file, pulls relative changes and restarts it on the remote machines. If an integer `index` is used, the job will be performed only on the VM corresponding to that index. - `../../bin/express-cli --update-heimdall [index]` - Fetches `heimdall` branch defined as `HEIMDALL_BRANCH` in `.env.devnet` file, pulls relative changes and restarts it on the remote machines. If an integer `index` is used, the job will be performed only on the VM corresponding to that index. - `../../bin/express-cli --restart-all [index]` - Restarts `bor` and `heimdall` on all the remote machines. If an integer `index` is used, the job will be performed only on the VM corresponding to that index. - `../../bin/express-cli --restart-bor [index]` - Restarts `bor` on all the remote machines. If an integer `index` is used, the job will be performed only on the VM corresponding to that index. - `../../bin/express-cli --restart-heimdall [index]` - Restarts `heimdall` on all the remote machines. If an integer `index` is used, the job will be performed only on the VM corresponding to that index. - `../../bin/express-cli --cleanup` - Cleans up `ganache`, `bor`, `heimdall` and `bridge`, redeploys all the contracts and restarts all the services The `express-cli` also provides additional testing commands, listed here. - `../../bin/express-cli --send-state-sync` - Create a `state-sync` transaction on the remote network - ` ../../bin/express-cli --monitor [exit]` - Monitors the reception of state-syncs and checkpoints to make sure the whole network is in a healthy state. If `--send-state-sync` hasn't been used before, only checkpoints will be detected. Monitor the setup. If `exit` string is passed the process terminates when at least one `stateSync` and one `checkpoint` are detected. - ` ../../bin/express-cli --instances-stop` - Stop the AWS EC2 VM instances associated with the deployed devnet. - ` ../../bin/express-cli --instances-start` - Start the (previously stopped) AWS EC2 VM instances associated with the deployed devnet. Also, it starts all services, such as ganache, heimdall, and bor - `../../bin/express-cli --stress [fund]` - Runs the stress tests on remote nodes. The string `fund` is needed when stress tests are ran for the first time, to fund the accounts - `../../bin/express-cli --setup-datadog` - Sets up datadog on the nodes and gets them ready to send metrics to Datadog Dashboard. `DD_API_KEY` env var is required for this. - `../../bin/express-cli --chaos [intensity]` - Adds dedicated chaos(de-peering) to the network. The `intensity` parameter is optional and can be set from `1` to `10`. If not set, `5` is used. - `../../bin/express-cli --rewind [numberOfBlocks]` - Rewinds the chain by a defined number of blocks (not greater than `128`). Default `numberOfBlocks` value is `100`. - `../../bin/express-cli --eip-1559-test [index]` - Executes a test to send EIP 1559 tx. In case of a non-dockerized devnet, if an integer [index] is specified, it will use that VM to send the tx. Otherwise, it will target the first VM. ## `matic-cli` `matic-cli` has to be installed on a `ubuntu` VM (_host_) and - through a config file - it will point to other VMs' IPs (_remotes_). - Host machine will run a Polygon node (`bor` and `heimdall`) and a layer 1 node (`ganache`) - Remote machines will only run a Polygon node each ### Requirements Please, make sure to install the following software/packages on the VMs. - Build Essentials (_host_ and _remotes_) ```bash sudo apt update sudo apt install build-essential ``` - Go 1.18+ (_host_ and _remotes_) ```bash wget https://raw.githubusercontent.com/maticnetwork/node-ansible/master/go-install.sh bash go-install.sh --remove bash go-install.sh ``` - Rabbitmq (_host_ and _remotes_) ```bash sudo apt install rabbitmq-server ``` - Docker (_host_ and _remotes_, only needed in case of a docker setup) - https://docs.docker.com/engine/install/ubuntu/ - https://docs.docker.com/engine/install/linux-postinstall/) - Node v16.17.1 (only _host_) ```bash curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash nvm install 16.17.1 ``` - Npm (only _host_) ```bash sudo apt update sudo apt install nodejs npm ``` - Python 2 (only _host_) ```bash sudo apt install python2 && alias python="/usr/bin/python2 ``` - Solc v0.5.16 (only _host_) ```bash sudo snap install solc ``` - Ganache CLI (only _host_) ```bash sudo npm install -g ganache ``` ### Usage On the _host_ machine, please run ```bash cd ~ git clone https://github.com/maticnetwork/matic-cli.git cd matic-cli npm i mkdir devnet cd devnet ``` #### Local dockerized network Adjust the [docker configs](configs/devnet/docker-setup-config.yaml) and run ```bash ../bin/matic-cli setup devnet -c ../configs/devnet/docker-setup-config.yaml ``` Once the setup is done, follow these steps for local docker deployment - Move to devnet folder ```bash cd matic-cli/devnet ``` - Start ganache ```bash bash docker-ganache-start.sh ``` - Start `heimdall` instances (it will run all services - rabbitmq, heimdall, bridge, server) ```bash bash docker-heimdall-start-all.sh ``` - Setup `bor` ```bash bash docker-bor-setup.sh ``` - Start bor ```bash bash docker-bor-start-all.sh ``` - Deploy contracts on Child chain ```bash bash ganache-deployment-bor.sh ``` - Sync contract addresses to Main chain ```bash bash ganache-deployment-sync.sh ``` Logs will be stored under `logs/` folder Note: in case of docker setup, we have provided [some additional scripts](src/setup/devnet/templates/docker/README.md) which might be helpful. #### Remote network Adjust the [remote configs](configs/devnet/remote-setup-config.yaml) and run ```bash ../bin/matic-cli setup devnet -c ../configs/devnet/remote-setup-config.yaml ``` Alternatively, this step can be executed interactively with ```bash ../bin/matic-cli setup devnet -i ``` Once the setup is done, follow these steps for remote deployment In this case, the stack is already running, you would just need to deploy/sync some contracts, as follows: - Move to devnet folder ```bash cd matic-cli/devnet ``` - Deploy contracts on Child chain ```bash bash ganache-deployment-bor.sh ``` - Sync contract addresses to Main chain ```bash bash ganache-deployment-sync.sh ``` #### Clean setup Stop al services, remove the `matic-cli/devnet` folder, and you can start the process once again #### Notes 1. The ganache URL hostname will be used for ganache `http://:9545` 2. Make sure that the _host_ machine has access to remote machines for transferring the data To persist ssh key for remote access, please run: ```bash eval "$(ssh-agent -s)" ssh-add `<.pem file>` ``` 3. We have provided the default config values [here](configs/devnet) to ensure smooth functioning of the process Please check the relative [README](configs/README.md) for more accurate description of such configs These files are used as templates and dynamically modified by `express-cli`, hence they should not be deleted nor any modification remotely pushed Therefore, they are under `.gitignore`, and in case you do not want those changes to be reflected in your local `git`, you can use the commands ```bash git update-index --assume-unchanged configs/devnet/remote-setup-config.yaml git update-index --assume-unchanged configs/devnet/docker-setup-config.yaml ``` to undo, please use ```bash git update-index --no-assume-unchanged configs/devnet/remote-setup-config.yaml git update-index --no-assume-unchanged configs/devnet/docker-setup-config.yaml ``` ## License MIT