## Advanced User Guide
For those who would like to have more control or need some customized setups, here are some step by step instructions of how to setup and deploy a solo network.

NOTE: for cleanup from previous runs, you may need to run the following command:
```
rm -Rf ~/.solo
kind delete cluster -n "${SOLO_CLUSTER_NAME}"
```

### Setup Kubernetes cluster

#### Remote cluster

* You may use remote kubernetes cluster. In this case, ensure kubernetes context is set up correctly.

```
kubectl config use-context <context-name>
```

#### Local cluster

* You may use [kind](https://kind.sigs.k8s.io/) or [microk8s](https://microk8s.io/) to create a cluster. In this case,
  ensure your Docker engine has enough resources (e.g. Memory >=8Gb, CPU: >=4). Below we show how you can use `kind` to create a cluster

First, use the following command to set up the environment variables:

```
export SOLO_CLUSTER_NAME=solo
export SOLO_NAMESPACE=solo
export SOLO_CLUSTER_SETUP_NAMESPACE=solo-cluster
```

Then run the following command to set the kubectl context to the new cluster:

```bash
kind create cluster -n "${SOLO_CLUSTER_NAME}"
```
Example output

```
Creating cluster "solo-update-readme-14261772714-1" ...
 ✓ Ensuring node image (kindest/node:v1.32.0) 🖼
 ✓ Preparing nodes 📦 
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-solo-update-readme-14261772714-1"
You can now use your cluster with:

kubectl cluster-info --context kind-solo-update-readme-14261772714-1

Thanks for using kind! 😊
```

You may now view pods in your cluster using `k9s -A` as below:


```
 Context: kind-solo                                <0> all   <a>       Attach       <ctr… ____  __.________
 Cluster: kind-solo                                          <ctrl-d>  Delete       <l>  |    |/ _/   __   \______
 User:    kind-solo                                          <d>       Describe     <p>  |      < \____    /  ___/
 K9s Rev: v0.32.5                                            <e>       Edit         <shif|    |  \   /    /\___ \
 K8s Rev: v1.27.3                                            <?>       Help         <z>  |____|__ \ /____//____  >
 CPU:     n/a                                                <shift-j> Jump Owner   <s>          \/            \/
 MEM:     n/a
┌───────────────────────────────────────────────── Pods(all)[11] ─────────────────────────────────────────────────┐
│ NAMESPACE↑          NAME                                        PF READY STATUS   RESTARTS IP          NODE     │
│ solo-setup     console-557956d575-4r5xm                    ●  1/1   Running         0 10.244.0.5  solo-con │
│ solo-setup     minio-operator-7d575c5f84-8shc9             ●  1/1   Running         0 10.244.0.6  solo-con │
│ kube-system         coredns-5d78c9869d-6cfbg                    ●  1/1   Running         0 10.244.0.4  solo-con │
│ kube-system         coredns-5d78c9869d-gxcjz                    ●  1/1   Running         0 10.244.0.3  solo-con │
│ kube-system         etcd-solo-control-plane                     ●  1/1   Running         0 172.18.0.2  solo-con │
│ kube-system         kindnet-k75z6                               ●  1/1   Running         0 172.18.0.2  solo-con │
│ kube-system         kube-apiserver-solo-control-plane           ●  1/1   Running         0 172.18.0.2  solo-con │
│ kube-system         kube-controller-manager-solo-control-plane  ●  1/1   Running         0 172.18.0.2  solo-con │
│ kube-system         kube-proxy-cct7t                            ●  1/1   Running         0 172.18.0.2  solo-con │
│ kube-system         kube-scheduler-solo-control-plane           ●  1/1   Running         0 172.18.0.2  solo-con │
│ local-path-storage  local-path-provisioner-6bc4bddd6b-gwdp6     ●  1/1   Running         0 10.244.0.2  solo-con │
│                                                                                                                 │
│                                                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```


### Step by Step Instructions

* Initialize `solo` directories:

```
# reset .solo directory
rm -rf ~/.solo

solo init
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: init
**********************************************************************************
✔ Setup home directory and cache
✔ Check dependency: helm [OS: linux, Release: 5.15.0-131-generic, Arch: x64]
✔ Check dependencies
✔ Setup chart manager

***************************************************************************************
Note: solo stores various artifacts (config, logs, keys etc.) in its home directory: /home/runner/.solo
If a full reset is needed, delete the directory or relevant sub-directories before running 'solo init'.
***************************************************************************************
✔ Copy templates in '/home/runner/.solo/cache'
```

* Generate `pem` formatted node keys

```
solo node keys --gossip-keys --tls-keys -i node1,node2,node3
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: node keys --gossip-keys --tls-keys --node-aliases node1,node2,node3
**********************************************************************************
✔ Initialize
✔ Backup old files
✔ Gossip key for node: node1
✔ Gossip key for node: node2
✔ Gossip key for node: node3
✔ Generate gossip keys
✔ Backup old files
✔ TLS key for node: node1
✔ TLS key for node: node2
✔ TLS key for node: node3
✔ Generate gRPC TLS Keys
✔ Finalize
```
PEM key files are generated in `~/.solo/keys` directory.
```
hedera-node1.crt    hedera-node3.crt    s-private-node1.pem s-public-node1.pem  unused-gossip-pem
hedera-node1.key    hedera-node3.key    s-private-node2.pem s-public-node2.pem  unused-tls
hedera-node2.crt    hedera-node4.crt    s-private-node3.pem s-public-node3.pem
hedera-node2.key    hedera-node4.key    s-private-node4.pem s-public-node4.pem
```

* Create a deployment in the specified clusters, generate RemoteConfig and LocalConfig objects.

```
solo deployment create -n "${SOLO_NAMESPACE}" --context kind-${SOLO_CLUSTER_NAME} --email "${SOLO_EMAIL}" --deployment-clusters kind-${SOLO_CLUSTER_NAME} --deployment "${SOLO_DEPLOYMENT}"
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: deployment create --node-aliases node1,node2,node3 --namespace solo --context kind-solo-update-readme-14261772714-1 --email john@doe.com --deployment-clusters kind-solo-update-readme-14261772714-1 --deployment solo-deployment
Kubernetes Namespace	: solo
**********************************************************************************
✔ Initialize
✔ Setup home directory
✔ Prompt local configuration
✔ Add new deployment to local config
✔ Resolve context for remote cluster
✔ Validate context- validated context kind-solo-update-readme-14261772714-1
✔ Update local configuration
✔ Testing connection to cluster: kind-solo-update-readme-14261772714-1
✔ Validate cluster connections
✔ Create remote config in cluster: kind-solo-update-readme-14261772714-1
✔ Create remoteConfig in clusters
```

* Setup cluster with shared components

```
solo cluster setup -s "${SOLO_CLUSTER_SETUP_NAMESPACE}"
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: cluster setup --cluster-setup-namespace solo-cluster
**********************************************************************************
✔ Initialize
✔ Prepare chart values
✔ Install 'solo-cluster-setup' chart
```

In a separate terminal, you may run `k9s` to view the pod status.

* Deploy helm chart with Hedera network components

It may take a while (5~15 minutes depending on your internet speed) to download various docker images and get the pods started.

If it fails, ensure you have enough resources allocated for Docker engine and retry the command.

```
solo network deploy -i node1,node2,node3 --deployment "${SOLO_DEPLOYMENT}"
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: network deploy --node-aliases node1,node2,node3 --deployment solo-deployment
**********************************************************************************
✔ Acquire lease - lease acquired successfully, attempt: 1/10
✔ Initialize
✔ Check if cluster setup chart is installed
✔ Copy Gossip keys to staging
✔ Copy gRPC TLS keys to staging
✔ Prepare staging directory
✔ Copy Gossip keys
✔ Node: node2, cluster: kind-solo-update-readme-14261772714-1
✔ Copy Gossip keys
✔ Node: node3, cluster: kind-solo-update-readme-14261772714-1
✔ Copy TLS keys
✔ Copy Gossip keys
✔ Node: node1, cluster: kind-solo-update-readme-14261772714-1
✔ Copy node keys to secrets
✔ Install chart 'solo-deployment'
✔ Check Node: node1, Cluster: kind-solo-update-readme-14261772714-1
✔ Check Node: node2, Cluster: kind-solo-update-readme-14261772714-1
✔ Check Node: node3, Cluster: kind-solo-update-readme-14261772714-1
✔ Check node pods are running
✔ Check HAProxy for: node3, cluster: kind-solo-update-readme-14261772714-1
✔ Check HAProxy for: node1, cluster: kind-solo-update-readme-14261772714-1
✔ Check Envoy Proxy for: node1, cluster: kind-solo-update-readme-14261772714-1
✔ Check HAProxy for: node2, cluster: kind-solo-update-readme-14261772714-1
✔ Check Envoy Proxy for: node2, cluster: kind-solo-update-readme-14261772714-1
✔ Check Envoy Proxy for: node3, cluster: kind-solo-update-readme-14261772714-1
✔ Check proxy pods are running
✔ Check MinIO
✔ Check auxiliary pods are ready
✔ Add node and proxies to remote config
```

* Setup node with Hedera platform software.
  * It may take a while as it download the hedera platform code from <https://builds.hedera.com/>

```
solo node setup -i node1,node2,node3 --deployment "${SOLO_DEPLOYMENT}"
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: node setup --node-aliases node1,node2,node3 --deployment solo-deployment
**********************************************************************************
✔ Acquire lease - lease acquired successfully, attempt: 1/10
✔ Initialize
✔ Validating state for node node1 - valid state: requested
✔ Validating state for node node2 - valid state: requested
✔ Validating state for node node3 - valid state: requested
✔ Validate nodes states
✔ Check network pod: node1
✔ Check network pod: node3
✔ Check network pod: node2
✔ Identify network pods
✔ Update node: node1 [ platformVersion = v0.58.10, context = kind-solo-update-readme-14261772714-1 ]
✔ Update node: node2 [ platformVersion = v0.58.10, context = kind-solo-update-readme-14261772714-1 ]
✔ Update node: node3 [ platformVersion = v0.58.10, context = kind-solo-update-readme-14261772714-1 ]
✔ Fetch platform software into network nodes
✔ Copy configuration files
✔ Copy configuration files
✔ Copy configuration files
✔ Set file permissions
✔ Node: node2
✔ Set file permissions
✔ Node: node3
✔ Set file permissions
✔ Node: node1
✔ Setup network nodes
✔ Change node state to setup in remote config
```

* Start the nodes

```
solo node start -i node1,node2,node3 --deployment "${SOLO_DEPLOYMENT}"
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: node start --node-aliases node1,node2,node3 --deployment solo-deployment
**********************************************************************************
✔ Acquire lease - lease acquired successfully, attempt: 1/10
✔ Initialize
✔ Validating state for node node1 - valid state: setup
✔ Validating state for node node2 - valid state: setup
✔ Validating state for node node3 - valid state: setup
✔ Validate nodes states
✔ Check network pod: node1
✔ Check network pod: node3
✔ Check network pod: node2
✔ Identify existing network nodes
✔ Start node: node2
✔ Start node: node1
✔ Start node: node3
✔ Starting nodes
✔ Check network pod: node1  - status ACTIVE, attempt: 18/300
✔ Check network pod: node3  - status ACTIVE, attempt: 18/300
✔ Check network pod: node2  - status ACTIVE, attempt: 18/300
✔ Check all nodes are ACTIVE
✔ Check proxy for node: node1
✔ Check proxy for node: node2
✔ Check proxy for node: node3
✔ Check node proxies are ACTIVE
✔ Change node state to started in remote config
✔ Adding stake for node: node1
✔ Adding stake for node: node2
✔ Adding stake for node: node3
✔ Add node stakes
```

* Deploy mirror node

```
solo mirror-node deploy --deployment "${SOLO_DEPLOYMENT}"
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: mirror-node deploy --deployment solo-deployment
**********************************************************************************
✔ Acquire lease - lease acquired successfully, attempt: 1/10
✔ Initialize
✔ Prepare address book
✔ Deploy mirror-node
✔ Enable mirror-node
✔ Check Postgres DB
✔ Check GRPC
✔ Check REST API
✔ Check Importer
✔ Check Monitor
✔ Check pods are ready
✔ Insert data in public.file_data
✔ Seed DB data
✔ Add mirror node to remote config
```

* Deploy Hedera Explorer

```
solo explorer deploy --deployment "${SOLO_DEPLOYMENT}" --cluster-ref kind-${SOLO_CLUSTER_NAME}
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: explorer deploy --deployment solo-deployment --cluster-ref kind-solo-update-readme-14261772714-1
**********************************************************************************
✔ Acquire lease - lease acquired successfully, attempt: 1/10
✔ Initialize
✔ Load remote config
✔ Install explorer
✔ Check explorer pod is ready
✔ Add explorer to remote config
```

* Deploy a JSON RPC relay

```
solo relay deploy -i node1,node2,node3 --deployment "${SOLO_DEPLOYMENT}"
```

* Example output

```

******************************* Solo *********************************************
Version			: 0.35.2
Kubernetes Context	: kind-solo-update-readme-14261772714-1
Kubernetes Cluster	: kind-solo-update-readme-14261772714-1
Current Command		: relay deploy --node-aliases node1,node2,node3 --deployment solo-deployment
**********************************************************************************
✔ Acquire lease - lease acquired successfully, attempt: 1/10
✔ Initialize
✔ Prepare chart values
✔ Deploy JSON RPC Relay
✔ Check relay is ready
✔ Add relay component in remote config
```

You may view the list of pods using `k9s` as below:

```
Context: kind-solo                                <0> all   <a>       Attach       <ctr… ____  __.________
 Cluster: kind-solo                                          <ctrl-d>  Delete       <l>  |    |/ _/   __   \______
 User:    kind-solo                                          <d>       Describe     <p>  |      < \____    /  ___/
 K9s Rev: v0.32.5                                            <e>       Edit         <shif|    |  \   /    /\___ \
 K8s Rev: v1.27.3                                            <?>       Help         <z>  |____|__ \ /____//____  >
 CPU:     n/a                                                <shift-j> Jump Owner   <s>          \/            \/
 MEM:     n/a
┌───────────────────────────────────────────────── Pods(all)[31] ─────────────────────────────────────────────────┐
│ NAMESPACE↑          NAME                                                           PF READY STATUS   RESTARTS I │
│ kube-system         coredns-5d78c9869d-994t4                                       ●  1/1   Running         0 1 │
│ kube-system         coredns-5d78c9869d-vgt4q                                       ●  1/1   Running         0 1 │
│ kube-system         etcd-solo-control-plane                                        ●  1/1   Running         0 1 │
│ kube-system         kindnet-q26c9                                                  ●  1/1   Running         0 1 │
│ kube-system         kube-apiserver-solo-control-plane                              ●  1/1   Running         0 1 │
│ kube-system         kube-controller-manager-solo-control-plane                     ●  1/1   Running         0 1 │
│ kube-system         kube-proxy-9b27j                                               ●  1/1   Running         0 1 │
│ kube-system         kube-scheduler-solo-control-plane                              ●  1/1   Running         0 1 │
│ local-path-storage  local-path-provisioner-6bc4bddd6b-4mv8c                        ●  1/1   Running         0 1 │
│ solo                envoy-proxy-node1-65f8879dcc-rwg97                             ●  1/1   Running         0 1 │
│ solo                envoy-proxy-node2-667f848689-628cx                             ●  1/1   Running         0 1 │
│ solo                envoy-proxy-node3-6bb4b4cbdf-dmwtr                             ●  1/1   Running         0 1 │
│ solo                solo-deployment-grpc-75bb9c6c55-l7kvt                     ●  1/1   Running         0 1 │
│ solo                solo-deployment-hedera-explorer-6565ccb4cb-9dbw2          ●  1/1   Running         0 1 │
│ solo                solo-deployment-importer-dd74fd466-vs4mb                  ●  1/1   Running         0 1 │
│ solo                solo-deployment-monitor-54b8f57db9-fn5qq                  ●  1/1   Running         0 1 │
│ solo                solo-deployment-postgres-postgresql-0                     ●  1/1   Running         0 1 │
│ solo                solo-deployment-redis-node-0                              ●  2/2   Running         0 1 │
│ solo                solo-deployment-rest-6d48f8dbfc-plbp2                     ●  1/1   Running         0 1 │
│ solo                solo-deployment-restjava-5d6c4cb648-r597f                 ●  1/1   Running         0 1 │
│ solo                solo-deployment-web3-55fdfbc7f7-lzhfl                     ●  1/1   Running         0 1 │
│ solo                haproxy-node1-785b9b6f9b-676mr                                 ●  1/1   Running         1 1 │
│ solo                haproxy-node2-644b8c76d-v9mg6                                  ●  1/1   Running         1 1 │
│ solo                haproxy-node3-fbffdb64-272t2                                   ●  1/1   Running         1 1 │
│ solo                minio-pool-1-0                                                 ●  2/2   Running         1 1 │
│ solo                network-node1-0                                                ●  5/5   Running         2 1 │
│ solo                network-node2-0                                                ●  5/5   Running         2 1 │
│ solo                network-node3-0                                                ●  5/5   Running         2 1 │
│ solo                relay-node1-node2-node3-hedera-json-rpc-relay-ddd4c8d8b-hdlpb  ●  1/1   Running         0 1 │
│ solo-cluster        console-557956d575-c5qp7                                       ●  1/1   Running         0 1 │
│ solo-cluster        minio-operator-7d575c5f84-xdwwz                                ●  1/1   Running         0 1 │
│                                                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```