# Rally Tools This documentation is currently WIP. Some sentences may randomly end and sections may be poorly organized. This repository provides multiple helpful tools for working within the SDVI Rally environment. Some features are: - Preset uploader - Rule uploader - Automated deployer - Remote diff checker - Code sync checker # Installation - `npm install -g rally-tools` - `rally` (to check if it works) - `rally config` # Library usage Most classes and functions are exposed through src/index.js. This means when you want to create a node plugin that uses this library, you should just use `import {Preset, Rule, rallyFunctions, ...etc} from "rally-tools"`. # Development - Clone this repo - Run `npm install` - Run `npm link` to symlink the `rally` executable - Start `npx rollup -cw` or `npm run watch` in the background to automatically compile code - Test your changes - Commit your changes and run `npm version ` to increment the version # Config Pass the `--config` option to read/write a different config file location. If including as a module, then you need to call ``` const rally = require("rally-tools"); rally.loadConfig(filename); // OR rally.setConfig({... config object here ...}) ``` Options: - chalk: allow colored output. - restrictUAT: Only allow GET requests from UAT, not POST/PUT/etc. - api: Your api keys and urls - repodir: The directory of your repository. Should have 3 folders: `silo-presets`, `silo-rules`, `silo-metadata`. - defaultEnv: Your development environment, usually DEV. # Usage To get started, run `rally` in any command prompt or terminal. If your config is setup, you will see environment data for each of your setup api keys. It will look like this: ``` Rally Tools vx.y.z CLI LOCAL: OK UAT: 200 OK DEV: Unconfigured PROD: 200 OK QA: 401 (Unauthorized) ``` Use `rally help` or `rally help [command]` to see all public commands and basic documentation. #### `rally preset` This command deals with preset actions such as creating, uploading, and downloading. `rally preset create` can be used to create a preset. When run without arguments, a command line UI will be given. If you wish to script this, the flags `--provider`, `--ext`, and `--name` can be given. The basic download usage is `rally preset list`, which lists all presets. Giving the `--resolve` flag will internal resolve the dynamic references in an object. You can then add these to the output using `--attach`. Ex. `rally preset list -e PROD --resolve --attach` `rally preset upload -e [env] -f [preset]` can be used to upload a file to a remote env. You can specify multiple -f arguments to upload multiple files. If the - argument is given (`rally preset upload -`) then the files are read from stdin. For example: `git diff HEAD..UAT --name-only | grep silo-presets | rally preset upload -` will upload all changed files using git as the reference. `rally preset diff -f [preset]` can be used to view the differences between a local file and a remote one. `--command` can be used to run a command other than diff. For example, `rally preset diff -f abc.xyz --command vimdiff -e PROD` would compare the local file abc.xyz to the remote version on prod using vimdiff. (make sure that zbc.xyz has a proper rally header/metadata or this will fail). `rally preset grab -f [preset]` will attempt to download the metadata file for this asset. The `--full` argument can be given to also download the `code`, too. #### `rally rule` This command is similar to `rally preset`, but for `Supply Chain Rules`. `rally rule create` can be used in the same fashion as `rally preset create`. To access the interactive rule creator, just run `rally rule create` with no arguments. To see all rules, use `rally rule list`. `--raw` available. #### `rally provider` See all providers. `rally provider list`. `--raw` available. #### `rally asset` This command allows you to create and launch workflows on assets. See `rally help asset` for a quick reference help menu. The first part of the command will be getting an asset context. You can either: - Use an asset id (ex. discovery.sdvi.com/content/[id]). - add the `--id [id]` argument - ex. `rally asset -e PROD --id 12345 launch ...` - Use an asset name - add the `--name [asset name]` argument - ex. `rally asset -e UAT --name 1232345_004_TCCS_123456_2 launch ...` - Create a new asset - add the argument "create" - supply a name using --name - `#` will be replaces with a random number. - ex. `rally asset create --name "TEST_FILE_#" -e UAT launch ...` - Use an anonymous context. (not supported by all commands) - add the `--anon` argument - ex. `rally asset --anon -e PROD launch ...` Once you have your target asset, you can run any of the following commands: #### `launch`, `launchEvaluate`: Launch a rule or evaluate on an asset. Works in anon contexts. Requires flag `--job-name`. Optional flag `--init-data` can supply data to the step in json format. It can load the json from a file, or receive it as a string, or read from stdin. `--priority` is planned as a flag, but rally currently does not support dynamic priority on started jobs. Launching as an evaluate means that no next steps will be ran. ie. - from file: `--init-data @filename.json` - from text: `--init-data '{"some": "json", "here": "yep"}` - from stdin: `--init-data -` Example: `rally asset ... launch --job-name "00 john sandbox" --init-data '{"transcode": "XDCAM"}' `rally asset ... launchEvaluate --job-name "00 john sandbox" --init-data '{"transcode": "XDCAM"}' #### `rally supply` This is probably the most complex command mechanically. `rally supply calc [starting rule]` will create a supply chain object in memory. Then, using other flags you can do something with this chain. - `--to [env]` will copy the supply chain onto the env, creating new rules and presets as needed. - `--check [env]` will do a diff on each file in the chain to the remote given `rally supply make -f [files]` #### `rally conifg` This command manages the "~/.rallyconfig" file, so that you don't need to edit it manually. `rally config` simply creates a new config walking through all the options. `rally config [key]` gives the config interactor for a single key. `rally config chalk` would bring up y/n menu for color. `rally config api` would bring up the configuration for all all the environments, but `rally config api.DEV` would let you modify just the DEV credentials. `rally config --raw` prints out the current config *including configs changed by command line options* #### `metadata`: This command prints metadata. This will have two top level keys: - `Workflow` contains the workflow metadata - `Metadata` contains the supply chain metadata - `AnalyzeInfo` contains a random analyze artifact. Usually, this will be the main file, but it is highly recommend to no rely on this data being accurate. The default print will use the internal node debug print. For full json, add the `--raw` argument. # Deployments Deployments using this tool are based around supply chains. At their core, supply chains are simply a group of rally objects, where an object is either a rule, preset, or notification. Although you can only deploy supply chains, there are many ways to construct the deployment you want. The first, recommended way is using `rally supply make`. `make` takes a list of identifiers and constructs a supply chain. Identifiers come in two forms: - Remote types - These point to a unique preset or rule on some remote environment. - `R-UAT-283` would mean rule id 283 on uat. - ex: `P-UAT-283: Something python`, `R-DEV-617: Some rule` - Local types - These point to a local file, relative to your repodir or absolute to your system - `./some/file/path` - `/user/someone/rally_repodir/some/file/path` - Displayed as `P-LOCAL: Something` You can take take list of local or remote identifiers to create a supply chain using `rally supply make`. Each file can be given by `-f`, or by using `-` to specify stdin. A shorthand for `rally supply make -` is `rally @`. This is the most used development command. Lets say you edited these 3 objects in DEV. ``` $ cat > changes.txt P-DEV-283: NL - EST - Util Library P-DEV-285: NL P1000 - MP - Non Linear Media Preparation Workflow R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow $ cat changes.txt | rally @ Reading from stdin Required notifications: Required rules: 1 R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow Required presets: 2 P-DEV-283: NL - EST - Util Library P-DEV-285: NL P1000 - MP - Non Linear Media Preparation Workflow ``` Rally tools by default will print out the current loaded supply chain object when using make. To do something else, you supply any number of post actions There are 3 available post actions: - `--to` - This is the bread and butter deployment action. `--to DEV` would deploy the current supply chain onto DEV, adding or creating Now you can treat this like any other supply chain, and deploy it. Remote to remote, or remote to local. However, this tool is built to integrate directly with git on your local filesystem. If you edited those 3 files locally, then commited to git, you should be able to see the diff with the git command `git diff HEAD HEAD^`. We are only interested in the names, so lets get those. Quick note: The shorthand for `rally supply make -` is `rally @`. ``` $ git diff HEAD HEAD^ --name-only silo-presets/NL - EST - Util Library silo-presets/NL P1000 - MP - Non Linear Media Preparation Workflow silo-rules/NL R1000 - MP - Non Linear Media Preparation Workflow $ #Passing those to make will produce a supply chain based on LOCAL $ git diff HEAD HEAD^ --name-only | rally @ Reading from stdin Required notifications: Required rules: 1 R-LOCAL: NL R1000 - MP - Non Linear Media Preparation Workflow Required presets: 2 P-LOCAL: NL - EST - Util Library P-LOCAL: NL P1000 - MP - Non Linear Media Preparation Workflow ``` There is another way to do deployments that has been deprecated: calc automatically generated links between rules and preset at a time where our git repos did not have all available presets. This is left here as a guide for older scripts, but may not work correctly on new versions: `rally supply calc [starting rule] [ending rule]` does the heavy lifting of parsing rules, finding notifications, linking the presets, creating metadata, etc. Using the E2 Supply chain as an example... ``` $ rally supply calc R1000 -e DEV ... Calculating Supply chain... R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow Done! Required notifications: N-21: SNS All - test Required rules: 8 R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow R-DEV-618: NL R2001 - MP - Make EST Media Articrafts by Split R-DEV-621: NL R3012 - MP - Make EST Media File using Media Convert R-DEV-622: NL R4001 - MP - Make EST Closed Caption File R-DEV-627: NL R3013 - MP - QC EST Media File Launcher R-DEV-623: NL R5001 - MP - Make EST Media ArtiCrafts by Join R-DEV-628: NL R3014 - MP - QC EST Media File using SimpleSDVIQC R-DEV-623: NL R5001 - MP - Make EST Media ArtiCrafts by Join Required presets: 10 P-DEV-285: NL P1000 - MP - Non Linear Media Preparation Workflow P-DEV-18: Fail P-DEV-283: NL - EST - Util Library P-DEV-284: NL - MP - Util Library P-DEV-286: NL P2001 - MP - Make EST Media Articrafts by Split P-DEV-289: NL P3012 - MP - Make EST Media File using MediaConvert P-DEV-290: NL P4001 - MP - Make EST Closed Caption File P-DEV-306: NL P3013 - MP - QC EST Media File Launcher P-DEV-291: NL P5001 - MP - Make EST Media ArtiCrafts by Join P-DEV-307: NL P3014 - MP - QC EST Media File using SimpleSDVIQC ``` This internally creates a supply chain object, which we can then apply an action to. An example action is `sync`, which is given by the `--to` arg. `rally supply calc R1000 -e DEV --to LOCAL` would sync this supply chain (based on DEV) to LOCAL. In order to move it to a protected envornment, add --no-protect. However, calc is limited by the fact that it is very rigid. Its best use is the inital setup of an environment, or to mass move supply chains. To fix this, lets move to `rally supply make` To give a generic approach, in order to deploy all the changes between 2 commits, run `git diff featureCommit baseCommit --name-only | rally @ --to DEV`. `rally` is currently stateless: It does not remember what is deployed, who deployed it or when. All this should be tracked through git. Therefore, tagging releases or using a release branch would allow for basic version control. ## Automated deployments Automated deployments should be constructed telling rally tools the list of changed presets and rules. Silo constants, notification presets, and silo metadata should be changed manually before an automatic deploy, as these can not be stored in source control. For example, if the previous deployment to prod was the tagged commit `v1.2.3`, and the new deployment will be version `v2.0.0`, then the deployment script will be `git diff v2.0.0 v1.2.3 --name-only | rally @ --to PROD --no-protect` This will need to be run on a computer with a .rallyconfig in the home directory, otherwise the config should be given by the --config flag to the rally command. # Examples Heres some other examples of common usage: Upload a preset `rally preset upload -e DEV -f "~/ORP/silo-presets/Audio Metadata Conditioner.py"` Look at all ffmpeg jobs `rally rule list -e DEV --resolve --attach | grep ffmpeg` Clone some ffmpeg jobs you want to edit (`vipe` optional) `rally rule list -e DEV --resolve --attach | grep ffmpeg | vipe | rally supply make - --to LOCAL` Create a new supply chain and print it `rally supply calc ORHIVE` # Header Parsing There are two types of headers in standard rally usage. The first is the rally docstring. It looks like this: ``` ''' name: (Name of the preset as it exists on the silo) autotest: (Name of movie for testing purposes) autotest: (Name of movie, supports and arbitrary nubmer) autotest: (Name of movie, ....) autotest: id: (id of movie to test) ''' ``` This docstring contains into about what the name of the preset on the enviornment should be and what tests should be run on upload. The docstring is parsed by rally tools on upload. It does not need to be at the top of the file, and can use either single-quotes `'`, double-qoutes `"`. Using `#` for the docstring is discouraged. This header is automatically generated when using `rally preset create`. `-` can be used to disable autotests temporarily. ``` # name: ok # autotest: ok // name: also works -autotest: will not run -- autotest: this will run ``` The second type of header is the deployment info header. On any upload from rally tools, we will look for a file named `bin/header.sh`. This will be inserted at the start of each python upload using info about deployment time, git info, and uploader name. On any automatic download, this header will be automatically stripped. You should never see this type of header in git locally. Here is an example header format: ``` # Built On: Wed 2020/01/02 12:53:13pm # Author: John Schmidt # Tag: releases/23 # Build: # Version: . # Branch: staging # Commit: 20ab14bc6d16a19e60962efbe213a33fc21bafb7 # Local File: YEP/silo-presets/COC.py ############################################################### ``` The deployment info header can be read with `rally preset info`. It is used like a rally preset upload command, where you supply the local file to be read from multiple environments. It will print dependencies and then show all the build info. Heres an example output: ``` $ rally preset info --file "YEP/silo-presets/COC.py" --e UAT,PROD - COC - Some Checkin Library - cool client lib - Silo Constants - client lib helpers - lib/common_vars - Other Library - Third Library - (seen) Other Library - (seen) lib/common_vars - (miss) Some Missing Preset ENV: UAT, updated ~8 hours ago Built on Wed 2020/09/02 04:10:39pm by John Schmidt From (unknown) on feature-1234 (20ab14bc6d16a19e60962efbe213a33fc21bafb7) ENV: PROD, updated ~8 days ago Built on Wed Aug 26 13:11:33 UTC 2020 by Other Dev From 124 on staging (20ab14bc6d16a19e60962efbe213a33fc21bafb7) ``` #### Atom integration Rally tools now supports a basic amount of atom integration including testing, uploading, downloading, and rule managment. Two plugins are used for this: process-pallete, and optionally, file-watcher. Please see the file `process-pallete.json` in `jderby/ONRAMP_WORKFLOW_PYTHON`. This should be copied into your base directory (same level as the silo-\* folders) This is still early in testing and does not support features like diffs and inline live test results #### Vim integration Anyone else use vim? Just me? Heres my config: all the file arguments are `^R%` where ^R is the CTRL-R sequence (register-insert-command mode). Use `CTRL-V` in insert mode to enter insert-escape mode, then press `CTRL-R` to type that sequence. ``` nnoremap u :!rally preset upload --file "%" -e UAT nnoremap U :!rally preset upload --file "%" -e PROD --no-protect nnoremap u :!rally supply make --file "%" --to UAT nnoremap i :!rally supply make --file "%" --to QA nnoremap U :!rally supply make --file "%" --to PROD --no-protect nnoremap k :!rally preset info --file "%" --e UAT,PROD nnoremap d :call Rallydiff("") nnoremap D :call Rallydiff("-e PROD") nnoremap c :call Rallydiff("-e QA") nnoremap C :call Rallydiff("-e DEV") nnoremap D :diffoff nnoremap Q :%!node ~/node-rally-tools/util/addMIOSupport.js nnoremap N :%!node ~/node-rally-tools/util/addDynamicNext.js set splitright function! Rallydiff(extra) let file = system("rally preset diff --only-new --file '" . bufname("%") . "' --raw " . a:extra) execute "silent vs" . file execute "silent windo diffthis" "echo file endfunction ``` # Troubleshooting #### Cannot acclimatize shelled preset Solution: Create the preset on the remote enviornment manually, or run `rally preset create` Under normal usage, presets will have an associated metadata file saved. This contains information like its provider type, input and output settings, or timestamps. `Preset#acclimatize` attempts to take this data from a generic format into an environment specific format so that it can be accuractly created when uploading. A file without any metadata is marked as "Shelled" and given some dummy data while limiting functionality. This functionality includes updating the code of a preset, or viewing the metadata of an enviornment. #### CLI Aborted: Protected enviorment Solution: Add the --no-protect flag, or run `rally config restrictUAT` to unprotect UAT (if the error is on UAT). Protected enviorments cannot recieve anything but get requests, so any kind of POST/PUT/PATCH will fail with this error. Internally, --no-protect is sets the --protect flag to false instead of true, which in turn sets the configObject.dangerModify flag to true. So if you really wish, you could add `"dangerModify": true,` to your config to allow unrestricted UAT/PROD posts, then use the --protect flag when you want safe calls. #### API Error Sometimes, the rally API simply wont work. Verify that all endpoints are active by running `rally`. Under normal circumstances, they should return a 2xx response. If that is ok, read the data that is returned by the API to see if it is a fixable error: ex. `401 Unauthorized` probably means that you have a bad API key, so run `rally config api` or `rally config api.UAT`. #### My problem isn't in the list Ask me on SDVI or discocomm slack @John Schmidt