1 | # Rally Tools
|
2 |
|
3 | This repository provides multiple helpful tools for working within the SDVI
|
4 | Rally enviornment. Some features are:
|
5 |
|
6 | - Preset uploader
|
7 | - Rule uploader
|
8 | - Automated deployer
|
9 | - Remote diff checker
|
10 | - Code sync checker
|
11 |
|
12 | ## Installation
|
13 |
|
14 | - `npm install -g rally-tools`
|
15 | - `rally` (to check if it works)
|
16 | - `rally config`
|
17 |
|
18 | ## Library usage
|
19 |
|
20 | Most classes and functions are exposed through src/index.js. This means when
|
21 | you want to create a node plugin that uses this library, you should just use
|
22 | `import {Preset, Rule, rallyFunctions, ...etc} from "rally-tools"`.
|
23 |
|
24 | ## Development
|
25 |
|
26 | - Clone this repo
|
27 | - Run `npm install`
|
28 | - Run `npm link` to symlink the `rally` executable
|
29 | - Start `rollup -cw` in the background to automatically compile code
|
30 | - Test your changes
|
31 | - Commit your changes and run `npm version <minor|patch>` to increment the
|
32 | version
|
33 |
|
34 | ## Config
|
35 |
|
36 | Pass the `--config` option to read/write a different config file location. If
|
37 | including as a module, then you need to call
|
38 |
|
39 | ```
|
40 | const rally = require("rally-tools");
|
41 |
|
42 | rally.loadConfig(filename);
|
43 | // OR
|
44 | rally.setConfig({... config object here ...})
|
45 | ```
|
46 |
|
47 | Options:
|
48 |
|
49 | - chalk: allow colored output.
|
50 | - restrictUAT: Only allow GET requests from UAT, not POST/PUT/etc.
|
51 | - api: Your api keys and urls
|
52 | - repodir: The directory of your repository. Should have 3 folders:
|
53 | `silo-presets`, `silo-rules`, `silo-metadata`.
|
54 | - defaultEnv: Your development enviornment, usually DEV.
|
55 |
|
56 | ## Usage
|
57 |
|
58 | Use `rally help` or `rally help [command]` to see all public commands and basic
|
59 | documentation.
|
60 |
|
61 | #### `rally preset`
|
62 |
|
63 | This command deals with preset actions such as creating, uploading, and
|
64 | downloading.
|
65 |
|
66 | `rally preset create` can be used to create a preset. When run without
|
67 | arguments, a command line UI will be given. If you wish to script this, the flags
|
68 | `--provider`, `--ext`, and `--name` can be given.
|
69 |
|
70 | The basic download usage is `rally preset list`, which lists all presets.
|
71 | Giving the `--resolve` flag will internall resolve the dynamic references in an
|
72 | object. You can then add these to the output using `--attach`. Ex. `rally
|
73 | preset list -e PROD --resolve --attach`
|
74 |
|
75 | `rally preset upload -e [env] -f [preset]` can be used to upload a file to a
|
76 | remote env. You can specify multiple -f arguments to upload multiple files. If
|
77 | the - argument is given (`rally preset upload -`) then the files are read from
|
78 | stdin. For example: `git diff HEAD..UAT --name-only | grep silo-presets |
|
79 | rally preset upload -` will upload all changed files using git as the reference.
|
80 |
|
81 | `rally preset diff -f [preset]` can be used to view the differences between a
|
82 | local file and a remote one. `--command` can be used to run a command other
|
83 | than diff. For example, `rally preset diff -f abc.xyz --command vimdiff -e PROD`
|
84 | would compare the local file abc.xyz to the remote version on prod using
|
85 | vimdiff. (make sure that zbc.xyz has a proper rally header/metadata or this
|
86 | will fail).
|
87 |
|
88 | `rally preset grab -f [preset]` will attempt to download the metadata file for
|
89 | this asset. The `--full` argument can be given to also download the `code`, too.
|
90 |
|
91 | #### `rally rule`
|
92 |
|
93 | See all rules. `rally rule list`. `--raw` available.
|
94 |
|
95 | #### `rally provider`
|
96 |
|
97 | See all providers. `rally provider list`. `--raw` available.
|
98 |
|
99 | #### `rally asset`
|
100 |
|
101 | This command allows you to create and launch workflows on assets.
|
102 |
|
103 | The first part of the command will be getting an asset context. You can either:
|
104 | - Use an asset id (ex. discovery.sdvi.com/content/[id]).
|
105 | - add the `--id [id]` argument
|
106 | - `rally asset --id 12345 launch ...`
|
107 | - Use an asset name
|
108 | - add the `--name [asset name]` argument
|
109 | - `rally asset --name 1232345_004_TCCS_123456_2 launch ...`
|
110 | - Create a new asset
|
111 | - add the argument "create"
|
112 | - supply a name using --name
|
113 | - `#` will be replaces with a random number.
|
114 | - `rally asset create --name "TEST_FILE_#" launch`
|
115 |
|
116 | #### `rally supply`
|
117 |
|
118 | This is probably the most complex command mechanically.
|
119 |
|
120 | `rally supply calc [starting rule]` will create a supply chain object in memory.
|
121 | Then, using other flags you can do something with this chain.
|
122 |
|
123 | - `--to [env]` will copy the supply chain onto the env, creating new rules and
|
124 | presets as needed.
|
125 | - `--check [env]` will do a diff on each file in the chain to the remote given
|
126 |
|
127 | `rally supply make -f [files]`
|
128 |
|
129 | #### `rally conifg`
|
130 |
|
131 | This command manages the "~/.rallyconfig" file, so that you dont need to edit
|
132 | it manually. `rally config` simply creates a new config walking through all the
|
133 | options.
|
134 |
|
135 | `rally config [key]` gives the config interactor for a single key. `rally
|
136 | config chalk` would bring up y/n menu for color. `rally config api` would bring
|
137 | up the configuration for all all the enviornments, but `rally config api.DEV`
|
138 | would let you modify just the DEV credentials.
|
139 |
|
140 | `rally config --raw` prints out the current config *including configs changed
|
141 | by command line options*
|
142 |
|
143 | #### Deployments
|
144 |
|
145 | Deployments using this tool are based around supply chains. At their core,
|
146 | supply chains are simply a group of rally objects, where an object is either a
|
147 | rule, preset, or notification.
|
148 |
|
149 | Although you can only deploy supply chains, there are many ways to construct
|
150 | the deployment you want.
|
151 |
|
152 | The first, and easiest way you'll probably encounter is through `rally supply
|
153 | calc [starting rule] [ending rule]`. This does the heavy lifting of parsing
|
154 | rules, finding notifications, linking the presets, creating metadata, etc.
|
155 |
|
156 | Using the E2 Supply chain as an example...
|
157 |
|
158 | ```
|
159 | $ rally supply calc R1000 -e DEV
|
160 | ...
|
161 | Calculating Supply chain... R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow
|
162 | Done!
|
163 | Required notifications:
|
164 | N-21: SNS All - test
|
165 | Required rules: 8
|
166 | R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow
|
167 | R-DEV-618: NL R2001 - MP - Make EST Media Articrafts by Split
|
168 | R-DEV-621: NL R3012 - MP - Make EST Media File using Media Convert
|
169 | R-DEV-622: NL R4001 - MP - Make EST Closed Caption File
|
170 | R-DEV-627: NL R3013 - MP - QC EST Media File Launcher
|
171 | R-DEV-623: NL R5001 - MP - Make EST Media ArtiCrafts by Join
|
172 | R-DEV-628: NL R3014 - MP - QC EST Media File using SimpleSDVIQC
|
173 | R-DEV-623: NL R5001 - MP - Make EST Media ArtiCrafts by Join
|
174 | Required presets: 10
|
175 | P-DEV-285: NL P1000 - MP - Non Linear Media Preparation Workflow
|
176 | P-DEV-18: Fail
|
177 | P-DEV-283: NL - EST - Util Library
|
178 | P-DEV-284: NL - MP - Util Library
|
179 | P-DEV-286: NL P2001 - MP - Make EST Media Articrafts by Split
|
180 | P-DEV-289: NL P3012 - MP - Make EST Media File using MediaConvert
|
181 | P-DEV-290: NL P4001 - MP - Make EST Closed Caption File
|
182 | P-DEV-306: NL P3013 - MP - QC EST Media File Launcher
|
183 | P-DEV-291: NL P5001 - MP - Make EST Media ArtiCrafts by Join
|
184 | P-DEV-307: NL P3014 - MP - QC EST Media File using SimpleSDVIQC
|
185 | ```
|
186 |
|
187 | This internally creates a supply chain object, which we can then apply an action to.
|
188 |
|
189 | An example action is `sync`, which is given by the `--to` arg. `rally supply
|
190 | calc R1000 -e DEV --to LOCAL` would sync this supply chain (based on DEV) to
|
191 | LOCAL. In order to move it to a protected envornment, add --no-protect.
|
192 |
|
193 | However, calc is limited by the fact that it is very rigid. Its best use is the
|
194 | inital setup of an environment, or to mass move supply chains. To fix this,
|
195 | lets move to `rally supply make`
|
196 |
|
197 | `make` takes a list of identifiers and constructs a supply chain. Identifiers
|
198 | are passed in by the `this.files` array. From the command line this can be
|
199 | given by suppling -f arguments or reading from stdin.
|
200 |
|
201 | Lets say you edited these 3 objects in DEV.
|
202 |
|
203 | ```
|
204 | $ cat > changes.txt
|
205 | P-DEV-283: NL - EST - Util Library
|
206 | P-DEV-285: NL P1000 - MP - Non Linear Media Preparation Workflow
|
207 | R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow
|
208 |
|
209 | $ cat changes.txt | rally supply make -
|
210 | Reading from stdin
|
211 | Required notifications:
|
212 | Required rules: 1
|
213 | R-DEV-617: NL R1000 - MP - Non Linear Media Preparation Workflow
|
214 | Required presets: 2
|
215 | P-DEV-283: NL - EST - Util Library
|
216 | P-DEV-285: NL P1000 - MP - Non Linear Media Preparation Workflow
|
217 | ```
|
218 |
|
219 | Now you can treat this like any other supply chain, and deploy it. Remote to
|
220 | remote, or remote to local. However, this tool is built to integrate directly
|
221 | with git on your local filesystem.
|
222 |
|
223 | If you edited those 3 files locally, then commited to git, you should be able
|
224 | to see the diff with the git command `git diff HEAD HEAD^`. We are only
|
225 | interested in the names, so lets get those.
|
226 |
|
227 | Quick note: The shorthand for `rally supply make -` is `rally @`.
|
228 |
|
229 | ```
|
230 | $ git diff HEAD HEAD^ --name-only
|
231 | silo-presets/NL - EST - Util Library
|
232 | silo-presets/NL P1000 - MP - Non Linear Media Preparation Workflow
|
233 | silo-rules/NL R1000 - MP - Non Linear Media Preparation Workflow
|
234 |
|
235 | $ #Passing those to make will produce a supply chain based on LOCAL
|
236 | $ git diff HEAD HEAD^ --name-only | rally @
|
237 | Reading from stdin
|
238 | Required notifications:
|
239 | Required rules: 1
|
240 | R-LOCAL: NL R1000 - MP - Non Linear Media Preparation Workflow
|
241 | Required presets: 2
|
242 | P-LOCAL: NL - EST - Util Library
|
243 | P-LOCAL: NL P1000 - MP - Non Linear Media Preparation Workflow
|
244 | ```
|
245 |
|
246 | To give a generic approach, in order to deploy all the changes between 2
|
247 | commits, run `git diff featureCommit baseCommit --name-only | rally @
|
248 | --to DEV`. `rally` is currently stateless: It does not remember what is
|
249 | deployed, who deployed it or when. All this should be tracked through git.
|
250 | Therefore, tagging releases or using a release branch would allow for basic
|
251 | version control.
|
252 |
|
253 | #### Automated deployments
|
254 |
|
255 | Automated deployments should be constructed telling rally tools the list of
|
256 | changed presets and rules. Silo constants, notification presets, and silo
|
257 | metadata should be changed manually before an automatic deploy, as these can not
|
258 | be stored in source control.
|
259 |
|
260 | For example, if the previous deployment to prod was the tagged commit `v1.2.3`,
|
261 | and the new deployment will be version `v2.0.0`, then the deployment script will
|
262 | be `git diff v2.0.0 v1.2.3 --name-only | rally @ --to PROD --no-protect`
|
263 |
|
264 | This will need to be run on a computer with a .rallyconfig in the home
|
265 | directory, otherwise the config should be given by the --config flag to the
|
266 | rally command.
|
267 |
|
268 | #### Examples
|
269 | Heres some other examples of common usage:
|
270 |
|
271 | Upload a preset
|
272 | `rally preset upload -e DEV -f "~/ORP/silo-presets/Audio Metadata Conditioner.py"`
|
273 |
|
274 | Look at all ffmpeg jobs
|
275 | `rally rule list -e DEV --resolve --attach | grep ffmpeg`
|
276 |
|
277 | Clone some ffmpeg jobs you want to edit (`vipe` optional)
|
278 | `rally rule list -e DEV --resolve --attach | grep ffmpeg | vipe | rally supply make - --to LOCAL`
|
279 |
|
280 | Create a new supply chain and print it
|
281 | `rally supply calc ORHIVE`
|
282 |
|
283 | #### Header Parsing
|
284 |
|
285 | Rally tools supports a standard header format. Preset name
|
286 |
|
287 | ```
|
288 | '''
|
289 | name: (Name of the preset as it exists on the silo)
|
290 | autotest: (Name of movie for testing purposes)
|
291 | autotest: (Name of movie, supports and arbitrary nubmer)
|
292 | autotest: (Name of movie, ....)
|
293 | autotest: id: (id of movie to test)
|
294 | '''
|
295 | ```
|
296 |
|
297 | This can also exist using any other comment symbol except `-` directly
|
298 | preceding a key. For this reason, `-` can be used to disable autotests
|
299 | temporarily.
|
300 |
|
301 | ```
|
302 | # name: ok
|
303 | # autotest: ok
|
304 | // name: also works
|
305 | -autotest: will not run
|
306 | -- autotest: this will run
|
307 | ```
|
308 |
|
309 | #### Atom integration
|
310 |
|
311 | Rally tools now supports a basic amount of atom integration including testing,
|
312 | uploading, downloading, and rule managment. Two plugins are used for this:
|
313 | process-pallete, and optionally, file-watcher. Please see the file
|
314 | `process-pallete.json` in `jderby/ONRAMP_WORKFLOW_PYTHON`.
|
315 |
|
316 | This should be copied into your base directory (same level as the silo-\*
|
317 | folders)
|
318 |
|
319 | This is still early in testing and does not support features like diffs and
|
320 | inline live test results
|
321 |
|
322 |
|
323 | ## Troubleshooting
|
324 |
|
325 | #### Cannot acclimatize shelled preset
|
326 |
|
327 | Solution: Create the preset on the remote enviornment manually, or run `rally
|
328 | preset create`
|
329 |
|
330 | Under normal usage, presets will have an associated metadata file saved. This
|
331 | contains information like its provider type, input and output settings, or
|
332 | timestamps. `Preset#acclimatize` attempts to take this data from a generic
|
333 | format into an environment specific format so that it can be accuractly created
|
334 | when uploading. A file without any metadata is marked as "Shelled" and given
|
335 | some dummy data while limiting functionality. This functionality includes
|
336 | updating the code of a preset, or viewing the metadata of an enviornment.
|
337 |
|
338 | #### CLI Aborted: Protected enviorment
|
339 |
|
340 | Solution: Add the --no-protect flag, or run `rally config restrictUAT` to
|
341 | unprotect UAT (if the error is on UAT).
|
342 |
|
343 | Protected enviorments cannot recieve anything but get requests, so any kind of
|
344 | POST/PUT/PATCH will fail with this error. Internally, --no-protect is sets the
|
345 | --protect flag to false instead of true, which in turn sets the
|
346 | configObject.dangerModify flag to true. So if you really wish, you could add
|
347 | `"dangerModify": true,` to your config to allow unrestricted UAT/PROD posts,
|
348 | then use the --protect flag when you want safe calls.
|
349 |
|
350 | #### API Error
|
351 |
|
352 | Sometimes, the rally API simply wont work. Verify that all endpoints are active
|
353 | by running `rally`. Under normal circumstances, they should return a 2xx
|
354 | response.
|
355 |
|
356 | If that is ok, read the data that is returned by the API to see if it is a
|
357 | fixable error: ex. `401 Unauthorized` probably means that you have a bad API
|
358 | key, so run `rally config api` or `rally config api.UAT`.
|
359 |
|
360 | #### My problem isn't in the list
|
361 |
|
362 | Ask me on SDVI or discocomm slack @John Schmidt
|