1 | # Project Operations
|
2 |
|
3 | Atomist enables you to inspect or edit the contents of projects. One
|
4 | of Atomist's distinguishing qualities is the ease with which you can
|
5 | work with code, as well as the data that surrounds code, such as
|
6 | builds, issues and deploys.
|
7 |
|
8 | ## Sourcing Projects to Operate On
|
9 | Use the `doWithAllRepos` helper function to work with many repos. Its signature is as follows:
|
10 |
|
11 | ```typescript
|
12 | export function doWithAllRepos<R, P>(ctx: HandlerContext,
|
13 | credentials: ProjectOperationCredentials,
|
14 | action: (p: Project, t: P) => Promise<R>,
|
15 | parameters: P,
|
16 | repoFinder: RepoFinder = allReposInTeam(),
|
17 | repoFilter: RepoFilter = AllRepos,
|
18 | repoLoader: RepoLoader =
|
19 | defaultRepoLoader(credentials.token)): Promise<R[]> {
|
20 | ```
|
21 |
|
22 | The `credentials` parameter usually contains the current GitHub token.
|
23 |
|
24 | The most important parameter is `action` which maps from the project and parameters to the return type `R`. The subsequent parameters are optional: Default behavior will be
|
25 | to load all GitHub repos associated with the current team. Pass in different functions for custom filtering, sourcing from different a source etc.
|
26 |
|
27 | ## Concepts
|
28 |
|
29 | The `Project` and `File` interfaces allow you to work with project
|
30 | content, regardless of where it's sourced from (e.g. GitHub, a local
|
31 | project, or in memory). The `GitProject` interface extends `Project`
|
32 | to add support for cloning projects and creating commits, branches and
|
33 | pull requests. Atomist secret management makes it easy to obtain the
|
34 | necessary tokens, respecting the role of the current user, in the
|
35 | event of a command handler.
|
36 |
|
37 | Three higher-level concepts: **reviewers**, **editors** and
|
38 | **generators**, make it easier to work with project content,
|
39 | performing cloning and git updates automatically.
|
40 |
|
41 | Microgrammar support
|
42 | allows sophisticated parsing and updates with clean diffs. [Path expression](PathExpressions.md) support makes it possible to drill into the structure of files within projects in a consistent way, using a variety of grammars.
|
43 |
|
44 | ## Project and File Interface Concepts: Sync, Async and defer
|
45 |
|
46 | The project and file interfaces represent a project (usually backed by
|
47 | a whole repository) and a file (a single artifact within a project).
|
48 |
|
49 | The two interfaces follow the same pattern, in being composed from
|
50 | finer-grained interfaces with different purposes.
|
51 |
|
52 | Consider the `Project` interface:
|
53 |
|
54 | ```typescript
|
55 | export interface Project extends ProjectAsync, ProjectSync {
|
56 | }
|
57 | ```
|
58 |
|
59 | Let's examine these interfaces in turn:
|
60 |
|
61 | - `ProjectSync`: Does what you expect: synchronous, blocking
|
62 | operations. Following `node` conventions, synchronous functions
|
63 | end with a `Sync` suffix. They should be avoided in production
|
64 | code, although they can be very handy during tests.
|
65 | - `ProjectAsync`: Functions that return TypeScript/ES6 promises or
|
66 | `node` streams. As they are the default choice in most cases, their names do not have any distinguishing prefix or suffix.
|
67 |
|
68 | ### Deferring operations
|
69 |
|
70 | Any function that returns a Promise can be deferred for later execution. This can be useful when you have many fine-grained steps and prefer the convenience of void returns.
|
71 |
|
72 | Do this by using the `defer` wrapper function. For example:
|
73 |
|
74 | ```typescript
|
75 | defer(project, project.addFile("thing", "1"));
|
76 | ```
|
77 |
|
78 | - After using `defer`, it's necessary to call
|
79 | `flush` on the `Project` or `File` to effect the changes. This takes
|
80 | what might be many promises and puts them into a single promise.
|
81 | - Until `flush` is called, changes made by scripting operations are not visible to subsequent scripting operations. However, `flush` can be called at any time to flush intermediate working, and repeated calls to `flush` are safe.
|
82 | - Deferred operations will ultimately be executed in the order in which they were queued.
|
83 | - **Do not** mix deferred operations with synchronous or promise-returning operations without first calling `flush`, as non-scripting operations will not see changes made by scripting operations.
|
84 |
|
85 | Methods on the `FileScripting` interface automatically defer. These method names typically begin with a `record` prefix. For example, these three code snippets are equivalent:
|
86 |
|
87 | ```typescript
|
88 | const f: File = ...
|
89 | f.replaceAll("foo", "bar") // Returns a promise
|
90 | .then(f => ...
|
91 | ```
|
92 |
|
93 | ```typescript
|
94 | const f: File = ...
|
95 | defer(f, f.replaceAll("foo", "bar"))
|
96 | .flush()
|
97 | .then(f => ...
|
98 | ```
|
99 |
|
100 | ```typescript
|
101 | const f: File = ...
|
102 | f.recordReplaceAll("foo", "bar")
|
103 | .flush() // Returns a promise
|
104 | .then(f => ...
|
105 | ```
|
106 |
|
107 | In this case, it would probably be simpler to use `replaceAll`.
|
108 |
|
109 | ## Files
|
110 |
|
111 | Files are lightweight objects that are lazily loaded. Thus keeping
|
112 | many files in memory is not normally a concern.
|
113 |
|
114 | ## Projects and globs
|
115 |
|
116 | Many methods on or related to `Project`
|
117 | use [glob patterns](https://en.wikipedia.org/wiki/Glob_(programming))
|
118 | to select files.
|
119 |
|
120 | For example:
|
121 |
|
122 | ```typescript
|
123 | streamFiles(...globPatterns: string[]): FileStream;
|
124 | ```
|
125 |
|
126 | If no glob patterns are specified, all files are matched.
|
127 |
|
128 | `streamFiles` uses default negative glob patterns to exclude content
|
129 | that should normally be excluded, such as the `.git` directory and the
|
130 | `node_module` and `target` directories found when working locally with
|
131 | JavaScript or Java projects. If you want complete control over the
|
132 | globs used, without any default exclusions, use the following
|
133 | lower-level method:
|
134 |
|
135 | ```typescript
|
136 | streamFilesRaw(globPatterns: string[], opts: {}): FileStream;
|
137 | ```
|
138 |
|
139 | The return type `FileStream` extends `node` `Stream`. An example of using the streaming API directly:
|
140 |
|
141 | ```typescript
|
142 | let count = 0;
|
143 | project.streamFiles()
|
144 | .on("data", (f: File) => {
|
145 | count++;
|
146 | },
|
147 | ).on("end", () => {
|
148 | console.log(`We saw ${count} files`);
|
149 | });
|
150 | ```
|
151 |
|
152 | If you find promises more convenient, use the helper `toPromise`
|
153 | method from `projectUtils`, which converts a stream to a promise, or
|
154 | use the helper functions described in the next section.
|
155 |
|
156 | ## Convenience functions
|
157 |
|
158 | `projectUtils` contains convenience methods for working with projects:
|
159 | For example, to apply the same function to many files, or to convert a
|
160 | stream of files into a promise.
|
161 |
|
162 | For example, replacement across all files matched by a glob is very simple:
|
163 |
|
164 | ```typescript
|
165 | import { doWithFiles } from "@atomist/automation-client/project/util/projectUtils";
|
166 |
|
167 | doWithFiles(p, "**/Thing", f => f.replace(/A-Z/, "alpha"))
|
168 | .run()
|
169 | .then(_ => {
|
170 | assert(p.findFileSync("Thing").getContentSync() === "alpha");
|
171 | });
|
172 | ```
|
173 |
|
174 | `parseUtils` integrates with microgrammars, and will be discussed later.
|
175 |
|
176 | ## Reviewers
|
177 |
|
178 | See [Project Reviewers](ProjectReviewers.md).
|
179 |
|
180 | ## Editors
|
181 |
|
182 | See [Project Editors](ProjectEditors.md).
|
183 |
|
184 | ## Generators
|
185 |
|
186 | See [Project Generators](ProjectGenerators.md).
|
187 |
|
188 | ## Local or remote operations
|
189 |
|
190 | All convenience superclasses, such as `EditorCommandSupport`, can work
|
191 | locally if passed a `local` parameter. In this case, they will look in
|
192 | the current working directory, for a two-tiered directory structure,
|
193 | of org and repo.
|
194 |
|
195 | ## Microgrammars
|
196 |
|
197 | Reviewer and editor implementations will often use microgrammars. This
|
198 | library provides an integration with
|
199 | the
|
200 | [Atomist microgrammar project](https://github.com/atomist/microgrammar). This
|
201 | enables you to pull pieces of content out of files and even modify
|
202 | them, with clean diffs.
|
203 |
|
204 | Consider the following microgrammar, which picks out `npm` dependencies:
|
205 |
|
206 | ```typescript
|
207 | export function dependencyGrammar(dependencyToReplace: string) {
|
208 | return Microgrammar.fromString<Dependency>('"${name}": "${version}"',
|
209 | {
|
210 | name: dependencyToReplace,
|
211 | version: /[0-9^.-]+/,
|
212 | },
|
213 | );
|
214 | }
|
215 |
|
216 | export interface Dependency {
|
217 | name: string;
|
218 | version: string;
|
219 | }
|
220 | ```
|
221 |
|
222 | It's possible to be able to work with matches as follows:
|
223 |
|
224 | ```typescript
|
225 | return doWithFileMatches(project, "package.json",
|
226 | dependencyGrammar(dependencyToReplace), f => {
|
227 | const m = f.matches[0] as any;
|
228 | m.name = newDependency;
|
229 | m.version = newDependencyVersion;
|
230 | })
|
231 | .run() // Return a promise
|
232 | .then(files => ...
|
233 | ```
|
234 |
|
235 | Because the grammar is strongly typed, `name` and `version` fields
|
236 | will be type checked. Assigning the values will result in an update to
|
237 | the files in which the matches occur.
|
238 |
|
239 | See `parseUtils.ts` for the various functions that integrate this
|
240 | library with `microgrammar`.
|
241 |
|
242 | All functions take a glob pattern as their second argument. In the
|
243 | above example, `package.json` is used because the location of an `npm`
|
244 | package file is well known. But it would be possible to look for all
|
245 | `.json` files with a glob pattern of `**/*.json`.
|
246 |
|
247 | *Previous versions of Rug offered path expressions. This microgrammar
|
248 | integration replaces many uses of that idiom. However, pure TypeScript
|
249 | equivalents of former "Rug types" may be added in future.*
|
250 |
|
251 | ## Testing
|
252 |
|
253 | Atomist automations are easily unit testable, which is a major design goal and benefit.
|
254 |
|
255 | This is easy with our project and file abstractions. `InMemoryProject`
|
256 | offers a convenient way of creating projects and making
|
257 | assertions. For example:
|
258 |
|
259 | ```typescript
|
260 | it("should not edit with no op editor", done => {
|
261 | const project = InMemoryProject.of(
|
262 | {path: "thing1", content: "1"},
|
263 | {path: "thing2", content: "2"}
|
264 | );
|
265 | const editor: ProjectEditor<EditResult> = p => Promise.resolve({ edited: false });
|
266 | editor(null, project, null)
|
267 | .then(r => {
|
268 | assert(!r.edited);
|
269 | done();
|
270 | });
|
271 | });
|
272 | ```
|