1 | ![jscpd logo](https://raw.githubusercontent.com/kucherenko/jscpd/master/assets/logo.svg?sanitize=true)
|
2 |
|
3 |
|
4 | ## jscpd
|
5 | [![npm](https://img.shields.io/npm/v/jscpd.svg?style=flat-square)](https://www.npmjs.com/package/jscpd)
|
6 | ![jscpd](https://raw.githubusercontent.com/kucherenko/jscpd/master/assets/jscpd-badge.svg?sanitize=true)
|
7 | [![license](https://img.shields.io/github/license/kucherenko/jscpd.svg?style=flat-square)](https://github.com/kucherenko/jscpd/blob/master/LICENSE)
|
8 | [![Travis](https://img.shields.io/travis/kucherenko/jscpd.svg?style=flat-square)](https://travis-ci.org/kucherenko/jscpd)
|
9 | [![npm](https://img.shields.io/npm/dw/jscpd.svg?style=flat-square)](https://www.npmjs.com/package/jscpd)
|
10 | [![Coveralls](https://img.shields.io/coveralls/kucherenko/jscpd.svg?style=flat-square)](https://coveralls.io/github/kucherenko/jscpd)
|
11 | [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fkucherenko%2Fjscpd.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fkucherenko%2Fjscpd?ref=badge_shield)
|
12 | [![Backers on Open Collective](https://opencollective.com/jscpd/backers/badge.svg)](#backers)
|
13 | [![Sponsors on Open Collective](https://opencollective.com/jscpd/sponsors/badge.svg)](#sponsors)
|
14 |
|
15 |
|
16 | > Copy/paste detector for programming source code, supports [150+ formats](docs/supported_formats.md).
|
17 |
|
18 | Copy/paste is a common technical debt on a lot of projects. The jscpd gives the ability to find duplicated blocks implemented on more than 150 programming languages and digital formats of documents.
|
19 | The jscpd tool implements [Rabin-Karp](https://en.wikipedia.org/wiki/Rabin%E2%80%93Karp_algorithm) algorithm for searching duplications.
|
20 |
|
21 | [![NPM](https://nodei.co/npm/jscpd.png)](https://nodei.co/npm/jscpd/)
|
22 |
|
23 | ## Features
|
24 | - Detect duplications in programming source code, use semantic of programing languages, can skip comments, empty lines etc.
|
25 | - Detect duplications in embedded blocks of code, like `<script>` or `<style>` sections in html
|
26 | - Blame authors of duplications
|
27 | - Generate XML report in pmd-cpd format, JSON report, [HTML report](http://kucherenko.github.io/jscpd-report.html)
|
28 |
|
29 | - Integrate with CI systems, use thresholds for level of duplications
|
30 | - The powerful [API](docs/api.md) for extend functionality and usage
|
31 |
|
32 | ## What is new in jscpd v1.0.0?
|
33 |
|
34 | - Powerful development [API](docs/api.md) written on TypeScript (no more CoffeeScript)
|
35 | - Supports more formats (moved source code tokenizer from CodeMirror to Prism.js)
|
36 | - New reporters: html, badge (badge reporter is separate package `jscpd-badge-reporter`)
|
37 | - Add blamed lines to JSON report
|
38 | - Default config file is `.jscpd.json`, no more `.cpd.yaml`
|
39 | - Detect different formats in one file, like js scripts in html tags
|
40 | - Allow to use multiple cli options for parameters like `jscpd --ignore tests,build`
|
41 | - Allow multiple paths for detection like `jscpd ./src ./tests ./docs`
|
42 | - Statistic of detection
|
43 | - Use patterns form `.gitignore` for ignoring detection
|
44 | - Ignored blocks in code
|
45 |
|
46 | ## 0.6.x
|
47 |
|
48 | The old version of jscpd [here](https://github.com/kucherenko/jscpd/tree/0.6.x)
|
49 |
|
50 | ## Getting started
|
51 |
|
52 | ### Usage
|
53 | ```bash
|
54 | $ npx jscpd /path/to/source
|
55 | ```
|
56 | or
|
57 |
|
58 | ```bash
|
59 | $ npm install -g jscpd
|
60 |
|
61 | $ jscpd /path/to/code
|
62 | ```
|
63 |
|
64 | ### Options
|
65 |
|
66 | #### Min Lines
|
67 |
|
68 | Minimal block size of code in lines. The block of code less than `min-lines` will be skipped.
|
69 |
|
70 | - Cli options: `--min-lines`, `-l`
|
71 | - Type: **number**
|
72 | - Default: **5**
|
73 | #### Max Lines
|
74 |
|
75 | Maximum file size in lines. The file bigger than `max-lines` will be skipped.
|
76 |
|
77 | - Cli options: `--max-lines`, `-x`
|
78 | - Type: **number**
|
79 | - Default: **500**
|
80 |
|
81 | #### Max Size
|
82 |
|
83 | Maximum file size in bytes. The file bigger than `max-size` will be skipped.
|
84 |
|
85 | - Cli options: `--max-size`, `-z`
|
86 | - Type: **string**
|
87 | - Default: **30kb**
|
88 | #### Threshold
|
89 |
|
90 | The threshold for duplication level, check if current level of duplications bigger than threshold jscpd exit with error.
|
91 |
|
92 | - Cli options: `--threshold`, `-t`
|
93 | - Type: **number**
|
94 | - Default: **null**
|
95 |
|
96 | #### Config
|
97 |
|
98 | The path to configuration file. The config should be in `json` format. Supported options in config file can be the same with cli options.
|
99 |
|
100 | - Cli options: `--config`, `-c`
|
101 | - Type: **path**
|
102 | - Default: **null**
|
103 |
|
104 | #### Ignore
|
105 |
|
106 | The option with glob patterns to ignore from analyze. For multiple globs you can use coma as separator.
|
107 | Example:
|
108 | ```bash
|
109 | $ jscpd --ignore "**/*.min.js,**/*.map" /path/to/files
|
110 | ```
|
111 | - Cli options: `--ignore`, `-i`
|
112 | - Type: **string**
|
113 | - Default: **null**
|
114 |
|
115 | #### Reporters
|
116 | The list of reporters. Reporters use for output information of clones and duplication process.
|
117 |
|
118 | Available reporters:
|
119 | - **console** - report about clones to console;
|
120 | - **consoleFull** - report about clones to console with blocks of code;
|
121 | - **json** - output `jscpd-report.json` file with clones report in json format;
|
122 | - **xml** - output `jscpd-report.xml` file with clones report in xml format;
|
123 | - **html** - output `jscpd-report.html` file with clones report;
|
124 | - **verbose** - output a lot of debug information to console;
|
125 | - **time** - output all time of execution;
|
126 |
|
127 | > Note: A reporter can be developed manually, see API section.
|
128 |
|
129 | - Cli options: `--reporters`, `-r`
|
130 | - Type: **string**
|
131 | - Default: **console,time**
|
132 |
|
133 | #### Output
|
134 |
|
135 | The path to directory for reports. JSON and XML reports will be saved there.
|
136 |
|
137 | - Cli options: `--output`, `-o`
|
138 | - Type: **path**
|
139 | - Default: **./report/**
|
140 |
|
141 | #### Mode
|
142 | The mode of detection quality.
|
143 | - `strict` - use all types of symbols as token, skip only blocks marked as ignored.
|
144 | - `mild` - skip blocks marked as ignored and new lines and empty symbols.
|
145 | - `weak` - skip blocks marked as ignored and new lines and empty symbols and comments.
|
146 |
|
147 | > Note: A mode can be developed manually, see API section.
|
148 |
|
149 | - Cli options: `--mode`, `-m`
|
150 | - Type: **string**
|
151 | - Default: **mild**
|
152 |
|
153 | #### Format
|
154 |
|
155 | The list of formats to detect for duplications. Available over [150 formats](docs/supported_formats.md).
|
156 |
|
157 | Example:
|
158 | ```bash
|
159 | $ jscpd --format "php,javascript,markup,css" /path/to/files
|
160 | ```
|
161 |
|
162 | - Cli options: `--format`, `-f`
|
163 | - Type: **string**
|
164 | - Default: **{all formats}**
|
165 |
|
166 | #### Blame
|
167 | Get information about authors and dates of duplicated blocks from git.
|
168 |
|
169 | - Cli options: `--blame`, `-b`
|
170 | - Type: **boolean**
|
171 | - Default: **false**
|
172 |
|
173 | #### Silent
|
174 | Don't write a lot of information to a console.
|
175 |
|
176 | Example:
|
177 | ```
|
178 | $ jscpd /path/to/source --silent
|
179 | Duplications detection: Found 60 exact clones with 3414(46.81%) duplicated lines in 100 (31 formats) files.
|
180 | Execution Time: 1381.759ms
|
181 | ```
|
182 | - Cli options: `--silent`, `-s`
|
183 | - Type: **boolean**
|
184 | - Default: **false**
|
185 |
|
186 | #### Absolute
|
187 | Use the absolute path in reports.
|
188 |
|
189 |
|
190 | - Cli options: `--absolute`, `-a`
|
191 | - Type: **boolean**
|
192 | - Default: **false**
|
193 |
|
194 | #### Formats Extensions
|
195 | Define the list of formats with file extensions. Available over [150 formats](docs/supported_formats.md).
|
196 |
|
197 | In following example jscpd will analyze files `*.es` and `*.es6` as javascript and `*.dt` files as dart:
|
198 | ```bash
|
199 | $ jscpd --formats-exts javascript:es,es6;dart:dt /path/to/code
|
200 | ```
|
201 | > Note: formats defined in the option redefine default configuration, you should define all need formats manually or create two configuration for run `jscpd`
|
202 |
|
203 | - Cli options: `--formats-exts`
|
204 | - Type: **string**
|
205 | - Default: **null**
|
206 |
|
207 | ## Config File and Config section in package.json
|
208 |
|
209 | Put `.jscpd.json` file in the root of the projects:
|
210 | ```json
|
211 | {
|
212 | "threshold": 0.1,
|
213 | "reporters": ["html", "console", "badge"],
|
214 | "ignore": ["**/__snapshots__/**"],
|
215 | "absolute": true
|
216 | }
|
217 | ```
|
218 |
|
219 | Also you can use section in `package.json`:
|
220 |
|
221 | ```json
|
222 | {
|
223 | ...
|
224 | "jscpd": {
|
225 | "threshold": 0.1,
|
226 | "reporters": ["html", "console", "badge"],
|
227 | "ignore": ["**/__snapshots__/**"],
|
228 | "absolute": true,
|
229 | "gitignore": true
|
230 | }
|
231 | ...
|
232 | }
|
233 |
|
234 |
|
235 | ```
|
236 |
|
237 | ## Ignored Blocks
|
238 |
|
239 | Mark blocks in code as ignored:
|
240 | ```javascript
|
241 | /* jscpd:ignore-start */
|
242 | import lodash from 'lodash';
|
243 | import React from 'react';
|
244 | import {User} from './models';
|
245 | import {UserService} from './services';
|
246 | /* jscpd:ignore-end */
|
247 | ```
|
248 |
|
249 | ```html
|
250 | <!--jscpd:ignore-start-->
|
251 | <meta data-react-helmet="true" name="theme-color" content="#cb3837"/>
|
252 | <link data-react-helmet="true" rel="stylesheet" href="https://static.npmjs.com/103af5b8a2b3c971cba419755f3a67bc.css"/>
|
253 | <link data-react-helmet="true" rel="stylesheet" href="https://static.npmjs.com/cms/flatpages.css"/>
|
254 | <link data-react-helmet="true" rel="apple-touch-icon" sizes="120x120" href="https://static.npmjs.com/58a19602036db1daee0d7863c94673a4.png"/>
|
255 | <link data-react-helmet="true" rel="apple-touch-icon" sizes="144x144" href="https://static.npmjs.com/7a7ffabbd910fc60161bc04f2cee4160.png"/>
|
256 | <link data-react-helmet="true" rel="apple-touch-icon" sizes="152x152" href="https://static.npmjs.com/34110fd7686e2c90a487ca98e7336e99.png"/>
|
257 | <link data-react-helmet="true" rel="apple-touch-icon" sizes="180x180" href="https://static.npmjs.com/3dc95981de4241b35cd55fe126ab6b2c.png"/>
|
258 | <link data-react-helmet="true" rel="icon" type="image/png" href="https://static.npmjs.com/b0f1a8318363185cc2ea6a40ac23eeb2.png" sizes="32x32"/>
|
259 | <!--jscpd:ignore-end-->
|
260 | ```
|
261 |
|
262 | ## Reporters
|
263 |
|
264 | ### HTML
|
265 |
|
266 | [Demo report](http://kucherenko.github.io/jscpd-report.html)
|
267 |
|
268 | ### Badge
|
269 |
|
270 | ![jscpd](assets/jscpd-badge.svg)
|
271 |
|
272 | More info [jscpd-badge-reporter](https://github.com/kucherenko/jscpd-badge-reporter)
|
273 |
|
274 | ### PMD CPD XML
|
275 | ```xml
|
276 | <?xml version="1.0" encoding="utf-8"?>
|
277 | <pmd-cpd>
|
278 | <duplication lines="10">
|
279 | <file path="/path/to/file" line="1">
|
280 | <codefragment><![CDATA[ ...first code fragment... ]]></codefragment>
|
281 | </file>
|
282 | <file path="/path/to/file" line="5">
|
283 | <codefragment><![CDATA[ ...second code fragment...}]]></codefragment>
|
284 | </file>
|
285 | <codefragment><![CDATA[ ...duplicated fragment... ]]></codefragment>
|
286 | </duplication>
|
287 | </pmd-cpd>
|
288 | ```
|
289 |
|
290 | ### JSON reporters
|
291 | ```json
|
292 | {
|
293 | "duplications": [{
|
294 | "format": "javascript",
|
295 | "lines": 27,
|
296 | "fragment": "...code fragment... ",
|
297 | "tokens": 0,
|
298 | "firstFile": {
|
299 | "name": "tests/fixtures/javascript/file2.js",
|
300 | "start": 1,
|
301 | "end": 27,
|
302 | "startLoc": {
|
303 | "line": 1,
|
304 | "column": 1
|
305 | },
|
306 | "endLoc": {
|
307 | "line": 27,
|
308 | "column": 2
|
309 | }
|
310 | },
|
311 | "secondFile": {
|
312 | "name": "tests/fixtures/javascript/file1.js",
|
313 | "start": 1,
|
314 | "end": 24,
|
315 | "startLoc": {
|
316 | "line": 1,
|
317 | "column": 1
|
318 | },
|
319 | "endLoc": {
|
320 | "line": 24,
|
321 | "column": 2
|
322 | }
|
323 | }
|
324 | }],
|
325 | "statistic": {
|
326 | "detectionDate": "2018-11-09T15:32:02.397Z",
|
327 | "formats": {
|
328 | "javascript": {
|
329 | "sources": {
|
330 | "/path/to/file": {
|
331 | "lines": 24,
|
332 | "sources": 1,
|
333 | "clones": 1,
|
334 | "duplicatedLines": 26,
|
335 | "percentage": 45.33,
|
336 | "newDuplicatedLines": 0,
|
337 | "newClones": 0
|
338 | }
|
339 | },
|
340 | "total": {
|
341 | "lines": 297,
|
342 | "sources": 1,
|
343 | "clones": 1,
|
344 | "duplicatedLines": 26,
|
345 | "percentage": 45.33,
|
346 | "newDuplicatedLines": 0,
|
347 | "newClones": 0
|
348 | }
|
349 | }
|
350 | },
|
351 | "total": {
|
352 | "lines": 297,
|
353 | "sources": 6,
|
354 | "clones": 5,
|
355 | "duplicatedLines": 26,
|
356 | "percentage": 45.33,
|
357 | "newDuplicatedLines": 0,
|
358 | "newClones": 0
|
359 | },
|
360 | "threshold": 10
|
361 | }
|
362 | }
|
363 | ```
|
364 |
|
365 | ## API
|
366 |
|
367 | ```typescript
|
368 | import {
|
369 | JSCPD,
|
370 | IClone,
|
371 | IOptions,
|
372 | MATCH_SOURCE_EVENT,
|
373 | CLONE_FOUND_EVENT,
|
374 | SOURCE_SKIPPED_EVENT,
|
375 | END_EVENT
|
376 | } from 'jscpd';
|
377 |
|
378 | const options: IOptions = {};
|
379 |
|
380 | const cpd = new JSCPD(options);
|
381 |
|
382 | const code = '...string with my code...';
|
383 |
|
384 | cpd.on(MATCH_SOURCE_EVENT, (source) => {
|
385 | // new source detection started
|
386 | console.log(source);
|
387 | });
|
388 |
|
389 | cpd.on(CLONE_FOUND_EVENT, (clone: IClone) => {
|
390 | // clone found event
|
391 | console.log(clone);
|
392 | });
|
393 |
|
394 | cpd.on(SOURCE_SKIPPED_EVENT, (stat) => {
|
395 | // skipped source due size (see max-size, min-lines and max-lines options)
|
396 | console.log(stat);
|
397 | });
|
398 |
|
399 | cpd.on(END_EVENT, (clones: IClone[]) => {
|
400 | // detection process finished
|
401 | console.log(clones);
|
402 | });
|
403 |
|
404 | cpd.detect(code, { id: 'test', format: 'markup' })
|
405 | .then((clones: IClone[]) => console.log(clones));
|
406 |
|
407 |
|
408 | cpd.detectInFiles(['./src', './tests'])
|
409 | .then((clones: IClone[]) => console.log(clones));
|
410 |
|
411 | ```
|
412 |
|
413 | [Progamming API](docs/api.md)
|
414 |
|
415 | ## Contributors
|
416 |
|
417 | This project exists thanks to all the people who contribute.
|
418 | <a href="https://github.com/kucherenko/jscpd/contributors"><img src="https://opencollective.com/jscpd/contributors.svg?width=890&button=false" /></a>
|
419 |
|
420 |
|
421 | ## Backers
|
422 |
|
423 | Thank you to all our backers! 🙏 [[Become a backer](https://opencollective.com/jscpd#backer)]
|
424 |
|
425 | <a href="https://opencollective.com/jscpd#backers" target="_blank"><img src="https://opencollective.com/jscpd/backers.svg?width=890"></a>
|
426 |
|
427 |
|
428 | ## Sponsors
|
429 |
|
430 | Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/jscpd#sponsor)]
|
431 |
|
432 | <a href="https://opencollective.com/jscpd/sponsor/0/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/0/avatar.svg"></a>
|
433 | <a href="https://opencollective.com/jscpd/sponsor/1/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/1/avatar.svg"></a>
|
434 | <a href="https://opencollective.com/jscpd/sponsor/2/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/2/avatar.svg"></a>
|
435 | <a href="https://opencollective.com/jscpd/sponsor/3/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/3/avatar.svg"></a>
|
436 | <a href="https://opencollective.com/jscpd/sponsor/4/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/4/avatar.svg"></a>
|
437 | <a href="https://opencollective.com/jscpd/sponsor/5/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/5/avatar.svg"></a>
|
438 | <a href="https://opencollective.com/jscpd/sponsor/6/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/6/avatar.svg"></a>
|
439 | <a href="https://opencollective.com/jscpd/sponsor/7/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/7/avatar.svg"></a>
|
440 | <a href="https://opencollective.com/jscpd/sponsor/8/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/8/avatar.svg"></a>
|
441 | <a href="https://opencollective.com/jscpd/sponsor/9/website" target="_blank"><img src="https://opencollective.com/jscpd/sponsor/9/avatar.svg"></a>
|
442 |
|
443 | ![ga tracker](https://www.google-analytics.com/collect?v=1&a=257770996&t=pageview&dl=https%3A%2F%2Fgithub.com%2Fkucherenko%2Fjscpd&ul=en-us&de=UTF-8&cid=978224512.1377738459&tid=UA-730549-17&z=887657232 "ga tracker")
|
444 |
|
445 | ## License
|
446 |
|
447 | [MIT](LICENSE) © Andrey Kucherenko
|