UNPKG

14.5 kBHTMLView Raw
1<!DOCTYPE html>
2<html lang="en">
3<head>
4 <meta charset="utf-8">
5 <title>JSDoc: Index</title>
6
7 <script src="scripts/prettify/prettify.js"> </script>
8 <script src="scripts/prettify/lang-css.js"> </script>
9 <!--[if lt IE 9]>
10 <script src="//html5shiv.googlecode.com/svn/trunk/html5.js"></script>
11 <![endif]-->
12 <link type="text/css" rel="stylesheet" href="styles/prettify-tomorrow.css">
13 <link type="text/css" rel="stylesheet" href="styles/jsdoc-default.css">
14</head>
15
16<body>
17
18<div id="main">
19
20 <h1 class="page-title">Index</h1>
21
22
23
24
25
26
27
28 <h3> </h3>
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43 <section>
44 <article><h1 id="node-goldwasher">node-goldwasher</h1>
45<p><a href="https://www.npmjs.org/package/goldwasher"><img src="http://img.shields.io/npm/v/goldwasher.svg" alt="npm version"></a>
46<a href="https://travis-ci.org/alexlangberg/node-goldwasher"><img src="http://img.shields.io/travis/alexlangberg/node-goldwasher.svg" alt="Build Status"></a>
47<a href="https://coveralls.io/r/alexlangberg/node-goldwasher?branch=master"><img src="http://img.shields.io/coveralls/alexlangberg/node-goldwasher.svg" alt="Coverage Status"></a>
48<a href="https://codeclimate.com/github/alexlangberg/node-goldwasher"><img src="http://img.shields.io/codeclimate/github/alexlangberg/node-goldwasher.svg" alt="Code Climate"></a></p>
49<p><a href="https://david-dm.org/alexlangberg/node-goldwasher"><img src="https://david-dm.org/alexlangberg/node-goldwasher.svg" alt="Dependency Status"></a>
50<a href="https://david-dm.org/alexlangberg/node-goldwasher#info=devDependencies"><img src="https://david-dm.org/alexlangberg/node-goldwasher/dev-status.svg" alt="devDependency Status"></a></p>
51<p>The purpose module is to extract text information from HTML, usually a website, which will often have to be sanitized and filtered to be useful. This module takes a pile of HTML and washes out the parts you need as small, golden nuggets of text and related metadata, the default options referred to as the <em>goldwasher format</em>:</p>
52<p>JSON format (see additional formats in the bottom):</p>
53<pre><code class="lang-javascript">{
54 timestamp: 1402847736380,
55 text: &quot;Oak is strong and also gives shade.&quot;,
56 keywords: [
57 {word: &quot;oak&quot;, count: 1},
58 {word: &quot;strong&quot;, count: 1},
59 {word: &quot;gives&quot;, count: 1},
60 {word: &quot;shade&quot;, count: 1}
61 ],
62 href: &quot;http://www.oakisstrong.com/oak/strong&quot;,
63 tag: &quot;h1&quot;,
64 position: 0,
65 total: 2,
66 uuid: &quot;808b7490-f743-11e4-90b2-df723554e9be&quot;,
67 batch: &quot;14eefda0-f762-11e4-a0b3-d5647c4f7651&quot;,
68 source: &quot;http://www.oakisstrong.com&quot;
69}
70</code></pre>
71<p>It works by passing it either pure HTML as a string (e.g. from <a href="https://www.npmjs.org/package/request">request</a>) or a <a href="https://www.npmjs.org/package/cheerio">cheerio</a> object, usually along with a <a href="https://www.npmjs.org/package/cheerio">cheerio</a> (jQuery) selector (HTML tags) from which the text should be extracted, along with other options. It will then return an array of nuggets (objects) of information - one per recognized tag. For each nugget, it will try to:</p>
72<ol>
73<li>Get the text of the tag and sanitize it, e.g. remove newlines.</li>
74<li>Optionally discard the nugget, if it matches an array of stop texts.</li>
75<li>Get the exact time of processing.</li>
76<li>Extract a count of each word in the text as keywords.</li>
77<li>Optionally discard keywords that match an array of stop words.</li>
78<li>Optionally discard keywords that match an external array of keywords (see the folder stop_words).</li>
79<li>Extract the nearest URL of the closest link.</li>
80<li>Extract the tag type of the matched target.</li>
81<li>Assign a unique identifier (UUID V1).</li>
82<li>Index the nugget position in the order it was found found.</li>
83<li>Add the total nugget count.</li>
84<li>Add the URL of the original source.</li>
85<li>Assign a unique identifier (UUID V1) that is similar for the entire batch of nuggets.</li>
86</ol>
87<p>The returned nuggets include the object properties:</p>
88<ul>
89<li><code>timestamp</code> - the exact time the tag was processed.</li>
90<li><code>text</code> - a sanitized version of the text of the tag.</li>
91<li><code>keywords</code> - a count of each word in the text, special characters removed.</li>
92<li><code>href</code> - the closest link, the first that matches one of:<ol>
93<li>Is the tag itself a link?</li>
94<li>Does the tag have a child node that is a link?</li>
95<li>Is there a link if we traverse up the DOM tree?</li>
96<li>Is there an adjecent (sibling) link?</li>
97</ol>
98</li>
99<li><code>tag</code> - the type of tag that was processed.</li>
100<li><code>position</code> - the position of the object, indicating the order in which tags were found. 0-based.</li>
101<li><code>total</code> - total number of nuggets in relation to the position. 1-based.</li>
102<li><code>uuid</code> - a unique identifier (UUID V1).</li>
103<li><code>batch</code> - a unique identifier (UUID V1) that is the same for the entire batch of nuggets.</li>
104<li><code>source</code> - a URL that was scraped, also the same for all nuggets.</li>
105</ul>
106<p>Alternatively, the output can be configured as XML, Atom or RSS format with the <code>output</code> option. The reason redundant information is included, such as the source, is that each returned nugget is supposed to be an atomic piece of information. As such, each nugget is to contain the information that &quot;somewhere, at some point in time, something was written (with a link to some place)&quot;.</p>
107<h2 id="installation">Installation</h2>
108<pre><code>npm install goldwasher
109</code></pre><h2 id="options">Options</h2>
110<ul>
111<li><code>selector</code> - cheerio (jQuery) selector, a selection of target tags.</li>
112<li><code>search</code> - only pick results containing these terms. Not case or special character sensitive (sluggified search).</li>
113<li><code>limit</code> - limit number of results.</li>
114<li><code>url</code> - base url of links, for sites that use relative urls.</li>
115<li><code>filterTexts</code> - stop texts that should be excluded.</li>
116<li><code>filterKeywords</code> - stop words that should be excluded as keywords.</li>
117<li><code>filterLocale</code> - stop words from external JSON file (see the folder stop_words).</li>
118<li><code>output</code> - output format (<code>json</code>, <code>xml</code>, <code>atom</code> or <code>rss</code>).</li>
119<li><code>contractAdjecent</code> - contracts adjecent targets. Useful for sites that split text, e.g. <code>&lt;div&gt;&lt;h1&gt;Hello&lt;/h1&gt;&lt;br&gt;&lt;h1&gt;World&lt;h1&gt;&lt;/div&gt;</code>.</li>
120</ul>
121<h2 id="example">Example</h2>
122<pre><code class="lang-javascript">var goldwasher = require(&#39;goldwasher&#39;);
123
124var html = &#39;&lt;a href=&quot;/oak/strong&quot;&gt;&lt;h1&gt;Oak is strong and also gives shade.&lt;/h1&gt;&lt;/a&gt;&#39;;
125 html += &#39;&lt;h2&gt;&lt;a href=&quot;http://www.catsanddogs.com/hate&quot;&gt;Cats and dogs each hate the other.&lt;/a&gt;&lt;/h2&gt;&#39;;
126 html += &#39;&lt;h2&gt;Some unwanted text.&lt;/h2&gt;&#39;;
127
128var options = {
129 selector: &#39;h1, h2&#39;,
130 url: &#39;http://www.oakisstrong.com&#39;,
131 filterTexts: [&#39;Some unwanted text.&#39;],
132 filterLocale: &#39;en&#39;,
133 filterKeywords: [&#39;also&#39;]
134}
135
136var result = goldwasher(html, options);
137
138/* result:
139[
140 {
141 timestamp: 1402847736380,
142 text: &quot;Oak is strong and also gives shade.&quot;,
143 keywords: [
144 {word: &quot;oak&quot;, count: 1},
145 {word: &quot;strong&quot;, count: 1},
146 {word: &quot;gives&quot;, count: 1},
147 {word: &quot;shade&quot;, count: 1}
148 ],
149 href: &quot;http://www.oakisstrong.com/oak/strong&quot;,
150 tag: &quot;h1&quot;,
151 position: 0,
152 total: 2,
153 uuid: &quot;808b7490-f743-11e4-90b2-df723554e9be&quot;,
154 batch: &quot;14eefda0-f762-11e4-a0b3-d5647c4f7651&quot;,
155 source: &quot;http://www.oakisstrong.com&quot;
156 },
157 {
158 timestamp: 1402847736381,
159 text: &quot;Cats and dogs each hate the other.&quot;,
160 keywords: [
161 {word: &quot;cats&quot;, count: 1},
162 {word: &quot;dogs&quot;, count: 1},
163 {word: &quot;hate&quot;, count: 1}
164 ],
165 href: &quot;http://www.catsanddogs.com/hate&quot;,
166 tag: &quot;h2&quot;,
167 position: 1,
168 total: 2,
169 uuid: &quot;a48fbb30-f743-11e4-96e6-7b423a412011&quot;,
170 batch: &quot;14eefda0-f762-11e4-a0b3-d5647c4f7651&quot;,
171 source: &quot;http://www.oakisstrong.com&quot;
172 }
173]
174*/
175</code></pre>
176<h2 id="output-formats">Output formats</h2>
177<p><strong>JSON:</strong></p>
178<pre><code class="lang-javascript">{
179 timestamp: 1402847736380,
180 text: &quot;Oak is strong and also gives shade.&quot;,
181 keywords: [
182 {word: &quot;oak&quot;, count: 1},
183 {word: &quot;strong&quot;, count: 1},
184 {word: &quot;gives&quot;, count: 1},
185 {word: &quot;shade&quot;, count: 1}
186 ],
187 href: &quot;http://www.oakisstrong.com/oak/strong&quot;,
188 tag: &quot;h1&quot;,
189 position: 0,
190 total: 2,
191 uuid: &quot;808b7490-f743-11e4-90b2-df723554e9be&quot;,
192 batch: &quot;14eefda0-f762-11e4-a0b3-d5647c4f7651&quot;,
193 source: &quot;http://www.oakisstrong.com&quot;
194}
195</code></pre>
196<p><strong>XML:</strong></p>
197<pre><code class="lang-xml">&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
198&lt;goldwasher&gt;
199 &lt;nugget&gt;
200 &lt;href&gt;/oak/strong&lt;/href&gt;
201 &lt;tag&gt;h1&lt;/tag&gt;
202 &lt;text&gt;Oak is strong and also gives shade.&lt;/text&gt;
203 &lt;position&gt;0&lt;/position&gt;
204 &lt;timestamp&gt;1431296135800&lt;/timestamp&gt;
205 &lt;uuid&gt;14eefda0-f762-11e4-a0b3-d5647c4f7651&lt;/uuid&gt;
206 &lt;total&gt;3&lt;/total&gt;
207 &lt;batch&gt;14eefda0-f762-11e4-a0b3-d5647c4f7651&lt;/batch&gt;
208 &lt;source&gt;http://www.oakisstrong.com&lt;/batch&gt;
209 &lt;keywords&gt;
210 &lt;keyword&gt;
211 &lt;word&gt;oak&lt;/word&gt;
212 &lt;count&gt;1&lt;/count&gt;
213 &lt;/keyword&gt;
214 &lt;keyword&gt;
215 &lt;word&gt;is&lt;/word&gt;
216 &lt;count&gt;1&lt;/count&gt;
217 &lt;/keyword&gt;
218 &lt;keyword&gt;
219 &lt;word&gt;strong&lt;/word&gt;
220 &lt;count&gt;1&lt;/count&gt;
221 &lt;/keyword&gt;
222 &lt;keyword&gt;
223 &lt;word&gt;and&lt;/word&gt;
224 &lt;count&gt;1&lt;/count&gt;
225 &lt;/keyword&gt;
226 &lt;keyword&gt;
227 &lt;word&gt;also&lt;/word&gt;
228 &lt;count&gt;1&lt;/count&gt;
229 &lt;/keyword&gt;
230 &lt;keyword&gt;
231 &lt;word&gt;gives&lt;/word&gt;
232 &lt;count&gt;1&lt;/count&gt;
233 &lt;/keyword&gt;
234 &lt;keyword&gt;
235 &lt;word&gt;shade&lt;/word&gt;
236 &lt;count&gt;1&lt;/count&gt;
237 &lt;/keyword&gt;
238 &lt;/keywords&gt;
239 &lt;/nugget&gt;
240&lt;/goldwasher&gt;
241</code></pre>
242<p><strong>Atom:</strong></p>
243<pre><code class="lang-xml">&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;
244&lt;feed xmlns=&quot;http://www.w3.org/2005/Atom&quot;&gt;
245 &lt;title&gt;foo title&lt;/title&gt;
246 &lt;link&gt;foo.com&lt;/link&gt;
247 &lt;updated&gt;2015-05-12T16:12:45Z&lt;/updated&gt;
248 &lt;link rel=&quot;alternate&quot; href=&quot;foo.com&quot;/&gt;
249 &lt;subtitle&gt;Foo Bar Baz&lt;/subtitle&gt;
250 &lt;generator&gt;Feed for Node.js&lt;/generator&gt;
251 &lt;category term=&quot;oak&quot;&gt;&lt;/category&gt;
252 &lt;category term=&quot;is&quot;&gt;&lt;/category&gt;
253 &lt;category term=&quot;strong&quot;&gt;&lt;/category&gt;
254 &lt;category term=&quot;and&quot;&gt;&lt;/category&gt;
255 &lt;category term=&quot;also&quot;&gt;&lt;/category&gt;
256 &lt;category term=&quot;gives&quot;&gt;&lt;/category&gt;
257 &lt;category term=&quot;shade&quot;&gt;&lt;/category&gt;
258 &lt;entry&gt;
259 &lt;title type=&quot;html&quot;&gt;&lt;![CDATA[Oak is strong and also gives shade.]]&gt;&lt;/title&gt;
260 &lt;id&gt;foo.com/oak/strong&lt;/id&gt;
261 &lt;link href=&quot;foo.com/oak/strong&quot;&gt;
262 &lt;/link&gt;
263 &lt;updated&gt;2015-05-12T16:12:45Z&lt;/updated&gt;
264 &lt;summary type=&quot;html&quot;&gt;&lt;![CDATA[Oak is strong and also gives shade.]]&gt;&lt;/summary&gt;
265 &lt;author&gt;
266 &lt;name&gt;Baz Barfoo&lt;/name&gt;
267 &lt;uri&gt;foo.com&lt;/uri&gt;
268 &lt;/author&gt;
269 &lt;/entry&gt;
270&lt;/feed&gt;
271</code></pre>
272<p><strong>RSS:</strong></p>
273<pre><code class="lang-xml">&lt;?xml version=&quot;1.0&quot; encoding=&quot;utf-8&quot;?&gt;
274&lt;rss version=&quot;2.0&quot;&gt;
275 &lt;channel&gt;
276 &lt;title&gt;foo title&lt;/title&gt;
277 &lt;description&gt;Foo Bar Baz&lt;/description&gt;
278 &lt;link&gt;foo.com&lt;/link&gt;
279 &lt;lastBuildDate&gt;Tue, 12 May 2015 16:15:03 GMT&lt;/lastBuildDate&gt;
280 &lt;docs&gt;http://blogs.law.harvard.edu/tech/rss&lt;/docs&gt;
281 &lt;generator&gt;Feed for Node.js&lt;/generator&gt;
282 &lt;category&gt;oak&lt;/category&gt;
283 &lt;category&gt;is&lt;/category&gt;
284 &lt;category&gt;strong&lt;/category&gt;
285 &lt;category&gt;and&lt;/category&gt;
286 &lt;category&gt;also&lt;/category&gt;
287 &lt;category&gt;gives&lt;/category&gt;
288 &lt;category&gt;shade&lt;/category&gt;
289 &lt;item&gt;
290 &lt;title&gt;&lt;![CDATA[Oak is strong and also gives shade.]]&gt;&lt;/title&gt;
291 &lt;link&gt;foo.com/oak/strong&lt;/link&gt;
292 &lt;guid&gt;foo.com/oak/strong&lt;/guid&gt;
293 &lt;pubDate&gt;Tue, 12 May 2015 16:15:03 GMT&lt;/pubDate&gt;
294 &lt;description&gt;&lt;![CDATA[Oak is strong and also gives shade.]]&gt;&lt;/description&gt;
295 &lt;content:encoded/&gt;
296 &lt;author&gt;
297 &lt;name&gt;Baz Barfoo&lt;/name&gt;
298 &lt;link&gt;foo.com&lt;/link&gt;
299 &lt;/author&gt;
300 &lt;/item&gt;
301 &lt;/channel&gt;
302&lt;/rss&gt;
303</code></pre>
304</article>
305 </section>
306
307
308
309
310
311
312</div>
313
314<nav>
315 <h2><a href="index.html">Index</a></h2>
316</nav>
317
318<br clear="both">
319
320<footer>
321 Documentation generated by <a href="https://github.com/jsdoc3/jsdoc">JSDoc 3.3.0-alpha5</a> on Tue Jan 10 2017 23:11:11 GMT+0100 (CET)
322</footer>
323
324<script> prettyPrint(); </script>
325<script src="scripts/linenumber.js"> </script>
326</body>
327</html>
\No newline at end of file