# Scraping Headlines Using Stew *This is a complete (but simple) example of using [Stew](https://github.com/rodw/stew) to extract content from the web. It is written as a "litcoffee" file, which is an executable/compilable file containing Markdown content with embedded CoffeeScript. ([Follow this link to go back to the README file.](../README.html))* In this example, we'll extract headlines from the venerable social-tech-news site [Slashdot](http://slashdot.org/). URL = 'http://slashdot.org/' If you examine the HTML of the Slashdot homepage carefully, you'll find that each headline is contained in an `h2` tag with the class `story`, and that within this heading there is an anchor (`a`) tag that contains the link. As a CSS selector, that looks like: SELECTOR = 'h2.story a' We'll use that selector to extract the headlines and links from the HTML print them to the console with the following function: print_headline = (node)-> headline = domutil.to_text(node) link = "http:#{node.attribs.href}" console.log "#{headline} <#{link}>" (`domutil` is an instance of Stew's `DOMUtil` type, which is imported below.) Now, given an `html` string, selecting and printing the headlines is as simple as this: select_and_print_headlines = (html)-> stew.select html, SELECTOR, (err,nodeset)-> for node in nodeset print_headline node That's really all there is to it. All of the Stew-specific code is found above. The rest of this file jumps through the hoops needed to download the HTML document from the web. ## Importing the Library When using Stew, you'll typically import the library using something like this: # This is what you'll typically do: # stew = new (require('stew-select')).Stew() # and/or # domutil = new (require('stew-select')).DOMUtil() but since this file is found *within* the Stew repository itself, we'll do things a little differently. Most readers can safely ignore these next few lines and use the simple `require` statement above instead. # You WON'T do the following. We're only doing it here because we # want to use the "local" implementation of Stew. fs = require 'fs' path = require 'path' HOMEDIR = path.join(__dirname,'..') LIB_COV_DIR = path.join(HOMEDIR,'lib-cov') LIB_DIR = if fs.existsSync(LIB_COV_DIR) then LIB_COV_DIR else path.join(HOMEDIR,'lib') stew = new (require(path.join(LIB_DIR,'stew'))).Stew() domutil = new (require(path.join(LIB_DIR,'stew'))).DOMUtil() ## Setting up the HTTP "Fetcher" Let's define a function that will fetch a web page and pass the resulting content to a callback function. We'll use the Node.js `http` library for this. http = require 'http' Our function will accept the `url` for the document to download and a `callback` function to invoke once the document is parsed. Following Node.js convention, we'll use the signature `callback(err,body)` for the callback function. fetch = (url,callback)-> Using `http`, we'll create an callback function to buffer the HTTP response: http_callback = (response)-> unless 200 <= response.statusCode <= 299 callback "Unexpected status code #{response.statusCode}" else buffer = "" response.setEncoding 'utf8' response.on 'data', (chunk)->buffer += chunk and, when the full response body has been recieved, pass it to the callback: response.on 'end', ()-> callback(null,buffer) Finally, we can trigger the actual request: http.get(url, http_callback).on('error', callback) Now our `fetch` method will download content from the URL and pass it to a callback function. ## Actual processing Now we can fetch the document and print the result using our `select_and_print` method: fetch URL, (err,body)-> if err? console.error "Error:", err else console.log '-----------------------------------------' console.log "CURRENT HEADLINES AT #{URL}" console.log '-----------------------------------------' select_and_print_headlines body console.log '-----------------------------------------' ## Running this script Now we can run this script by typing: ```console coffee docs/example.litcoffee ``` and see output like the following: ```console ----------------------------------------- CURRENT HEADLINES AT http://slashdot.org/ ----------------------------------------- DRM: How Book Publishers Failed To Learn From the Music Industry Small Black Holes: Cloudy With a Chance of Better Visibility No, the Tesla Model S Doesn't Pollute More Than an SUV The Case For a Government Bug Bounty Program When Smart Developers Generate Crappy Code New York City Wants To Revive Old Voting Machines Big Asteroid (With Its Own Moon) To Have Closest Approach With Earth Today Google Maps Used To Find Tax Cheats Judge Orders Google To Comply With FBI's Warrantless NSL Requests Ask Slashdot: How Important Is Advanced Math In a CS Degree? Badgers Block British Broadband Buildout Confirmed: Water Once Flowed On Mars Motorola Developing Pill and Tattoo Authentication Methods Seeing Atomic Bonds Before and After Reactions U.S. Authorizes Sales of American Communication Tech To Iran ----------------------------------------- ``` *([Follow this link to go back to the README file.](../README.html))*