- [Alpha One](#alpha-one) - [Tagging](#tagging) - [Limiting](#limiting) - [Browser Console / ANSI Colors](#browser-console--ansi-colors) - [Options](#options) - [Encryption](#encryption) - [General Data](#general-data) - [User Authentication & Management](#user-authentication-&-management) - ['Easy DB'](#'easy-db') - [NPM & GitHub Publishing Helpers, Module Administration](#npm-&-github-publishing-helpers-module-administration) - [Persisting, Responsive Long Running Processes](#persisting-responsive-long-running-processes) - [Jizura Data Queries](#jizura-data-queries) - [Paper Publishing](#paper-publishing) - [Publication Helpers / Build Tools](#publication-helpers--build-tools) - [Git Automation](#git-automation) - [Rejected](#rejected) - [NodeGit](#nodegit) - [Jake](#jake) - [NodeJS Version Management](#nodejs-version-management) - [Application Architecture & Design](#application-architecture-&-design) - [Control Flow](#control-flow) - [And the Winner is: Yield!](#and-the-winner-is-yield!) - [The Journey Continues](#the-journey-continues) - [Making Things Synchronous](#making-things-synchronous) - [Rejected](#rejected-1) - [Promises](#promises) - [Untested](#untested) - [Standalone Apps & Isomorphic Modules](#standalone-apps-&-isomorphic-modules) - [Untested](#untested-1) - [(Web) Application (Backend) Design](#web-application-backend-design) - [The Stack](#the-stack) - [Modern Wire Communication Protocols](#modern-wire-communication-protocols) - [Chunked HTTP](#chunked-http) - [WebSockets](#websockets) - [Web Application Frontend Design](#web-application-frontend-design) - [Icons &c](#icons-&c) - [Web Application Deployment](#web-application-deployment) - [Untested](#untested-2) - [HTLM5 & CSS3](#htlm5-&-css3) - [Flexboxes](#flexboxes) - [Custom HTML Elements](#custom-html-elements) > **Table of Contents** *generated with [DocToc](http://doctoc.herokuapp.com/)* # Alpha One **About**: This is a meta-project to collect scattered thoughts about and working solutions for recurring tasks in Web- and Backend-Application building. Do not expect polished, working code, but rather fragmentary, sketchy hints. The project takes its name from the 1970s British TV series [Moonbase Alpha / Space: 1999](http://en.wikipedia.org/wiki/Moonbase_Alpha_%28Space:_1999%29). ## Tagging * [CoffeeNode TagTool](https://github.com/loveencounterflow/coffeenode-tagtool) ## Limiting **Rationale**: We often want to restrict access to the ressources a web application publishes; for example, you want subscribers to have unlimited access, guest users to enjoy a realistic peek into your offerings without handing over the full power of your application and the data therein, and at the same time deter any automated downloads. One part of the equation that can make this happen is User Authentication; the other is identifying anonymous visitors and selectively throttle or block their access, based on automatically collected behavior details and / or manual black- and whitelisting. * A limiter should be completely agnostic as to the task it is limiting—it should help to generally put limits to ressource usage, not, say, be built to specifically limit HTTP requests (a drop-in middleware could do that–one that uses a generic limiter). * Because of its genericity, a limiter should work with arbitrary ID tokens that represent clients—these could be any piece of data (an internal user ID, an IP address, a session ID) that is fit to identify a client in the sense of the application. * Since a limiter should be able to base *its* behavior on the past behavior of a given client *and* on the client's ranking (say, customer vs. guest vs. rogue), it must either entertain a suitable datastructure to keep such group affiliation and usage data (or else be fed (some of) these details when called). * Whether or not a limiter should take care of data persistence is an open question; ideally, server restart or redirection to another server process should not impact limiter behavior (this consideration would appear to favor Redis-like data persistence plans). * A limiter should distinguish between and be configurable for at least three behavioral patterns: full access, deferred (throttled) access, and denial of service (the last two with timeouts and permanent). [CoffeeNode Limit](https://github.com/loveencounterflow/coffeenode-limit) is an sttempt to bring all of the above points to the backend. It currently works with a slightly patched version of [node-rate-limiter](https://github.com/jhurliman/node-rate-limiter). ## Browser Console / ANSI Colors * https://github.com/chjj/tty.js/ * SCRATCH/alpha-one/src/ANSI256.coffee * https://github.com/Guilda/console-output-colorizer/blob/master/src/jquery.colorizeConsoleOutput.js * SCRATCH/includer/index.html * http://replit.github.io/jq-console/ (relies on an outdated version of jQuery; doesn't decode ANSI colors correctly; makes strange assumption about browser console element styling) ## Options ### Command Line Option Parsing * http://docopt.org/ * coffeenode-options Using `path-extra` to get the canonical OS-dependent location for user-level application option files: ```coffeescript ### https://github.com/jprichardson/node-path-extra ### njs_path = require 'path-extra' info njs_path.tempdir() info njs_path.homedir() info njs_path.datadir 'app-name-here' ``` ## Encryption See discussion of initialization vector (iv): * http://stackoverflow.com/questions/19938470/why-cant-i-decrypt-a-file-with-nodejs-that-i-encrypted-with-openssl * http://www.matasano.com/articles/javascript-cryptography/ * http://stackoverflow.com/questions/18111176/javascript-for-end-to-end-browser-encryption-for-privacy-anonymity * src/encryption.coffee ### General Data Probably use openpgpjs or sjcl: * https://github.com/chengxianga2008/node-cryptojs-aes * http://openpgpjs.org/ * http://bitwiseshiftleft.github.io/sjcl/ * https://github.com/srijs/rusha * https://github.com/h2non/jshashes ### User Authentication & Management * http://codahale.com/how-to-safely-store-a-password/ The specific purpose of password encryption should probably be done using bcrypt, for which there are 100% JS implementations available. It does have the advantage of adaptable algorithmic complexity. APIs typically do not offer decryption, only comparison of data. #### https://github.com/jed/authom Jed Schmidt writes in the [README](https://github.com/jed/authom/blob/master/README.md): > since it doesn't handle logins, persistence, sessions, or anything past authentication, it is **more of a > tool and less of a framework** ## 'Easy DB' * SCRATCH/test-nedb.coffee * https://github.com/louischatriot/nedb ## NPM & GitHub Publishing Helpers, Module Administration Our erstwhile solution to preparing CoffeeScript source files for both development and production, [larq](https://github.com/loveencounterflow/larq), is very much on the way out. It still has one distinctive feature, though, and that is `larq dev`, which compiles a JavaScript file for each CoffeeScript file that does not contain the actual code, but rather dynamically loads (and compiles) the CoffeeScript source; in this way, the CoffeeScript sources may change any time, while to outside consumers, everything appears as though code was actually loaded from JavaScript files, from the locations where those JavaScript loader files are located. Thus, every effort is made to make the existence of CoffeeScript sources transparent; the result is convenience during development and assurance that production versions of the same code (produced with `larq pub` or `coffee --compile`) are fully functional (even in the absence of the CS sources—the plan is to publish those, too, though, so potential contributors get a chance to work on either JS or CS, whatever they feel most comfotable with). I started [larq](https://github.com/loveencounterflow/larq) ('Language-Aware ReQuire') [when I](https://github.com/joyent/node/issues/5430) [realized](https://groups.google.com/forum/#!topic/nodejs/QsOEWvptQpA) that the global `require.extensions` hook in NodeJS has huge problems—long story short, it's a global hook, so if any of your dependencies have different ideas on how to resolve a particular filename extensions, you're BFE. It would be fine if it was some very local thing, but as it stands, its not, and care must be exercised. There are [some](https://groups.google.com/forum/#!topic/nodejs/o3L_Rtq8Ujk) thoughts on [how to repair this](https://github.com/michaelficarra/CoffeeScriptRedux/wiki/Intentional-Deviations-From-jashkenas-coffee-script), but so far no overarching facility or best-practices solution has emerged, short of the advisory to compile your 3rd-party-language sources down to the standard ones (`*.js`, `*.css`, `*.html`) and only ever deal with these compilation targets in production. A possible, maybe makeshift solution is to use [CoffeeNode Monitor](https://github.com/loveencounterflow/coffeenode-monitor) or similar to watch all sources and compile on change; this may prove a better alternative than trying to do the same, but in more convoluted ways, using yet another tool ([larq](https://github.com/loveencounterflow/larq)). ## Persisting, Responsive Long Running Processes **Rationale**: When we want to have a single HTTP serving process to run on a production server, we can accomplish that by daemonizing the server process. Having more than one process and processes that are resumed after abortion requires more work. On a development machine, we want to typically have a single HTTP server to be run explicitly (not automatically on system startup), and, when any of its dependencies have been modified, we want to re-run any build steps and then restart the serving process. In case of syntax or runtime errors, we typically would like the monitoring instance to wait for any subsequent changes to the codebase and then try and re-run the build steps. A server is a Long Running Process—as distinguished from, say, a backup script that runs once a day for a minute or so. A Responsive Process is one that responds to (rather, is managed so it can respond to) changes in source files. A Persisting Process is one that is resumed on termination. * coffeenode-monitor ## Jizura Data Queries * coffeenode-mingkwai * coffeenode-solr * coffeenode-mojikura ## Paper Publishing * XeLaTex * https://www.google.de/search?q=xelatex+fraktur&ie=utf-8&oe=utf-8&rls=org.mozilla:en-US:official&client=firefox-a&gws_rd=cr&ei=LyBkUqKRF8GYtQbznIHYBA * coffeenode-tex ## Publication Helpers / Build Tools Generate TOC for your `README.md` with `npm install --global doctoc`: * https://github.com/thlorenz/doctoc-web * cake (part of CoffeeScript) * https://github.com/ricardobeat/cake-async * http://ricardo.cc/cake-flour/ Tested: * http://brunch.io/ * OSX GitHub app ### Git Automation Git Automation ##### Rejected ###### NodeGit * http://www.nodegit.org/, https://github.com/nodegit/nodegit (sadly, does not compile under NodeJS 0.11.7) -1 for not updating their examples on their homepage. -1 for the Pyramids of Doom in their examples. -1 for an API that makes the programmatical equivalent of `git add --all; commit -m "foo"` look like rocket science. -1 for embracing asynchronicity so damn hard you'll get a hundred callbacks when you ask for a commit history. -1 for embracing Object-Oriented Programming so damn hard that you'll never get an answer to your question but an object whose methods you must call—to asynchronously receive another object on which to call more methods. ###### Jake * https://github.com/mde/jake Cake is better. ## NodeJS Version Management **Rationale**: With the advent of breaking changes in NodeJS unstable 0.11.x, many modules relying on `gyp` / `waf` do not compile any more. Some tools (such as browserify) compile with 0.11.7 but not with 0.11.8. Consequently we must be able to switch between NodeJS versions in a swift and painless manner; [n by visionmedia](https://github.com/visionmedia/n) is one way to do that. After the painless install: `npm install --global n`, you get a new executable `n` that lets you install and / or use new versions with commands like `n latest`, `n stable` or `n 0.11.7`. Simply executing `n` yields a menu in the terminal that lets you choose one of the installed versions. The global `node` command is always re-bound to the last chosen version with `n`; in order to get commands for specific versions, `cd` into a directory on the path and do something along the lines of: ```bash ln -s /usr/local/n/versions/0.11.7/bin/node node0117 ln -s /usr/local/n/versions/0.11.7/bin/node node-latest ln -s /usr/local/n/versions/0.10.22/bin/node node-stable ``` ## Application Architecture & Design * https://github.com/mikaelbr/node-notifier * brew install terminal-notifier ### Control Flow #### And the Winner is: Yield! * https://github.com/loveencounterflow/coffy-script **NB** CoffyScript is still (highly) experimental; its use in underlying libraries is especially strongly discouraged. CyS is currently implemented as a random patch of (an earlier version of) CoffeeScript 1.6.3 which has been shown to posses some subtle bugs (especially faulty error reporting that does not point out the source of the error). There was an attempt to upgrade to the latest bugfix editions of CoffeeScript, but the code had already suffieciently changed to make simple copy-and-paste of the changes unviable. The [CS Redux project](https://github.com/michaelficarra/CoffeeScriptRedux) is, unfortunately, not yet advanced / adopted enough for me switch over, so i decided to just wait it out: wait for `yield` to become the available in a stable Node release (i.e. 0.12 or 1.x) and wait for CS Redux to land in the master branch of CoffeeScript. With all the caveats, CoffyScript *is* already usable and has greatly helped to write some very readable, working code that feels magnitudes better than the Promises/A+ ridden code it replaced. #### The Journey Continues * caolan async ##### Making Things Synchronous Sometimes the solution to async woes lies in choosing an underlying functionality provider that is synchronous—for example, when doing application startup stuff like compiling sources and so on, you might choose `fs.writeFileSync` over `fs.writeFile` because (1) it lets you write linear code, (2) you can't run your app before this step has completed anyhow. Using synchronous IO is an antipattern in online serving, but its much more of a best practice for a startup sequence. In the same vein, it may be desirable to execute shell commands in a synchronous fashion. Sadly, [this piece of functionality is not included in NodeJS proper](http://stackoverflow.com/questions/4443597/node-js-execute-system-command-synchronously); fortunately, there are modules to do that, like [execSync](https://github.com/mgutz/execSync). It compiles with warnings under NodeJS 0.11.7 on my OSX box but appears to work, so now you can do `TRM = require 'coffeenode-trm'`, `result = TRM.execute 'ls'` alongside with `TRM.execute 'ls', ( error, result ) -> ...`. ##### Rejected ###### Promises Promises, with all the [neat `/A+` marketing ribbons tacked unto them](http://promises-aplus.github.io/promises-spec/), are horrible IMHO. They come **fraught with conceptual burden** (eg. promises vs. deferreds), force boilerplate on you (never forget to call `.done()` when you're done), and they **urge you to wrap *all* of your callback-based functions** so as to make them play nice (which is not nice—callbacks were there first, and they are the conceptually simpler mechanism. You just threw out the trusty hammer, now you got a beeping flashy gadget). And: **The API** of the [leading promises implementation](https://github.com/kriskowal/q) **is huge**. If you insist on doing promises, i'll ask you to memorize this 60+ items list: `Q.all`, `Q.allSettled`, `Q.async`, `Q.defer`, `Q.delay`, `Q.denodeify`, `Q.getUnhandledReasons`, `Q.isPromise`, `Q.longStackSupport`, `Q.makePromise`, `Q.nbind`, `Q.nextTick`, `Q.nfapply`, `Q.nfcall`, `Q.ninvoke`, `Q.npost`, `Q.onerror`, `Q.promise`, `Q.promised`, `Q.reject`, `Q.resetUnhandledRejections`, `Q.spawn`, `Q.stopUnhandledRejectionTracking`, `promise.all`, `promise.allSettled`, `promise.catch`, `promise.delay`, `promise.delete`, `promise.dispatch`, `promise.done`, `promise.fapply`, `promise.fbind`, `promise.fcall`, `promise.finally`, `promise.get`, `promise.inspect`, `promise.invoke`, `promise.isFulfilled`, `promise.isPending`, `promise.isRejected`, `promise.keys`, `promise.nodeify`, `promise.post`, `promise.progress`, `promise.set`, `promise.spread`, `promise.then`, `promise.thenReject`, `promise.thenResolve`, `promise.timeout`, `deferred.promise`, `deferred.resolve`, `deferred.reject`, `deferred.notify`, `deferred.makeNodeResolver`. You're of course not done yet—care to learn that (from the [API docs](https://github.com/kriskowal/q/wiki/API-Reference)) "`Q.when( 5, onFulfilled )` is equivalent to `Q( 5 ).then( onFulfilled )`"? I never wanted to know this. One might argue that this particular API is just the result of some particular diligent programmer going somewhat over the top, but that doesn't devalidate the argument that there is more than a fair number of promises libraries out there, some of them broken, others with uncertain merits. Finding out how the [~30 libraries listed alongside the /A+ specs](https://github.com/kriskowal/q/wiki/API-Reference) interact and compare in terms of features, performance and memory consumption is not something i want to spend my time with. And interaction between promises libraries is a real concern. To cite [a proponent of promises](http://domenic.me/2012/10/14/youre-missing-the-point-of-promises/) (emphases mine): > As authors of Promises/A-consuming libraries, we would like to assume this statement to be true: that > something that is “thenable” [meaning you can say `x.then()`on it] actually behaves as a > Promises/A promise, with all the power that entails. > > If you can make this assumption, you can write very extensive libraries that are entirely agnostic to the > implementation of the promises they accept! Whether they be from Q, when.js, or even WinJS, you can use the > simple composition rules of the Promises/A spec to build on promise behavior. For example, here's a > generalized retry function that works with any Promises/A implementation. > > Unfortunately, **libraries like jQuery break this**. This **necessitates ugly hacks** to detect the presence of > objects masquerading as promises, and who call themselves in their API documentation promises, but aren't > really Promises/A promises. If the consumers of your API start trying to pass you jQuery promises, you have > two choices: **fail in mysterious and hard-to-decipher ways** when your compositional techniques fail, or fail > up-front and block them from using your library entirely. **This sucks.** Interesting to hear that the jQuery team tried to implement promises and failed. Interesting, too, that (some) promises library authors (according to the above statement) appearently try to live up to the [duck typing dream](https://en.wikipedia.org/wiki/Duck_typing)—i.e. they rationalize "this must be a promises object!" when they get a return value `x` where `x.then` happens to be a function. That's about as reliable as saying "this must be a cinema!" because "it has seats inside". Frankly, as much as asynchronous calls and Pyramids of Doom tend to be a PITA, promises look and feel wrong. ##### Untested * https://github.com/aponxi/npm-execxi ### Standalone Apps & Isomorphic Modules * https://github.com/spikebrehm/isomorphic-tutorial * SCRATCH/includer/index.html * https://github.com/rogerwang/node-webkit * /Volumes/Storage/temp/nw-sample-apps * https://github.com/alloy/terminal-notifier **Rationale**. NodeJS makes it possible to use the same code on the server and in the browser; tools like `node-webkit` make it possible to run one-file downloads as no-installation / 'portable' desktop apps (with the VM (runtime) either bundled or installed globally on the target machine). This spells good-bye to all (or at least most) of the many competing GUI toolkits (if and where a 70MB minimum size for the VM is acceptable, where a given platform is supported, and where performance is sufficient). Tested: * https://github.com/substack/node-browserify ##### Untested * http://bower.io/ ### (Web) Application (Backend) Design #### To Framework or Not To Framework? Tero Piirainen [writes under the title Technology lock-in](https://moot.it/blog/technology/frameworkless-javascript.html): > If you look at the history of frameworks in any programming language, it's a history of failures. > Frameworks come and go. #### The Stack * https://github.com/senchalabs/connect Ordered from the earliest, most general, down to the latest, most specific steps, the 'stack of actions' in a web application may be sketched as follows: * **Preparation** * Limit (blocking) * Routing & Query Parsing * Favicon * Static * Authentication * Limit (throttle) * **App** * Response Initiation * Session Handling * DB Access * Template (View) Building * **Finalization** * Response Finalization ### Modern Wire Communication Protocols #### Chunked HTTP * what it means for templating #### WebSockets * http://jinzhang.me/posts/2013/sockjs-redis-nodejs-tutorial/ * http://pyvideo.org/video/1798/make-more-responsive-web-applications-with-socket * http://strongloop.com/strongblog/real-time-engines-in-node-js/ ### Web Application Frontend Design #### Icons &c * http://fortawesome.github.io/Font-Awesome ### Web Application Deployment ##### Untested * https://www.codeship.io/ #### HTLM5 & CSS3 ##### Flexboxes Probably better than using `float`: * http://the-echoplex.net/flexyboxes/ ##### Custom HTML Elements **Rationale**: getting away from `
`, moving towards ``. Probably need a polyfill for IE compatibility; without namespacing, custom tags may face problems with future updates to HTML5 standard. ## Alpha-One Design Philosophy (Sketches) ### Templates, Layouts, Views A **template** is a function that accepts a `request` and a `response` argument and returns HTML as text (or as a POD—not yet implemented). A **layout** is a function that accepts a `request`, a `response`, and a `content` argument; it returns HTML as text (or as a POD—not yet implemented). A **view** is a middleware method that accepts a `request`, a `response`, and a `done` argument; it gets bound to a constant or parametrized HTTP route. Views are somewhat special in that they can act either in a synchronous or a asynchronous fashion. * **Asynchronous views must return `undefined` or `null`** to signal that output should not yet be sent to the client (this is expecially important to keep in mind when using CoffeeScript, as functions may be compiled with an explicit `return` statement, so it's good practice to end asynchronous view methods with an explicit `return null`). * **Asynchronous views must call `done content` when they're ready to, otherwise the client will hang on waiting indefinitely for a response. * **Synchronous views must return a text**; no other return values are allowed at this point in time, although it conceivably makes sense to return an object that represents some content and may be manipulated before beign serialized and sent to the client. After a view has returned a text, it must not call `done`; this is only allowed once. Conversely, once `done` has been called, returning anything but `undefined` or `null` is not permitted. * We currently use [teacup](https://github.com/goodeggs/teacup) as a templating engine, but this may change in the future. * Typically, we have one separate file for templates called, unsurprisingly, `templates.coffee`; within that module, there is a boilerplate header for name_ of teacup eval "#{name_.toUpperCase()} = teacup[ #{rpr name_} ]" that 'pulls' all teacup methods as uppercase variables into the module-local namespace. * Templates and views do not use `teacup.renderable`; instead, they typically have, near the top of the function, a line `return RENDER => ...`; this has proven to work better for our purposes (it's also less surprising than `RENDERABLE`). * There is some boilerplate near the top of each template and layout: O = get_options request page_style = O[ 'page-style' ] ? 'plain' title = O[ 'title' ] ? 'welcome' and a module-local function get_options = ( request ) -> return request[ app_key ] ? {} providing a way to get view options from the request object. `app_key` is a text variable shared between the provider (the middleware stack) and the consumers (the templates and layouts) of per-request data; it should be set to a string that can be assumed to be unique within the given middleware stack (in the example, we chose `alpha-one`). * As a matter of convention, `O[ 'layout' ]` may contain a function that should be used instead of `templates.layout`. * Here is how templates are turned into views.—In `app/main`, there is a function `view`: view = ( view_name ) -> return ( request, response ) -> http_ok request, response response.write templates[ view_name ] request response.end() It is used like this: app.get '/', view 'homepage' app.get '/welcome', view 'welcome' app.get '/goodbye', view 'goodbye' **TAINT** As becomes immediately apparent, there is **no support** to emit any other HTTP status code but `200` at this point in time—no redirects, no 404s are possible with this kind of setup; 'all sales are final'; when the given route matches, the templates is executed and its result are sent back with an HTTP `200 OK`. Also, no `next` method to skip the view is present. Such functionality may or may not get integrated into the `view` function at some point in the future; right now the way to do it is to implement a middleware function yourself (it is easy) and **not** use `view` for this purpose. ### User Authentication ### Redirect, Bounce, and Back-To There are three functions concerned with HTTP redirection: `redirect`, `bounce` , and `back_to`. * `redirect request, response, location` is just the plain Express `response.redirect` function (it is conceivable that some functionality might be added to this function so maybe prefer it over the Express method). * `bounce request, response, location` is like `redirect`, but it **(1)** issues a flash informing the user about the redirection (which your app may choose to silently ignore), and, **(2)** more importantly, sets a cookie value that contains the location from where the redirect was issued. * `back_to request, response, location` is like `redirect`, but it tries to retrieve the original location from the cookie set by `bounce`; if none can be found, it redirects to `location`. So say you go to a restricted are `http://app/nogo`, a `bounce` to `/login` will occur; after successfully entering credentials, you will go `back_to` `/nogo`. If you went straight to `/login` or you intentionally deleted the cookie in the process and the `login` middleware ends with the line `back_to request, response, '/welcome'`, you will end up at a catch-all welcome page instead. (Implementation details discussed here may change; it is conceivably cleaner to store the `bounce` address in a session instead of in a cookie).