# Symlink Behavior In Broccoli Plugins

**Summary:** We will soon be changing the contract between Broccoli plugins to
mandate that plugins follow symbolic links inside their input trees. This
includes recursing into symlinked directories.

## Background

Broccoli plugins often need to pass files through from their input trees to
their output trees. For instance, the CoffeeScript plugin (based on
broccoli-filter) will copy any files that do not end in `.coffee` verbatim
from its input tree to its output tree; and the merge-trees plugin will
successively copy the files in all its input trees to its output tree.

In the beginning, we used hardlinks to "copy" all the files. Hardlinking a
file takes a very small constant amount of time, regardless of file size. We
created a lot of hardlinks on each rebuild, but because hardlinks are fast,
the performance was adequate on most project sizes.

However, we later discovered that hardlinks can [cause data loss on OS
X](https://github.com/broccolijs/broccoli/blob/master/docs/hardlink-issue.md),
and as a stop-gap immediately switched all plugins to copying files
byte-by-byte rather than hardlinking.

Unfortunately, copying files turns out to be too slow. On a typical project,
it adds seconds or even tens of seconds to the rebuild time.

## Upcoming Change

To resolve this performance issue, we will be switching to symlinking files
instead of copying them. At the end of the build process, Broccoli will
automatically dereference all symlinks in the final tree, so that the output
generated by `broccoli build` only contains regular files and directories.

To make this possible, we will soon start requiring plugins to follow
(dereference) symlinks inside their input trees - that is, to treat symlinked
files the same as regular files, and recurse into symlinked directories. Until
now, we had left undefined how plugins deal with symlinks. In practice, most
plugin currently do not follow symlinks, but rather copy symlinks verbatim or
similar.

This is a change in the expected behavior - the "contract" between plugins, if
you will - rather than in the programmatic API.

The change will happen in two parts.

### Part 1: Transparently Follow Symlinks

The first set of changes is making plugins follow symlinks consistently.

This part is currently underway.

This change should be mostly non-breaking. Breakage can occur when there are
broken symlinks in source trees, which will now result in build failures; and
also when an application relies on plugins ignoring symlinked files or on
plugins not recursing into symlinked directories.

To implement this change, fortunately, not much code is necessary. Most file
system functions (like `readFile` and `readdir`) transparently follow
symlinks, both on Node and in external libraries. For example, calling
`readdir` on a symlink to a directory behaves the same as calling `readdir` on
the directory directly.

Notable places where we need to make changes are:

#### Use `stat` Instead Of `lstat`

The `stat` and `lstat` functions ([syscall
documentation](http://linux.die.net/man/2/stat), [Node
documentation](http://nodejs.org/api/fs.html#fs_fs_stat_path_callback)) behave
differently with regard to symlinks: `stat` follows symlinks, whereas `lstat`
returns information on the symlink itself. For instance:

```js
// Treat symlinks differently (old behavior):
var lstats = fs.lstatSync(somePath);
if (lstats.isFile()) {
  ...
} else if (lstats.isDirectory()) {
  ...
} else if (lstats.isSymlink()) {
  // Here be special symlink handling code.
  ...
} else {
  throw new Error('Unexpected file type'); // socket, device, or similar
}

// Transparently follow symlinks (new behavior):
var stats = fs.statSync(somePath);
if (stats.isFile()) {
  // Could be file, or symlink pointing to file.
  ...
} else if (stats.isDirectory()) {
  // Could be directory, or symlink pointing to directory.
  ...
} else {
  throw new Error('Unexpected file type'); // socket, device, or similar
}
```

Plugins should be sure to use `fs.statSync`, rather than `fs.lstatSync`.

Note that stat'ing a broken symlink (`fs.statSync('does-not-exist')`) throws
"Error: ENOENT, no such file or directory 'does-not-exist'". We will typically
let these errors propagate and not try to handle them. It seems acceptable to
fail when we encounter broken symlinks.

#### Helper Packages

The node-walk-sync and broccoli-kitchen-sink-helpers helper packages currently
do not follow symlinks. We will soon release updated versions that do. If your
plugin uses either of those, make sure that you use a [floating version
spec](https://www.npmjs.org/doc/misc/semver.html#ranges) with "~" or "^" to
automatically get the update, like so:

```js
"dependencies": {
  "walk-sync": "^0.1.2",
  "broccoli-kitchen-sink-helpers": "^0.2.4"
}
```

#### Auto-Dereference Symlinks

Once we start using symlinks, the output trees generated by plugins may
contain symlinks.

Broccoli will start automatically dereferencing symlinks in a soon-to-be
released version, so that the output generated by `broccoli build` will only
contain regular files and directories.

If you have custom builder code, you may need to invoke
[node-copy-dereference](https://github.com/broccolijs/node-copy-dereference)
on the final build output yourself.

### Part 2: Emit Symlinks As An Optimization

* broccoli-merge-trees
* later: broccoli-filter

## Emacs Lock Files

## Performance Gains

...