| 1 | # DwebTransports
|
| 2 | General transport library for Decentralized Web handles multiple underlying transports.
|
| 3 |
|
| 4 | ## Background
|
| 5 | This library is part of a general project at the Internet Archive (archive.org)
|
| 6 | to support the decentralized web.
|
| 7 |
|
| 8 | ### Goals
|
| 9 | * to allow a single API that can be used for most basic interactions with
|
| 10 | decentralized transports.
|
| 11 | * to support multiple URLs, on different transports, so that the (current) underlying unreliability
|
| 12 | is hidden.
|
| 13 |
|
| 14 | ### Node Installation
|
| 15 | * Clone this repo.
|
| 16 | * Until this is in npm, add the line
|
| 17 | `"@internetarchive/dweb-transports.git": "latest",`
|
| 18 | to your package.json file in the dependencies section.
|
| 19 | * `npm install @internetarchive/dweb-transports` will install the dependencies including IPFS & WebTorrent
|
| 20 |
|
| 21 | `const DwebTransports = require("@internetarchive/dweb-transports")` will add all Transports to a Javascript file.
|
| 22 | * TODO-API writeup how to require only some of the transports.
|
| 23 | * Then see usage API below
|
| 24 |
|
| 25 | ### Installation and usage in the Browser
|
| 26 | * Install npm & node
|
| 27 | * Clone this repo and cd to it.
|
| 28 | * `npm run build` will create dist/dweb_transports_bundle.js
|
| 29 | * Add `<SCRIPT type="text/javascript" src="dweb_transports_bundle.js"></SCRIPT>` to your `<HEAD>`
|
| 30 |
|
| 31 | Then code like this should work.
|
| 32 |
|
| 33 | ```
|
| 34 | async function main(url) {
|
| 35 | try {
|
| 36 | // and if not found will use the defaulttransports specified here.
|
| 37 | await DwebTransports.p_connect({
|
| 38 | statuselement: document.getElementById("statuselement"), // Where to build status indicator
|
| 39 | defaulttransports: ["HTTP","IPFS"], // Default transports if not specified
|
| 40 | transports: searchparams.getAll("transport") // Allow override default from URL parameters
|
| 41 | });
|
| 42 | // Any code you want to run after connected to transports goes here.
|
| 43 | } catch(err) {
|
| 44 | console.log("App Error:", err);
|
| 45 | alert(err.message);
|
| 46 | }
|
| 47 | }
|
| 48 | var searchparams = new URL(window.location.href).searchParams;
|
| 49 | ```
|
| 50 | ## Notes on implemntation
|
| 51 |
|
| 52 | ### Implementation on HTTP (TransportHTTP.js)
|
| 53 | The HTTP interface is pretty simple, a standard extensible gateway (dweb-gateway) we wrote is used.
|
| 54 |
|
| 55 | fetch is a straightforward HTTP GET to a URL that includes the multihash
|
| 56 | store is a POST to the same URL
|
| 57 |
|
| 58 | Lists are implemented via append-only files on the HTTP server using URLs that contain the same hashes.
|
| 59 | “Add” appends to this file, “list” retrieves the file.
|
| 60 |
|
| 61 | Listmonitor isn’t supported, and can’t really be as there is no open channel to the client.
|
| 62 |
|
| 63 | ### Implementation on IPFS (TransportIPFS.js)
|
| 64 |
|
| 65 | This section will only make sense if you understand something about IPFS.
|
| 66 |
|
| 67 | See TransportIPFS.js in the repository for details for code.
|
| 68 |
|
| 69 | IPFS has two Javascript versions, both of which currently implement only a subset of IPFS (JS-IPFS is missing IPNS, and JS-IPFS-API is missing pubsub).
|
| 70 | We are mostly using JS-IPFS because JS-IPFS-API creates a centralisation point of failure at a known HTTP host,
|
| 71 | and because JS-IPFS-API has issues connecting to a local IPFS peer because of some odd security choices by IPFS.
|
| 72 |
|
| 73 | IPFS is initialized via creating a IPFS object with a configuration. We use the Websockets connctor since the alternative, but its got a single point of failure. WebRTC is an alternative but is seriously broken (crashes both chrome and firefox)
|
| 74 |
|
| 75 | Blocks are stored and retrieved via ipfs.files.get and ipfs.files.add.
|
| 76 |
|
| 77 | For lists and tables - see YJS which uses IPFS.
|
| 78 |
|
| 79 | #### Issues with IPFS
|
| 80 |
|
| 81 | Error feedback is a little fuzzy.
|
| 82 |
|
| 83 | There are issues with IPFS swarms that we haven’t been able to figure out about how to ensure that “put”ting to IPFS creates an object that can be read at all other browsers, and persists. See DT issue#2
|
| 84 |
|
| 85 | Naming hasn’t been implemented in IPFS yet, partly because IPNS is not available in the JS-IPFS, and partly because IPNS has serious problems:
|
| 86 | (requirement to rebroadcast every 24 house so not persistent; merkle tree so change at leaf changes top level; doesnt work in JS-IPFS;)
|
| 87 | We implemented naming outside of IPFS (in Naming.js) to get it to work.
|
| 88 |
|
| 89 | To install IPFS for Node (and this needs testing)
|
| 90 | ```
|
| 91 | yarn add ipfs ipfs-http-client
|
| 92 | ```
|
| 93 | This will get overridden by an update of dweb-mirror, so its probably you will want
|
| 94 | this as a dependency of whatever is using dweb-transports instead.
|
| 95 |
|
| 96 | To use IPFS pass "IPFS" in during the "connect" step
|
| 97 |
|
| 98 | ### Implementation on WebTorrent
|
| 99 | WebTorrent implements the BitTorrent protocol in the browser. It will work for retrieval of objects and currently has the fastest/most-reliable stream interface.
|
| 100 |
|
| 101 | We also have a modified Seeder/Tracker which are currently (Sept2018) in testing on our gateway.
|
| 102 |
|
| 103 | To install WebTorrent for Node (and this needs testing)
|
| 104 | ```
|
| 105 | yarn add webtorrent
|
| 106 | ```
|
| 107 | This will get overridden by an update of dweb-mirror, so its probably you will want
|
| 108 | this as a dependency of whatever is using dweb-transports instead.
|
| 109 |
|
| 110 | To use WebTorrent pass "WEBTORRENT" in during the "connect" step
|
| 111 |
|
| 112 | ### Implementation on YJS (TransportYJS.js)
|
| 113 |
|
| 114 | YJS implements a decentralized database over a number of transports including IPFS. It supports several modes of which we only use “Arrays” to implement append-only logs and "Map" to implement key-value tables.
|
| 115 |
|
| 116 | There is no authentication built into YJS but If using via the higher level CommonList (CL) object,
|
| 117 | the authentication isnt required since the CL will validate anything sent.
|
| 118 |
|
| 119 | To install YJS for Node (and this needs testing)
|
| 120 | ```
|
| 121 | yarn add yjs
|
| 122 | ```
|
| 123 | This will get overridden by an update of dweb-mirror, so its probably you will want
|
| 124 | this as a dependency of whatever is using dweb-transports instead.
|
| 125 |
|
| 126 | To use YJS pass "YJS" in during the "connect" step
|
| 127 |
|
| 128 | ### Implementation on GUN
|
| 129 |
|
| 130 | GUN implements a decentralized database and we have mostly migrated to it (from YJS) because there is some support and an active team.
|
| 131 |
|
| 132 | Our tables and Lists are mapped as JSON objects inside GUN nodes due to some limitations in GUN's architecture for multi-level objects.
|
| 133 |
|
| 134 | Still (as of Sept2018) working on Authentiction, and some reliability/bug issues.
|
| 135 |
|
| 136 | To install GUN for Node (and this needs testing)
|
| 137 | ```
|
| 138 | yarn add gun
|
| 139 | ```
|
| 140 | This will get overridden by an update of dweb-mirror, so its probably you will want
|
| 141 | this as a dependency of whatever is using dweb-transports instead.
|
| 142 |
|
| 143 | To use GUN pass "GUN" in during the "connect" step
|
| 144 |
|
| 145 | ### Implementation on WOLK
|
| 146 |
|
| 147 | WOLK has implemented and maintain there own shim which is part of dweb-transports
|
| 148 |
|
| 149 | To install WOLK for Node (and this needs testing)
|
| 150 | ```
|
| 151 | yarn add "git://github.com/wolkdb/wolkjs.git#master"
|
| 152 | ```
|
| 153 | This will get overridden by an update of dweb-mirror, so its probably you will want
|
| 154 | this as a dependency of whatever is using dweb-transports instead.
|
| 155 |
|
| 156 | To use WOLK pass "WOLK" in during the "connect" step
|
| 157 |
|
| 158 | ### Implementation on FLUENCE
|
| 159 |
|
| 160 | FLUENCE has implemented and maintain there own shim which is part of dweb-transports
|
| 161 |
|
| 162 | To install FLUENCE for Node (and this needs testing)
|
| 163 | ```
|
| 164 | yarn add fluence
|
| 165 | ```
|
| 166 | This will get overridden by an update of dweb-mirror, so its probably you will want
|
| 167 | this as a dependency of whatever is using dweb-transports instead.
|
| 168 |
|
| 169 | To use FLUENCE pass "FLUENCE" in during the "connect" step
|
| 170 |
|
| 171 | ### Implementation of ContentHash
|
| 172 |
|
| 173 | We have a simple Contenthash store/fetch that supports lists and key-value databases,
|
| 174 | and knows about retrieving content by sha1 hash from the Archive
|
| 175 |
|
| 176 | No installation is required - it builds on the HTTP transport
|
| 177 |
|
| 178 | To use, pass "HASH" in during the "connect" step
|
| 179 |
|
| 180 | ### See also
|
| 181 |
|
| 182 | See [example_block.html](./example_block.html) for an example of connecting, storing and retrieving.
|
| 183 |
|
| 184 | See [API.md](./API.md) for the detailed API.
|
| 185 |
|
| 186 | See [Dweb document index](./DOCUMENTINDEX.md) for a list of the repos that make up the Internet Archive's Dweb project, and an index of other documents.
|
| 187 |
|
| 188 | ### Release Notes
|
| 189 |
|
| 190 | * 0.2.6: Move require of wrtc into TransportWEBTORRENT
|
| 191 | * 0.2.5: Remove dependencies on ipfs-http-client and wolkjs which need separate installation now, update README to clarify
|
| 192 | * 0.2.4: Lighter Fluence transport and Wolk scripting issue
|
| 193 | * 0.2.3: Add Fluence transport, split HASH (contenthash) out of HTTP
|
| 194 | * 0.2.2: Add Wolk, YJS, and add function for node loading
|
| 195 | * 0.2.1: Move script loading into Transports
|
| 196 | * 0.2.0: Start moving transport dependencies to consumer, specifically IPFS, GUN, WebTorrent
|
| 197 | ------
|
| 198 | * 0.1.63: Move naming internal
|
| 199 | * 0.1.62: Upgrade dependencies
|
| 200 | * 0.1.61: loopguard to return correct error from queuedFetch; add info to statuses
|
| 201 | * 0.1.60: Back on master release of webtorrent
|
| 202 | * 0.1.59: Add DwebTransports.statuses synchronous API
|
| 203 | * 0.1.58: Upgrade GUN and IPFS to v0.35.0 (0.36 breaks webpack as currently configured)
|
| 204 | * 0.1.57: Fix heartbeat timer stopping
|
| 205 | * 0.1.56: tweak http queue; changes to stop, status, info to support heartbeat on HTTP
|
| 206 | * 0.1.55: tweaking queue concurrency
|
| 207 | * 0.1.54: queue for http; disable fallback from IPFS to http://ipfs.io
|
| 208 | * 0.1.53: Reenable Wolk as fixed
|
| 209 | * 0.1.52: Gun fixes but off by default https://github.com/internetarchive/dweb-archive/issues/106
|
| 210 | * 0.1.51: Temporary fix to GUN failures by adding wait:2000
|
| 211 | * 0.1.50: Update to GUN's new bizarre version numbering.
|
| 212 | * 0.1.49: Disable WOLK - currently failing
|
| 213 | * 0.1.48: Support for noCache
|
| 214 | * 0.1.47: Update dependencies (see yarn.lock)
|
| 215 | * 0.1.46: Correctly recognize /arc/archive.org urls
|
| 216 | * 0.1.45: Fix mergeoptions and update ipfs cache id
|
| 217 | * 0.1.44: hooks to allow react-based UI in dweb-archive (via IAUX)
|
| 218 | * 0.1.43: Add WebTorrent seeding
|
| 219 | * 0.1.42: Better behavior when cant see gateway
|
| 220 | * 0.1.41: Remove createReadStream for browser (it was added for node in 0.1.40), add fetch(url,opts,cb)
|
| 221 | * 0.1.40: Bug fix in httpfetch({count=0}),
|
| 222 | * 0.1.40: Added support for "seed" and tested in IPFS
|
| 223 | * 0.1.40: WOLK - moved to their production sys and master branch
|
| 224 | * 0.1.39: WOLK - updated wolk.js module to fix bugs
|
| 225 | * 0.1.38: httptools - adds retries
|
| 226 | * 0.1.38: WOLK - added to the library
|
| 227 | * 0.1.37: IPFS - dont stop it if we didnt start it (were stopping via API)
|
| 228 | * 0.1.37: Start move to unpromisify pattern v5
|
| 229 | * 0.1.37: IPFS - updated to (significant) v0.34.0 API changes
|
| 230 | * 0.1.36: Made httptools accessable at Transports.httptools so it doesnt have to be separately 'require'd
|
| 231 | * 0.1.35: package update (note wont work with latest versions of yjs or uglify)
|
| 232 | * 0.1.33: Bug fixes; support for gatewayUrls (for dweb-mirror)
|