This past September, my team at TheoremOne embarked on a research project. Our mission was to evaluate the maturity of IPFS’s network and tools with the current frontend development tech stack and to weigh strategies and approaches for building decentralized apps.
To achieve this we planned on building a clone of Vine using Offline First and Distributed Web technologies. Vine was a short-form video hosting service where users could share six-second-long looping video clips. You can think of it as the “Instagram for GIFs.” If the necessary networks and protocols were mature enough, we would deliver a decentralized app where users could create GIFs and share them with friends. Decentralized Apps can also be called Distributed Apps, which people refer to as “Dapps.”
What is a Dapp?
Most people are familiar with the term “application” as it pertains to software. There are millions of software applications currently in use, and the vast majority of web software applications follow a centralized server-client model. A Dapp has its backend code running on a decentralized peer-to-peer network. Contrast this with an app where the backend code is running on centralized servers. Dapps will power the next wave of computing.
As explained in my coworker Chad’s talk, a Dapp is an app that works in the most extreme network conditions. Can you imagine horrible network conditions such as incredibly high latency, extremely slow connectivity from being very far from the server? Picture yourself on Mars if you want to easily imagine such conditions, apps built in this way would even work on a future Mars colony!
What exactly is decentralized about them? Two things: Data and Compute.
- Data: Rather than storing all of your data on one company’s servers, and always fetching all of that data from said company, Dapps allow you to store your own data, or to fetch it from whoever already has it nearby.
- Compute: Rather than using one company’s “cloud” for doing data processing, transactional logic, and other computation, Dapps make use of emerging new networks that don’t rely on the goodwill and stability of any single actor, but instead leverage the computing resources of the entire network.
For our GIF-based research, we planned mostly to explore the Data side of building a Dapp. We especially wanted to explore using IPFS — the InterPlanetary File System.
Our design goals:
- Build an app that runs in the browser, and requires no additional downloads (no need for extensions, native apps and the like)
- Store all data locally, on users’ individual machines
- Broadcast data directly from one user to another, with no servers intermediating
The second design goal clearly marks this project as research — we expect the data used by our website to grow too quickly for this storage strategy to be practical for a consumer app. Long-term, there are ways to work around this problem — we could offer users the opportunity to back up their own data on a device plugged into their home router, or we could allow them to pay other people to store their data on the IPFS network using Filecoin. But those sorts of long-term solutions aren’t ready yet, and are outside the scope of our research.
“Beware of the dragons”
What is IPFS, anyway?
There’s a lot to say, and they’ll tell you about it better than we can. But to start out, it’s important to understand that IPFS is a new protocol. A protocol is not a library; it is not a single tool. Instead, the protocol can be implemented in any language you care to write it in.
The IPFS team’s primary implementation is written in Go, a language that would work well on servers or on a desktop via a downloaded app. But per our design goals, we need the whole thing to work in a browser, no downloads required. To this end, we can use
IPFS as a whole is still in alpha, with known bugs and missing features. And though it is also built and maintained by the core IPFS team, the JS implementation is even less mature than the Go implementation. On the Project Status section of their README, they say explicitly that there’s “lots of development happening” and to “beware of the dragons.”
Well! I was named after Saint George, slayer of dragons. Bring ‘em on!
Our first dragon: the build process
- Created a new React app with
- GIF creation with
- Storing GIF data locally with
- Broadcasting GIF data to others using the app with
- Merge real-time interactions and solve potential conflicts using the library
It was working on our local machines and looking great.
And then: we tried to deploy it.
npm run buildfails to minify
You may occasionally find a package you depend on needs compiled or ships code for a non-browser environment. This is considered poor practice in the ecosystem and does not have an escape hatch in Create React App.
To resolve this:
Open an issue on the dependency’s issue tracker and ask that the package be published pre-compiled (retaining ES6 Modules). Fork the package and publish a corrected version yourself. If the dependency is small enough, copy it to your src/ folder and treat it as application code.
Apparently we were using a dependency that didn’t play nicely with
How to slay a dragon
create-react-app runs its
build script to prepare code for a production environment, one of the things it does is use
UglifyJS2 to make the code as small as possible. The version of
es6 syntax. Only
es5 syntax and older.
Indeed, when we investigated where exactly our build script was failing, it failed on keywords like
We tried to solve this in two different ways:
- Transpile all dependencies to ES5 syntax ourselves
- Remove minification altogether
Strategy 1: Transpile all dependencies to ES5 syntax ourselves
And I started adding them.
And each time, I’d get a little further. The build process would fail at a different spot. But with the same error.
And each time, one by one, I continued. I had to find the last dependency that needed transpiling. With each try I was sure I had reached the last one but the error kept showing up. Eventually the error stopped. I had built up a list of 255 files that needed to be transpiled! From 52 different dependencies!
We didn’t exhaustively investigate all of them, but most of these problem libraries are part of the IPFS ecosystem. It appears that most of these use a tool called
AEgir — also built by the IPFS team — for their build process. This would explain why so many of our dependencies fail.
I persevered. 225 problematic files later, I got our build script to get past that problem…
…And onto a new one.
We had actually come across a Github issue about this one before, when researching our initial problem. Now we could finally make sense of it.
It turns out that transpiling our dependencies to ES5 wasn’t enough. The minification step also removes constructor names, which causes the problems explained here.
So then. Strategy 2.
Strategy 2: Remove minification altogether
In order to solve this we had to
create-react-app wraps up various complex logic related to
babel, and other tools. It’s great. When you run
create-react-app eject, it stops hiding all that logic. It dumps Webpack and Babel configuration files into your codebase.
This gave us the flexibility we needed to remove minification altogether.
As you might expect, this increased the size of our built bundle by quite a bit. In fact, from 1.5MB to 3.8MB.
But it worked. We can at least deploy our app now.
We didn’t like this solution. We tried to avoid it. We tried to use the latest version of the minify tool for webpack (
UglifyJS2 version 3) with a set parameter to maintain classnames (
keep_classnames). It would get us around the
constructor name problem but another one would show up:
Uncaught SyntaxError: Unexpected token (. We did not reach far trying to debug this new error.
We decided to cut our losses, and move on with an unminified build.
The two headed dragon
Perhaps we’re being generous with ourselves. Maybe what we just described wasn’t so much slaying a dragon, but merely running away from it. Well, we felt like we had slain a dragon!
But just when we thought we were past it, another error appeared in our consoles:
So, something to do with our app connecting to an insecure WebSocket (
WS) instead of a secure one (
We found out we were dealing with a two headed monster! We tried some quick hacking, but nothing. We had wounded the creature, but it had resurrected.
In addition to this error, our app felt slower than it used to! After creating a new GIF and clicking “save”, it would take half a minute before it showed up in the list. It would take half a minute to show up on our friend’s devices. Why was it so slow? Did it have something to do with this
Then we found this comment. The team at IPFS are constantly updating their code, we probably had cached our IPFS repo settings from an old codebase and that caused the error to appear.
We applied the suggested solution and the beast finally surrendered. Our beleaguered app was working again. We could share GIFs with each other, with no servers in between us.
But it was still slower than it used to be…
IPFS, their network, and tools are a work in progress.
As mentioned in the above-linked forum post:
Performance improvements are being actively developed, but these things take time.
These are the problems we found during our Dapp development:
js-ipfs will produce a
build error during the minify step. Such error happens because the tool used to minify (
UglifyJS version 2) is not able to understand code written in
es6 syntax -
js-ipfs and some of its dependencies have code written in
- Transpiling the code written in
es5syntax will make the minify step succeed, thus the
buildscript will succeed too.
- Using another minify tool (one that is able to understand code written in
es6syntax such as
UglifyJS version 3or
babili) will make the minify step and the
buildscript succeed too.
- Using either the
js-ipfstranspiled or minified code will generate an error. Such error happens because the minify step removes
classnamesfrom the source code and some IPFS dependencies have checks in place for specific
- By setting the corresponding configuration parameters (
keepFnNames) the resulting code will pass the
- By setting the corresponding configuration parameters (
- A generic, very hard to debug error will appear if you’ve reached this far:
Uncaught SyntaxError: Unexpected token (
- Using either the
- Removing the minify step from the
buildscript will make the process succeed.
- The app will most likely more than double in size.
To sum up, follow these steps to
build a Dapp if you want to use the IPFS toolset and modern frontend tools:
- Execute the
- Remove the minify step from the production
webpackconfiguration file (this file can be found under the
Even when we used an older version of our codebase, from before removing the minification step, it was still sluggish. But… we hadn’t changed anything. Why did the app performance change, even with the same codebase?
While we were starting this project in late September, a political situation was becoming tense outside my window here in Barcelona, Catalonia, Spain. A planned independence referendum was about to take place: Catalonians were going to choose whether or not they would continue to be part of Spain or undertake their own autonomous governance. Spain’s central government had other plans. Disruption would take over.
On the days prior to the event some measures where taken: on Wednesday morning, police entered the offices of the
.cat internet registry’s headquarters in Barcelona and seized all of its computers. They arrested six members of the staff and held four of them for two days. Finally the CTO was accused of sedition.
It turns out, the way the Catalan government used IPFS to skirt Spanish censorship had attracted some attention. The IPFS network grew quickly from having about 1000 nodes to having about 6000.
But neither the Go implementation nor the JS implementation are built to deal with this increased scale. At least not yet. That’s coming!
We were very happy to see how these tools can help in real situations happening, like the Catalonia referendum. But there is more work to be done.
We hope our tale of the journey through the difficult
build process — the
uglify, and the rest — can help other people experimenting with these technologies. We also hope it might spur library owners to make their tools play more nicely with others in the wider ecosystem.
We encourage others to try out these new technologies. Let us know what dragons you find!