Things that can go wrong when downloading

When I get a little bit too emotional about my current baby, the itch.io app, there’s always a timely support ticket reminding me that it is currently, still a glorified game downloader.

However true that is, that doesn’t mean it’s easy! In the past year, I’ve had to account for a bunch of failure conditions that can happen, some of which I didn’t realize were even possible. Let’s review them, for fun!

Your computer may refuse to run butler

butler is a little helper program for the itch.io app and developers. It serves as both an uploader, and a downloader (and a differ, a patcher, and various little utility commands).

It’s coded in the Go language, which is great for compatibility, because golang executables will run pretty much anywhere - and it links (almost) everything statically, so there’s no dependency hell, just “whoops your exe is 17MB” hell.

small

There’s this GitHub issue that’s puzzling both the reporter and me: a Linux install that’s seemingly 64-bit, but refuses to run the 64-bit build of butler. It’ll run the 32-bit build fine though!

As far as I can tell, it’s an isolated case. butler runs on many flavors of Linux, both 32 and 64-bit, without any issues. Maybe we’ll find the answer to this riddle in the next few months, or maybe we won’t! Computers are fun.

Your API key may be expired/revoked

When the itch.io app runs butler, it passes your API key, and a download key, if applicable. But as a user, you’re free to revoke it any time from your user settings.

If it’s revoked, API calls will fail. This not a recoverable error - the only fix is to log in again from the itch.io app and try again. It’s not a particularly bizarre error condition, but it’s not necessarily the first thing you think about either!

small

You may be using a proxy

Thankfully, that’s (almost) an easy one. The itch.io app detects your system’s proxy settings and passes them along to butler, which uses them.

But proxies love to break HTTP in funny ways. Especially when you have long, streaming requests. Or any kind of caching.

And when someone reports a proxy problem, it’s not usually one of these open-source, easy-to-configure proxies you can just spin up locally to see what’s going on. They’re usually these closed-source corporate/university proxies, and then presto, you-the-developer-of-the-tool-that-doesnt-work-for-some-user are starring in an episode of Sherlock - if the writers had hired a technical consultant or ten.

small

DNS resolution might misbehave

Before making any API requests or downloads, we need to find the address of, respectively, the API server, and the CDN node we’ll be talking to. That’s where DNS comes in. If we can’t find their address, even we could reach them - we just don’t know who to reach, so the whole thing falls flat.

In 2017, it’s easy to get solid DNS. Just surrender your eternal soul to the Google overlord (set your DNS servers to 8.8.8.8 and 8.8.4.4) and, if you’re not extremely unlucky, you’ll have fast, accurate DNS resolution.

But if you’re just a regular Joe itching to play some games, you might inadvertently have lent your soul to your ISP instead. Or maybe you’re running some Linux distribution that has dnsmasq set up by default - and then butler will see some funny error conditions.

dnsmasq for example, relays DNS requests to another DNS server. If your internet access is temporarily cut, it’ll accept the request, try to relay it, wait, eventually fail and time out. That’s a different failure condition than “not being able to open a connection to 8.8.8.8 because the network is down”. So we have to account for that.

medium

Your computer might think you have IPv6, but you don’t

Once upon a time, it looked like all the RFC we’ll ever need were already written. IPv6 was the answer to that1. But it turns out deployment took a little while. It’s not uncommon for part of your internet route to support IPv6, while other parts don’t know what to do with it.

Thankfully, there’s a solution to that, and it even has a cute name: Happy Eyeballs. The gist is to attempt both IPv6 and IPv4 requests, and see what works best for us.

That’s not something I personally had to deal with, but it’s quirky enough that it warranted a mention in this list.

small

The HTTP(S) server might reject our request

As of late 2016, all itch.io downloads that aren’t external uploads are served by our CDN. Aside from the fact that CDNs are, uh, a bit of an oligopoly, it’s great:

  • Everyone gets fast downloads, no matter their region
  • It saves costs, since bandwidth is cheaper from the CDN than from the storage server
  • It’s a well-behaved, reliable HTTP server that supports Range requests and a bunch of other cool stuff

But there’s a few games still using external uploads, which means butler might be talking to any HTTP server at all.

For example, it might be talking to Github releases, which:

  • Redirects any requests to a US-based Amazon S3 bucket
  • …which is slow for anyone outside the US
  • …refuses HEAD requests (only accepts GET)
  • …and requires us to forward headers there when we’re doing Range requests

It gets worse - we might be talking to a self-hosted HTTP server that not only won’t scale but will also be pretty poor at HTTP. For example, it might not set the Content-Length header, so we won’t know how large the file we’re downloading is. Or it might reply to Range requests with 200, sending us the whole file every time (which means we can’t resume downloads if they’re paused/stopped/abruptly shut down).

And finally, it might not be a static file server but one of those terrible we’ll-make-you-wait-in-a-queue-while-you-watch-ads sites that eventually let you download at 80KiB/s, but only from a browser. There’s no hope for these. The jdownloader team spent a decade writing crawlers for all these, so, use that. I won’t!

Note: if you’re using external uploads, please don’t. Make yourself and your users a favor and upload directly to itch.io, whether using web (better), or butler (best). You can even automate uploads if you’re doing CI, it generates small patchs for itch.io app users and fixes a bunch of other problems. More on that later!

The HTTP(S) server might close the connection

Even the best of HTTP servers sometimes decide a connection has been going on for too darn long and decide to bail out of here before you stick your toothbrush in their bathroom. If they support Range requests, we can simply pick up where we left off. If not, we’re out of luck.

Your computer might close the connection

Maybe it’s the full moon, maybe you just installed VirtualBox’s network drivers, maybe it was something else, I’m not here to judge - but sometimes, your OS will just close a connection. Same dance - Range support makes for a happy butler, lack of it means we have to start over from the beginning.

Data might just stop coming in

Closing a connection isn’t something to be celebrated, but at least we know something’s up. Sometimes, data will just stop flowing. To fight against that, we have timeouts - if the connection is idle for too long, we give it up and try again with a fresh one (given - you guessed it - that we have Range support).

medium

A bit might get flipped

TCP/IP has a bunch of measures to avoid data corruption - but given all the agents involved, once in a very rare while, a bit might get flipped. This means the server thinks it’s sending the right data, but it’s sending something very slightly different - and then there’s a few scenarios:

  • Some check will catch it (for example, the corrupted data results in an invalid zip)
  • Nothing catches it, it extracts fine, and you end up with a corrupted copy of the game

The latter is even rarer because, even if by some freakish lack of luck, the resulting file can be extracted (zip headers aren’t corrupted, etc.), the zip format has built-in checksums, and it’ll notice on extraction that the checksum and the content doesn’t match (and then you get to try again, yay!).

But maybe it corrupted something else, like a file name - so you end up with Level02 instead of level0, and the game won’t start.

small

Writing to disk may fail

So, all the network operations are going fine, we’re downloading at full speed, and suddenly, writing to disk fails!

There could be a bunch of reasons:

  • A network disk was disconnected (because the host went down, for exmaple)
  • An external disk was disconnected (because someone tripped on the USB/SATA cable, or you’re Jesus and your SATA controller hates you)
  • You don’t have permissions to write to a folder
  • You’re trying to write to a file that’s locked3.

small

What you’re downloading may be funny

Sometimes, new developers think an .exe can run anywhere - so they add Linux and macOS tags, and call it a day. But, of course, it won’t run!4

Sometimes, developers using löve aren’t quite sure how to make macOS or Linux builds, so they just upload the .love file5 and hope it’ll run. (But of course, it won’t)

Or sometimes, it is a proper Windows/Linux/macOS build, but the uploaded .zip file was compressed using some software that decided standard zips are too good for them, and shove another compression format in there, like LZMA (or cthulhu forbid, Wavpack). butler’s unzipper doesn’t really handle that (yet!)

Maybe extraction goes perfectly fine, but then, on macOS/Linux, your executable is missing the right permissions (maybe because you compressed it from Windows!). Luckily, the itch.io app fixes that. But, your users downloading from web will definitely be staring at a broken game, unless they know what they’re doing.

Note: butler fixes permissions at upload time, solving the problem both for web users and app users!

Or maybe it’s a Windows build but it needs some dependency to run, like the MSVC runtime, or .NET, or… in which case, the itch.io app can install it for your users, if you tell it to.

small

Closing words

These are only some of the most entertaining failure conditions that can happen when downloading something.

Thanks for reading, I hope you enjoyed the article, and I wish I could say my little sister drew the illustrations but unfortunately it was all me.

If you liked this, maybe you’ll enjoy trying out the itch.io app - that’s what I spend most of my time working on, either on the visible or invisible parts of it.

Until next time, take care!

Footnotes

  1. I kid, I kid. IPv6 was the answer to IPv4 address exhaustion. The big promise was that every device could get its own IP. Making IoT DDoS botnets more scalable! Ahhh sorry, I’m snarking again :)

  2. Yes, flipping a given bit of an ASCII character can turn lowercase into uppercase and vice versa. That’s on purpose! Aren’t text encodings great? Don’t try to do that on emojis though!

  3. In an early version of the itch.io app, Adam sent me a fun bug report. “Amos, sometimes the game runs fine, but once you quit it won’t launch again.” Turns out the app was updating the game while it was running. Turns out it’s not such a hot idea. Linux and macOS will happily let you do that, while Windows is going to complain about you writing to a running .exe pretty hard.

  4. And that can be solved - we can detect what kind of files an archive contains, and we could tag it automatically. I’ll get around to it any day now. Not now though, I’m in the middle of writing a rant.

  5. Fun fact, .love files are just zip files. For recent versions of love, they even include a version number somewhere in a .lua file inside the zip. So there is a way