Compression is Witchcraft
I had a mild fascination with data compression when I was learning my way around a PC. Especially lossless compression, as commonly used in file archivers.
It felt like witchcraft.
People would try to help by explaining it in simple, high-level terms. For example:
“The compressor finds repeated bytes and replaces them with a token. By storing a dictionary of tokens alongside the processed stream, you can save space.”
Sure. That makes sense.
Except that when I then begin to think about it, it breaks down quickly. How big are the tokens? For that matter, how do you distinguish a token from non-token data? How much space does the dictionary take up?
So many questions, and every one makes it seem more complicated. More confusing. More advanced.
And you know what they say about advanced technologies and their distinguishability from magic.
Hence my adoption of a firm belief: Compression is witchcraft.
Invoking a compressor was like invoking magic. You utter the incantation, and voila! - a smaller file. Unless the file was already compressed, or had been cursed by a northwind, or sent to you via a widdershins route, in which case you got a slightly bigger file.
Magic can be unreliable like that.
Actually that’s a lie. Magic is incredibly reliable. But it’s also incredibly fragile - there are many, many edge cases that alter its behaviour, and many of those edge conditions are almost invisible to human perception. This gives the appearance of unreliability when actually magic is completely and deterministically reliable. But that’s something to discuss another day.
Anyway, back to the witchcraft of compression.
Despite its obvious unreliability, I tinkered with compression, trying to understand it. I gained a solid, practical grounding in using compression tools. But the underlying compression algorithms and processes? Witchcraft. No matter how many manuals, technical notes or primers I read, compression remained witchcraft.
This experimentation was being done on an 8086 with 640Kb of RAM, and a hard disk that moved slower than molasses going uphill in winter. The incantations did not present instant results - far from it. A particularly good incantation could result in the machine being occupied by magical meanderings for several minutes…
My preferred archiver for a long time was ARJ, which never really seemed to make the jump from DOS to Windows. For me ARJ had replaced LHA, whose major claim to fame was probably being the compressor used by the DOOM installer. (When you’re into witchcraft, you can’t help but see the signs of it being used.)
Hardware moved on. Minutes became seconds, and then milliseconds.
Compression is now something everyone uses almost every day that they use a computer. Modern office documents use it. Your connection to most webservers uses it. Updates to apps on your phone use it. It’s deeply baked into modern computing.
Imagine that. Computing, relying on witchcraft.
11 years ago, Google put out a video series called Compressor Head. This was in recognition of how pervasive compression had become, and it was designed to help developers understand it better. You can watch it here.
I now understand compression. It’s no longer witchcraft.
On the one hand, I am happy I understand this. On the other, the magic and mystery is gone. A part of me misses my rather happy ignorance.
Fortunately for me, there’s still lossy compression, as used in image/video/audio formats. That’s absolutely witchcraft. Freaky, freaky witchcraft…