Skip to content

Tag: torrent

  • Building Flume, a cross-platform BitTorrent client

    Building Flume, a cross-platform BitTorrent client

    The Engine was the Easy Part

    Most torrent clients are either powerful and unpleasant, or pretty and crippled. That observation is old enough to be boring, and it is still true. The category has spent twenty years accumulating features — media players, RSS schedulers, bundled search, cryptocurrency wallets, ad panels in the sidebar — while the part you actually touch every day stayed roughly where it was in 2008.

    So I built Flume: a general-purpose BitTorrent client, and deliberately only that. No chat, no RSS automation, no bundled search, no paid tier. A Tauri v2 shell around a Next.js frontend, with librqbit — a production-grade Rust engine — doing the actual torrenting.

    The bet behind it is simple: the engine is a solved problem. DHT bootstrapping, the peer wire protocol, piece picking, fast resume — librqbit already does all of that, and does it well. Which means every hour of effort can go into the part nobody has solved, which is the experience. Not “the UI.” The experience: what the app tells you, how confident it sounds when it tells you, and what it does with what it learns about you.

    That turned out to be a much more interesting problem than I expected, because being honest with a user requires knowing things the available APIs would not tell me.

    The question a peer count cannot answer

    Here is the situation every torrent client puts you in and none of them resolve. A download sits at 94%. It has been at 94% for two hours. The client cheerfully reports forty peers connected. Is it going to finish?

    Forty peers is not an answer. Forty peers who all stopped at 94% themselves is a download that will never complete, no matter how long you leave it running. Six peers who between them hold every piece is a download that finishes tonight. The number every client shows you — the peer count — is uncorrelated with the only question you actually have.

    The right figure is the rarest piece: of all the pieces you still need, how many connected peers hold the least-held one? If that number is zero, the torrent cannot finish from this swarm however fast the rest arrives. That is a verdict worth putting in front of someone.

    librqbit tracks per-peer bitfields internally — it has to, for piece picking — but did not expose them. So I opened an issue upstream asking for a per-peer count of held pieces, which seemed like the smallest possible ask.

    It was also the wrong ask, and I only noticed after starting the implementation. A per-peer count gives you the mean copies per piece. The verdict needs the minimum. Two peers holding 500 pieces each may overlap completely or not at all — identical averages, and only one of those torrents can finish. I had asked a maintainer to add an API that could not answer my question.

    The corrected version needed the bitfields themselves, which is a bigger ask, so it came with the condition that made it acceptable: the whole thing is gated behind an opt-in flag, off by default, so existing librqbit callers compute nothing and pay nothing. That went upstream as ikatson/rqbit#644, and Flume runs a patched fork until it lands in a release.

    The lesson generalises past torrents: work a concrete case all the way through before you ask anyone for an API. “Two peers, 500 pieces each, do they overlap?” would have taken ninety seconds on paper and saved a wrong request to someone else’s issue tracker.

    There is a smaller trap inside the same feature, and it is the kind of thing that only shows up in adversarial thinking. Bitfields are byte-padded — a torrent with 1,001 pieces ships 126 bytes, and the last seven bits are spare. A peer is not obliged to zero them, and librqbit does not enforce that it did. Count naively over the whole bitfield and a hostile peer inflates its apparent holdings for free. Both the patch and Flume’s own analysis slice to the real piece count before counting anything.

    Refusing to guess

    The availability work produced a rule that ended up governing far more of the app than the feature it came from:

    Verdicts the data cannot support are not invented.

    Flume’s swarm health has six states, and two of them exist purely to admit ignorance. Unknown means there were no bitfields to judge from — no live peers yet, or metadata that has not resolved far enough to know the piece count. None means no reachable peer holds the remainder, which is a real and defensible negative.

    What Flume will not do is split the difference. It would be trivially easy to look at “twelve peers, no bitfields yet” and render Thin or Healthy on vibes. Every client does some version of this. It would also be a confident wrong answer in front of someone deciding whether to leave their machine on overnight, and a confident wrong answer is worse than a blank. The UI renders Unknown as “Connected” and shows the peer counts instead — a fact rather than a judgement.

    The same discipline shows up in the diagnostics report. When a usage batch gets no response, Flume reports exactly that: no answer. It does not say “you’re offline,” because being offline, a DNS block, a certificate problem and a proxy in the way are not distinguishable from inside the process. Picking one would be a verdict the data cannot support.

    There is one genuinely interesting exception, where refusing to guess required adding a guess. The design specifies healthy as “every piece on three or more peers.” Taken literally, that is unreachable below three peers — two peers who are both seeds hold every piece twice over and would read as Thin forever, which is nonsense. The threshold scales: rarest >= min(3, live_peers). Below three peers, a swarm is judged on coverage rather than punished for its size. Rules that produce absurd results at the boundary are not rules, they are unfinished.

    Privacy as an architectural constraint

    Telemetry in a BitTorrent client is a special case, and pretending otherwise is how you become the top comment in every thread about your own software, permanently. qBittorrent ships none. That is the baseline you are measured against.

    Flume’s usage reporting is opt-in, asked exactly once at first run, and off unless you say yes. That costs sample rate, which is a real cost. Opt-out would cost the audience, which is not recoverable.

    But consent is the easy half. The hard half is making the promise structural, so that keeping it does not depend on everyone who touches the code remembering to. Three constraints do that work:

    All network egress originates in Rust. The webview’s content security policy is connect-src 'self' ipc: http://ipc.localhost and it is never widened. This is what makes the promise auditable rather than aspirational. Dropping any frontend analytics or crash-reporting SDK into the app would require relaxing that CSP — which would hand every piece of UI code an egress path it does not currently have, including anything that renders an attacker-controlled torrent name into the DOM. There is exactly one file in the codebase that talks to the outside world, and kjeeping it that way means there is exactly one file to audit.

    The wire format is a closed enum. Every field of every usage event is an enum or a boolean. There is no free-text field anywhere, and that is a hard constraint rather than a stylistic preference. librqbit’s error strings embed
    tracker URLs and filesystem paths — so a String reason field would exfiltrate precisely what the app promises not to collect, on the day someone reasonably decides that reporting why something failed would be useful. Failure reasons are a fixed list instead. A test pins the Rust enum against the collector’s schema, and the collector rejects any batch containing a value it does not recognise rather than storing it.

    Nothing that identifies a download leaves the machine. No info hashes, torrent or file names, tracker URLs, peer addresses, download paths, or proxy addressesURLs — not in a usage event, not in a diagnostics report.

    That last one has an honest limitation, which the privacy documentation states outright rather than burying. Diagnostics redaction strips torrent names by matching them against your current library, so a log line naming a torrent you already removed has nothing to match against. Flume no longer writes torrent
    names to its log at all, but librqbit is not under the same discipline. Which is exactly why the report is displayed on screen before you copy it, not after: you are the only person who can recognise a name that got through.

    A privacy policy that describes what you wish were true is a marketing document. The interesting version documents the gap.

    The bug that only exists on one platform

    A representative afternoon: on Windows, adding a completed torrent that another application already had open would fail outright.

    The cause was three layers down. librqbit’s storage opens every file for read and write when overwrite is permitted — including a finished torrent that will only ever be served to peers. On Windows, a write open requires the current holder to have granted FILE_SHARE_WRITE. An application holding the file with only FILE_SHARE_READ therefore made the add fail, even though seeding never writes a byte. On Unix the same code is fine, because an open handle does not restrict other opens.

    The patch retries read-only on a sharing violation and logs the degraded mode. The part I am more pleased with is the test: it separates the fixable case from the genuinely unfixable one — a file open with no sharing granted at all
    cannot be read by anyone, and asserting that Flume fails there is more useful than pretending a fix exists.

    One more decision, made by arithmetic

    A last example of the genre, because it is the kind of call that usually gets made by reflex. macOS builds are signed with a Developer ID certificate and notarized through Apple’s notary service, so Gatekeeper stays quiet. Windows builds are deliberately not signed at all.

    That looks like an inconsistency and is actually the same reasoning applied twice. SmartScreen reputation accrues per certificate, earned through download volume — so an ordinary OV certificate would buy a free app at this distribution volume essentially nothing, while still costing money every year. An EV certificate clears the warning immediately and costs hundreds annually, and since 2023 its keys must live on hardware, so the tidy CI story that makes macOS signing work does not transfer. The honest position is to skip it and document the warning users will see, which is what the platform notes do.

    Shipping decisions deserve the same treatment as technical ones: work out what the thing actually buys before paying for it.

    Where it landed

    Flume 1.0 ships for macOS, Windows, and Debian- and RHEL-family Linux. It is Apache-2.0, which matches librqbit and carries an express patent grant that matters for peer-to-peer software.

    The engine really was the easy part. What took the work was deciding what the app is allowed to claim — about a swarm it can only partially see, and about a user it deliberately declines to learn anything about.