security research, software archaeology, geek of all trades
564 stories

“Hello world” is slower in C++ than in C (Linux)


A simple C program might print ‘hello world’ on screen:

#include <stdio.h>
#include <stdlib.h>

int main() {
    printf("hello world\n");
    return EXIT_SUCCESS;

You can write the equivalent in C++:

#include <iostream>
#include <stdlib.h>

int main() {
    std::cout << "hello world" << std::endl;
    return EXIT_SUCCESS;

How fast do these programs run? We can check using a benchmarking tool such as hyperfine. Such tools handle various factors such as shell starting time and so forth.

I do not believe that printing ‘hello world’ itself should be slower or faster in C++, at least not significantly. What we are testing by running these programs is the overhead due to the choice of programming language.

Under Linux when using the standard C++ library (libstdc++), we can asked that the standard C++ be linked with the executable. The result is a much larger binary executable, but it may provide faster starting time.

Hyperfine tells me that the C executable runs considerably faster:

C 0.5 ms
C++ (dynamic) 1.4 ms
C++ (static) 0.7 ms

My source code and Makefile are available. I get these numbers on Ubuntu 22.04 LTS using an AWS node (Graviton 3).

There might be methodological issues having to do with hyperfine. Yet if these numbers are to be believed, there is a significant penalty due to C++ for tiny program executions, under Linux.

One millisecond of overhead, if it is indeed correct, is a huge penalty.

Read the whole story
6 days ago
I wonder how much improvement C++ sees if you start it off with std::cin.tie(0); std::ios::sync_with_stdio(0);

But also, benchmarking “hello world” is pointless… there *are* programs (usually in the form of shell scripts) where process startup and short output overhead dominates; benchmark one of those, lest you waste a lot of time optimizing the wrong thing.
Pittsburgh, PA
6 days ago
That's true, but if you have an exec()-heavy workflow, it all adds up. This is super common in compiled language build workflows. Say you need to invoke the compiler 10,000 times. Lemire's .9ms difference becomes 9 seconds. Hopefully, those are spread across multiple cores but it could still add up to 1s or more extra time per build. I checked and "ccache gcc" in the cache-hit case for a simple input file can be under 3ms, so looking at it this way (ccache is a C++ program) the extra overhead of c++ would perhaps become 30% of your total build process. (this on my machine where hello stdio takes 0.4ms so not far off from lemire)
Share this story
1 public comment
6 days ago
OK if you think 0.5ms is not much ... `musl-gcc -static` is 0.2ms. A version which uses `write(2)` instead of stdio is 0.1ms (again using musl). Eliminating all libraries and just putting two syscalls (sys_write and sys_exit) in _start is 0.1ms.

Edited to add: It's too fast for hyperfine's UI to show things properly, it doesn't go smaller than 0.1ms. The json output reveals a median time of just 58us for the very fastest version, enough to round up to 0.1ms. That's crazy fast for starting & exiting a whole UNIX process!
Earth, Sol system, Western spiral arm

A Walkable Internet

1 Share

Sometimes I think that popular media’s fascination with counterintuitive propositions is a big contributor to what got us into this mess. I use the word “media” there to mean more than just major publications, but we’ll get to that later. Also, sometimes, I like to think up counterintuitive propositions myself, like software doesn’t mean “code,” it means “a system for consolidating control of the means of production.” Or maybe the Internet can be defined as “that which will promise you what you want.”

Lucy Bellwood presenting a slide with the text Photo by Stefan Shepherd from Lucy’s extraordinary September 2016 talk, which I think about at least every other day.

I don’t offer these takes with any intention to defend them. I just think they’re useful mental calisthenics, valuable as alternative modes of thought to the definitions that creep into common idiomatic use: things like the Internet can be defined as “the most active population of large social media platforms.” I certainly use that shorthand myself, often in a scornful tone, despite my own attempts to stretch the popular conception of the Internet around the deconglomerated approaches that people these days call “IndieWeb.” One of the writers I admire, and linked to back in March when talking about this stuff, is Derek Guy of Die, Workwear and Put This On. Sometimes I sneak onto Twitter to see his dorky fashion memes, and today I discovered this, one of his more popular tweets of late. It has, as of this writing, numbers underneath it that far exceed its author’s follower count.

This is a gentle proposition, almost to the point of being anodyne. Maybe you disagree with it. I happen to agree, myself, as someone who has spent a number of years enjoying such a lifestyle; I agree in particular that it is luxurious, which is to say a luxury. One way I define luxury is an ephemeral privilege not to be taken for granted. Many people are systematically deprived of the privilege of walkability by the way that capital and its frequent servant, municipal policy, prioritize car travel and inherited wealth to create housing insecurity and food deserts. To me, that understanding seems built into the way these two sentences are constructed.

Three days after it was posted, I can sit here and watch the retweet and quote numbers on the post tick upward every time I breathe. I don’t think that’s due to positive attention.

I’m not here to write about how Twitter Is Bad. Even Twitter, as a body, agrees that Twitter Is Bad. I’ve written variations on that theme for ten years as of next month, and I can’t declare myself piously abstemious from social media when I’m quoting social media posts in my post about social media. The interests of capital demand that Twitter makes its graph lines go up; the simplest mechanism to make them go up is to incentivize conflict; the capital circulates through organizations until the system’s design iterates toward conflict optimization. Primed for bickering, just as the man says. The story of social media is the story of how billionaires made other people figure out how they could extract money from the late-twentieth-century invention of the real-time flame war.

I feel bad for Guy because I like his work and have a bit of a parasocial relationship with him: he is, more or less, the person who taught me how to enjoy shopping and wearing clothes. (I know many other people are subject to worse for less online, every day. I mean it when I say it’s Bad out there.) If not for Die, Workwear, I don’t think I would ever have chosen to take this series of self-portraits, a couple years back, wearing things I bought and liked just for myself.


I posted those photos on Flickr, even though I have my own IndieWeb site where I can host as many photos as I want. Flickr is a social media platform. It’s a rarity, not in that it did not generate for its acquiring capitalists the graph numbers they wanted, but in that it was then left to molder in neglect instead of being defenestrated for its failure. I have strong disagreements about some recent choices by its current owners, whatever their best intentions. But at least it’s not Instagram. Flickr has, for many years, retained an interface bent toward the humane and curious, instead of capitulating to the wind-tunnel abrasion of those who value human life less than the ascendance of the line on the graph.

Another thing I posted on Flickr, back in 2018, was the set of photos I took with Kat on our trip to Budapest together. One of the places we visited was Szimpla Kert, a romkocsma or “ruin bar,” built almost twenty years ago in what was once the city’s Jewish quarter by people in its neighborhood who wanted to make something new out of something old. It was once a condemned shell of a building; now it’s a huge draw, with thousands of visitors on a given night, most of whom are tourists like us. Locals might disagree, but I did not find that its charm was diminished by the crowd. It was idiosyncratic, vibrant, complex, and unique. Hungary—like my country, and like the Internet—is a more worrisome place to live than it was a few years ago. But Szimpla seems to be thriving, in large part because it is knit tightly into its local community.

Szimpla Kert

“Szimpla Kert” translates to “simple garden.” I have a little experience with the allure of gardening, and also how much work a garden takes to maintain; I’m sure the people running Szimpla work very hard. But an interesting way of looking at a garden, to me, is a space for growth that you can only attempt to control.

In the middle of drafting this increasingly long post, Kat asked me if I wanted to take a walk up to her garden bed, which is part of a community plot a ways to the north of us. I was glad to agree. I helped water the tomatoes and the kale, and ate a sugar snap pea Kat grew herself right off its vine, and on the way back I picked up dinner from our favorite tiny takeout noodle place. It took over an hour to make the full loop and return home, and I was grateful for every step. An unhurried walk was exactly what my summer evening needed. I luxuriated in its languidness, because I could.

When you put something in a wind tunnel, you’re not doing so because you value the languid. I am far from the first person to say that maybe we could use a little more friction in the paths we take to interact with each other online. Friction can be hindering or even damaging, and certainly annoying; I’m not talking about the way we’ve somehow reinvented popup ads as newsletter bugboxes and notification permission requests. I just want to point out that friction is also how our feet (or assistive devices) interact with the ground. We can’t move ourselves forward without it.

It’s a privilege to have the skills, money, time and wherewithal to garden. You need all those kinds of privilege to run your own website, too. I think social media platforms sold us on the idea that they were making that privilege more equitable—that reducing friction was the same thing as increasing access. I don’t buy that anymore. I don’t want the path between my house and the noodle restaurant to be a conveyor belt or a water slide; I just want an urban design that means it’s not too far from me, with level pavement and curb cuts and some streets closed to cars on the way. I want a neighborhood that values its residents and itself.

This is why I’m as just interested in edifices like Szimpla Kert and Flickr as I am in the tildeverse and social CMS plugins and building the IndieWeb anew. Portland is the most walkable city I’ve lived in, and it ended up that way kind of by accident: the founders optimized for extra corner lots out of capitalist greed, but the emergent effect was a porous grid that leaves more space for walkers and wheelchairs and buses and bikes. The street finds its own uses for things, and people find their own uses for the street. Sometimes people close a street to traffic, at least for a little while. And sometimes people grow things there.

I don’t expect the Internet we know will ever stop pumping out accelerants for flame wars directed at people who just felt like saying something nice about a walk to the grocery store. That paradigm is working for the owners of the means of production, for now, though it’s also unsustainable in a frightening way. (I will never again look at a seething crowd, online or off, without thinking twice about the word “viral.”) But if someone who lives in Chicago can’t entirely ignore what suburban white people get up to in the Loop on St. Patrick’s Day, then one doesn’t have to go out of one’s way to join in, either.

I’m ready to move on from the Information Superhighway. I don’t even like regular superhighways. The Internet where I want to spend my time and attention is one that considers the pedestrian and unscaled, with well-knit links between the old and the new, with exploration and reclamation of disused spaces, and with affordances built to serve our digital neighbors. I’m willing to walk to get there.

A front-end developer and former colleague I admire once said, in a meeting, “I believe my first responsibility is to the network.” It was a striking statement, and one I have thought about often in the years since. That mode of thought has some solid reasoning behind it, including a finite drag-reduction plan I can support: winnowing redundant HTTP requests increases accessibility for people with limited bandwidth. But it’s also a useful mental calisthenic when applied to one’s own community, physical or digital. Each of us is a knot tying others together. The maintenance of those bonds is a job we can use machines to help with, but it is not a job I think we should cede to any platform whose interests are not our own.

The Internet will promise you what you want, and the Internet will not give it to you. Here I am, on the Internet, promising you that people wielding picnics have put a stop to superhighways before.

IncompletePhoto by Diego Jimenez; all rights reserved.

Fifteen years ago this summer, I was exercising a tremendous privilege by living and working in London, in the spare room of an apartment that belonged to friends I met online. They were part of a group that met regularly to walk between subway stations, tracing the tunnel route overground, which they called Tube Walks. There was no purpose to the trips except to get some fresh air, see some things one might not otherwise have seen, and post the photos one took on Flickr.

My five months south of the Thames were my first real experience of a walkable life. I grew up in suburbs, struggled without a car in Louisville, and then, for the first time, discovered a place where I could amble fifteen minutes to the little library, or the great big park, or the neighborhood market, which would sell me just enough groceries for a single dinner. Battersea is not a bourgeois neighborhood, but it’s rich in growth and in history. It changed what I wanted from my life.

London, like Budapest, like Chicago, is a city that has burned down before. People built it back up again, and they didn’t always improve things when they did. But it’s still there, still made up of neighborhoods, still full of old things and new things you could spend a lifetime discovering. And small things, too, growing out of the cracks, just to see how far they can get.

Not sure where this little guy thinks he's going

Daniel Burnham, who bears responsibility for much of the shape of post-fire Chicago, was posthumously accorded the famous imperative to “make no little plans.” But I like little plans, defined as the plans I can see myself actually following.

I didn’t know where this post was going where I started it, and now it’s the longest thing I’ve ever published on this blog. If you read the whole thing, then please take a moment of your day and write me to tell me about a website that you make, or that you like, or that you want to exist. I’ll write back. More than ever, I want to reclaim my friendships from the machinery of media, and acknowledge directly the value that you give to my days.

Read the whole story
43 days ago
Pittsburgh, PA
Share this story

SPAs: theory versus practice

1 Comment and 2 Shares

I’ve been thinking a lot recently about Single-Page Apps (SPAs) and Multi-Page Apps (MPAs). I’ve been thinking about how MPAs have improved over the years, and where SPAs still have an edge. I’ve been thinking about how complexity creeps into software, and why a developer may choose a more complex but powerful technology at the expense of a simpler but less capable technology.

I think this core dilemma – complexity vs simplicity, capability vs maintainability – is at the heart of a lot of the debates about web app architecture. Unfortunately, these debates are so often tied up in other factors (a kind of web dev culture war, Twitter-stoked conflicts, maybe even a generational gap) that it can be hard to see clearly what the debate is even about.

At the risk of grossly oversimplifying things, I propose that the core of the debate can be summed up by these truisms:

  1. The best SPA is better than the best MPA.
  2. The average SPA is worse than the average MPA.

The first statement should be clear to most seasoned web developers. Show me an MPA, and I can show you how to make it better with JavaScript. Added too much JavaScript? I can show you some clever ways to minimize, defer, and multi-thread that JavaScript. Ran into some bugs, because now you’ve deviated from the browser’s built-in behavior? There are always ways to fix it! You’ve got JavaScript.

Whereas with an MPA, you are delegating some responsibility to the browser. Want to animate navigations between pages? You can’t (yet). Want to avoid the flash of white? You can’t, until Chrome fixes it (and it’s not perfect yet). Want to avoid re-rendering the whole page, when there’s only a small subset that actually needs to change? You can’t; it’s a “full page refresh.”

My second truism may be more controversial than the first. But I think time and experience have shown that, whatever the promises of SPAs, the reality has been less convincing. It’s not hard to find examples of poorly-built SPAs that score badly on a variety of metrics (performance, accessibility, reliability), and which could have been built better and more cheaply as a bog-standard MPA.

Example: subsequent navigations

To illustrate, let’s consider one of the main value propositions of an SPA: making subsequent navigations faster.

Rich Harris recently offered an example of using the SvelteKit website (SPA) compared to the Astro website (MPA), showing that page navigations on the Svelte site were faster.

Now, to be clear, this is a bit of an unfair comparison: the Svelte site is preloading content when you hover over links, so there’s no network call by the time you click. (Nice optimization!) Whereas the Astro site is not using a Service Worker or other offlining – if you throttle to 3G, it’s even slower relative to the Svelte site.

But I totally believe Rich is right! Even with a Service Worker, Astro would have a hard time beating SvelteKit. The amount of DOM being updated here is small and static, and doing the minimal updates in JavaScript should be faster than asking the browser to re-render the full HTML. It’s hard to beat element.innerHTML = '...'.

However, in many ways this site represents the ideal conditions for an SPA navigation: it’s small, it’s lightweight, it’s built by the kind of experts who build their own JavaScript framework, and those experts are also keen to get performance right – since this website is, in part, a showcase for the framework they’re offering. What about real-world websites that aren’t built by JavaScript framework authors?

Anthony Ricaud recently gave a talk (in French – apologies to non-Francophones) where he analyzed the performance of real-world SPAs. In the talk, he asks: What if these sites used standard MPA navigations?

To answer this, he built a proxy that strips the site of its first-party JavaScript (leaving the kinds of ads and trackers that, sadly, many teams are not allowed to forgo), as well as another version of the proxy that doesn’t strip any JavaScript. Then, he scripted WebPageTest to click an internal link, measuring the load times for both versions (on throttled 4G).

So which was faster? Well, out of the three sites he tested, on both mobile (Moto G4) and desktop, the MPA was either just as fast or faster, every time. In some cases, the WebPageTest filmstrips even showed that the MPA version was faster by several seconds. (Note again: these are subsequent navigations.)

On top of that, the MPA sites gave immediate feedback to the user when clicking – showing a loading indicator in the browser chrome. Whereas some of the SPAs didn’t even manage to show a “skeleton” screen before the MPA had already finished loading.

Screenshot from conference talk showing a speaker on the left and a WebPageTest filmstrip on the right. The filmstrip compares two sites: the first takes 5.5 seconds and the second takes 2.5 seconds

Screenshot from Anthony Ricaud’s talk. The SPA version is on top (5.5s), and the MPA version is on bottom (2.5s).

Now, I don’t think this experiment is perfect. As Anthony admits, removing inline <script>s removes some third-party JavaScript as well (the kind that injects itself into the DOM). Also, removing first-party JavaScript removes some non-SPA-related JavaScript that you’d need to make the site interactive, and removing any render-blocking inline <script>s would inherently improve the visual completeness time.

Even with a perfect experiment, there are a lot of variables that could change the outcome for other sites:

  • How fast is the SSR?
  • Is the HTML streamed?
  • How much of the DOM needs to be updated?
  • Is a network request required at all?
  • What JavaScript framework is being used?
  • How fast is the client CPU?
  • Etc.

Still, it’s pretty gobsmacking that JavaScript was slowing these sites down, even in the one case (subsequent navigations) where JavaScript should be making things faster.

Exhausted developers and clever developers

Now, let’s return to my truisms from the start of the post:

  1. The best SPA is better than the best MPA.
  2. The average SPA is worse than the average MPA.

The cause of so much debate, I think, is that two groups of developers may look at this situation, agree on the facts on the ground, but come to two different conclusions:

“The average SPA sucks? Well okay, I should stop building SPAs then. Problem solved.” – Exhausted developer


“The average SPA sucks? That’s just because people haven’t tried hard enough! I can think of 10 ways to fix it.” – Clever developer

Let’s call these two archetypes the exhausted developer and the clever developer.

The exhausted developer has had enough with managing the complexity of “modern” web sites and web applications. Too many build tools, too many code paths, too much to think about and maintain. They have JavaScript fatigue. Throw it all away and simplify!

The clever developer is similarly frustrated by the state of modern web development. But they also deeply understand how the web works. So when a tool breaks or a framework does something in a sub-optimal way, it irks them, because they can think of a better way. Why can’t a framework or a tool fix this problem? So they set out to find a new tool, or to build it themselves.

The thing is, I think both of these perspectives are right. Clever developers can always improve upon the status quo. Exhausted developers can always save time and effort by simplifying. And one group can even help the other: for instance, maybe Parcel is approachable for those exhausted by Webpack, but a clever developer had to go and build Parcel first.


The disparity between the best and the average SPA has been around since the birth of SPAs. In the mid-2000s, people wanted to build SPAs because they saw how amazing GMail was. What they didn’t consider is that Google had a crack team of experts monitoring every possible problem with SPAs, right down to esoteric topics like memory leaks. (Do you have a team like that?)

Ever since then, JavaScript framework and tooling authors have been trying to democratize SPA tooling, bringing us the kinds of optimizations previously only available to the Googles and the Facebooks of the world. Their intentions have been admirable (I would put my own fuite on that pile), but I think it’s fair to say the results have been mixed.

An expert developer can stand up on a conference stage and show off the amazing scores for their site (perfect performance! perfect accessibility! perfect SEO!), and then an excited conference-goer returns to their team, convinces them to use the same tooling, and two years later they’ve built a monstrosity. When this happens enough times, the same conference-goer may start to distrust the next dazzling demo they see.

And yet… the web dev community marches forward. Today I can grab any number of “starter” app toolkits and build something that comes out-of-the-box with code-splitting, Service Workers, tree-shaking, a thousand different little micro-optimizations that I don’t even have to know the names of, because someone else has already thought of it and gift-wrapped it for me. That is a miracle, and we should be grateful for it.

Given enough innovation in this space, it is possible that, someday, the average SPA could be pretty great. If it came batteries-included with proper scroll, focus, and screen reader announcements, tooling to identify performance problems (including memory leaks), progressive DOM rendering (e.g. Jake Archibald’s hack), and a bunch of other optimizations, it’s possible that developers would fall into the “pit of success” and consistently make SPAs that outclass the equivalent MPA. I remain skeptical that we’ll get there, and even the best SPA would still have problems (complexity, performance on slow clients, etc.), but I can’t fault people for trying.

At the same time, browsers never stop taking the lessons from userland and upstreaming them into the browser itself, giving us more lines of code we can potentially delete. This is why it’s important to periodically re-evaluate the assumptions baked into our tooling.

Today, I think the core dilemma between SPAs and MPAs remains unresolved, and will maybe never be resolved. Both SPAs and MPAs have their strengths and weaknesses, and the right tool for the job will vary with the size and skills of the team and the product they’re trying to build. It will also vary over time, as browsers evolve. The important thing, I think, is to remain open-minded, skeptical, and analytical, and to accept that everything in software development has tradeoffs, and none of those tradeoffs are set in stone.


Download video:

Download video:

Download video:
Read the whole story
48 days ago
Semi-counterpoint: I’m going to argue that “delegating function to the browser” is exactly the *strength* of MPAs (or, as we used to call them, “web sites”).
Pittsburgh, PA
Share this story

On the Dangers of Cryptocurrencies and the Uselessness of Blockchain

6 Comments and 12 Shares

Earlier this month, I and others wrote a letter to Congress, basically saying that cryptocurrencies are an complete and total disaster, and urging them to regulate the space. Nothing in that letter is out of the ordinary, and is in line with what I wrote about blockchain in 2019. In response, Matthew Green has written—not really a rebuttal—but a “a general response to some of the more common spurious objections…people make to public blockchain systems.” In it, he makes several broad points:

  1. Yes, current proof-of-work blockchains like bitcoin are terrible for the environment. But there are other modes like proof-of-stake that are not.
  2. Yes, a blockchain is an immutable ledger making it impossible to undo specific transactions. But that doesn’t mean there can’t be some governance system on top of the blockchain that enables reversals.
  3. Yes, bitcoin doesn’t scale and the fees are too high. But that’s nothing inherent in blockchain technology—that’s just a bunch of bad design choices bitcoin made.
  4. Blockchain systems can have a little or a lot of privacy, depending on how they are designed and implemented.

There’s nothing on that list that I disagree with. (We can argue about whether proof-of-stake is actually an improvement. I am skeptical of systems that enshrine a “they who have the gold make the rules” system of governance. And to the extent any of those scaling solutions work, they undo the decentralization blockchain claims to have.) But I also think that these defenses largely miss the point. To me, the problem isn’t that blockchain systems can be made slightly less awful than they are today. The problem is that they don’t do anything their proponents claim they do. In some very important ways, they’re not secure. They doesn’t replace trust with code; in fact, in many ways they are far less trustworthy than non-blockchain systems. They’re not decentralized, and their inevitable centralization is harmful because it’s largely emergent and ill-defined. They still have trusted intermediaries, often with more power and less oversight than non-blockchain systems. They still require governance. They still require regulation. (These things are what I wrote about here.) The problem with blockchain is that it’s not an improvement to any system—and often makes things worse.

In our letter, we write: “By its very design, blockchain technology is poorly suited for just about every purpose currently touted as a present or potential source of public benefit. From its inception, this technology has been a solution in search of a problem and has now latched onto concepts such as financial inclusion and data transparency to justify its existence, despite far better solutions to these issues already in use. Despite more than thirteen years of development, it has severe limitations and design flaws that preclude almost all applications that deal with public customer data and regulated financial transactions and are not an improvement on existing non-blockchain solutions.”

Green responds: “‘Public blockchain’ technology enables many stupid things: today’s cryptocurrency schemes can be venal, corrupt, overpromised. But the core technology is absolutely not useless. In fact, I think there are some pretty exciting things happening in the field, even if most of them are further away from reality than their boosters would admit.” I have yet to see one. More ore specifically, I can’t find a blockchain application whose value has anything to do with the blockchain part, that wouldn’t be made safer, more secure, more reliable, and just plain better by removing the blockchain part. I postulate that no one has ever said “Here is a problem that I have. Oh look, blockchain is a good solution.” In every case, the order has been: “I have a blockchain. Oh look, there is a problem I can apply it to.” And in no cases does it actually help.

Someone, please show me an application where blockchain is essential. That is, a problem that could not have been solved without blockchain that can now be solved with it. (And “ransomware couldn’t exist because criminals are blocked from using the conventional financial networks, and cash payments aren’t feasible” does not count.)

For example, Green complains that “credit card merchant fees are similar, or have actually risen in the United States since the 1990s.” This is true, but has little to do with technological inefficiencies or existing trust relationships in the industry. It’s because pretty much everyone who can and is paying attention gets 1% back on their purchases: in cash, frequent flier miles, or other affinity points. Green is right about how unfair this is. It’s a regressive subsidy, “since these fees are baked into the cost of most retail goods and thus fall heavily on the working poor (who pay them even if they use cash).” But that has nothing to do with the lack of blockchain, and solving it isn’t helped by adding a blockchain. It’s a regulatory problem; with a few exceptions, credit card companies have successfully pressured merchants into charging the same prices, whether someone pays in cash or with a credit card. Peer-to-peer payment systems like PayPal, Venmo, MPesa, and AliPay all get around those high transaction fees, and none of them use blockchain.

This is my basic argument: blockchain does nothing to solve any existing problem with financial (or other) systems. Those problems are inherently economic and political, and have nothing to do with technology. And, more importantly, technology can’t solve economic and political problems. Which is good, because adding blockchain causes a whole slew of new problems and makes all of these systems much, much worse.

Green writes: “I have no problem with the idea of legislators (intelligently) passing laws to regulate cryptocurrency. Indeed, given the level of insanity and the number of outright scams that are happening in this area, it’s pretty obvious that our current regulatory framework is not up to the task.” But when you remove the insanity and the scams, what’s left?

EDITED TO ADD: Nicholas Weaver is also adamant about this. David Rosenthal is good, too.

Read the whole story
52 days ago
52 days ago
Pittsburgh, PA
Share this story
6 public comments
52 days ago
Crypto is one of the rare cases where if we burn it to the ground it will help our species survive.
Jersey City, NJ
52 days ago
"This is my basic argument: blockchain does nothing to solve any existing problem with financial (or other) systems. Those problems are inherently economic and political, and have nothing to do with technology. And, more importantly, technology can’t solve economic and political problems. Which is good, because adding blockchain causes a whole slew of new problems and makes all of these systems much, much worse."
52 days ago
52 days ago
If we can just move all of the fraud into the blockchain, maybe then it can have purpose - keeping the scammers busy in crypto and leaving us outside of it alone.
52 days ago
Green's “rebuttal” was disappointingly weak — to be honest, I read it expecting the end to be that he'd picked up some lucrative consulting work from a cryptocurrency company.
Washington, DC
52 days ago
Well said!

I'm All-In on Server-Side SQLite

1 Comment and 2 Shares

I'm Ben Johnson. I wrote BoltDB, an embedded database that is the backend for systems like etcd. Now I work at, on Litestream. Litestream is an open-source project that makes SQLite tenable for full-stack applications through the power of ✨replication✨. If you can set up a SQLite database, you can get Litestream working in less than 10 minutes.

The conventional wisdom of full-stack applications is the n-tier architecture, which is now so common that it's easy to forget it even has a name. It's what you're doing when you run an "application server" like Rails, Django, or Remix alongside a "database server" like Postgres. According to the conventional wisdom, SQLite has a place in this architecture: as a place to run unit tests.

The conventional wisdom could use some updating. I think that for many applications – production applications, with large numbers of users and high availability requirements – SQLite has a better place, in the center of the stack, as the core of your data and persistence layer.

It's a big claim. It may not hold for your application. But you should consider it, and I'm here to tell you why.

A Brief History of Application Databases

50 years is not a long time. In that time, we've seen a staggering amount of change in how our software manages data.

In the beginning of our story, back in the '70s, there were Codd's rules, defining what we now call "relational databases", also known today as "databases". You know them, even if you don't: all data lives in tables; tables have columns, and rows are addressable with keys; C.R.U.D.; schemas; a textual language to convey these concepts. The language, of course, is SQL, which prompted a Cambrian explosion of SQL databases, from Oracle to DB2 to Postgres to MySQL, throughout the '80s and '90s.

It hasn't all been good. The 2000s got us XML databases. But our industry atoned by building some great columnar databases during the same time. By the 2010s, we saw dozens of large-scale, open-source distributed database projects come to market. Now anyone can spin up a cluster and query terabytes of data.

As databases evolved, so too did the strategies we use to plug them in to our applications. Almost since Codd, we've divided those apps into tiers. First came the database tier. Later, with memcached and Redis, we got the caching tier. We've got background job tiers and we've got routing tiers and distribution tiers. The tutorials pretend that there are 3 tiers, but we all know it's called "n-tier" because nobody can predict how many tiers we're going to end up with.

You know where we're going with this. Our scientists were so preoccupied with whether or not they could, and so on.

See, over these same five decades, we've also seen CPUs, memory, & disks become hundreds of times faster and cheaper. A term that practically defines database innovation in the 2010s is "big data". But hardware improvements have made that concept slippery in the 2020s. Managing a 1 GB database in 1996? A big deal. In 2022? Run it on your laptop, or a t3.micro.

When we think about new database architectures, we're hypnotized by scaling limits. If it can't handle petabytes, or at least terabytes, it's not in the conversation. But most applications will never see a terabyte of data, even if they're successful. We're using jackhammers to drive finish nails.

The Sweet Release of SQLite

There's a database that bucks a lot of these trends. It's one of the most popular SQL databases in the world, so standardized it's an official archival format of the Library of Congress, it's renowned for its reliability and its unfathomably encompassing test suite, and its performance is so good that citing its metrics on a message board invariably starts an argument about whether it should be disqualified. I probably don't have to name it for you, but, for the one person in the back with their hand raised, I'm talking about SQLite.

SQLite is an embedded database. It doesn't live in a conventional architectural tier; it's just a library, linked into your application server's process. It's the standard bearer of the "single process application": the server that runs on its own, without relying on nine other sidecar servers to function.

I got interested in these kinds of applications because I build databases. I wrote BoltDB, which is a popular embedded K/V store in the Go ecosystem. BoltDB is reliable and, as you'd expect from an in-process database, it performs like a nitro-burning funny car. But BoltDB has limitations: its schema is defined in Go code, and so it's hard to migrate databases. You have to build your own tooling for it; there isn't even a REPL.

If you're careful, using this kind of database can get you a lot of performance. But for general-purpose use, you don't want to run your database off the open headers like a funny car. I thought about the kind of work I'd have to do to make BoltDB viable for more applications, and the conclusion I quickly reached was: that's what SQLite is for.

SQLite, as you are no doubt already typing into the message board comment, is not without its own limitations. The biggest of them is that a single-process application has a single point of failure: if you lose the server, you've lost the database. That's not a flaw in SQLite; it's just inherent to the design.

Enter Litestream

There are two big reasons everyone doesn't default to SQLite. The first is resilience to storage failures, and the second is concurrency at scale. Litestream has something to say about both concerns.

How Litestream works is that it takes control of SQLite's WAL-mode journaling. In WAL mode, write operations append to a log file stored alongside SQLite's main database file. Readers check both the WAL file and the main database to satisfy queries. Normally, SQLite automatically checkpoints pages from the WAL back to the main database. Litestream steps in the middle of this: we open an indefinite read transaction that prevents automatic checkpoints. We then capture WAL updates ourselves, replicate them, and trigger the checkpointing ourselves.

The most important thing you should understand about Litestream is that it's just SQLite. Your application uses standard SQLite, with whatever your standard SQLite libraries are. We're not parsing your queries or proxying your transactions, or even adding a new library dependency. We're just taking advantage of the journaling and concurrency features SQLite already has, in a tool that runs alongside your application. For the most part, your code can be oblivious to Litestream's existence.

Or, think of it this way: you can build a Remix application backed by Litestream-replicated SQLite, and, while it's running, crack open the database using the standard sqlite3 REPL and make some changes. It'll just work.

You can read more about how this works here.

It sounds complicated, but it's incredibly simple in practice, and if you play with it you'll see that it "just works". You run the Litestream binary on the server your database lives on in "replicate" mode:

litestream replicate fruits.db s3://my-bukkit:9000/fruits.db

And then you can "restore" it to another location:

litestream restore -o fruits-replica.db s3://my-bukkit:9000/fruits.db

Now commit a change to your database; if you restore again then you'll see the change on your new copy.

The ordinary way people use Litestream today is to replicate their SQLite database to S3 (it's remarkably cheap for most SQLite databases to live-replicate to S3). That, by itself, is a huge operational win: your database is as resilient as you ask it to be, and easily moved, migrated, or mucked with.

But you can do more than that with Litestream. The upcoming release of Litestream will let you live-replicate SQLite directly between databases, which means you can set up a write-leader database with distributed read replicas. Read replicas can catch writes and redirect them to the leader; most applications are read-heavy, and this setup gives those applications a globally scalable database.

Litestream SQLite, Postgres, CockroachDB, or any other database

They all work on; we do built-in persistent storage and private networking for painless clustering, so it's easy to try new stuff out.

Try Fly  

You Should Take This Option More Seriously

One of my first jobs in tech in the early 2000s was as an Oracle Database Administrator (DBA) for an Oracle9i database. I remember spending hours poring over books and documentation to learn the ins and outs of the Oracle database. And there were a lot. The administration guide was almost a thousand pages—and that was just one of over a hundred documentation guides.

Learning what knobs to turn to optimize queries or to improve writes could make a big difference back then. We had disk drives that could only read tens of megabytes per second so utilizing a better index could change a 5-minute query into a 30 second query.

But database optimization has become less important for typical applications. If you have a 1 GB database, an NVMe disk can slurp the whole thing into memory in under a second. As much as I love tuning SQL queries, it's becoming a dying art for most application developers. Even poorly tuned queries can execute in under a second for ordinary databases.

Modern Postgres is a miracle. I've learned a ton by reading its code over the years. It includes a slew of features like a genetic query optimizer, row-level security policies, and a half dozen different types of indexes. If you need those features, you need them. But most of you probably don't.

And if you don't need the Postgres features, they're a liability. For example, even if you don't use multiple user accounts, you'll still need to configure and debug host-based authentication. You have to firewall off your Postgres server. And more features mean more documentation, which makes it difficult to understand the software you're running. The documentation for Postgres 14 is nearly 3,000 pages.

SQLite has a subset of the Postgres feature set. But that subset is 99.9% of what I typically need. Great SQL support, windowing, CTEs, full-text search, JSON. And when it lacks a feature, the data is already next to my application. So there's little overhead to pull it in and process it in my code.

Meanwhile, the complicated problems I really need to solve aren't really addressed by core database functions. Instead, I want to optimize for just two things: latency & developer experience.

So one reason to take SQLite seriously is that it's operationally much simpler. You spend your time writing application code, not designing intricate database tiers. But then there's the other problem.

The Light Is Too Damn Slow

We're beginning to hit theoretical limits. In a vacuum, light travels about 186 miles in 1 millisecond. That's the distance from Philadelphia to New York City and back. Add in layers of network switches, firewalls, and application protocols and the latency increases further.

The per-query latency overhead for a Postgres query within a single AWS region can be up to a millisecond. That's not Postgres being slow—it's you hitting the limits of how fast data can travel. Now, handle an HTTP request in a modern application. A dozen database queries and you've burned over 10ms before business logic or rendering.

There's a magic number for application latency: responses in 100ms or less feel instantaneous. Snappy applications make happy users. 100ms seems like a lot, but it's easy to carelessly chew it up. The 100ms threshold is so important that people pre-render their pages and post them on CDNs just to reduce latency.

We'd rather just move our data close to our application. How much closer? Really close.

SQLite isn't just on the same machine as your application, but actually built into your application process. When you put your data right next to your application, you can see per-query latency drop to 10-20 microseconds. That's micro, with a μ. A 50-100x improvement over an intra-region Postgres query.

But wait, there's more. We've effectively eliminated per-query latency. Our application is fast, but it's also simpler. We can break up larger queries into many smaller, more manageable queries, and spend the time we've been using to hunt down corner-casey N+1 patterns building new features.

Minimizing latency isn't just for production either. Running integration tests with a traditional client/server database easily grows to take minutes locally and the pain continues once you push to CI. Reducing the feedback loop from code change to test completion doesn't just save time but also preserves our focus while developing. A one-line change to SQLite will let you run it in-memory so you can run integration tests in seconds or less.

Small, Fast, Reliable, Globally Distributed: Choose Any Four

Litestream is distributed and replicated and, most importantly, still easy to get your head around. Seriously, go try it. There's just not much to know.

My claim is this: by building reliable, easy-to-use replication for SQLite, we make it attractive for all kinds of full-stack applications to run entirely on SQLite. It was reasonable to overlook this option 170 years ago, when the Rails Blog Tutorial was first written. But SQLite today can keep up with the write load of most applications, and replicas can scale reads out to as many instances as you choose to load-balance across.

Litestream has limitations. I built it for single-node applications, so it won't work well on ephemeral, serverless platforms or when using rolling deployments. It needs to restore all changes sequentially which can make database restores take minutes to complete. We're rolling out live replication, but the separate-process model restricts us to course-grained control over replication guarantees.

We can do better. For the past year, what I've been doing is nailing down the core of Litestream and keeping a focus on correctness. I'm happy with where we've landed. It started as a simple, streaming back up tool but it's slowly evolving into a reliable, distributed database. Now it's time to make it faster and more seamless, which is my whole job at There are improvements coming to Litestream — improvements that aren't at all tied to! — that I'm psyched to share.

Litestream has a new home at, but it is and always will be an open-source project. My plan for the next several years is to keep making it more useful, no matter where your application runs, and see just how far we can take the SQLite model of how databases can work.

Read the whole story
97 days ago
On the one hand, I am fully in favor of application architectures that avoid needing a database server.

On the other hand, I will be happy if I never need to touch SQLite’s bastardized typeless SQL ever again.
Pittsburgh, PA
96 days ago
That does make me curious about the adoption rate for strict tables
96 days ago
That's a new feature since the last time I looked at SQLite. Reading the docs, though, it seems to me that drh doesn't actually understand why people want this feature, and that makes me not trust it.
Share this story

How I Got from Mastodon’t to Mastodon


I finally wrapped my head around Mastodon, a social media platform, this past week. On Monday, April 25, I was beyond annoyed by how confusing I found Mastodon to be — and a similar exasperation was expressed by numerous friends of mine. For a while, I embraced this camaraderie of disinclination. But the more I worked to understand Mastodon, the more my perception changed, and my attitude along with it.

Tuesday was still more of the same. By Wednesday afternoon, however, I was quite active on Mastodon, and I began to run into some of those same friends, as well as familiar avatars from other social media platforms. I also met, in internet terms, new folks — and new-ish folks (one introduced themselves as the person who wrote a bot I interact with on another social media platform). That bot-to-human incident is just one anecdote, but anecdotes can be orienting, even if only as stories. The story here was that I’d traversed from a highly public social network to a relatively more circumspect one, and upon arrival I met not a bot but the person behind the bot.

By Friday, April 28, I had emerged as something resembling a Mastodonian. I’d moved through the three common stages of digital adoption: from annoyed through engaged to engrossed. That evening, when a friend casually asked, via a group email thread, if Mastodon was worth paying attention to, I began to reply — and I only finished after unexpectedly writing a roughly 2,000-word explanation to help my friend, along with the other participants in the thread, understand how Mastodon functions. Or more to the point, how I understand Mastodon to function, and why I think Mastodon might matter.

Grains of Salt
To begin with, I can’t say with assuredness that I’ll be sticking around on Mastodon. My general rule of thumb with online tools is to simply sign up and see if it sticks. I’ve tried so many social media tools, and very few have stuck. I quickly ditched Mastodon twice in the past, but it certainly makes more sense to me now than it did then. And since I found Mastodon difficult to make sense of, I wanted to share here my sense of what Mastodon is, why it can be hard to initially comprehend, and how one might go about both comprehending and engaging with it.

Yes, I know the complaint: if a social media platform requires a 2,000-word explanation (more like 4,500 words, as of this essay, which expands upon my original email), it is doomed to fail. I’m not here to say Mastodon is the future. I’m just here to say Mastodon is very interesting — and that while a lot of the perceived bugs may be bugs, and a lot of the conundrums are just subpar design and inefficient communication, some of those seeming bugs are features (or the residue of features), and much of that subpar communication is because of just how different Mastodon is from the current dominant forms of social media. In other words: Don’t miss the paradigm forest due to the bug trees.

If Mastodon succeeds (define success as you wish), it won’t simply be because the service became popular. It won’t even be because a significant number of people got over the same conceptual hump I did in order to understand Mastodon. It will be because an even more significant number of people won’t ever recognize the conceptual hump, because what right now, at the start of May 2022, seems downright odd about Mastodon actually will have become the new normal. That potential outcome is quite interesting.

And if you want to experience Mastodon before reading my attempt at an explanation, check it out at

Reminiscing About the Early Pliocene Era of Computer Communication
Some personal context might help. And you can skip this section entirely. It’s just background on who wrote this thing you’re reading.

I’ve been on enough social media platforms that it feels as if their combined logos could fill a yearbook. My first experience online, broadly defined, was a nascent form of social media: a dial-up BBS, or bulletin board system. This would have been roughly around the time The Empire Strikes Back was released. Back then, I didn’t think much about the “self-enclosed-ness” of the BBS. The notion of dialing into a system and then communicating directly with people on the other end, and only those who had likewise dialed in, mapped easily to the idea of a phone call, even if we were communicating by typing rather than speaking.

The mental mapping from BBS to phone call was all the more easy to comprehend because an actual phone line was required to hook the computer — a RadioShack TRS-80, in my case — up to the world outside one’s home. (This wasn’t my home. This was a friend’s. An extra phone line cost real money, as did the phone call itself. Such expenses were beyond my childhood home’s norms for decision-making. My parents were not entirely clear on this BBS concept at first, but they did tell me about the emergence of phones in their own youth. The idea of a “party line” — or “party wire,” vis-à-vis the Normal Rockwell illustration of that name — helped all of us understand the BBS more than we might have otherwise.)

Then high school and college happened, and I didn’t log on again until the early 1990s (not counting the limited school network, which was just for programming, when I was an undergraduate flirting with being — and then being flummoxed by the demands of — a computer science major). If I had to put a date on it, I imagine I logged on for the first time in April or May of 1993 — so almost exactly 29 years ago. This would have been the direct result of the debut issue of Wired magazine. If archaic phone systems helped me understand social media, then it was paper that helped me go digital.

Two Steps to Understanding Mastodon
As I said at the opening, I had already tried Mastodon previously, since it launched in 2016. Back then, though, I wasn’t frustrated by it. I was simply unenthusiastic. Mastodon’s interface felt as if a long-running food co-op tried to recreate Twitter or Facebook: it all sorta worked, but was utilitarian at best, and mired in complex systems at worst. You could almost smell the carob brownies. The benefits of Mastodon were unclear to me. At that early phase of my adoption, Mastodon reminded me of so many wannabe SoundCloud replacements whose sole apparent purpose was to replace SoundCloud. “SoundCloud done right” is a self-denuding rallying cry. They brought nothing new to the party, and few if any of them gained steam.

I was also reminded of a certain geek ethos, the one in which a computer-minded individual expresses interest in, say, having a blog, but actually takes far more active interest in creating, from scratch, their own blogging software. They never end up blogging. Mastodon felt, initially, to me like it might have been made by people with more interest in making a micro-blog network platform than in actually micro-blogging themselves.

This past week, however, was quite different. This past week I wasn’t unenthusiastic; this time, actual frustration kicked in. And while frustration is, well, frustrating, it can also be an engine of intrigue. I had not been that confused online for some time. It was sort of intoxicating. I’d like to say I simply put concerted effort into “getting” Mastodon, but that wasn’t quite how it played out. At first, all I did was complain, and the variety of responses to my complaints informed my experience. I’m fortunate to have a lot of patient and informed online friends.

Also helping in the process of getting acclimated: user error on my part. I ended up somehow with two different Mastodon accounts. In part this was a hassle, because their URLs were just similar enough that I took one to be an abbreviation for the other. But having two Mastodon accounts, each with its own unique URL, helped me understand something that had not, to me, been obvious previously: there are numerous Mastodon URLs. There is no or for Mastodon. The concept of Mastodon doesn’t merely contain — as Walt Whitman taught us to verbalize — multitudes, but is founded on them.

The interface can be maddening as you come up to speed. If privacy is a concern, you might find yourself wondering why you can change a public account’s individual posts private or but not a private accounts individual posts public. You might change an account from private to public, and then wonder why your earlier posts remain private. When you try to figure out how your posts show up on some other instances, you may end up looking at a chart, one that a friend has rightly likened to something out of the brain-frying time-travel film Primer (note: I love the movie, and it fried my brain). All these things eventually make sense, but the difference from the widely experienced, carefully designed chutes and ladders of Twitter and Facebook is palpable. I’ll get more into this in the next section, but suffice to say: people would maybe less often confuse Mastodon’s posts with Twitter’s tweets if Mastodon didn’t refer to its posts as “toots.”

Indeed, Mastodon’s current communications really don’t help matters. As of this writing, when you sign up for a new account on the main Mastodon URL, you are immediately asked to choose one of myriad “servers,” which are broken into “categories.” What is not clear is that all those servers are in effect communities and that they are each separate “instances” of Mastodon. (This is stated on the page, but “stated” is different from “clear,” and clear is different from “apparent,” let alone “self-evident.”) Much of the rest of this article will involve unpacking that single word: “instance.” Once I got that word, that concept, everything about Mastodon that had previously been frustrating began, instead, to make sense. I then deleted my two conflicting Mastodon accounts and I started a new one.

As whenever you make it through a thick conceptual window, this experience of finally “getting” Mastodon was fulfilling. For the first two days, my attitude was: this is the stupidest interface I’ve ever used. And then it made sense. To explain how it came to make sense, I retraced my steps. What felt at the time like an extended process of trial and error could, in fact, be reduced considerably. Partially that is because numerous of my steps were missteps, such as those recounted up above. In the end, I think there are two steps to understanding why Mastodon is special.

Step 1 of 2: Mastodon Looks like Twitter but It’s More Like WordPress
It’s very important to not think of Mastodon as simply a replacement for Twitter. Why? Because Twitter is a single globe-spanning instance of software that every user is inside together. Mastodon, however, is software more along the lines of the way provides software. When you install WordPress’s open-source software at your own URL, it’s its own self-contained instance of WordPress. WordPress is software in a practical sense, whereas Twitter is software only in the sense that it’s a digital service. My own website,, is on WordPress (I am vaguely familiar with the geek ethos mentioned earlier: from 1996, when I founded, until 2007, when I commissioned someone to port the site to WordPress, I published the entire site with hand-coded static files, every single .html page, even the RSS feed). If someone posts a comment on, that’s happening in my specific instance of WordPress, not on “WordPress as a single globe-spanning platform.”

So, let’s break this down: Make sure you get the difference between WordPress and Twitter. Now, imagine Twitter not as a company with a single platform, but as an installable-on-the-internet piece of software like WordPress. That’s a step toward understanding Mastodon. Mastodon lets you set up your own self-contained instance of the software, just like WordPress does, and you can run it on your own (server use costs money, and the more users you have, the more it costs; it’s more expensive than WordPress). No one can join your Mastodon instance whom you don’t want as a member. You can set the rules as you like. You can make it open to anyone who wants to read it or wall it off entirely — and even if you make it open to anyone who wants to read, you can allow each of your instance’s individual users to choose to hide their own posts from anyone but the people they choose to see it. (If you’re handy with code, you can even fork Mastodon and make your own version — so long as you post the source code online, per the open-source licensing agreement.) Also, you don’t need to set up Mastodon yourself. You can just join a pre-existing server/community.

This took days to comprehend, and then even when I got it, it took a while to grok it. My head hurt. I got angry. Then suddenly it clicked. A big reason I got angry is there are a lot of know-it-all Mastodon-heads out there who condescendingly ask regularly, “Why aren’t you just on Mastodon?” when people complain about Twitter and Facebook. The answer to that question, as it turns out, isn’t just “Mastodon isn’t easy to understand.” It isn’t even “Mastodon isn’t as clean and efficient as those heavily funded websites that are literally designed to algorithmically reflect parts of our consciousness we’re not even aware of.” No, the more full answer is, “To really use Mastodon, you have to step through a conceptual window that’s akin, perhaps, to, long ago, someone who’s only ever used AOL then trying to use the Internet. Except even harder to comprehend, unless someone is patient and takes the time to explain it.” I’m trying to explain it, first to myself, and then to anyone who wants to read this.

Step 2 of 2: Mastodon Communities Can Easily (if Currently Clumsily) Connect with Each Other
This is where Mastodon gets interesting — like, really interesting. It’d be enough if Mastodon were just “WordPress for self-contained social media groups.” But before talking about Mastodon’s built-in interconnectedness, let’s return to the concept of blog comments above.

Do you remember a piece of once ubiquitous online software called Disqus? (I’m not sure how broadly utilized it is anymore.) Disqus provided connective commenting between separate blogs and websites. For example, if I went to some experimental-music blog, and someone said something interesting in the comments, I could click on their avatar, and I’d see other stuff they’d commented on all around the internet. So if they had commented on another blog, I could then click through and see what they had commented on. Maybe I’d discover another experimental-music blog, or maybe I’d find out they also like recipes for Estonian cuisine, or maybe I’d come upon the music made by the very person who possesses that avatar.

The phenomenon of Disqus was more than blogs cross-linking through so-called “blogrolls.” Disqus was also more than a portfolio of blogs owned by one company and using a shared platform. This was seemingly truly (but not actually, as I’ll explain in a moment) ad hoc — and it was exciting. Disqus just happened: you show up on one blog, and there’s your avatar — you show up on another, same. (Now, it wasn’t quite as easy as I describe, which is part of the reason it didn’t take off as much as it might have. Which is part of why what I’m getting around to describing about Mastodon is so interesting.)

I once saw one of Disqus’ two founders, Daniel Ha, give a talk, early on in the company’s existence, and he made a comment I think about a lot to this day. He said something along the lines of how comments people made online were just as valid a form of publishing, of self-expression, as was the writing of a post or article. That’s not quite how he put it, but I feel like much of the subsequent explosive growth of social media shows just how accurate his observation was. (If this seems self-evident to you, I will note this was not a widespread perception at the time.)

You may be thinking, “Well, that’s cool, but how is that blog commenting scenario different from Mastodon?” The thing with Disqus was it was centralized. You had all these different blogs, but the only way they connected was through Disqus. You had little to no control as a Disqus commenter. If someone started saying crappy stuff to you or just crappy or inconsequential stuff in general, you couldn’t unfollow them or hide them on blogs where you might stumble on them (at least when I used the service — it may have gained such functionality). There were issues for blog owners, too, but let’s just pause there and move on. The key thing was it was centralized: if Disqus went down, all of Disqus went down. If Disqus made a big change, it immediately impacted the entire network. Had Disqus ever gone under (which it hasn’t), it might well have disappeared.

A cool thing about Mastodon is the software is created so that anyone on any single Mastodon instance (like, say,, which appears to be the biggest one, or, where I eventually signed up, despite me not totally liking the somewhat creepy tone of the word “lurk”) can still communicate with people on other Mastodon instances. Even as I type this, I can’t quite understand how it works, but it does. (A friend explained to me helpfully that the underlying protocol, ActivityPub, which Mastodon and other online services, can be thought of as “kind of like two-way RSS,” which is to say the protocol most of us know as a way to track a bunch of blogs through one tool, such as Feedly, Inoreader, or the sadly defunct Google Reader. I don’t know much about ActivityPub, but I’ve been reading up. And I put this section in parentheses to emphasize that when you start seeing terms like “RSS” and “ActivityPub,” it’s a bit beyond the technical literacy — even the technical curiosity — I’ve assumed for a reader of his essay.) If I log onto in the morning, I might see replies from other Mastodon accounts at places like or or or or or or or, all real unique Mastodon instances, and I can communicate individuals who call such places home. I can even, in a subtly signaled way, see who in my feed is part of “my” home instance (i.e., and who isn’t: accounts that share my instance appear by their avatar names, whereas accounts from other instances appear with their avatar name appended by the name of their alternate instance (e.g., I appear as on the feed of someone at any Mastodon instance other than; for anyone on, I appear simply as @disquiet).

If these other accounts turn out to be bots or merely inconsequential to what I’m interested in focusing on, I can mute them. If I find that a particular instance of Mastodon (like ihate.ambient — not a real instance) is filled with bots or hateful humans, I can save myself the Whack-a-Mole effort and just mute the whole instance — and, this is another clincher, I can do so as a user. Read the previous clause again: as a user. I don’t need to depend on the Mastodon instance in which I am located to filter whom I communicate with.

Think about it this way: each Mastodon instance can become its own little community without necessarily being cut off from the broader world. (The term for this sort of arrangement is “federated.” The word, which predates Mastodon, is one that the service features repeatedly on its home page, even though the same page offers no definition for curious newcomers.) The managers of a given instance can certainly say, “You can only chat here, and the rest of the internet can’t see in unless they have an account.” However, the real power of Mastodon is how you can have your own little instance for a distributed community of individuals to discuss folk dancing, or living at sea, or modular synthesizers, or vintage sports equipment — likewise, you could have one for your family, or your college class, or your neighborhood volunteer clean-up group — and the participants can connect with each other as well as with users beyond your instance, as each user sees fit.

Witnessing these varied instances of Mastodon communicate with each other is kind of amazing. I do a lot of stuff online, and I love being online. I still think of IMAP, an internet standard protocol that powers a lot of email, as magical. Mastodon is cool on that order of magnitude. It’s science-fiction cool.

The Next Steps
That was really helpful for me to type out, because doing so helped me understand Mastodon more clearly through explaining it to myself. This documents my experience and perception. Like I said, I passed through a conceptual window this week, as far as Mastodon is concerned. And a funny thing happens after you pass through a conceptual window: you can’t always see clearly back through it. It took almost as much effort to retrace my steps as it did to take those steps in the first place, albeit minus any of the frustration. (Fortunately, I have my sequence of tweets from that week, and the trajectory is pretty clearly delineated if you read them in order.)

So, will Mastodon take off? It’s done well during the current Twitter-evacuation, or at least current “Twitter trial separation,” but Mastodon still needs to do a lot of hard work. It needs to work on that interface. It needs to infuse its “federated” underpinning with deeper meaning and purpose so that the term is unifying and clarifying rather than merely vaguely differentiating. And Mastodon needs to do a much better job of explaining to new users how it works. It needs to help newcomers start off. As mentioned earlier, when you show up you have to blindly choose a community — and it doesn’t explain clearly that it’s an arbitrary choice, to some degree, because you can communicate across instances. The whole concept of inherently interconnected instances is not self-evident, or easy to immediately comprehend. To understand the solution, users must first appreciate the problem. “Getting off Twitter and Facebook” is a problem for many, but it’s not really the problem that Mastodon is trying to solve. Per my comment about SoundCloud earlier, it doesn’t do justice to what Mastodon (along with other experiments in federated and decentralized social networks) is pushing toward.

The issues aren’t merely about language. If you’re on and I’m on post.lurk,org, and I “follow” you, this is how it plays out: first, I jump through a few somewhat opaque hoops to follow you, and then on it shows that I’m following you. However, anytime I happen to find myself back on your page, I’ll still see a big “follow” button, which naturally makes me wonder whether or not I’m following you. This is not a big problem at first, but I don’t know how sustainable it can be in the long run when I and a growing number of people are following a lot of accounts. This sort of disconnect may just become an accepted online norm, or it may provide just the sort of cognitive dissonance that keeps a service from reaching a broader audience.

And that about covers it. As is clear, after these nearly 4,500-ish words, those being a revision of a nearly 2,000-word email, the qualities of Mastodon hold a lot of promise and appeal to me. I spend a lot of time online, and I don’t do so alone. I joke regularly that Facebook is where I realize how little I have in common with my friends, while Twitter is where I realize how much I have in common with people I don’t know. I’m not sure where Mastodon fits in that formulation, and I’m slowly sorting out that a whole new formulation may be required.

A lot of my online imagination is tied up in the Disquiet Junto, an online community I’ve moderated since 2012, and it was preceded by a half decade spent organizing online collaborations between musicians. The Junto isn’t a “place,” not even a virtual one in the sense we think of virtual places currently. It exists on numerous platforms, key among them: SoundCloud, Slack, Twitter, and, the latter an instance of Discourse, another online discussion platform. (This platform diaspora, so to speak, largely occurred following the suddenness with which SoundCloud, many years ago, removed its “groups” functionality.) Using Mastodon has helped me understand how that current constellation of online Junto locales may not be truly “federated.” Part of me wonders if a Disquiet Junto instance of Mastodon might be worth pursuing, but right now the onboarding process (both practical and conceptual) is too arduous. I want the Junto to be welcoming, and Mastodon isn’t welcoming — at least not enough, and at least not yet.

Both through speculative interest and practical application, online networks are where I spend a lot of time. Six years into its existence, Mastodon registers as a potentially important step forward. Perhaps some service other than Mastodon will have eventual widespread, ubiquity-equivalent success with this “federated” model. Perhaps some even more autonomous identity — closer to an email address or phone number — will arise in the process. (This lengthy post is not in any way comprehensive, but if a lingering question is “Would it help to have more than one Mastodon account?” then the answer may relate to the question “Do you need more than one phone number?” Not everyone does, but there are work and life circumstances when it may be useful, and even necessary.) Perhaps the internet will achieve something even more “decentralized” than a “merely” “federated” model — which is to say, a situation in which no one need “join” a server, and can simply participate (one hedge would be a groundswell, I imagine, of usage such that everyone has their own individual Mastodon instance, but that feels more like a hack than an intentional system).

No matter what comes in this regard, it will have been Mastodon that helped rewire my brain for such things. Rewiring can be a painful procedure, but it was worth the effort.

In any case, if you do join Mastodon, you can find me, at least for the time being, at:

Acknowledgements: Special thanks to Todd Elliott, Kamen Nedev, Matt Nish-Lapidus, C. Reider, and Jason Wehmhoener, among others, who helped me get on Mastodon, helped me sort out Mastodon, and/or read this at some stage of draft form, and to Bart Beaty for having asked the initial question via email. Any broken metaphors or just plain incorrect information is my fault alone.

Read the whole story
104 days ago
Pittsburgh, PA
Share this story
Next Page of Stories