Any serious theory of communism needs to have not only an account of how and why communism has always failed, but also an account of how and why it could work differently now. Ideally, a parsimonious theory would solve both of these puzzles at once: the specification of some key macro-systemic variable(s) that explain why communism has always failed in all of its previous instantiations — a variable which, it just so happens, recently changed in a direction unlocking the communist possibility.
Ever since the rise of the world wide web, we've seen a thriving cottage industry of thinkers who claim that the world wide web might be one of these magic variables. In a way, I suspect that the works of this industry might end up being vindicated to some degree, but they're mostly premonitions rather than blueprints. There's been a sense that this revolutionary event in the history of information-communication technology appears to take place on a vector toward something beyond capitalism, but the construction of its engineering diagram is always postponed.
I believe there are two broad sets of reasons why previous communist patches have failed in the past. It seems to me that the solution of each one is now in reach, but only time, and experimentation, will tell. Whether there exist human beings still desirous, able, and willing to take it — that's another question. It might be the case that human being as such has been pacified to such a degree that even a workable and immediately available blueprint for the achievement of the communist dream will not be taken up.
This post will focus on only one of the broad sets of reasons why communism has failed. I have a draft on the second one, but this took all of my free time for this evening. I'll either post it later or include it in some future, longer volume of some kind. If you're an editor at a major press, I will consider forwarding you a private draft, but if too many of you write me at once I might not be able to respond to all of you.
This first basket of reasons we might simply call "the rational-choice critique." Rational-choice theory and game-theory are roughly synonyms. For a few different reasons, typically communism is just not a game-theoretic equilibrium. The big kahuna of this class of problems is the basic prisoner's dilemma (any situation where cooperation would be better than mutual cheating, but cheating when others cooperate would be the best). For the most productive producers, it will always be in their rational self-interest to quit the commune and choose to profit on the open market—even though a society of generalized competitive brutality is least desirable. In the rational-choice literature, liberal rule of law with a little bit of welfare redistribution is generally seen as the best you can get, because a well-fed and stable populace is in the interests of Capital. Try to take more from Capital, and they'll leave you. The common view tends to see the Capitalists as the evil defectors from the common good, but it's not really this simple. In a recent but not-popularly-discussed book by political economist Carles Boix, we learn about another major problem, which is that the masses cannot credibly commit to not take everything from Capital. That is, if the masses could somehow tie their hands, to ensure that they will not absolutely fleece the Capitalists and leave them dead in a ditch, they could potentially get way more redistribution than they do today. It's this rational fear that always makes Capital want to escape greater social accountability.
Another sub-problem roughly in the rational-choice perspective is self-selection: modern people who are willing to embark on some crazy life experiment to alter the basic parameters of their existence are going to be disproportionately desperate people. This camp certainly includes people who are quite skilled and capable — but, almost by definition, they are never in the top, top percentiles of money makers. There are some in the top percentiles who profess communistic sympathies no doubt, but they rarely if ever risk their wealth on joint schemes with the larger bulk of desperate folks (they fall in the camp, discussed above, of Capitalists afraid of grabbing hands that cannot credibly limit themselves). This has always been one of the more magical aspects of capitalism, that it tends to pay people just enough that it would be irrational for them to play any game other than capitalism; it pays them according to a kind of weighted function of precisely those traits that would allow them to contribute to alternative ways of life, for the simple reason that real human powers are real human powers, whether they're given to capitalism, communism, or an idiosyncratic new-age death cult.
Today "communism" is championed almost exclusively by the desperate, those who — quite transparently — are barely able to participate in frank speech, call their mothers on a semi-regular basis, or keep their rooms clean, let alone engineer an alternative way of life.
(By the way, I'm not money- or status-desperate, and I don't really care about making or keeping bullshit friends, which is why I can help solve communism — I'm doing pretty well under capitalism to be honest, so you can trust I'm not reaching for any words that can get me some crumbs of money or status. For me, it's just a fun challenge that seems like a beautiful and noble goal that I would like to be the first person in history to achieve, for glory. I'm not going for crumbs, I'm going for the huge status reward, the one that only comes from being super real, from going hard and fast after what's real for a really long time. This is a motive that certainly has its pitfalls, and you should watch me vigilantly and judge me accordingly for my vulnerability to those pitfalls, but for the purpose of thinking honestly and seriously about communism, my incentives and motives are well aligned for you to take me very seriously.)
Today, class consciousness refers to members of the desperate horde selling each other goods and services that make them feel like they can engineer something. Typically, the implicit rationale is that the quantity of their ranks will make up for their lackluster skills and productivity relative to the evil capitalist overlords. How many mentally ill people with humanities degrees does it take to equal the political-engineering power of Elon Musk? My guess is that the reigning left-wing thought leaders really do believe that there exists an answer to this question — and the entire extent of their implicit engineering diagram, the one on which they operate behaviorally, is to merely corral this number of followers. One piece of evidence for this inference is that, generally, the concrete engineering questions are usually deferred until some critical popular mass is reached. This often doubles as an anti-authoritarian reassurance as well: "We can't engineer it until we have the participation of enough people. If I could give you the engineering diagram right now, that would make me like Stalin!" OK, so can you give me the engineering diagram showing how a certain critical mass of followers should trigger some reduction of the search space—or whatever it is you think it will unlock? Crickets. Thus, I must believe that "build the ranks" is the engineering diagram.
I've digressed, but the basic point up there is that communist movements are generally composed of those who can't make communism work, and capitalism wins the strongest loyalty from those who are the very best at making things work. This also explains why the major modern efforts to create communism have been led by extreme authoritarian personalities. People like Stalin almost certainly would have had the intelligence to get on at least as well as the median person without totalitarian revolution, but their psychopathic tendencies would be at odds with their self-advancement through peaceful trade. Their psychopathic traits put them in the 99th percentile of "leading huge desperate hordes in murderous personality cults," but maybe only in the 65th percentile of being a normal civilized person. (If it's not obvious, I'm pulling these numbers out of thin air just to sketch a hypothesis.) I am guessing that the only strong, smart, and powerful people who opt into communism tend to be somewhat evil, "Dark Triad" types. After the disgrace of modern Communism, the desperate horde-base for communism became so small and desperate that it's not even worth it to potential psychopathic leaders. Psychopaths are now free enough to thrive under globalized cybercapitalism; for anyone at all above average in intelligence and conscientiousness, even in the 65th percentiles with some Dark Triad traits, the gains from trade are now way larger than the gains from leading a small and factious horde of deplorables. Sorry, I meant to say desperates.
Although this class of problems is correctly placed under the label of rational-choice or game-theoretic problems, it's crucial to understand that there are very strong emotional stakes involved. As we've seen, for highly productive people, and people with very profitable skills in fields such as engineering or computer science, capitalism will pay you very well and at least leave you alone to work your 60-hour weeks in "peace." Because the whole point of communism is to pay for the life of the desperate horde (and this is good, indeed it is the definition of Nobility), you're obviously not going to be paid as much, but in modern Communism — adding insult to injury — you're also going to be positively denied the social status proportionate to your contributions. Often, you will have to submit to the disingenuous, confused, often bitterly resentful, and sometimes downright psychotic attitudes and behaviors of many damaged people, the kind that self-select into communism. But as communists will tell you themselves: to be systematically denied the recognition you deserve is to suffer literal violence. Whatever you want to call it, the pain is real and deep, and nobody will submit to it for long, including capitalists.
I think this constitutes a decently charitable and effective summary of why and how communism has always failed in the past, at least for one class of reasons. It just so happens that I've already sketched a positive solution, meeting the criteria I set out at the top: that the solution must be based on something new, that was not available in all previous implementations of communism. I admit I do some hand-waving of my own, in that I do think the digital revolution is a relevant game-changer for tying much of this together, without fully specifying how, although I have specified somewhat. In short, I believe this set of problems could be solved by an engineering blueprint that mimics the medieval arrangement known as nobless oblige, although the seeming absurdity and generally unfashionable nature of the idea helps to explain why uptake has not been widespread and immediate. Communists don't want communism badly enough to submit to the reality-disciplining it would require.