Something went very wrong yesterday. Well a few things really, which happened to involve some of the most important people alive. Yet the resulting journalism has mostly been curiously thin (and in two major cases fundamentally irreconcilable).
Speaking as a member of the reading public, there are three things we really needed to see clearly established:
How it happened
What Twitter needs to disclose
(Quick note before we go on: everything in this newsletter is subject to a corrections policy. Catch something? Say something! I’ll either send you cash or make a donation in your name. The internet needs to be less wrong, and that requires reader help.)
[EDIT: 08-06-21 - Coming back almost a year later to make a marked correction and two footnotes, and to link this larger update / retrospective.]
#1: What Happened
Let’s start with what everyone agrees on:
An unknown number of high-profile Twitter accounts were taken over yesterday, some 30 of which posted tweets pushing various crypto scams, beginning a little after noon and continuing to about 3pm (all times here PT).
Twitter acknowledged this at 2:45pm.
At 3:18pm Twitter added that some accounts were being preemptively barred from both tweeting and resetting passwords (the latter because the attacks were premised on swapping out the email addresses used for password recovery).
At 5:41pm they announced that most things were back to normal.
Some two hours later a few preliminary explanations followed.
As for what the perpetrators actually did, accounts vary:
Twitter said the attackers used their access to send rogue tweets, and added that “we’re looking into what other malicious activity they may have conducted”.
VICE claims to have spoken to two of the perps, whose narrative was that a Twitter employee sold them access to internal tools, which they used to both run the crypto scams and gain ownership of some desirable Twitter handles. (The obvious tension between those two schemes wasn’t commented on.)
TechCrunch ran a largely similar story (as far as what was done, though with a different theory as to how the attackers got in), sourced to some unknown figure “in the underground hacking scene” with an unknown relationship to the perp(s).
The NYT, seemingly uninterested in giving any credence to the inside man theory, voiced their relief that “[t]he hackers did not use their access to take aim at any important institutions or infrastructure — instead just asking for Bitcoin.”
Well, here’s the rub: what no one seems to have noticed is that Twitter quietly made a change yesterday. At some point between ~3pm and 4:54pm they killed access to the tool that lets users download an archive of their collected data.
If you try to access said tool now:
Now, there are two obvious possibilities here:
Twitter disabled the feature out of a general abundance of caution
Twitter disabled the feature because their audit logs suggested that the attackers had been downloading user archives as they gained control of new accounts
What's the harm in an archive, you ask? If it’s just a collection of tweets that are already public, who cares?
In seeking to establish whether said archives included private DMs, I stumbled upon the below thread. Note that the bottom comment was left this past Saturday:
(Note: I reached out to the commenter to see if they’d give me a redacted screencap of their ZIP’s file structure. I have yet to hear back, but will update here if/as I do. Though even if the archives don’t include DMs, there is code on GitHub that logged-in users can run to scrape said content. So Twitter has to rule out both possibilities.)
Now, to be clear, none of this is proof positive of either contention. It’s possible that Twitter was indeed just acting out of general caution, and it’s possible that the commenter (somehow?) misunderstood what his archive contained. But these are questions we need answers to. Because the difference between “some accounts were used to promote scams of relatively low value” and “an unknown number of high-profile accounts had their private messages scraped” is obviously enormous.
(While a lot of attention has been given to the big names involved, less has been given to the question of how many total accounts were accessed where tweets weren’t sent. The latter is especially important in context of nearly every US politician having an account, where the sum total of their DMs would represent powerful potential kompromat going into a contentious election season. Again, it really could be that the attackers were just bumblers too shortsighted to see beyond the small-dollar stuff. But the likelihood of that depends on which outlet you believe as to who was involved.)
#2: How It Happened
This is where the competing narratives become wholly irreconcilable.
Using two major outlets as champions of the two main interpretations (where others mostly tried to somehow straddle the gulf between them):
VICE believes that they’ve spoken to two hackers directly involved, who were unequivocal in their claim that this attack was premised on inside help.
The NYT believes that this was a classic outside-in hack (i.e., no inside help), likely initiated by a state actor.
[Aug ‘21 EDIT: This second point above was wrong. I corrected it prominently in a sequel that I published four days after this post, but somehow forgot to loop back here to link it and make an in-line edit per my normal practice. Mea culpa.]
A Twitter insider was responsible for a wave of high profile account takeovers on Wednesday, according to leaked screenshots obtained by Motherboard [VICE] and two sources who took over accounts.
Twitter’s investigation into the breach revealed that several employees who had access to internal systems had their accounts compromised in a “coordinated social engineering attack,” a spokesman said, referring to attacks that trick people into giving up their credentials.
These takes clearly can’t both be true. Either (i) this was a case of rogue low-level actors bribing an employee to help them with a scheme worth perhaps hundreds of thousands of dollars1, or (ii) this was a case of a serious intrusion by a party that managed to take control of multiple employee accounts without inside help.
(You could also argue a variant of the first case where a turncoat was bribed as part of some yet-to-be-realized scheme much larger than the crypto stuff. But this still doesn’t reconcile the two positions, as either a Twitter employee was or wasn’t involved2.)
As to the evidence that might help us adjudicate between these narratives:
Tyler Winklevoss, CEO of Gemini (one of the first accounts used to send a scam tweet), was clear that their account had 2FA enabled. This means that either Twitter’s internal panel allows admins to unilaterally bypass this check or the attackers had access to something like root-level privileges outside the panel.
The NYT says “Twitter quickly removed many of the messages, but in some cases similar tweets were sent again from the same accounts, suggesting that Twitter was powerless to regain control.” This would imply that either it wasn’t as simple as a corrupt employee swapping recovery email addresses or that Twitter was somehow very, very slow to identify and suspend said employee’s account (who was constantly resetting their resetting attempts and getting away with it?).
So far as establishing that two of their sources were actually involved, VICE seems to have leaned heavily on screencaps provided of the internal panel (at least one of which showed an account we know was involved). While these screencaps likely were taken by the attackers, this doesn’t necessarily mean that the people VICE spoke to were said individuals. (Not to say definitively that they weren’t. Only that it doesn’t automatically follow, as said images could have been circulated to other folks who decided to take credit for whatever reasons.)
VICE says “[Twitter] is still investigating whether the employee hijacked the account themselves”, whereas The NYT says “several employees who had access to internal systems had their accounts compromised” (emphasis mine in both). Twitter itself hasn’t been clear here, with their sole public comment being that the attackers targeted “some of our employees”, which you can parse either way based on how successful those attacks were (but which in either reading doesn’t seem to allow for any employees to have been cooperatively involved).
While not technically evidence, one would have to explain why a Twitter employee (generally quite a good-paying job) would risk the overwhelming penalties involved just to make a cut of a haul that might itself equal a year’s salary in the US. If Twitter has any internal logging, the chances of getting away with this are vanishingly small. (The best explanation I can conceive here is that said employee could be in a low-income country that isn’t likely to either prosecute or extradite — which is possible, I guess.)
If no employee was involved, the challenge of gaining that access was much higher. Would a party who went through all that work throw away their gained advantage on something so relatively trivial as $100-200k? It’s also not impossible. But it does again stretch the imagination a little.
I’ll leave it to the reader to judge. But regardless of which narrative seems more plausible to you, there are major questions we still need to be asking here.
#3: What Twitter Needs to Disclose
A non-exhaustive list:
What’s the point of 2FA if an employee can just override it? (Or, if you prefer, which employees should be granted those kinds of extraordinary powers in relation to accounts belonging to notably powerful people?)
Is Twitter able to track all archive-download requests? What about DM scraping attempts by logged-in users? Will users be allowed to see a history of either?
What do those archives actually contain? And what (if any) of it is encrypted with a key other than the user’s (possibly just reset) password?
How did it take Twitter so long to recognize and respond to what was going on? The attacks continued for some three hours in total.
Two Twitter employees were caught leaking user info to the Saudis back in 2015. And then, in 2017, a contractor was able to temporarily delete Donald Trump’s account. What controls were established in response? (Regardless of whether an inside employee was guilty of abetting here, surely those prior events would have called for sophisticated audit logs and internal flagging mechanisms?)
(While this is tangential to the main narrative here, the screencaps that VICE shared seemed to indicate that Twitter admins have a set of tools that can limit distribution / discovery for specific users. While there’s nothing inherently objectionable here, and while Twitter has kinda disclosed this in vague terms before, it is worth asking about process here. Some accounts are bad and deserve a measure of quarantine, sure. But who decides? And how? And how might a user either discover that such a penalty has been applied or appeal it?)
Getting some of these answers might require subpoena powers that newspapers don’t have. But there are some questions they can ask — both of Twitter and of themselves.
Kind thanks to Yair Livne and others for reviewing and improving my initial draft.
If you place a value on this sort of thing, please consider subscribing. It’s $5/mo for all posts, and for the next two months I’m donating all revenues (pro-rated for annual) to the Thurgood Marshall College Fund, which helps Black students get the equal education they deserve.
What wasn’t clear at the time was that major crypto exchanges blocked the vast majority of attempted transfers to the attackers’ Bitcoin addresses. Had that not happened, the haul would have been something like $1.5-$1.6m, though that figure includes payments for some stolen usernames as well, which I think were in the tens of thousands.
No Twitter employee was involved that we know of. But Kirk, the self-assigned name for the group that attacked Twitter, had presented themselves as a Twitter employee to members of the OGUsers community. So I think VICE was just under-skeptical there, taking the word of said members at face value.