Second, I think you're missing some important bits of the AI story.
1. Go look at Ethan Mollick's recent experiments (https://www.oneusefulthing.org/p/it-is-starting-to-get-strange). Lots of niche or local journalism could benefit from more quantitative analysis in a way that isn't financially feasible if you have to hire a data scientist. Lots of journalism could also benefit from better display of its quantitative results (Tufte, etc). Seems like AI is ready to do both, more or less today!
2. Progress-utility isn't only about the leading edge, it's also (at least as much) about catch-up growth: poor countries becoming middle-income, boring industries adopting computers, software verticals moving to the cloud, etcetc. The audience for work aiming at this is, nearly by definition, not at the upper end of the literacy curve. As you note, such lower-literacy audiences are not very good at pulling in context from other work, transferring knowledge between domains, etc. AI is really good at tasks like (literal) language translation, providing context, transferring commonplaces into the argot of a different industry or place, etc. These are also extremely progress-useful things to do!
1. Agree on data presentation. That's an easy win. But data analysis? AI isn't there yet imo. You end up needing someone who understands the field well enough to verify if the output is actually legit, which at least partially ruins the point of the exercise. And it's difficult to know just how long it will take to close that gap to a reasonable level of confidence? Like if your AI analyst hallucinates 10% of the time, that's obviously too much. But what about 1 or 2%? My guess is rewards here go to larger pubs, as they can hire analysts on salary to once-over Ai-generated output at scale.
2. Yeah that's certainly all true (and probably understated in net importance). Was just a bit orthogonal to my focus here. AI will certainly do a lot of good in the world so far as patching over the costs of low literacy (and just like making the world more accessible!). But not sure that it will do much about getting substantially more people to Levels 4/5 where they can meaningfully parse written text to say "hmm no this specific bit is wrong, I can safely discard it". Like whenever I use GPT I'm struck by how close it is to being really insightful. But there's always one detail or another (often a central one) that's completely coked up, and you have to know a fair amount about the thing you're querying to spot it (which presumably most can't, or they wouldn't be asking the question in the first place). Getting people to the state where they can spot those mistakes consistently is super hard, and my best guess would be that over-reliance on GPT etc will actually decrease the number of people we get to this stage. Because, as with bad journalism, it's super easy to discount the net impact of all these mistakes going uncaught, and very easy to over-focus on "hey, this is really fun and I'm learning a lot" (even where some % of what you're learning is actually bullshit, and even if none of it is truly learning at all).
1. That's not clear to me. Yes, of course hallucination is a problem, but as we know it's also a problem when reading real academic studies--a shocking percentage of the journal output in some fields is so bad at statistics and so good at p-hacking that even the sign might be wrong. The tide is turning in high-prestige-psych but we're nowhere close in education research, traffic engineering, or many other disciplines important to journalism. Furthermore, in the fragmentation era, there are a lot of pretty clued-up but low-resourced outlets like your substack, where there is a steady and reasonable hand to make sure the AI isn't being crazy. Mollick has really been showing the way on this, eg by asking the AI to produce summary stats or cross-tabs that easily verify its more exciting claims.
2. A lot of this seems to depend on the prompt engineering. If you just say "explain X" well, odds not so great. But if you say "Take X's theory of Y and translate it into Z domain" or better yet "take this PDF and provide a useful abstract for someone from Romania" you get massively better results. I thought your point here was about the impact of literacy on progress-utility, not just literacy as an end-good, in which case this seems super-relevant.
3. It therefore matters a lot whether progress-utility is a strongest-link or weakest-link problem (or, realistically, what % of the two). To the extent that it's a strongest-link problem, AI seems likely to improve productivity of the sharpest people, who can use it for what it's good at, ignore what it's not, and have reasonable mechanisms to filter what they read. To the extent that it's a weakest-link problem in general, then it should be dominated by the catch-up growth rather than leading-edge effects, since weak links were never at the leading edge. So then the question becomes how flooding the zone with (even more) crap weighs against the positives in these two areas. That's an empirical question that will be hard to answer.
First, I agree very strongly with the first part, but it's interesting to think of historical examples where this has gone better. It seems like journalism really did improve between Hearst's day and the 90's NYT, whatever its flaws. How, and why, if not the bundling theory favored by Yglesias et al?
Another case study would be the transition between Abbe Migne's publishing machine (https://en.wikipedia.org/wiki/Jacques_Paul_Migne) which supplied new academic libraries all over the world with subscriptions to horrifically edited and sometimes outright forged texts and the modern university presses, which again whatever their flaws seem to do much better. Migne was already a bundler, so that can't be the issue there.
In these two cases is it that literacy just genuinely went up, perhaps because of public education in the former case and accreditation of tertiary education in the latter? If so that seems really important to achieve!
Hey Ryan! Always appreciate the thoughtful feedback!
The rising tide of literacy certainly helped boost journalism's demand, which at least meant more money available to spend. Plus there was the decreasing latency of information, along with better indexing / easier querying of that information via libraries and commercial data services. But even adding that all together, did average output really improve? Was journalism from 1935 notably better or worse from 1995 if we compared outlets by say accuracy quartile? I'm skeptical. Some outlets have always tried pretty hard to get it right; some have never cared at all. So it goes, so it always has. I suppose it became easier to get some quantifiable aspects right? Just so, bundling was certainly good in so far as it provided the kind of revenues that made good journalism *possible*. But whether this potential was ever meaningfully exploited, I'm...yeah, skeptical? (Tricky thing to measure empirically though.)
But hard agree that lifting up literacy is massive. Even a very modest lift can have outsized effects everywhere across an economy. Few public policy goals are more important.
Well, always appreciate your thoughtful posts! Two thoughts:
1. While serious longitudinal analysis is thin (most articles in "Journal of Newspaper Research" seem to be either quantitative or historical, not both), the bits I've been able to find suggest that things really did improve. This impressive dissertation (https://research.rug.nl/en/publications/between-personal-experience-and-detached-information-the-developm) argues that the "normative objectivity regime"--which sounds a lot like "progress-useful"--peaked between the 1960s and 1990s.
2. This article (https://www.aeaweb.org/articles?id=10.1257/aer.104.10.3073) seems to support the Yglesias theory of high distribution costs -> bundling -> monopolistic two-sided-market -> good for hard news. Since the 90s were probably a competitive minimum (after newspaper consolidation, but before Fox News and blogs), that also suggests an actual golden-age of progress-useful journalism. The article is taking consumer preference as a given and looking at newspaper response, but of course we can treat consumer preference as itself endogenous with literacy, etc.
My pet theory is that the radical simplicity and formulaic bent of his writing is actually intentional, in that it optimizes his stuff for the widest possible audience? That or he’s participating in an escalating wager about just how much viewers will excuse bad writing if the wizardy is wizard-y enough.
Ok, I split the difference and just gave $10 to GiveWell in your name (confirmation # 250948). IMO it's important for me that I honor even small mistakes (where it's an oversight of thinking vs. eg just a misspelled word), as it reinforces my incentive to be super careful on details.
Second, I think you're missing some important bits of the AI story.
1. Go look at Ethan Mollick's recent experiments (https://www.oneusefulthing.org/p/it-is-starting-to-get-strange). Lots of niche or local journalism could benefit from more quantitative analysis in a way that isn't financially feasible if you have to hire a data scientist. Lots of journalism could also benefit from better display of its quantitative results (Tufte, etc). Seems like AI is ready to do both, more or less today!
2. Progress-utility isn't only about the leading edge, it's also (at least as much) about catch-up growth: poor countries becoming middle-income, boring industries adopting computers, software verticals moving to the cloud, etcetc. The audience for work aiming at this is, nearly by definition, not at the upper end of the literacy curve. As you note, such lower-literacy audiences are not very good at pulling in context from other work, transferring knowledge between domains, etc. AI is really good at tasks like (literal) language translation, providing context, transferring commonplaces into the argot of a different industry or place, etc. These are also extremely progress-useful things to do!
Taking these in turn:
1. Agree on data presentation. That's an easy win. But data analysis? AI isn't there yet imo. You end up needing someone who understands the field well enough to verify if the output is actually legit, which at least partially ruins the point of the exercise. And it's difficult to know just how long it will take to close that gap to a reasonable level of confidence? Like if your AI analyst hallucinates 10% of the time, that's obviously too much. But what about 1 or 2%? My guess is rewards here go to larger pubs, as they can hire analysts on salary to once-over Ai-generated output at scale.
2. Yeah that's certainly all true (and probably understated in net importance). Was just a bit orthogonal to my focus here. AI will certainly do a lot of good in the world so far as patching over the costs of low literacy (and just like making the world more accessible!). But not sure that it will do much about getting substantially more people to Levels 4/5 where they can meaningfully parse written text to say "hmm no this specific bit is wrong, I can safely discard it". Like whenever I use GPT I'm struck by how close it is to being really insightful. But there's always one detail or another (often a central one) that's completely coked up, and you have to know a fair amount about the thing you're querying to spot it (which presumably most can't, or they wouldn't be asking the question in the first place). Getting people to the state where they can spot those mistakes consistently is super hard, and my best guess would be that over-reliance on GPT etc will actually decrease the number of people we get to this stage. Because, as with bad journalism, it's super easy to discount the net impact of all these mistakes going uncaught, and very easy to over-focus on "hey, this is really fun and I'm learning a lot" (even where some % of what you're learning is actually bullshit, and even if none of it is truly learning at all).
1. That's not clear to me. Yes, of course hallucination is a problem, but as we know it's also a problem when reading real academic studies--a shocking percentage of the journal output in some fields is so bad at statistics and so good at p-hacking that even the sign might be wrong. The tide is turning in high-prestige-psych but we're nowhere close in education research, traffic engineering, or many other disciplines important to journalism. Furthermore, in the fragmentation era, there are a lot of pretty clued-up but low-resourced outlets like your substack, where there is a steady and reasonable hand to make sure the AI isn't being crazy. Mollick has really been showing the way on this, eg by asking the AI to produce summary stats or cross-tabs that easily verify its more exciting claims.
2. A lot of this seems to depend on the prompt engineering. If you just say "explain X" well, odds not so great. But if you say "Take X's theory of Y and translate it into Z domain" or better yet "take this PDF and provide a useful abstract for someone from Romania" you get massively better results. I thought your point here was about the impact of literacy on progress-utility, not just literacy as an end-good, in which case this seems super-relevant.
3. It therefore matters a lot whether progress-utility is a strongest-link or weakest-link problem (or, realistically, what % of the two). To the extent that it's a strongest-link problem, AI seems likely to improve productivity of the sharpest people, who can use it for what it's good at, ignore what it's not, and have reasonable mechanisms to filter what they read. To the extent that it's a weakest-link problem in general, then it should be dominated by the catch-up growth rather than leading-edge effects, since weak links were never at the leading edge. So then the question becomes how flooding the zone with (even more) crap weighs against the positives in these two areas. That's an empirical question that will be hard to answer.
First, I agree very strongly with the first part, but it's interesting to think of historical examples where this has gone better. It seems like journalism really did improve between Hearst's day and the 90's NYT, whatever its flaws. How, and why, if not the bundling theory favored by Yglesias et al?
Another case study would be the transition between Abbe Migne's publishing machine (https://en.wikipedia.org/wiki/Jacques_Paul_Migne) which supplied new academic libraries all over the world with subscriptions to horrifically edited and sometimes outright forged texts and the modern university presses, which again whatever their flaws seem to do much better. Migne was already a bundler, so that can't be the issue there.
In these two cases is it that literacy just genuinely went up, perhaps because of public education in the former case and accreditation of tertiary education in the latter? If so that seems really important to achieve!
Hey Ryan! Always appreciate the thoughtful feedback!
The rising tide of literacy certainly helped boost journalism's demand, which at least meant more money available to spend. Plus there was the decreasing latency of information, along with better indexing / easier querying of that information via libraries and commercial data services. But even adding that all together, did average output really improve? Was journalism from 1935 notably better or worse from 1995 if we compared outlets by say accuracy quartile? I'm skeptical. Some outlets have always tried pretty hard to get it right; some have never cared at all. So it goes, so it always has. I suppose it became easier to get some quantifiable aspects right? Just so, bundling was certainly good in so far as it provided the kind of revenues that made good journalism *possible*. But whether this potential was ever meaningfully exploited, I'm...yeah, skeptical? (Tricky thing to measure empirically though.)
But hard agree that lifting up literacy is massive. Even a very modest lift can have outsized effects everywhere across an economy. Few public policy goals are more important.
Well, always appreciate your thoughtful posts! Two thoughts:
1. While serious longitudinal analysis is thin (most articles in "Journal of Newspaper Research" seem to be either quantitative or historical, not both), the bits I've been able to find suggest that things really did improve. This impressive dissertation (https://research.rug.nl/en/publications/between-personal-experience-and-detached-information-the-developm) argues that the "normative objectivity regime"--which sounds a lot like "progress-useful"--peaked between the 1960s and 1990s.
2. This article (https://www.aeaweb.org/articles?id=10.1257/aer.104.10.3073) seems to support the Yglesias theory of high distribution costs -> bundling -> monopolistic two-sided-market -> good for hard news. Since the 90s were probably a competitive minimum (after newspaper consolidation, but before Fox News and blogs), that also suggests an actual golden-age of progress-useful journalism. The article is taking consumer preference as a given and looking at newspaper response, but of course we can treat consumer preference as itself endogenous with literacy, etc.
Stratechery just posted something relevant: https://stratechery.com/2023/the-unified-content-business-model/
7th footnote is great!
My pet theory is that the radical simplicity and formulaic bent of his writing is actually intentional, in that it optimizes his stuff for the widest possible audience? That or he’s participating in an escalating wager about just how much viewers will excuse bad writing if the wizardy is wizard-y enough.
Good essay.
Nit: "an arms race pitting John Henry and his axe against the team with the steam-powered drill" should be a hammer— not an axe.
*Bows head in shame.* I suppose in my head I meant pickaxe. Which would still be wrong, but at least less hilariously wrong.
Anyway, in the spirit of my corrections policy will send you a small something for catching it!
De nada— I admire your consistency, but this was basically a typo.
Ok, I split the difference and just gave $10 to GiveWell in your name (confirmation # 250948). IMO it's important for me that I honor even small mistakes (where it's an oversight of thinking vs. eg just a misspelled word), as it reinforces my incentive to be super careful on details.