As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.
It really hit home for me when Google shipped their 'writing help' feature or whatever in gmail. Pretty much every sentence I wrote, it had useful tips for me like "make this more concise" and "reduce run-on sentence". No joke, I averaged about 1.3 suggestions per sentence.
And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.
> And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
I remember when I first used Microsoft Word maybe some 31 years ago (Windows 95 was that long ago, huh), and I had squigglies everywhere because of some grammar checker. "Don't use passive voice!"
At least back in those days, it was advice that I could research more about, and decide whether I wanted to act on.
Many people take AI's output at face value, even if merely used as a proofreader.
As you’ve described it nothing has changed. Word used to add squiggles and people who didn’t research the meaning blindly accepted the recommendation. Same shit. Different day.
"Google shipped their" -> "Google shipped its" (I use 'their' to make it clear I'm talking about the people who build features at Google, and not merely about a faceless corporation)
"feature or whatever" -> "feature" ('or whatever' is chosen to tell everyone that I'm not actually sure what the name of the feature is, and I can't really be bothered to figure it out)
"gmail" -> "Gmail" (meh)
"I wrote, it had useful tips" -> "I wrote received useful tips" (if you remove the comma, the pacing changes, the reader doesn't linger on the word 'wrote', and you invite some ambiguity around the writing and the receiving of useful tips. I feel like this suggestion is particularly brain damaged, I simply can't understand my own writing when I make this change)
(and so on. I now write emails in notepad, and copy-paste them into Gmail when I'm done, that's how much the suggestions annoy me).
Pacing is extremely important in writing. The way one speaks out loud is often not the way that they actually read in their head. Speaking out loud has three parts to it in conception, translation, and vocation. Conception as to what the thought is, translation of that thought to language, and vocation of the language through visuals or sound. Reading only shares the translation portion, but also has organization added to it where you need to organize the information as you ingest it. As such, it's important to format whatever you're writing so that it can be organized first and translated second. People look at the words linearly but don't ingest the information linearly. This is a principle many literary jokes are built upon.
ChatGPT, Gemini, and whatever else do not in fact have the capability to understand that interaction between organization and translation. They process the information linearly, organize it linearly, and translate it linearly. That's what makes it so strange to read. A must lead to B which must lead to C. A cannot connect to C and then circle back to B which is connected to both A and C obliquely. It's a failure in understanding language and how it is used.
In the UK it's common to refer to companies in the plural. "Nike have announced..." etc. . I like it. It's a reminder that they are in fact companies of people, not abstract entities.
LOL. The trick with Ai is to write whatever you want in any which way you like and it understands it no matter how shitty it is. Good writers should avoid asking Ai for suggestions. Bad ones like me, don't bother asking anyways. Too lazy to fix it all.
I learned in school (English is my 2nd language) that you can never use "their" or "they" about something inanimate like a corporation (or a chair, for example). You would use "its" or have to rephrase your sentence; "the developers at Google shipped their feature."
The suggestions are valid and correct, we (generic "we") are just used to not following proper grammar. This is how languages evolve but it is also what puts these assistance tools in a peculiar spot, because they help with accuracy, not everyday expressions.
When one interprets grammar rules as prescriptive rather than descriptive, it is easy to fall into the trap of believing in "correct" and "incorrect", when expressing intent is the single most important thing. Expressing intent is not a thing AI can do, thus far, as they don't want anything...there is no intention.
I'm certain there are folks out there who have a hard time with written English that could benefit from an AI editor, but I would almost always rather read casual English written by a human, even if idiosyncratic and "incorrect" sometimes.
And, to be clear: Hard to read English can never be correct, no matter how carefully it follows grammatical rules, and AI often writes hard to read English.
I think that is an interesting observation. You're comparing the "why" we have a language vs. the "how" the language is structured, and for sure the former is more important than the latter.
A side effect of studying programming languages in my youth was that I began understanding grammar better. Suddenly words belonged to "classes" and when thinking about "object references" in sentences, and the rules that followed, it helped me read and write more clearly.
I think I might be slightly opposite of you, but not very far opposite. I prefer reading a precise and accurate sentence that - when understood - clearly describes the intent without a shred of doubt much more than an easy to understand sentence that I find ambiguous because, to understand it, it requires context and a social agreement that nobody is trying to trick people.
I had a friend once that kept using open-ended phrasings and ambiguous descriptions because (I think) he enjoyed the feeling of power or control that the confusion and miscommunication introduced. Like if he were to "run to the stores" you could ask what he was buying and he'd follow up with a "oh, I didn't say I was buying anything, I'm just going for a run". If I have to guess which of the variations of meanings is the intended one, then the communication has failed in my book, regardless of how easy it was to understand the sentence.
In British English it’s entirely standard (and common) for grammatically singular collective nouns to take a plural verb and be referred to with a plural pronoun:
"The ManU have won three out of their last four matches."
"The Parliament have had heated debates over the NIH budget."
"Tesla introduced their newest EV model last week."
Any competent English teacher should know this.
Also, there’s no "correct" language outside of how it’s actually used. Linguists are the first to explain that their job is to describe, not to prescribe.
- When all the members of a collective noun are performing an action as a unit (and that’s usually the case), use a singular verb.
- The chamber orchestra often plays at the Art Centre.
When to use a plural verb
When the members of a collective noun are performing an action as individuals, use a plural verb. In this case, all or some members of the group are doing something independently of the other members; the group is not acting together as a unit.
That applies to NA English. And even there, as the reference says, notional agreement (verb/pronouns in plural) can be used when it’s more about the individual members of the collective doing something. So GP’s rule that only grammatical agreement (verb/pronouns in singular) is ever correct is wrong even in NA. But in BrE notional agreement is the most idiomatic choice while grammatical agreement is certainly also fine, just not standard style.
"The army are advancing", if I wrote that as a kid, I would get an F and if a British wrote it, they would get an A.. For me "the army is advancing". Anyways I learned a new thing today.
Well, I think you've more or less explained my point exactly! The purpose of language is to convey meaning, and my using "their" for Google is precisely and intentionally in defiance of the idea that a corporation is inanimate. If you change that part of my writing, you actually change the entire voice and personality behind the writing!
And that's why it's so difficult to read the things that LLMs write. Because they follow the rules of language without embracing the chaotic nature of the fact that it's meant to convey information rather than to "be correct".
Another way to view this is as metonymy. "Google" is a metonym for "Google Staff", so using "their" to refer to the humans-making-up-the-corporation (as referenced by the metonym) is appropriate.
I agree it's correct English to say "Google have launched their new product".
This is prescriptive vs descriptive grammar, where descriptive grammar describes how people actually use language. Using "they" in that context is only wrong because someone or group decided that must be the rule.
In Swedish -
> Vi går till mitt hus
"Let's go to my house" - this is correct but not really said
> Vi går hem till mig
"Let's go to my place" - incorrect but the actual way to say it
> I learned in school (English is my 2nd language) that you can never use "their" or "they" about something inanimate like a corporation (or a chair, for example).
Definitely not a chair, but a corporation is also "just" a group of people. "Their" makes perfect sense for a corp, even if it isn't perfectly accurate
So, again, are we aiming for high correctness or high comprehension? "Company" makes sense because you can have a "company of five participants" and they can do something, e.g., select a wine pairing menu at a restaurant, but a corporation is typically "a thing".
Everyone who understands English at a professional level (and probably even lower levels, too - or perhaps especially those) will understand what you mean if you say "the corporation announced their financial report last weekend", but it won't be the correct use of "their", as far as I'm aware.
Similarly you can't, as far as I understand, append an "s" to describe a relationship of an inanimate thing, e.g., this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing. But on the flipside phrases like "the employees of the corporation went to..." sound a bit too - what's the word - posh?
What is "a thing", exactly ? Corporations are certainly ambiguous in that regard: they have legal personhood[1] and even free speech[2] in US, apparently.
In the other hand, no corporation ever really "announce its financial report". Someone does it on behalf of the corporation, technically.
> this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing
You can easily say "the corporation's financial report for Q4 was worse than predicted".
I think you're focusing too much on specific and hyper-specific rigid rules in a single textbook. As others said, prescriptivism vs. descriptivism. Descriptivism is always better, as it describes how language works and may evolve. Prescriptivism assumes languages are rigid unchanging things that can be precisely described.
The full rules for a working language take people years to learn at universities, and they are never as simple as "an inanimate object can't have a possessive form with 's".
Prescriptivism also works in schools/learning as it helps people not to drift too much from the norm. But too many take it to heart.
It likely that no response at all or "This looks good, no suggestions" are not possible responses. They haven't figured out how to detect when a conversation should just end or that no response is necessary and include that in the training data.
The critique of AI systems that lives rent-free in my head is that they "hit the target but miss the point." In writing, especially technical writing, the target is conciseness, word choice, tone, etc, while the point is to effectively communicate authorial intent.
Yea the hypothesis seems the exact opposite of what I find with LLM writing. It’s not that it’s overly dense instead I find that it’s hard to read because it’s so information sparse. Like you’re saying it feels optimized to “good writing” not to conveying information.
I find that it’s both: filled to the brim with facts and numbers, but they’re completely useless and irrelevant. For instance I ask it to come up with some ad copy for an Unreal UI plugin I made and it focuses on things like “Supports 64-bit Windows” or “allows transparent rendering” or lists out-of-context performance numbers.
Of course it supports 64-bit Windows - every Unreal plugin does! They don’t even make 32-bit Windows anymore!
Yeah; whatever the AIs are doing I wouldn't call it "information dense" like the abstract does. I'd prefer the hypothesis that there are complex patterns in speech related to successfully communication information to a limited human brain and the AIs aren't complex enough to model them yet. Or the signal is being obscured somehow.
I'm a writer, a technical writer, and a highly technical person in the executive space. Communication has always been one of my strong points, and the blathering on that AI does drives me to distraction. It's impossible to really tailor the output of LLMs very much. I value information dense, direct writing when it comes to technical things and they're always embellishing and going off on side quests and it's obnoxious.
As both a writer and SWE the problem with LLM output is the same. They can achieve localized goals but have trouble juggling multiple concerns at the same time.
Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.
This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.
Speaking of remembering, i just asked Astra about what temp to use and when to put on a pork shoulder for BBQ. It told me a temp and time and then went through and then did long winded explanation of all the variables that go into it's assessment and ended "in conclusion I recommendation X" where X was a totally different conclusion that in had earlier.
It also insisted that I start an hour early even though it was adding in 3 hours of rest time at the end.
No. The conclusion was not totally different. It recommended one of the explained options and recommended one extra hour as a buffer. I feel like this is a distinct reading comprehension issue if you think the final recommendation came out of left field.
Yeah but that entire wall of text was not needed. The user wants a simple answer that they can take action on.
This happens to me way too often. I just want a simple answer. I don't need a wall of text with 6 different options and a lengthy explanation. Just. Tell. Me. The. Answer.
> As a technical writer, I believe default AI responses are optimized for what people _think_ is good writing rather than what actually is good writing.
At least in the case of ChatGPT - it was (and mostly still is) objectively optimized for engagement. Since ChatGPT was leading the way, and much distilling has occurred, there are at least partial remnants of that problem everywhere.
The Chinese models may or may not really care about this at all, but they're doing a lot of distilling, so they're getting it whether they want it or not.
The American models for the most part care about engagement - some a lot more than others.
dividefuel 12 hours ago
My guess is years of RLHF optimized for rater's first impressions of how well-written something was. It uses lists! It has headings! It uses fun phrases like 'smoking gun'! It's long and informative! These things are relatively fast and easy to identify and select as a preference.
But none of this really covers whether the response is _useful for the audience_. If you're not the audience, it can be hard to tell what is and isn't useful.
Taek 12 hours ago
And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.
And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.
ValentineC 9 hours ago
I remember when I first used Microsoft Word maybe some 31 years ago (Windows 95 was that long ago, huh), and I had squigglies everywhere because of some grammar checker. "Don't use passive voice!"
At least back in those days, it was advice that I could research more about, and decide whether I wanted to act on.
Many people take AI's output at face value, even if merely used as a proofreader.
tstrimple 6 hours ago
Taek 12 hours ago
"Google shipped their" -> "Google shipped its" (I use 'their' to make it clear I'm talking about the people who build features at Google, and not merely about a faceless corporation)
"feature or whatever" -> "feature" ('or whatever' is chosen to tell everyone that I'm not actually sure what the name of the feature is, and I can't really be bothered to figure it out)
"gmail" -> "Gmail" (meh)
"I wrote, it had useful tips" -> "I wrote received useful tips" (if you remove the comma, the pacing changes, the reader doesn't linger on the word 'wrote', and you invite some ambiguity around the writing and the receiving of useful tips. I feel like this suggestion is particularly brain damaged, I simply can't understand my own writing when I make this change)
(and so on. I now write emails in notepad, and copy-paste them into Gmail when I'm done, that's how much the suggestions annoy me).
Tanoc 11 hours ago
ChatGPT, Gemini, and whatever else do not in fact have the capability to understand that interaction between organization and translation. They process the information linearly, organize it linearly, and translate it linearly. That's what makes it so strange to read. A must lead to B which must lead to C. A cannot connect to C and then circle back to B which is connected to both A and C obliquely. It's a failure in understanding language and how it is used.
CalRobert 1 hour ago
silverFork 11 hours ago
Izmaki 11 hours ago
The suggestions are valid and correct, we (generic "we") are just used to not following proper grammar. This is how languages evolve but it is also what puts these assistance tools in a peculiar spot, because they help with accuracy, not everyday expressions.
SwellJoe 11 hours ago
I'm certain there are folks out there who have a hard time with written English that could benefit from an AI editor, but I would almost always rather read casual English written by a human, even if idiosyncratic and "incorrect" sometimes.
And, to be clear: Hard to read English can never be correct, no matter how carefully it follows grammatical rules, and AI often writes hard to read English.
Izmaki 10 hours ago
A side effect of studying programming languages in my youth was that I began understanding grammar better. Suddenly words belonged to "classes" and when thinking about "object references" in sentences, and the rules that followed, it helped me read and write more clearly.
I think I might be slightly opposite of you, but not very far opposite. I prefer reading a precise and accurate sentence that - when understood - clearly describes the intent without a shred of doubt much more than an easy to understand sentence that I find ambiguous because, to understand it, it requires context and a social agreement that nobody is trying to trick people.
I had a friend once that kept using open-ended phrasings and ambiguous descriptions because (I think) he enjoyed the feeling of power or control that the confusion and miscommunication introduced. Like if he were to "run to the stores" you could ask what he was buying and he'd follow up with a "oh, I didn't say I was buying anything, I'm just going for a run". If I have to guess which of the variations of meanings is the intended one, then the communication has failed in my book, regardless of how easy it was to understand the sentence.
SwellJoe 10 hours ago
A failure to express intent, not a failure of grammar. Perfect grammar can mislead just as easily.
Sharlin 10 hours ago
"The ManU have won three out of their last four matches."
"The Parliament have had heated debates over the NIH budget."
"Tesla introduced their newest EV model last week."
Any competent English teacher should know this.
Also, there’s no "correct" language outside of how it’s actually used. Linguists are the first to explain that their job is to describe, not to prescribe.
silverFork 10 hours ago
"When to use a singular verb
- When all the members of a collective noun are performing an action as a unit (and that’s usually the case), use a singular verb.
- The chamber orchestra often plays at the Art Centre.
When to use a plural verb
When the members of a collective noun are performing an action as individuals, use a plural verb. In this case, all or some members of the group are doing something independently of the other members; the group is not acting together as a unit.
- The orchestra are tuning their instruments.
- The cast have been practising their lines."
https://our-languages.canada.ca/en/writing-tips-plus/verb-ag...
Sharlin 10 hours ago
silverFork 9 hours ago
Taek 10 hours ago
And that's why it's so difficult to read the things that LLMs write. Because they follow the rules of language without embracing the chaotic nature of the fact that it's meant to convey information rather than to "be correct".
Borealid 9 hours ago
I agree it's correct English to say "Google have launched their new product".
triyambakam 10 hours ago
In Swedish -
> Vi går till mitt hus
"Let's go to my house" - this is correct but not really said
> Vi går hem till mig
"Let's go to my place" - incorrect but the actual way to say it
bluefirebrand 11 hours ago
Definitely not a chair, but a corporation is also "just" a group of people. "Their" makes perfect sense for a corp, even if it isn't perfectly accurate
Izmaki 11 hours ago
Everyone who understands English at a professional level (and probably even lower levels, too - or perhaps especially those) will understand what you mean if you say "the corporation announced their financial report last weekend", but it won't be the correct use of "their", as far as I'm aware.
Similarly you can't, as far as I understand, append an "s" to describe a relationship of an inanimate thing, e.g., this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing. But on the flipside phrases like "the employees of the corporation went to..." sound a bit too - what's the word - posh?
sdeframond 10 hours ago
What is "a thing", exactly ? Corporations are certainly ambiguous in that regard: they have legal personhood[1] and even free speech[2] in US, apparently.
In the other hand, no corporation ever really "announce its financial report". Someone does it on behalf of the corporation, technically.
[1] https://en.wikipedia.org/wiki/Corporate_personhood [2] https://en.wikipedia.org/wiki/Citizens_United_v._FEC
bluefirebrand 10 hours ago
troupo 10 hours ago
Yes, yes it will be.
See collective nouns, and "Metonymic merging of grammatical number" https://en.wikipedia.org/wiki/Collective_noun
> this would be incorrect: "the corporations employees went to the summer party" again because "corporation" is a thing
You can easily say "the corporation's financial report for Q4 was worse than predicted".
I think you're focusing too much on specific and hyper-specific rigid rules in a single textbook. As others said, prescriptivism vs. descriptivism. Descriptivism is always better, as it describes how language works and may evolve. Prescriptivism assumes languages are rigid unchanging things that can be precisely described.
The full rules for a working language take people years to learn at universities, and they are never as simple as "an inanimate object can't have a possessive form with 's".
Prescriptivism also works in schools/learning as it helps people not to drift too much from the norm. But too many take it to heart.
thwarted 10 hours ago
It likely that no response at all or "This looks good, no suggestions" are not possible responses. They haven't figured out how to detect when a conversation should just end or that no response is necessary and include that in the training data.
WesolyKubeczek 9 hours ago
ThrustVectoring 9 hours ago
trescenzi 12 hours ago
joegibbs 5 hours ago
Of course it supports 64-bit Windows - every Unreal plugin does! They don’t even make 32-bit Windows anymore!
roenxi 11 hours ago
bulder 9 hours ago
IsTom 10 hours ago
burnte 11 hours ago
Izmaki 11 hours ago
(Sorry. I blame Claude.)
lubujackson 12 hours ago
Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.
This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.
malfist 11 hours ago
It also insisted that I start an hour early even though it was adding in 3 hours of rest time at the end.
Human judgment it is not
Chat: https://chatgpt.com/share/6add218a-3bc8-83ea-a2d7-613444eb7c...
tstrimple 6 hours ago
rolosa 2 hours ago
This happens to me way too often. I just want a simple answer. I don't need a wall of text with 6 different options and a lengthy explanation. Just. Tell. Me. The. Answer.
onlyrealcuzzo 12 hours ago
At least in the case of ChatGPT - it was (and mostly still is) objectively optimized for engagement. Since ChatGPT was leading the way, and much distilling has occurred, there are at least partial remnants of that problem everywhere.
The Chinese models may or may not really care about this at all, but they're doing a lot of distilling, so they're getting it whether they want it or not.
The American models for the most part care about engagement - some a lot more than others.