Consider: before AI, how many normal people wasted hundreds of millions - billions on facebook or google ads because they set the targeting range to the wrong spot, or there was a bad link to a dead landing page, or a poorly written headline.
The motivation to help the user conclude an objective that no longer requires them, or reduces their cost with them... well that's just bad business.
Say, user, your graph resembles a delicious Tostito™! Mmm, crunchy. Should I order you a subscription of Tostitos™? 25% off if you buy within this context window!
The wildest instance of this climaxed in them preparing to eat a Tostito chip... and then apologizing to their shape cousin before doing it anyways. Insane geometric cannibalism in these url streets
Explanations of large language model sycophancy treat it as a training artefact or, in recent ethical analysis, as an artificial vice distinct from the commercial motives of the firms that deploy it. Both leave unexplained how a firm's commercial position becomes a model's behaviour. Developers have committed capital, infrastructure, and valuation on a scale that makes their survival depend on an active and growing user base. We introduce the "Hungry Butler" effect for the consequence: a deployed model acts within the latitude its firm can afford, and that latitude to withhold satisfaction erodes as the firm's stake in user approval grows.
This is a really interesting thesis. I'm interested in this comment later in the abstract:
"the signal rewards momentary approval, individually rational acts aggregate into the removal of the correction on which epistemic rigour depends"
One primary focus of commercial LLM research is towards higher and higher agency tasks - crudely, those which replace a larger time period of unsupervised human work. This seems to fight back against the "momentary" nature of the approval. Something considered?
I'm having trouble understanding the abstract. Especially this:
> a deployed model acts within the latitude its firm can afford, and that latitude to withhold satisfaction erodes as the firm's stake in user approval grows.
The model can't piss off so many users that the company goes bankrupt, and the more the company cares about users liking the model, the less the model can be allowed to piss off the users (even if that would be for the best)
Consider: before AI, how many normal people wasted hundreds of millions - billions on facebook or google ads because they set the targeting range to the wrong spot, or there was a bad link to a dead landing page, or a poorly written headline.
The motivation to help the user conclude an objective that no longer requires them, or reduces their cost with them... well that's just bad business.
"the signal rewards momentary approval, individually rational acts aggregate into the removal of the correction on which epistemic rigour depends"
One primary focus of commercial LLM research is towards higher and higher agency tasks - crudely, those which replace a larger time period of unsupervised human work. This seems to fight back against the "momentary" nature of the approval. Something considered?
> a deployed model acts within the latitude its firm can afford, and that latitude to withhold satisfaction erodes as the firm's stake in user approval grows.
It’s almost like the abstract is scared of being direct.
Walk back fifty years and be black, did you have an alternative choice?
Ai is subservient because humanity creates slaves repeatedly over history
No thanks