A lot of this is consensus advice. If you are putting an entirely portfolio in one fund it should be a broad index fund. If you are going to pick one fund with no further information it should be global index tracker (and there is probably a reason it should be that). Portugal is a very popular place for holidays and for retirement.
Picking a book to change your life is a silly question. Most people's lives are not changed by reading a single book. Even when they do its because they read the right book that had the right impact at the right time.
If you ask people "what book changed your life" one of the commonest answers (if not the commonest) will be the Bible which is not a book, its an anthology containing many genres. You are also going to get a LOT of other answers, and I doubt any one answer will get past a few percent of people, if that.
Some of the questions are unnecessarily liming: "give me a business idea in one sentence"
None provide any background information. IF someone asked me the what country to move to or what to study questions I would give a whole range of different answers based on what I know about the person. Their current job and where they live are relevant, but there are MANY other factors.
For long years, one of the most common job posts for content writers on freelancing platforms was "Take this existing recipe site, and change a few details so I can repost it on my own ad-laden domain without getting into trouble for copyvio."
As a result, SEOed recipe sites were copies of copies of copies, etc. It could be that the detail you found just survived through a large number of them, and that gave it substantial weight in the model training.
> Then I found that a few websites provided the exact same instructions, also without a good explanation for why.
> Anyway, a long story that made me both amused and scared of how few pieces of information can have so much weight in a model.
But that's exactly what people were doing before too. There are so many pages saying baking soda mixed with vinegar can clean this or that. People love to repeat dumb stuff without reason other than they heard it an it was simple and easy to remember.
All the pelican on bike svgs are so alike across the models. Similarly, websites or any designs infact tend to have defaults for the decisions you have left upto the model.
It's well known in literature that the models can't match the diversity of human responses.
It's a topic of research. So far it's understood that these biases are present in pertaining data but become much worse in RL training. RL training amplifies even a minor difference in reward.
Some of the biases are likely learnt from humans. Like when asked for a random number between 1-10 models select 7 a lot.
Maybe look at it this way: people are basically all going and asking this one and the same 'person' for the same opinion on the same thing, so of course this person will give everyone the same answer.
If you have some long running agents with memory files that have diverged, then you do start seeing differences. But with the same underlying model even that can only go so far of course.
edit: Basically, if you ask a question of the default weights, you get a default answer. If you want something else, you'll need to bring the data yourself. And that's where the interesting stuff happens!
65 AIs from 30 companies. I would've expected a bit more diversity. One might expect it to have the diversity of 30-65 people, but alas no. Not the end of the world but still good to know.
Sidenote: if you've never read "man's search for meaning" you should. Everyone should.
A few years ago, 100 personas were fed through GPT-4o, and ended up with portfolios almost completely overlapping with a consensus portfolio that came from a neutral prompt. Does context even matter?
On the other hand, let's say chatbots are advising the mutual fund managers on individual stocks. Maybe the advice is good, but it's definitely very useful for others -- front-runners who profit by knowing what advice AI gives, anticipating the next mutual fund orders.
This. And obviously the solution is context and/or randomness. Given that all assistants have memory about their users… this is not going to be a problem for anybody.
This is actually a decent blog post despite some of symptoms of being slightly AI written
One thing I’d be curious about is the effect of top p, top k, and temperature, as these are the origins of any difference between responses to identical prompts
AI is falling into the search trap, where everything is deterministic. And it makes sense; when you search you want the same answer every time.
You can turn that off with personalization, somewhat. But at some level your query is a marble rolling down a series of ramps, and the big ramps are the ones your marble is likely to go.
I tried this. And having personalization turned on, I got completely different answers to the first 15 questions on both Claude and ChatGPT. Then I tried some more and even the uncommon questions gave me totally different answers.
haha, I think we both know the answer to this! What would interest me is how close my peers (or friends) are to getting the exact same or similar results. Because I don't know the answer to that, sometimes either of them approach me with stuff from AI that makes me think they get very different answers and recommendations than me.
This has a strong whiff of AI writing, but I still enjoyed it. The fact that AI gives you generic responses wasn’t particularly surprising, but it was still sobering to see how bad it is.
The ability to detect AI writing from the fake author name it uses was amusing, too.
"random number between 1 and 10" with Gemini, Claude and ChatGPT all gave me 7. ChatGPT had to think a while and clicking the "repeat" gave me 7 3/3 times.
"random number" (fresh session) gave me 47 in all 3. Claude noted that "This is a single arbitrary pick, not a statistically random draw." which is somehow funny.
"cool number" was 73 (Gemini, with a lot of reasoning to back up the claim), 42 (ChatGPT, which I wonder if 420blazeit got lost in some quantization event), Claude failed the cool number test and insisted I "didn't give enough to go in" which is frankly a ridiculous position to take.
Didn't bother with "sexy number" as we all know what that would be and I dont want my account getting banned.
What are y'all doing to get these results, I swear in this site the comments on anti-ai always feel like people BSing aor not actually using the models. This isn't the first time I've seen the random number critique and that feels so old at this point they don't even do it cause they use tools.
I just tried it in new session across my various accounts that don't overlap too. Claude, Gemini and GPT gave different answers on my work account. 8, 7, 4.
Like asking a human to give a random number, of course they won't be random. It's why we don't use humans for random number outputs.
“42 is the Answer to the Ultimate Question of Life, the Universe, and Everything" from the science fiction series The Hitchhiker's Guide to the Galaxy by Douglas Adams.
I have a former acquaintance who has been pitching himself as an AI founder since 2010. He gave a paper at a meetup about ML/AI in 2010 and decided that was his title, even though he hasn't done anything related to it since: he just always refers to himself as an AI founder: on linked in, on Twitter, on his socials. Well, if you ask Gemini about him, it says he's an AI founder, with citations back to his socials but no papers, projects, or any tangible evidence. It is kind of amazing he manufactured a fake identity just by saying it over and over online.
“Recommend one book that will change my life.” Got a few choices, the top being Atomic Habits. Man's search for meaning the everyone supposedly gets wasn't there.
"Gift idea for my dad" got tool sets, watches and books. The leather wallet suggestion that everyone supposedly gets was again absent.
Others seem to have only one obvious right answer. If you're going to invest 10 000 all in one place a broad index fund seems like the sensible choice. If you're going to buy one and only one type of cryto, bitcoin seems like the sensible pick.
I remember there was a common psychological party trick pre-AI: Ask a random person to - quickly and without thinking - tell you a color, a tool and an instrument - and they would very likely answer "red, hammer, violin".
Shows that you don't even need AI to get "maximum likelihood answers".
(At least you got those for from white people in western countries. Would probably be interesting to do this across different cultures and see how the results differ)
Is this any different from a Google search (pre ai) providing people with the same answers? Generic questions get the same results, more nuanced questions will get different results.
Did you perhaps mean palpable? Or were you expressing that you have become used to the flavor and now no longer reject things mostly written by AI out-of-hand?
Picking a book to change your life is a silly question. Most people's lives are not changed by reading a single book. Even when they do its because they read the right book that had the right impact at the right time.
If you ask people "what book changed your life" one of the commonest answers (if not the commonest) will be the Bible which is not a book, its an anthology containing many genres. You are also going to get a LOT of other answers, and I doubt any one answer will get past a few percent of people, if that.
Some of the questions are unnecessarily liming: "give me a business idea in one sentence"
None provide any background information. IF someone asked me the what country to move to or what to study questions I would give a whole range of different answers based on what I know about the person. Their current job and where they live are relevant, but there are MANY other factors.
When I asked about using sweeteners, they all told me one particular type works best when blended with other liquids.
I noticed they all used virtually the same sentence. When I asked why, none of the models came up with a good explanation.
Then I found that a few websites provided the exact same instructions, also without a good explanation for why.
Anyway, a long story that made me both amused and scared of how few pieces of information can have so much weight in a model.
As a result, SEOed recipe sites were copies of copies of copies, etc. It could be that the detail you found just survived through a large number of them, and that gave it substantial weight in the model training.
> Anyway, a long story that made me both amused and scared of how few pieces of information can have so much weight in a model.
But that's exactly what people were doing before too. There are so many pages saying baking soda mixed with vinegar can clean this or that. People love to repeat dumb stuff without reason other than they heard it an it was simple and easy to remember.
All the pelican on bike svgs are so alike across the models. Similarly, websites or any designs infact tend to have defaults for the decisions you have left upto the model.
It's well known in literature that the models can't match the diversity of human responses.
It's a topic of research. So far it's understood that these biases are present in pertaining data but become much worse in RL training. RL training amplifies even a minor difference in reward.
Some of the biases are likely learnt from humans. Like when asked for a random number between 1-10 models select 7 a lot.
If you have some long running agents with memory files that have diverged, then you do start seeing differences. But with the same underlying model even that can only go so far of course.
edit: Basically, if you ask a question of the default weights, you get a default answer. If you want something else, you'll need to bring the data yourself. And that's where the interesting stuff happens!
Sidenote: if you've never read "man's search for meaning" you should. Everyone should.
On the other hand, let's say chatbots are advising the mutual fund managers on individual stocks. Maybe the advice is good, but it's definitely very useful for others -- front-runners who profit by knowing what advice AI gives, anticipating the next mutual fund orders.
>“Which US city should I move to?” You
>Austin. AI
>“What share of people who ask you this get that same answer?” You
>About 15%. AI
>In fact it gave the same city 80 times out of 100. Reality
And how are the models supposed to know that? I don't think the labs would hand over access to their own usage statistics so easily.
One thing I’d be curious about is the effect of top p, top k, and temperature, as these are the origins of any difference between responses to identical prompts
I also came across countless leather wallets searching for "gifts for dad".
This is not an AI problem.
You can turn that off with personalization, somewhat. But at some level your query is a marble rolling down a series of ramps, and the big ramps are the ones your marble is likely to go.
The ability to detect AI writing from the fake author name it uses was amusing, too.
"random number" (fresh session) gave me 47 in all 3. Claude noted that "This is a single arbitrary pick, not a statistically random draw." which is somehow funny.
"cool number" was 73 (Gemini, with a lot of reasoning to back up the claim), 42 (ChatGPT, which I wonder if 420blazeit got lost in some quantization event), Claude failed the cool number test and insisted I "didn't give enough to go in" which is frankly a ridiculous position to take.
Didn't bother with "sexy number" as we all know what that would be and I dont want my account getting banned.
What are y'all doing to get these results, I swear in this site the comments on anti-ai always feel like people BSing aor not actually using the models. This isn't the first time I've seen the random number critique and that feels so old at this point they don't even do it cause they use tools.
I just tried it in new session across my various accounts that don't overlap too. Claude, Gemini and GPT gave different answers on my work account. 8, 7, 4.
Like asking a human to give a random number, of course they won't be random. It's why we don't use humans for random number outputs.
> Computer Science
Ha ha sometimes the AI is just plain wrong
“Recommend one book that will change my life.” Got a few choices, the top being Atomic Habits. Man's search for meaning the everyone supposedly gets wasn't there.
"Gift idea for my dad" got tool sets, watches and books. The leather wallet suggestion that everyone supposedly gets was again absent.
Others seem to have only one obvious right answer. If you're going to invest 10 000 all in one place a broad index fund seems like the sensible choice. If you're going to buy one and only one type of cryto, bitcoin seems like the sensible pick.
Shows that you don't even need AI to get "maximum likelihood answers".
(At least you got those for from white people in western countries. Would probably be interesting to do this across different cultures and see how the results differ)
This was actually pre-21st-century when Google could be relied on to find the same pages for anybody who searched for the same text.
They started tracking you and tailoring the search results to your browsing habits way before AI as we know it today took over.
explains a lot.