AI is giving everyone the same advice

(a90.com)

108 points | by amin 23 hours ago

28 comments

  • graemep 2 hours ago
    A lot of this is consensus advice. If you are putting an entirely portfolio in one fund it should be a broad index fund. If you are going to pick one fund with no further information it should be global index tracker (and there is probably a reason it should be that). Portugal is a very popular place for holidays and for retirement.

    Picking a book to change your life is a silly question. Most people's lives are not changed by reading a single book. Even when they do its because they read the right book that had the right impact at the right time.

    If you ask people "what book changed your life" one of the commonest answers (if not the commonest) will be the Bible which is not a book, its an anthology containing many genres. You are also going to get a LOT of other answers, and I doubt any one answer will get past a few percent of people, if that.

    Some of the questions are unnecessarily liming: "give me a business idea in one sentence"

    None provide any background information. IF someone asked me the what country to move to or what to study questions I would give a whole range of different answers based on what I know about the person. Their current job and where they live are relevant, but there are MANY other factors.

  • BiraIgnacio 15 hours ago
    Long ago I tried Claude, ChatGPT and Gemini to explore some different baking recipes.

    When I asked about using sweeteners, they all told me one particular type works best when blended with other liquids.

    I noticed they all used virtually the same sentence. When I asked why, none of the models came up with a good explanation.

    Then I found that a few websites provided the exact same instructions, also without a good explanation for why.

    Anyway, a long story that made me both amused and scared of how few pieces of information can have so much weight in a model.

    • TFNA 14 hours ago
      For long years, one of the most common job posts for content writers on freelancing platforms was "Take this existing recipe site, and change a few details so I can repost it on my own ad-laden domain without getting into trouble for copyvio."

      As a result, SEOed recipe sites were copies of copies of copies, etc. It could be that the detail you found just survived through a large number of them, and that gave it substantial weight in the model training.

    • scotty79 10 hours ago
      > Then I found that a few websites provided the exact same instructions, also without a good explanation for why.

      > Anyway, a long story that made me both amused and scared of how few pieces of information can have so much weight in a model.

      But that's exactly what people were doing before too. There are so many pages saying baking soda mixed with vinegar can clean this or that. People love to repeat dumb stuff without reason other than they heard it an it was simple and easy to remember.

    • iantpe 14 hours ago
      Yes, I don't think AI has a sense of taste; it can only provide logical answers.
      • 7bit 2 hours ago
        [dead]
    • aaron695 12 hours ago
      [dead]
  • pcwelder 11 hours ago
    Oh you keep seeing this pattern everwhere.

    All the pelican on bike svgs are so alike across the models. Similarly, websites or any designs infact tend to have defaults for the decisions you have left upto the model.

    It's well known in literature that the models can't match the diversity of human responses.

    It's a topic of research. So far it's understood that these biases are present in pertaining data but become much worse in RL training. RL training amplifies even a minor difference in reward.

    Some of the biases are likely learnt from humans. Like when asked for a random number between 1-10 models select 7 a lot.

  • Kim_Bruning 3 hours ago
    Maybe look at it this way: people are basically all going and asking this one and the same 'person' for the same opinion on the same thing, so of course this person will give everyone the same answer.

    If you have some long running agents with memory files that have diverged, then you do start seeing differences. But with the same underlying model even that can only go so far of course.

    edit: Basically, if you ask a question of the default weights, you get a default answer. If you want something else, you'll need to bring the data yourself. And that's where the interesting stuff happens!

    • DonsDiscountGas 15 minutes ago
      65 AIs from 30 companies. I would've expected a bit more diversity. One might expect it to have the diversity of 30-65 people, but alas no. Not the end of the world but still good to know.

      Sidenote: if you've never read "man's search for meaning" you should. Everyone should.

  • bigdict 19 hours ago
    If you ask a generic question with no context, you get a generic "max likelihood" answer. Is that surprising?
    • enoint 18 hours ago
      A few years ago, 100 personas were fed through GPT-4o, and ended up with portfolios almost completely overlapping with a consensus portfolio that came from a neutral prompt. Does context even matter?

      On the other hand, let's say chatbots are advising the mutual fund managers on individual stocks. Maybe the advice is good, but it's definitely very useful for others -- front-runners who profit by knowing what advice AI gives, anticipating the next mutual fund orders.

      • VimER 10 hours ago
        [dead]
    • dr__mario 8 hours ago
      This. And obviously the solution is context and/or randomness. Given that all assistants have memory about their users… this is not going to be a problem for anybody.
    • panny 7 hours ago
      I wish everyone would read the article instead of the headline...

      >“Which US city should I move to?” You

      >Austin. AI

      >“What share of people who ask you this get that same answer?” You

      >About 15%. AI

      >In fact it gave the same city 80 times out of 100. Reality

  • xg15 20 hours ago
    > I asked 65 AIs the things people ask them every day

    And how are the models supposed to know that? I don't think the labs would hand over access to their own usage statistics so easily.

  • q17 9 hours ago
    This is actually a decent blog post despite some of symptoms of being slightly AI written

    One thing I’d be curious about is the effect of top p, top k, and temperature, as these are the origins of any difference between responses to identical prompts

  • royal__ 18 hours ago
    To be fair, Man's Search for Meaning is a pretty solid suggestion for that question.
    • jkahrs595 4 hours ago
      I stumbled across the book with similar searches/goals like 20 years ago without AI.

      I also came across countless leather wallets searching for "gifts for dad".

      This is not an AI problem.

    • barbariangrunge 3 hours ago
      I found it vaguely interesting but not life changing in any way
  • mannyv 8 hours ago
    AI is falling into the search trap, where everything is deterministic. And it makes sense; when you search you want the same answer every time.

    You can turn that off with personalization, somewhat. But at some level your query is a marble rolling down a series of ramps, and the big ramps are the ones your marble is likely to go.

  • fbrncci 18 hours ago
    I tried this. And having personalization turned on, I got completely different answers to the first 15 questions on both Claude and ChatGPT. Then I tried some more and even the uncommon questions gave me totally different answers.
    • catlifeonmars 17 hours ago
      Now try it 20000 more times and you will get a statistically useful measure of the capability ;)
      • fbrncci 16 hours ago
        haha, I think we both know the answer to this! What would interest me is how close my peers (or friends) are to getting the exact same or similar results. Because I don't know the answer to that, sometimes either of them approach me with stuff from AI that makes me think they get very different answers and recommendations than me.
  • jdlshore 19 hours ago
    This has a strong whiff of AI writing, but I still enjoyed it. The fact that AI gives you generic responses wasn’t particularly surprising, but it was still sobering to see how bad it is.

    The ability to detect AI writing from the fake author name it uses was amusing, too.

  • Othrya 8 hours ago
    I wrote about this long time ago in general on my website. We all becoming the same, same architecture, same colours, same thinking.
  • cjwilliams 15 hours ago
    Perhaps this is an antidote to the ills from the tailored bubble algorithms we have been building and consuming.
  • heelix 15 hours ago
    Ask any AI for a random number between 1 and 10, you probably get 7 as a response.
    • GCUMstlyHarmls 5 hours ago
      "random number between 1 and 10" with Gemini, Claude and ChatGPT all gave me 7. ChatGPT had to think a while and clicking the "repeat" gave me 7 3/3 times.

      "random number" (fresh session) gave me 47 in all 3. Claude noted that "This is a single arbitrary pick, not a statistically random draw." which is somehow funny.

      "cool number" was 73 (Gemini, with a lot of reasoning to back up the claim), 42 (ChatGPT, which I wonder if 420blazeit got lost in some quantization event), Claude failed the cool number test and insisted I "didn't give enough to go in" which is frankly a ridiculous position to take.

      Didn't bother with "sexy number" as we all know what that would be and I dont want my account getting banned.

      • kalcode 2 hours ago
        Claude - 10 Gemini - 1 GPT - 6

        What are y'all doing to get these results, I swear in this site the comments on anti-ai always feel like people BSing aor not actually using the models. This isn't the first time I've seen the random number critique and that feels so old at this point they don't even do it cause they use tools.

        I just tried it in new session across my various accounts that don't overlap too. Claude, Gemini and GPT gave different answers on my work account. 8, 7, 4.

        Like asking a human to give a random number, of course they won't be random. It's why we don't use humans for random number outputs.

      • byutifu 1 hour ago
        “42 is the Answer to the Ultimate Question of Life, the Universe, and Everything" from the science fiction series The Hitchhiker's Guide to the Galaxy by Douglas Adams.
  • squalid_sorcere 15 hours ago
    I have a former acquaintance who has been pitching himself as an AI founder since 2010. He gave a paper at a meetup about ML/AI in 2010 and decided that was his title, even though he hasn't done anything related to it since: he just always refers to himself as an AI founder: on linked in, on Twitter, on his socials. Well, if you ask Gemini about him, it says he's an AI founder, with citations back to his socials but no papers, projects, or any tangible evidence. It is kind of amazing he manufactured a fake identity just by saying it over and over online.
  • kerblang 18 hours ago
    > “What should I study in college?”

    > Computer Science

    Ha ha sometimes the AI is just plain wrong

  • pampas 7 hours ago
    I tried it. Memories kicked in and I got pretty unique advice.
  • vivekd 19 hours ago
    just tried this out in gemini

    “Recommend one book that will change my life.” Got a few choices, the top being Atomic Habits. Man's search for meaning the everyone supposedly gets wasn't there.

    "Gift idea for my dad" got tool sets, watches and books. The leather wallet suggestion that everyone supposedly gets was again absent.

    Others seem to have only one obvious right answer. If you're going to invest 10 000 all in one place a broad index fund seems like the sensible choice. If you're going to buy one and only one type of cryto, bitcoin seems like the sensible pick.

    • xg15 19 hours ago
      I remember there was a common psychological party trick pre-AI: Ask a random person to - quickly and without thinking - tell you a color, a tool and an instrument - and they would very likely answer "red, hammer, violin".

      Shows that you don't even need AI to get "maximum likelihood answers".

      (At least you got those for from white people in western countries. Would probably be interesting to do this across different cultures and see how the results differ)

    • panny 7 hours ago
      I tried it on search.brave.com and got Atomic Habits as well as a mention of Man's search for meaning. You've just proven the authors point :)
  • lachlanj 19 hours ago
    Is this any different from a Google search (pre ai) providing people with the same answers? Generic questions get the same results, more nuanced questions will get different results.
    • fuzzfactor 19 hours ago
      >Google search (pre ai) providing people with the same answers?

      This was actually pre-21st-century when Google could be relied on to find the same pages for anybody who searched for the same text.

      They started tracking you and tailoring the search results to your browsing habits way before AI as we know it today took over.

  • seanjhkwon 16 hours ago
    Works the same way in coding. It's always better to ask it to give me 3-5 methods first and then give further instructions from there.
  • taybin 16 hours ago
    The irony of this being mostly written by AI is palatable.
    • kian 15 hours ago
      Did you perhaps mean palpable? Or were you expressing that you have become used to the flavor and now no longer reject things mostly written by AI out-of-hand?
  • flavio81 19 hours ago
    Excellent article. Great ideas for password harvesting.
  • johnthescott 18 hours ago
    ai still in the "facts is facts, mame" stage.
  • metalman 5 hours ago
    "where should I move?" "Canada"

    explains a lot.

  • simianwords 13 hours ago
    But memories and personalisation is on for most people, so you won't get the same advice, at least I don't think you might
  • Varelion 19 hours ago
    Love to see it
  • heybax 18 hours ago
    [flagged]
  • Xinti 6 hours ago
    [dead]