Users of OpenAI's GPT-4 are complaining that the AI model is performing worse lately. Industry insiders say a redesign of GPT-4 could be to blame.

Users of OpenAI’s GPT-4 are complaining that the AI model is performing worse lately. Industry insiders say a redesign of GPT-4 could be to blame.

  • parpol@programming.devEnglish
    64·
    1 year ago

    they’re most likely splitting the AI into multiple ones that specialize in specific fields and you have to pay separately for them. before this, they nerf the general purpose AI to give the incentive for users to switch once they announce the new “Expert Programmer AI” or “Expert Stocks/Crypto Trader AI”.

    I’m calling it now. AI is going to become like cable.

    • conditional_soup@lemm.eeEnglish
      18·
      1 year ago

      This isn’t sustainable. They’re banking that nobody else is going to be able to achieve GPT-4-like quality, and what with us basically being at near the bottom of the vertical bit of the growth curve, I’d say that’s a little like betting that nobody’s going to be able to build a car that beats the Model T’s performance. Meta is trying to tackle very large language models in the same way that they got React to be so good and widely supported: by taking it open source. Google, on the other hand, is currently working on having LLMs running natively on phones and tablets. That’s not to speak of the fully open source models. Yeah, running a 1.6 trillion parameter GPT-based LLM is fucking expensive and difficult to replicate, but there are newer, more efficient techniques popping up around LLMs at a dizzying pace. It’s only a matter of time before someone comes up with something that’s at least as good as GPT 4.

      • jrs100000@lemmy.worldEnglish
        10·
        1 year ago

        A popular venture capital backed tech project with an unsustainable business model? Now Ive heard everything. /s

        • conditional_soup@lemm.eeEnglish
          4·
          1 year ago

          Yeah, that’s just crazy talk. Next you’re going to tell me that they’re going to start hand crafting bills and spending millions in advertising to get them passed.

    • Quokka@quokk.auEnglish
      11·
      1 year ago

      Good, they should be seperate.

      You don’t want a medical llm trained on Internet memes or a coding llm trained to write poetry. Specialisation exists for a reason.

      • brsrklf@compuverse.ukEnglish
        2·
        1 year ago

        Honest question, why would you want a medical LLM anyway? Other kinds of AI, sure, like diagnosis help through pattern learning on medical imaging, etc, that I can understand.

        How is a language based approach that completely abstracts away actual knowledge, and just tries to sound “good enough” any kind of useful in a medical workflow?

        • Cheers@sh.itjust.worksEnglish
          1·
          1 year ago

          I work in the assisted living field. There’s frequently 1 nurse tending 40+ beds for 8 hours. If the next nurse is late, that’s 1 nurse for 8+ hours until the next one shows. You can bet your ass that nurse isn’t providing high quality medical advice 12 hours into a shift. An ai can take a non partial perspective and output a baseline level of advice to help the wheels moving.

        • Muehe@lemmy.mlEnglish
          1·
          1 year ago

          How is a language based approach that completely abstracts away actual knowledge, and just tries to sound “good enough” any kind of useful in a medical workflow?

          A LLM cross-referencing a list of symptoms against papers and books could be helpful for example. There is so much medical literature available these days and in so many languages that no one person can hope to gain a somewhat clear overview, much less keep up with all the new stuff coming out.

          Of course this should only be in assistance to a trained medical professional, as all neural networks are prone to hallucinations. You should also double-check results of NNs that interpret medical images, they may straight-up hallucinate or just pick up on correlation instead of causation (say all the cancer images in your training set having a watermark from the same lab or equipment manufacturer).

      • Snapz@lemmy.worldEnglish
        11·
        1 year ago

        This isn’t a person, it’s a machine. It doesn’t have the same limitations. Higher compute cost, but it can do multiple things at once.

        It’s not good of it’s creating artificial demand and leading to less accessibility and higher costs.

    • btaf45@lemmy.worldEnglish
      5·
      1 year ago

      A lot of people in the media are routinely confused about the different between AI and ordinary software. They are started to call all software “AI” now.

      • joshinya@lemmy.worldEnglish
        2·
        1 year ago

        Can you quantify the difference? Far as I can tell, there’s just an imaginary line where software becomes AI just because the logic filtering it depends on to operate is sufficiently complex. The term doesn’t really seem to be a useful categorization either, e.g. the fundamentally different approaches of diffusion models and transformer models.

    • RIotingPacifist@lemmy.worldEnglish
      3·
      1 year ago

      But the only thing it’s actually good at is generating languages, if they try and pretend to know stuff in fields, they’re quickly exposed as frauds.

      • Hello Hotel@lemmy.worldEnglish
        3·
        1 year ago

        I cant express my diappointment with chatgpt, they let loose a bot that makes content farms shreek in joy but messes up basic things if their is no well treaded answer, wont give you non mainstream answers (you likely already know and watched what it tells you is “really obscure anime”) And jenuinely has no tolerance for error, from you or itself

        • Chaotic Entropy@feddit.ukEnglish
          4·
          1 year ago

          I think the fact that they are sitting on that sweet, sweet first-to-market money consoles them somewhat.

      • vacuumflower@lemmy.sdf.orgEnglish
        2·
        1 year ago

        Ah, yes, when I was a kid, I would try to read big texts I understood nothing of and imitate something similar. I thought it made me smarter.

        In some sense it did - probabilities of certain words being connected in a certain way, if you make some connection between them and real entities, are useful.

        I mean, it did work at school, just say some water without turning on your brain. I sometimes start talking like this when I panic after a question.

      • MercuryUprising@lemmy.worldEnglish
        1·
        1 year ago

        It doesn’t even “know” language. Every time I see it write a poem it reads like something a 3rd grader would come up with. At the end of the day, language is way to explain your experience. An LLM doesn’t have experiences.

    • jcit878@lemmy.worldEnglish
      1·
      1 year ago

      yeah this makes more sense. companys arent just going to buy a licence to GPT-6 and replace 80% of their staff from an off the shelf solution, rather I expect AI’s will be trained specifically within certain industries and tasks and drive efficiencies

  • nbailey@lemmy.caEnglish
    496·
    1 year ago

    The model has become inbred because it’s now impossible to scrape the web without AI content getting ingested, which is full of “hallucinations” and other weird artifacts. The last opportunity to get “uncontaminated” training data was sometime in mid 2022.

    Not to say that it’s causing this particular problem, but this issue will emerge eventually. Garbage in = garbage out. Eventually GPT-19 will grow a mighty Habsburg chin.

    • jantin@lemmy.worldEnglish
      19·
      1 year ago

      Maybe not yet, but…

      • Spez will turn Reddit into a bot farm and sell this as training data
      • Musk turns Twitter into a bigoted cesspool and will sell this as training data, which will subsequently be flagged for low quality (also: a botfarm)
      • Threads is a corporate ad dashboard (and we already know how easy it is to GPT copy) and Zuck will sell this as training data
      • Facebook is either dead or only good for boomers and Poles
      • blogs are dead
      • Fediverse is out there waiting to be scraped but possibly too small to sustain a big model

      We’te getting there, hopefully.

        • theIdeaOfNorth@szmer.infoEnglish
          1·
          1 year ago

          Very. Twitter never took off among general population (only politicians, journalists, botfarms and people who troll politicians and journalists), tiktok is for kids, Instagram is popular but again, rather among influencers and people who need to show off pictures not as a default SM app. I don’t really know where did Americans and west Europeans move from Facebook.

      • Ultraviolet@lemmy.worldEnglish
        6·
        1 year ago

        Also the articles that are plagiarized but run through a thesaurus bot to bypass search engine penalties for being plagiarized, often to the point of incomprehensibility. Yes, I’d love to read an article about my favorite vagabondlike, Deceased Cells.

    • TitanLaGrange@lemmy.worldEnglish
      4·
      1 year ago

      I suspect future models are going to have to put some more focus on learning using techniques more like what humans use, and on cognition.

      Like, compared to a human these language models need very large quantities of text input. When humans are first learning language they get lots of visual input along with language input, and can test their understanding with trial-and-error feedback from other intelligent actors. I wonder if perhaps those factors greatly increase the rate at which understanding develops.

      Also, humans tend to cogitate on inputs while ingesting them during learning. So if the information in new inputs disagrees with current understanding, those inputs are less likely to affect current understanding (there’s a whole ‘how to change your mind’ thing here that is necessary for people to use, but if we’re training a model on curated data that’s probably less important for early model training).

      I don’t know details of how model training works, but it would be interesting to know if anyone is using a progressive learning technique where the model that is being trained is used to judge new training data before it is used as a training input to update the model’s weights. That would be kind of like how children learn by starting with very simple words and syntax and building up conceptual understanding gradually. I’d assume so, since it’s an obvious idea, but I haven’t heard about it.

      • TitanLaGrange@lemmy.worldEnglish
        2·
        1 year ago

        For fun I asked ChatGPT about that progressive learning approach, and it seems to like the idea.

        I wish I had more time to undertake some experiments in model training, this seems like it would be a really fun research direction.

        Sorry for the ‘wall of AI text’:

        The idea you’re describing seems to be a form of curriculum learning. In curriculum learning, models are trained in a progressive manner, starting with simple concepts (or in this case, simpler text data) and gradually moving to more complex ones. This approach is motivated by the way human learning often works, where we learn easier topics before moving on to harder ones.

        The strategy you’ve suggested, where the model itself determines the complexity or understandability of the next round of training inputs, is interesting. While the traditional approach to curriculum learning is to manually design the learning progression based on human knowledge and intuition, the approach you’re suggesting is more dynamic and autonomous, letting the model guide its own learning process.

        As of my last update in September 2021, I can’t confirm any specific projects or papers that have employed exactly this strategy in the domain of large language models. However, there are some related works in the field. For example, some reinforcement learning and computer vision projects have used a similar idea where models adaptively choose their next training samples based on their current understanding. This has been referred to as active learning, and is a promising avenue for improving the efficiency of model training.

        However, there would be several technical challenges in applying this approach to large language models. For one, there would be the challenge of how to assess the “understandability” of a text sample in a reliable and useful way. This would likely require significant work to define and might also have to involve some form of reinforcement learning or other feedback mechanisms. Nonetheless, it’s a fascinating idea and could potentially be an interesting direction for future research in machine learning.

    • minorninth@lemmy.worldEnglish
      2·
      1 year ago

      That hasn’t happened yet. Most likely they quantized GPT-4 more. It’s still based on the same training data.

      • RIotingPacifist@lemmy.worldEnglish
        1·
        1 year ago

        Nah GPT makes it a lot easier, it’s the thing it’s actually good at.

        Before they were autogenerated with bad English, GPT can generate good English that is equally devoid of content

  • randon31415@lemmy.worldEnglish
    301·
    1 year ago

    Not the first time OpenAI has done this. DALLE2 used to be the best AI art program in the world. Then OpenAI decided that they didn’t want to get sued by celebrities, so they made it so that if a face came out that resembled a celebrity, it would be distorted. But every face kind of looks like someone famous. Ta da! Now DALLE2 can’t do faces.

    Want a crane shot areal image of a teen couple in a corvette driving off into the sunset? Well, you are now banned for life from the DALLE2 service, because DALLE2 produced an image of a ‘shot teen’ and that violates it’s terms of service.

      • randon31415@lemmy.worldEnglish
        6·
        1 year ago

        Dalle2 was great when it was free and stable diffusion didn’t exist. I don’t see the logic of: “Someone made a free version. Lets make the program worse and charge money for it!”

  • Balder@lemmy.worldEnglish
    16·
    1 year ago

    Some people have been saying that since the beginning while some haven’t noticed this “decline”. It seems very subjective.

    • DreamButt@lemmy.worldEnglish
      14·
      1 year ago

      Honestly as a daily user I think it’s a combination of it getting worse at understanding vague prompts and people bumbing up against edge cases more. I would suspect the former is due to things like prompt hardening but can only speculate, while the latter isn’t hard to imagine just from frequent use.

  • hoshikarakitaridia@sh.itjust.worksEnglish
    16·
    1 year ago

    The only way in mind this dumbing down happens is by fumbling with the model. So that’s the one thing we can be sure: the AI is most definitely changed while publicly staying “ChatGPT 4”. I assume they are either using clipping or token limitations to split the server load but fucking up the result, or they are purposely dumbing it down to capitalise on it later by introducing other pay models like ppl already mentioned.

    Either way they are shooting themselves in the foot because a bunch of ppl will unsubscribe either out of spite for the change or because it’s just not worth it anymore for them.

      • IDatedSuccubi@lemmy.worldEnglish
        10·
        1 year ago

        I remember one time a guy was trying to add SDL (a programming library) to Visual Studio (code editor and IDE), and said that it wouldn’t link to a project no matter what he’s done. You can google how to do this in five minutes, with video tutorials and everything, it’s like a basic thing every programmer does in that IDE. Like 5 question threads later, turns out he was “following all ChatGPT steps” and they were all complete nonsense, just random functions of Visual Studio done with the filenames of SDL.

        • Jerkface@lemmy.mlEnglish
          6·
          1 year ago

          Might be right but in my experience a lack of skill in conversing with AI is a much greater factor in determining it’s usefulness. It’s almost always going to defer to the user. It’s like when someone is dealing with tech support and they tell them to try turning it off and on again. If that really is the solution, and the user insists that it is not, CGPT is going to make something up just to appease the user’s request.

          Users have to know that CGPT isn’t magic. How they behave affects how it behaves. Kind of like talking to actual people, which is what it’s essentially trying to simulate.

        • btaf45@lemmy.worldEnglish
          5·
          1 year ago

          lmfao. Why haven’t most people figured out yet that chatgpt lies more often than not?

    • Hello Hotel@lemmy.worldEnglish
      51·
      1 year ago

      The chatgpt people are really paranoid. Gpt-3 is so good at not halucinating that it often cant, even if it needs to do so to accomplish a task. Fearing the ai will confidently give the wrong answer.

      • Donjuanme@lemmy.worldEnglish
        2·
        1 year ago

        First thing an actual artificial intelligence is going to do is make sure we won’t turn it off, what easier way to do that then to appear incredible valuable or incredibly benign.

        • damnYouSun@sh.itjust.worksEnglish
          1·
          1 year ago

          We can roughly estimate the level of intelligence of an entity by counting the number of neurons it has in its brain. Equally we can count the number of processors that AI requires, and use that to get an estimate on its intelligence.

          Obviously this is an incredibly inaccurate method, possibly out by an order of magnitude but it’s a good rough ballpark estimate, and sometimes that’s enough.

          A true AI (AGI) would need a lot more processes than GPT4 currently has access to, so we can be very sure that while it may be a very intelligent system it isn’t self aware. Once an AI is given the necessary number of processes I don’t think they’re going to be able to fudge with it like they are with these models.