• loathsome dongeaterA
    link
    fedilink
    English
    arrow-up
    4
    ·
    8 months ago

    This research has been done a lot of a times but I don’t see the point of it. Exams are something I would expect LLMs, especially the higher end ones, to do well because of their nature. But it says next to nothing about how reliable the LLM as an actual doctor.

    • gregorum@lemm.ee
      link
      fedilink
      English
      arrow-up
      2
      ·
      edit-2
      8 months ago

      Even those who do well in testing of wrote knowledge can perform poorly in practical exercises. That’s why medical doctors have to train and qualify through several years of supervised residency before being allowed to practice even basic medicine.

      GPT-4 can’t do even that.

    • beardown@lemm.ee
      link
      fedilink
      arrow-up
      1
      arrow-down
      1
      ·
      8 months ago

      But it says next to nothing about how reliable the LLM as an actual doctor.

      Yet these tests say anything about how a human would be as an actual doctor?

      • loathsome dongeaterA
        link
        fedilink
        English
        arrow-up
        1
        ·
        8 months ago

        It says as much as it does for an LLM but doctors have to have a lot of field experience after passing these tests before they get certified as doctors.

        • beardown@lemm.ee
          link
          fedilink
          arrow-up
          1
          arrow-down
          1
          ·
          8 months ago

          Then we should remove such tests and, if anything, increase such field experience

            • beardown@lemm.ee
              link
              fedilink
              arrow-up
              1
              arrow-down
              1
              ·
              8 months ago

              Because clearly passing such tests doesn’t matter. If it did matter then it would be noteworthy and have implications for the labor value of doctors that gpt could pass the tests to a better extent than many of them

              • my_hat_stinks@programming.dev
                link
                fedilink
                arrow-up
                1
                ·
                8 months ago

                If you can’t read a licence plate at 20 metres you can’t safely drive. Being able to read a licence plate at 20 metres does not make you a safe driver. The test still matters.

              • loathsome dongeaterA
                link
                fedilink
                English
                arrow-up
                1
                ·
                7 months ago

                Tests are meant to gatekeep who gets to get the field training required to become a doctor. Sending every jabroni into residency willy-nilly is probably gonna collapse the healthcare system completely.

                • beardown@lemm.ee
                  link
                  fedilink
                  arrow-up
                  1
                  ·
                  7 months ago

                  That wouldn’t collapse the health care system, it would devalue the salaries of doctors which would be good for everyone else as it would lower costs. Which is what has happened to practically every other profession