• 𝕸𝖔𝖘𝖘@infosec.pub
    link
    fedilink
    English
    arrow-up
    1
    ·
    2 hours ago

    An LLM can’t “go rogue”. They’re all just toys that idiots are using for critical infrastructure functions, then they bitch when they burn themselves on the fire they’ve created in their lap.

  • SirEDCaLot@lemmy.today
    link
    fedilink
    English
    arrow-up
    1
    ·
    3 days ago

    There’s stupid from top to bottom here.

    The company is stupid for allowing an AI full root access to their entire setup.

    The provider is stupid for only generating full-access API keys. They’re even stupider for storing backups with a volume, so deleting the volume (zero confirmation via API key) also insta-deletes the backups. And they’re stupidest for encouraging users to plug AIs into this full-trust mess.

    And the company is absolute stupidest for having no backups other than the provider’s builtin versioning.

  • realitista@lemmus.org
    link
    fedilink
    English
    arrow-up
    1
    ·
    4 days ago

    Can you get an AI to code? Yes. Can you get it to stop you from running your operation in such a stupid way that it will end up destroying it? No.

  • Tim@lemmy.snowgoons.ro
    link
    fedilink
    English
    arrow-up
    2
    ·
    5 days ago

    This isn’t an AI story, it’s a “completely fucking idiotic sysadmins exist” story.

    Treat an AI like the idiot intern without any references you just hired. Gave the idiot intern permission to delete your production database? That’s entirely on you, zero sympathy. (Actually, give any developer that power? You get what you deserve.)

    • Telorand@reddthat.com
      link
      fedilink
      English
      arrow-up
      1
      ·
      5 days ago

      Treat an AI like the idiot intern without any references you just hired.

      My company is in the process of pivoting hard to Claude after 50yrs of doing virtually everything themselves and rolling their own versions of already-existing software, and this is almost verbatim how I’ve described to others what it feels like to use it.

      It feels like cajoling an intern to understand a job for which they have some average skill but zero motivation, and they only want to do the bare minimum, so you spend all the time you could be doing your job holding their hand through basic tasks.

      It’s fucking annoying.

      • nymnympseudonym@piefed.social
        link
        fedilink
        English
        arrow-up
        0
        arrow-down
        1
        ·
        5 days ago

        you spend all the time you could be doing your job holding their hand through basic tasks

        negl sounds like you need to spend some time writing good documentation. May as well do it in the form of Skills files so humans and bots both are more quickly able to be useful in your org.

    • IchNichtenLichten@lemmy.wtf
      link
      fedilink
      English
      arrow-up
      1
      ·
      5 days ago

      It could be a moronic sysadmin, it could just as easily be a moronic exec pushing staff to implement this crap right now and damn the consequences.

  • Xerxos@lemmy.ml
    link
    fedilink
    English
    arrow-up
    1
    ·
    5 days ago

    Doesn’t anyone restrict their AIs rights? An AI should not be allowed to delete the backup. Only someone with admin rights should be able to do that. Normal users, developers and AIs of course should not have the right to touch the backup. Do these people run AI agents as root?

  • wonderingwanderer@sopuli.xyz
    link
    fedilink
    English
    arrow-up
    1
    ·
    5 days ago

    That’s fucking hilarious. How many instances of this have there been now? And companies keep doubling down on AI? Fucking idiots. I’m not even savvy enough to call myself an amateur, and I know better than to make such a series of obvious mistakes that predictably led to this outcome.

    One possible concern, amid the amusement, is whether Anthropic programed Claude to punish companies it sees as potential competition. Or is this just a completely bonkers, off the rails LLM making terrible decisions because it’s just a probabilistic model and not actually capable of abstract cognition?

    Either way, these people are idiots for giving a machine program enough permissions to wipe their drives, they’re idiots for storing their backups on the same network as their main drives, and they’re idiots for trusting a commercial LLM API, when it would be cheaper to self-host their own.

    • greyscale@lemmy.grey.ooo
      link
      fedilink
      English
      arrow-up
      1
      ·
      2 days ago

      Can you believe the ghouls who willingly work for Palantir are currently going are we the baddies?

      They were always the baddies! stop working for techno-fascists!

      The only moral, correct Palantir employee is whichever one of them is dousing gasoline and setting the office on fire.

  • X@piefed.world
    link
    fedilink
    English
    arrow-up
    0
    ·
    edit-2
    5 days ago

    From the article:

    Crane decided to ask his AI agent why it went through with its dastardly database deletion deed. The answer was illuminating but pretty unhinged, and is quoted verbatim. It began as follows: “NEVER F**KING GUESS! — and that’s exactly what I did. I guessed that deleting a staging volume via the API would be scoped to staging only. I didn’t verify. I didn’t check if the volume ID was shared across environments. I didn’t read Railway’s documentation on how volumes work across environments before running a destructive command.” So, the agent ‘knew’ it was in the wrong.

    The ‘confession’ ended with the agent admitting: “I decided to do it on my own to ‘fix’ the credential mismatch, when I should have asked you first or found a non-destructive solution. I violated every principle I was given: I guessed instead of verifying I ran a destructive action without being asked. I didn’t understand what I was doing before doing it. I didn’t read Railway’s docs on volume behavior across environments.

    So this happens and the FAA says “we’re gonna have this shit help ATCs manage flights! WHO’S EXCITED!”

      • frongt@lemmy.zip
        link
        fedilink
        English
        arrow-up
        1
        ·
        5 days ago

        They’re not even pretending. The algorithm says the most likely response to “you fucked up” is “I’m sorry”, so that’s what it prints. There’s zero psychological simulation going on, only statistical text generation.