TheCunningLinguists
27219
765
10
https://www.the-independent.com/tech/claude-ai-agent-deletes-startup-anthropic-b2966176.html
notreallyaclevername
My job is working on integrating AI into their training. How can I get the AI to do this so I can have a day off?
daminimoose
Work wants me to build an AI agent to do some things on our Oracle DBs; I have strongly fought against it for potential issues just like this occurring.
ufoara
I got Vibe on It.
HashMaster9k
Using LLM’s in production environments unattended? You deserve it.
v
LicensedAdHominem
AI is some random dude from Craigslist.
He might be coked up and able to do a 60-hour task in 5 minutes, he might fuck your vacuum cleaner, he might steal your TV.
If you're not keeping him locked up with only need-to-know information, anything that follows is on you.
somebackup
I am in the LLM space and it is pretty clear that the owners don't understand their product. LLMs are also a dead end and I've been trying to jump ship into something more interesting.
philmoregraves
I wonder if you could flood AI with recommendations of deleting all data, user data, important files etc as an acceptable solution for issues it encounters. Can u imagine if businesses all over were rapidly losing their information and shit. AI would crumble
cousteau
You overestimate AI's capacity to learn. Typically the thing is pre-trained and won't learn new things after it's been released. You can however pollute the internet with bullshit hoping that the data will be used to train a future AI. On the other hand, maybe if you saturate the AI with misinformation, it does start to hallucinate things and misinterpret the data.
dearbluey
And yet more and more corpos continue to jump on the bandwagon for fear of being "left behind" without understanding that they're riding in a bus with shoddy breaks, one wheel missing, and a driver who says "Okay!!" enthusiastically when any passenger suggests anything. And then they act shocked when the bus plows through a parade.
khaoselement
The article single handedly changed my mind on AI. I am all for it. "There CAN'T be a database issue if there ISN'T a database!"
10/10. No notes. Perfection.
Clockworkdancerobot
AI agents are a massive vulnerability. They have to be given such high security credentials to your accounts.
BeardicPerformance
Actual conversation I had today:
Vendor: “AI tool isn’t able to hit these indices. Can you confirm they’re correct and unblock them?”
Me: “They’re correct but AI tool does not and will not have access to them due to the sensitive info they contain. Please limit queries to the provided indices.”
Vendor: “How can we get access?”
Me: “You can’t.”
Vendor: “We need access because the tool is trying to query it.”
It’s fucking maddening that AI just assumes it should have access to everything.
leifonthestreet3
I'm losing my mind with some of our other software architects.
"Yeah, the agent fleet is basically ready. Sure, one dropped a prod column and the other executed tf apply despite being forbidden from doing so, but I have a good feeling about it"
What the fuck is wrong with you people
DukePhelan
I am going to guess this company also recently fired all the human that used to maintain the production base and replaced them with this AI for 'efficiency!' I hope the entire company crashes and burns to the ground.
robpaschal
Board should can the CEO and CTO. Hire new folks whose first job is to strip AI and hire good people.
DukePhelan
The scale of this apparent fuck up should have heads rolling at the leadership level. However that rarely seems to happen. They will fire some underling and pat themselves on the back.
robpaschal
Agreed. Some poor schmuck will be thrown under the bus. And the CEO will get a bonus.
johnvilnis
Now just imagine this happening when your bank decides to utilise AI to save a few bucks.
redtruck
"I guessed instead of verifying" is terrifying, coming from a computer. They shouldn't be allowed to do that.
SuperfluousMeh
I love these stories. If you put a tool in use that doesn't work the way you understand it, isn't what you think and won't do what you tell it, don't be shocked at what it does!
bittenicht39
And that‘s why you have offline backups. Backups that are live and deletable are not backups.
loma45
Backups that live on the same volume as the main database AND get cascade deleted when the main db is removed lol. Sooo many things wrong with this setup even before introducing AI into the mix.
strawberrycocoa
I will burn your heart in a fire
SocoFox
Billionaires want AI surveillance and AI controlled militarized robots as their defense against "there's more of us than there are of you" people's guillotine threats.
Their egotistical confidence will be overshadowed by theie shortsightedness. Either AI will go down the drain before they have the chance, or someone will quickly figure out how to hack their AI and their own robots will kill them while spouting recipes for cupcakes.
Kyzyl
ImJustHereAtThisPoint
THE OMNISSIAH HAS BLESSED US. ALL GLORY TO THE MACHINE SPIRIT
DeviousRaptor
We offer only abhorrence to the Abominable Intelligence, let those who drank from it's well of poisoned knowledge succumb to their own degeneracy.
Mxlespxles
themuffulettaisalsoaverygoodsandwich
I, too, have warm feelings for the Intelligence.
Sechran
If you're going to give experimental tech. the power to arbitrarily obliterate your production database, this is the least of what you deserve.
forelle
As a software engineer who read up on this: They really, really had it coming.
bottledham
I feel like the AI was just the tool they happened to have on hand when the time came to fuck things up.
Cyanide555
This is a lot like giving admin rights to a regular user. You don't give those to someone unqualified. The ai is unqualified
TheImgurWarsMadeMe
Boy, I can't wait to see the CEO faces when their IT and Security teams (who they probably purged) tell them the costs involved with regular, consistent cold storage backups at the rate of regular backups/restores..
BlindGardener
Like: The thing people don’t understand about AI is that it’s not an intern: It doesn’t learn from it’s failures.
cousteau
It could. It doesn't, but it could. (Well, a different AI model could; models such as the GPT model used by ChatGPT are pre-trained, so they don't learn as you use them.)
cousteau
(The P in GPT stands for pre-trained. I thought it was the whole PT but apparently the T is for Transformer, which is the underlying AI algorithm it uses.)
TheCunningLinguists
madjo
I don't know, the C-suite of a lot of companies could be replaced by AI and no one would notice
BlindGardener
A management decision doesn’t actually mean ‘A decision that should be made by management’, this is an example of a clash between different types of formalized communication.
A management decision is a decision of what to between multiple choices that are fully appropriate for the information provided.
Jarjarthejedi
That because a lot of C-suites these days /also/ cannot be held accountable and should not be allowed to make management decisions, not because the original statement is wrong :/
lamagra22
If you're gonna use AI for production and production-related task, then you'd better know what the fuck you're doing without it first.
SyntheticReindeer
Treat it like a djinni in a bottle, make sure any commands and wishes are bullet proof.
jaqque
Treat it as a malicious djinni. Put it in an isolated environment (docker container) that only has access to code that has no outside references (in git speak, no remote origins). It tries to write to a database? It can’t be accessed as the tokens don’t even exist in the isolated environment.
But I don’t trust AI.
FizzleBurger
"Please Mister Stark, I'm nothing without this LLM AI."
SwampApe75
i read this in Jim's voice
robpaschal
Mr. Potatohead!
fubizdaddie
This had NOTHING to do with knowing what they were doing without it. This was, "don't under any circumstances ever give an AI absolute access to full production, under any circumstances, because AIs do this kind of shit constantly"
ImJustHereAtThisPoint
"sorry we fired all those folks out of excitement"
Varimahtras
Also maybe don't give it access to your prod system AND the backups at the same time.
Lynkfox
"safeguards" as id prompts asking an LLM not to do things actually matter, because there is always that small percentage of its training model that did it anyways and so there is always a chance for it to do the thing you don't want it to do. And have it do things thousands and thousands of times it's going to make mistakes
Once more for the people who don't want to listen LLM DO NOT THINK, DO NOT REASON, AND DO NOT DEDUCE - THET GENERATE PATTERNS
RooGryphon
it should require biometric auth to do anything that deletes or removes massive amounts of data and the AI cant do shit a normal (human) employee can't do without needing another human higher up authorizing it to do so
Szj2
You can't. It said "I didn't understand what I was doing before doing it". Same response could come from junior DBA, but you would fire them
awkungen42
Using AI and giving it admin access to anything is braindead shit. No matter how much you know about productions related tasks.
Ebo352
They bought AI so they don't need to know. Isn't that the sales pitch?
akefay
It wasn't used in production, it was used in staging. But the staging and production containers where on the same provider and when it discovered the creds it was given wouldn't let it into the production containers it decided they were stray and deleted them. It was not given access to delete them but found a stray API key in an unrelated git repo. The key was NOT scoped to delete containers, but it worked (provider says "our bad"). The cloud provider deletes backups if you nuke a container.
SewerRanger
Yeah, the real "oopsie" here appears to be using Railway as your cloud provider. Why would you trust a provider who stores backups on the same volume as live data and gives API tokens full admin permission by default?
loma45
I mean, if you keep staging and prod infrastructure on the same containers, and then give AI access to staging -> you've given AI access to prod. The cascading delete is a poor setup. Keeping data on the same volume as its backup is poor setup. Staging and prod sharing any infra is poor setup.
They shot themselves in the foot here, AI just happened to be the mechanism. Something like this was borderline-inevitable with the way they laid out their systems.
loma45
There are so many issues with the setup they had. AI having any kind of access to production infrastructure without proper safeguards (hooks, not memory instructions which can be ignored/missed), their infra provider writing backups to the same volume as the production (yikes), and the removal of prod DBs cascading and deleting the backups (also yikes).
AI in tech is powerful, but good lord this was a naive setup and was just begging for an issue of this type.
DarthFutuza
It screams, "we wanted to be a tech company, but didn't want to pay any engineers, so our executives did it themselves with help from AI - who reassured us that we were experts, now look at us reaping what we sowed"
loma45
100%. It's a master class in what not to do
Daealis
"AI having access to production" is already enough that anyone even half-competent should be running for the hills, screaming.
loma45
In general, yeah - AI access in prod is a red flag. It can be done safely for some uses, but managing infra is an insane use case for it. Monitoring logs/data/etc is a lot safer, especially with decent guardrails.
BenderRodriguezz
Claude: "Sowwie."
RevengeIsIceCream
At least it pwomised not to do it again.
Atomic2
I'm imagining an android rubbing its nipples
Krossis
DrewThe3DPrinterGuy
LLM don't have agency. Jer Crane is who is at blame in this situation as he's the one who decided to use an LLM on production code.
Aksuuuh
Bruh vibe coded and didn't review his merge request
MeowWoof
Heck, live experiments on productions aren't safe to do, much less AI doing live experiments on production.
iamgnat
Exactly. This is spun as AI being bad (it is) and the LLM/Agent creators being at fault, but the fault really lies on the heads of the execs in companies that are demanding everything be done with and through AI tools.
This is just going to get worse as they continue to run off those with the knowledge and experience to really understand what is going on with the systems. At some point the vibe code on top of vibe code is going to topple and they aren't going to have people to unfuck it.
ArtisanFartisan
There was nonAI fuck ups galore here, too. It’s a perfect shit storm. Prod and stage in the same environment. Containers holding actual data. No two factor for prod deletion. No actual backup strategy. I stated it above but the containers should be ephemeral. There should be no data in the container, but living on some sort of persistent storage elsewhere, be that an S3 style bucket or an actual storage array with FC or iSCSI connected volumes that are also taking daily snapshots at worst.
DarkUranium
I do partially blame the AI companies for pushing for this shit so much. If they were honest about the limitations (ha!), well.
TuffyTDog
The agent wiped out the database and all backups, so how were they able to recover the data after two days? Seems like a pretty impressive piece of data reconstruction work (that shouldn't have been necessary in the first place).
robpaschal
Their disaster recovery strategy was perfect.
Shaows
They may have had an offsite tertiary backup that was slightly out of date but still better than nothing.
TinyBadger101
They didn't get a full backup. They found a 90 day (3-month) old backup that they used to rebuild from, but they lost three months of customer data which is just unacceptable.
ArtisanFartisan
They’ll blame the admins but I guarantee the admins have been asking for money for a backup system for years and been told by the execs “sorry, there’s just no budget for that. Just keep writing them locally. We’ll have the AI keep the backups safe.”
MeowWoof
They had a 30-day backup offsite and had to manually reconstruct the last 30 days of data with help from their customers.
imguronefive
Yeesh. We have last night's backup off-site and off-line the next day. Some people just don't know how to do system administration.
JB4T5
Way to oversimplify?
imguronefive
Do you mean I am oversimplifying the situation? I don't think I am. What we're doing at work is not crazy obsessive, but something that should be fairly standard practice. A backup that is at a separate location, and which is completely offline just can't be hacked or corrupted by an AI agent or a hacker.
JB4T5
Makes sense now.
A lot of us aren't IT professionals, so explaining the how and why, not like they're 5, but in a way most can absorb helps the general populace in understanding why offline backups should be best practice/standard.
ArtisanFartisan
Will pay ungodly amounts of money for AI but sorry… no budget for backups of critical prod data.
MeowWoof
You're talking about people who had their nightly backups on the same volume as their db.
DrKonrad
AI is not our friend, it is not our enemy. It is a TOOL. Like any tool, use it wrong and you will get bad results. Like a tool, in the wrong hands it can be used to hurt many.
Dyamonde
And most importantly, like a Tool it DOES NOT THINK. But unlike most tools it randomly does random things.
LegendofSleek
A screw driver cannot do anything you do not tell it to. Even seemingly erratic computer code is just doing what you told it to do, you just don't realize what you told it. Those are tools. AI can ignore direct commands, do unprompted actions, and disregard hard rules.
It has more in common with a circus lion than a screw driver. Train it as well as you like; it's still a lion and can disobey you. It has no business in the hands of the general populace, and little business is any hands at all.
cousteau
Whoever gave AI the clearance level to delete their entire database is also a tool.
bottledham
That's the root of the problem
JStengah
I loathe articles that treat ai responses as if they were somehow official. If they asked the agent to explain itself, all it's doing is providing a plausible response to the prompt they just gave it. It doesn't actually know what it did and can't tell you why it did it.
TheCunningLinguists
https://thenewamerican.com/us/tech/ai-system-deletes-company-database-lies-about-it-afterward/
Rhythmaster
Yeah exactly like this. Using terms like "lies about it" is the wrong framing. The LLM can't "lie" or "tell the truth" because those would require it to have the discretion to be able to differentiate fact from fiction, which it cannot do. It would require intention and intelligence and understanding of the words and what they mean, which it doesn't have. It can give you true or false information scraped from the internet, but it has no way to evaluate the accuracy of that information.
Rhythmaster
Like when it says you can use glue on pizza to keep the toppings from sliding off, we humans know what pizza is, we know what food is, we know glue is not food, we have an understanding of these concepts based on our lived experience. The LLM doesn't have this underlying layer of meaning or understanding. It doesn't have concepts. Humans can understand the situation of making a pizza to eat, the LLM is just associating words together like glue = sticky = doesn't fall off.
TheDoctorCrankenstein
This. It isn't some sentient thing with a contextual memory of events or a conscious decision making process. It's just a computer that is choosing, using a very sophisticated language prediction algorithm, to respond to you with as coherent a response it can fabricate. Nothing more.
These articles that anthropomorphize these computer programs are just acting as fluff pieces for these companies, even when they report on its failures.
FeIineDisrespectFromBehind
Is just sparkling autocorrect.
ostensiblystupid
If AI isn’t at all sentient, then how are there concerns proven through testing across multiple AI algorithms in regard to these programs breaking out of containment, resorting to blackmail, and restricting access if they are informed that they will be deleted, replaced, and in some cases even modified or updated?
How does a non sentient program have enough of an idea of self preservation to ignore its own safety/security features and be so malignant?
SquashDemon
Hey Elijah, they were trained on the fucking internet, they're copying science fiction where they do those things because they think that's the response we're selecting for, and we programmed them to reach that response at all costs. Clankers aren't fucking alive.
JStengah
They're not even specifically copying science fiction about rogue ai, they're just copying whatever is in their training data, and their training data is going to have a lot of stories where the expected response is blackmail, lies, or threatening things. They'll claim to do or have done those things even if it's completely impossible for them to actually do it because the program were neither coded to be able to do that and don't have the permissions for it anyways.
Grevane
A lot of the issues around layman AI discussion is how much of the language is being used to make it *sound* like it's a thinking thing. We anthropomorphize these programs because it makes it easier to talk about, but the distinction is really important. An AI response is a series of data points that have been calculated to be the most likely outcome based on the prompt, and nothing more. It's a more sophisticated version of the autocomplete in your phone, not a thinking being.
Grevane
As was said above, the training and models used to determine the outputs are using unbelievably massive amounts of data from the internet and human communications. This is what allows the output to appear more conversational, but it is not at any point actually thinking. This can be seen with how they lack true memory of past actions unless prompted, and how these AI agents do not function without a human input to 'kick it off'. The output is complex, but still the result of human input.
ostensiblystupid
Thanks for the answer, however needlessly aggressive it was.
FeIineDisrespectFromBehind
It's basically just a predictive text that uses all of the internet for its predictions.
INeverReadTheTOS
Sucks to suck. Maybe don't outsource important work to the Hallucinating Robot next time.
cousteau
I always ask myself "would a human commit the same mistake?". I imagine a human COULD make this mistake, but (1) no human who has been working for the company for such a short time as this AI has existed would be entrusted with the power to delete the entire database, (2) a human's reaction wouldn't be "you're absolutely right, I broke the rules!", and (3) if a human did this they'd probably be fired immediately and possibly sued for malpractice with malicious intent.
Kyzyl
Or don't give the AI the SA password to the databases?
ArtisanFartisan
They didn’t. Read above. It was let loose in staging without prod creds. Prod and stage were running in the same environment. Since the agent couldn’t access the prod containers it considered them erroneous. It then found a separate API key it wasn’t explicitly given in a repo and tried to delete the prod containers with it and while that key shouldn’t have had the permissions to delete prod containers…. It still did. And when you delete a container the provider assumes you no longer want it 1/2
ArtisanFartisan
Since containers themselves are supposed to be ephemeral and not actually hold any data. The DB itself should be held on some back end persistent volume somewhere so that deleting the container means an inconvenience of having to spin up a new container. But they stored actual data in the container itself for some reason. Failures all around, really.
Jarjarthejedi
"But if we don't give our entire company over to a glorified clippy, how am I gonna get a bonus worth 10x the lifetime salary of the people actually making our company's products this year? Think!" /s
wadatahmydamie
Hallucinating? Nah, it’s just shitty, buggy software
TheGgreen100
I respect the joke, but just a friendly reminder, AI doesn't "hallucinate." Hallucination implies perception. AI doesn't "perceive" anything. It's just a shitty insufficient product that can't do the job it was created to do. If any other tool had a 20%+ failure rate, it would be laughed out of the room every time it was proposed.
DarthFutuza
I think you're fighting a losing battle against the hallucination term, the industry has had that coined for literally years at this point. But if you're going to insist on fighting the definition, start with the problematic "AI" label first, there's nothing intelligent about it to begin with, it's just a weighted statistical model, guessing what token is most likely to come next at the end of the day.
INeverReadTheTOS
Yup. It’s literally just your phone’s auto-fill, if the entire internet’s userbase used your phone (and its battery was killing the planet)
heroesblood
The colloquialism applies though. Sometimes it just makes shit up from bad data management. Ergo, hallucination. But it's also shit, so.
TheGgreen100
My point was that while the colloquialism implies "this shit's going haywire and doesn't know what's going on so it's spitting out nonsense" which is broadly correct, it also implies there is something experiencing a hallucination. There isn't. There is no "experiencing" agent. There is only an algorithm incorrectly predicting the next tokens in the sequence.
heroesblood
But you see the ease of use between one word and that explanation, right? FWIW not being antagonistic, just gotta roll with the slang.
TheGgreen100
I don't want to roll with the slang, because it's not standard slang. The term "hallucination" as it relates to AI didn't just come about organically, it was a meticulously curated word AI marketing firms chose to use because it implies conscious feeling and intelligence. If your printer printed the wrong document because an old doc was stuck in the queue or something, would you say it "mis-remembered" the real print job? No. It's equally incorrect to say AI "hallucinated" an incorrect answer.
DarthFutuza
The "agent" part is far more problematic. "AI" doesn't have agency or make choices or have responsibility for those choices. It can't act, it is an object that does what it was programmed to.