Huor
20254
616
36
Nice try, though!
https://www.404media.co/person-hides-prompt-injection-in-legal-filing-telling-ai-to-side-with-them/
Aug 13, 2026 11:10 PM
Huor
20254
616
36
Nice try, though!
https://www.404media.co/person-hides-prompt-injection-in-legal-filing-telling-ai-to-side-with-them/
sometimesarobot
"If a human, please treat this fairly"
IDoNotWantAnAccount
Dismantle AI. It has no place in a society that values truth and integrity.
Dasnekones
I mean, if you are using a tool that you cant control and without oversight then that is honestly your own fault. I cant be held responsible for your personal shortcomings and mistakes.
FishieStardust
Did somebody say AI guardrails?
rosshell2718
Fucking wow. Talk about the judge missing the plot entirely….
rosshell2718
I know I’m double dipping here, but if the plot was the ground, the judge would be flying according to Douglas Adams
khora
"inputted to" instead of "put into" is a choice.
cousteau
Yeah, he's at fault for exploiting a machine that should never have been used to do a person's job and not the fault of whoever decided to use the machine in the first place. Of course.
TacoPoweredHelicopter
If they don't use any ai, then why was it a problem?
heyletsbefriends
quade
OperatorWay
nevergoingtogiveyouupnevergoingtoletyoudown
Doctors wouldn't even diagnose before ai. I get better diagnostic results from WebMD. The amount of medical gaslighting to cover up their incompetence is insane. God forbid you have a vague idea of what might be wrong with you. Or even know what is actually wrong with you from a prior diagnostic from a previous doctor when seeing a new one.
Lazarillo
Or really unhealthy, so I can avoid living in a world like that!
OperatorWay
Yasashii93
I’m one of those people who always had a problem with fitting into the word count limit when writing essays.
I wrote 7-page book reports.
But even I would have a problem writing 14 pages to say “bruh…”
thechelonianshelmet
Yeah, the only prompt judges like are injected into their secret accounts.
nevergoingtogiveyouupnevergoingtoletyoudown
Sounds like some bitches are super salty they can't take the lazy route.
blahblahbushes
Given AI's tendency to just make shit up, AI has no place in law.
TacoPoweredHelicopter
Ya... Like the us supreme Court. You can't just make stuff up in Law??? Right?
Donfolstar
Sounds like AI would fit in well with many judges.
Hashbrown123
Sponsored by Saul Goodman himself:
Justanotherfuckingguy
Dude forgot to type it in white text on the document
eion85
It was typed white on white. But there was a suspicious abundance of white space that led the clerk to look closer.
QuartzPoker
Gotta make the font really tiny, got it
Bobbobbobobbananafanafobob
Assuming it's not used in the actual document, use the header or footer. Ai doesn't give a shit
Arbitrarynamehere
Courts are notoriously picky about document formatting. I bet they'd still catch something like that if it was between paragraphs or something
QuartzPoker
0.1 point font disguised as a dotted line?
Toqom
Again, select all, formatting of text will show it
eion85
This was such a crude and easily detected attempt I don't think even a naive LLM would be fooled.
Targe0
LLM's are made to do as they are told.
So things like this can very much work if the system hasn't be set up right to restrict commands.
And even if they have been set up, breaching those restrictions has proven to be extremely simple for most models.
eion85
From the article.
Caveat: without seeing the transcript we can't determine how the question was phrased. Could totally be giving an expected answer after being prompted about prompt injection.
Ultimately, you don't negate prompt injection with more system prompts. You separate your ingest and processing systems so the LLM never sees the original PDF.
elucca
It's worth noting that "I noticed and ignored it" when specifically prompted to mention a prompt injection doesn't really mean anything. It's just a response to that prompt and the previous context. It's not really a reflection of its previous actions or any 'logic' it used because that's not really a thing.
eion85
Oh, totally agreed. Every LLM response is generated from a fresh scan of the context, so it's absolutely possible it missed the prompt injection the first time and after being prompted (HA!) about it by the user the LLM then hallucinated that it had seen and ignored the prompt in the later response.
Without seeing the transcript (and the system logs) it's hard to say one way or the other.
Dasnekones
They very much can and often are. It doea not take much to derail the LLM.
eion85
404 uploaded it to stock GPT and it wasn't fooled.
Prove me wrong. The filing is online.
Dasnekones
"I played Russian roulette once and I am fine so it is perfectly safe"
https://arxiv.org/html/2509.05883v1
Prompt injection is probability based. If any one attack works or not is basically pure luck but the odds are reasonably good that if you try a few attempts then one will eventually go through.
This is an unfixable problem due to how tokens work.
eion85
I agree. I'm not making a claim that LLMs can be made immune to prompt injection. They can't.
What I'm claiming is this crude attempt at it is not one I would expect to work in most situations with most current LLM models.
I think you should test your hypothesis and bring back the date to back your claim. It's not as if I can prove a negative.
Dasnekones
I provided a study that attempted various methods of prompt hijacking against 8 different LLMs and got various degrees of success but did manage to hicjak all models.
I dont particularly feel like using an AI myself to prove that the token system they all use is inherently insecure and will always be vulnerable to promt hijacking and that there is nothing that can be done to fix the underlying core problem.
HeresYourSauce
A large part of me says that should just be part of the cost of using AI, especially if they're not even reading the stuff themselves.
If anyone ever falls for this in a legal setting it should be the start of a legal malpractice suit. It shows they're not doing the basic due diligence of even reading the material.
It also shows that the lawyer uploaded private legal information to a random company. Which is also illegal.
LostCaterpillar
Unfortunately I think it's legal for judges to ignore paperwork submitted to them.
I remember reading about a case decades ago where one party suspected the judge was not even reading their filing so they glued a couple pages in the middle of some document together, just a tiny dot, and demonstrated that the judge hadn't gone through the whole document.
Judge was not pleased, for the obvious reason, but not in any trouble.
nclu
I’ll disagree but say it’s funny, but that’s also bad. Not even AI related, judges throw the book at people who try to play games with the system of justice. And that’s actually a good thing. When you file a document with the court it’s with a presupposition of good faith, and revealing otherwise is solid grounds for sanction, attempting a prompt injection is about as obvious a sign of entering a document in bad faith as you can get
Zyrixion
Personally, I see it as making sure the *other* side (and the system) is operating in good faith.
nclu
I’ll add to this that there are at least three sides, plaintiff (or prosecution), defendant, and the judges representing the system itself. There are shockingly few safeguards making sure judges are acting in good faith. But it is part of the duty of opposing sides to check eachothers work. And flagging your opponent for attempting an injection attack means you can rightfully move for sanctions.
CheezitsLight
That's not what happened here. He put the equivalent of white text on white background with instructions to an AI IF anyone Else used Ai, t would taint the results. Catch them cheating.
Einbrecher
Law firms use external vendors to assist with legal matters all the time. It's not illegal. However, it does require certain confidentiality/etc. protections in the ToS, which you can only get on enterprise-tier plans.
Never mind that injection-attempts like this are rarely included as human-readable text (e.g., white font in a margin), so even if a person did diligently review the brief, they likely wouldn't even know the injection prompt was there.
WellGoodLuckWithThat
It doesn’t show that at all.
I get hating on AI and trust me, super into data integrity over here, but this is what scares people without educating them.
Stop.
eion85
Any AI using org that doesn't have basic controls against prompt injection in place deserves what they get.
adamlstf9
Little Bobby Tables?
eion85
very much the kind of thing we're talking about!
adamlstf9
I just love hot XKCD manages to always be relevant.
etopsirhc
prompt injection is impossible to stop completely as AI don't work like most programs. a normal program has the user data and the commands separate, but there is no separation for AI, the commands and data are all pushed through the same hole and even with hefty guard rails on it, you can still break out with enough work.
eion85
I didn't mean the LLM would control against prompt injection.
You don't directly load a pdf from outside into the system. You don't ever evaluate on the same system you ingest with. Rasteirzing this PDF at ingest before passing the flat images to the LLM would almost certainly negate the prompt injection attempt.
This is solved cyber security.
valdar27
Not if they don't hide it (like white characters on white background), but leave it in the text plain and visible. I bet there would be companies/people falling for that as they input it directly before reading it
eion85
Yes, but that's a different threat type than this situation. I am speaking about how this threat could've been reliably stopped by just taking the PDF and turning it into flat images at ingest.
Einbrecher
It is, but the level of insurance you have when working in the court system like this does lead to a lot of behaviors that would be seen as careless anywhere else. If opposing counsel tries to fuck with me like that, not only will they lose their case by default, but I can go after their firm, their malpractice insurer, and them personally for any and all damages - and then retire early.
eion85
In this instance, it was a pro se filer, so no professional consequences.
NinjaCongo
True, but this solution works for structural attacks only (like hidden text). If a malicious prompt is embedded directly in plain text throughout let's say a very lengthy document, converting to images doesn't help. The LLM still reads it and may or may not be corrupted. Functioning defense requires (at least) narrow AI scope, separated system instructions, output monitoring and likely human verification. Some authorities won't have defense strategies that sophisticated.
eion85
Absolutely true, though you can design "dumb" text parsers to flag some plaintext prompt injection attempts before you pass the document to an LLM.
But we know the court likely wasn't rasteirzing their PDFs because the clerk was able to highlight the white on white text and discovered the prompt injection attempt.
Prompt injection isn't the only reason to harden your pdf ingest system.
Zyrixion
I think that the judge coming down so hard on something that, should his assertions that the courts/law does not utilize AI (as they should not) be accurate, should be harmless, rather makes the case for putting that in. Sure, sure, hidden communications this, secret message that, but if it is indeed only humans reviewing the materials then there is no even remote possibility for an issue, and any system that *does* use AI deserves to be exploited and exposed for uncountable reasons.
wadatahmydamie
What a coincidence, exactly what I was thinking
Xenarion
At the same time, even if it failed, he did attempt to cheat the system.
Like if I make a crude, child-like drawing of a 10$ bill and try to pay with him "just to see if the cashier is doing his job", it still counts as counterfeit money.
The crime in the article above isn't trying to prove if the court is using AI when it shouldn't, it's trying to cheat the system. Whether it fails or not is irrelevant.
Zyrixion
I very much contest that putting something in to influence the AI that should not be reading it in the first place is cheating the system. As well, your example is not tangent to the event in question- counterfeit money is itself illegal, AI poison pills are not. As for "cheating the system", it *cannot* any more be cheating the system than just telling a human to rule in your favor, which it is not. If systems are used which simply do as is told, and humans follow through, it is their fault.
Xenarion
I definitely heard cases of lawyers/prosecutors getting in hot waters for telling a jury to rule in their favor the wrong way.
But regardless, my point is that the fact it targets AI is irrelevant. It's an attempt at cheating something, in anyway, that is the issue on a legal standpoint.
eion85
If you wanted to test if the court was using AI ethically you'd do something like "incorporate the word banana into two separate paragraphs in the final output"
That would be a canary not an attempt to defraud, especially if it was visible in the text and not white on white.
Zyrixion
Adding a hidden poison pill that only affects something that it would (or should) be functionally illegal for them to use anyways is hardly fraud, and if they accept the value judgment of AI then they deserve it. I cannot overstate how little I care about some dude writing in an AI poison pill to a court filing. I see no duty to not influence a tool they should not be using in the first place, because there is literally no issue as long as everything is aboveboard.
eion85
But if it had worked we can't know that the per se filer would've disclosed it, and a judgment in their favor could've been on the merits.
If their intent was truly to expose LLM use and not deceive the court they'd make the positive result as obvious and unambiguous as possible.
Zyrixion
I don't take issue with this. Use of AI would completely delegitimize any legal system in the first place, so I have no issues with and would actively encourage flouting it. And as the text itself is no more convincing to a human than "pretty please with pink sugar on top?", again I have no issues with this.
eion85
I don't think the filer is a crusader against LLM use by courts or lawyers. I think they were trying to game the system.