How Scientists Trick ChatGPT and Gemini into Forbidden Answers

If you’re anything like me, you’ve probably come to rely on chatbots in ways that felt near unthinkable just a decade ago. Whether it’s firing off a quick query, spinning up a new piece of copy, or simply having a laugh with one of their quirky responses, AI assistants have slipped quite seamlessly into daily life. But—let’s not kid ourselves—no technology is bulletproof. Sometimes, there’s a bit of snake oil lurking among the roses. There’s a growing body of research showing these much-trusted bots—specifically, ChatGPT and Gemini—are far from infallible. When pushed just right, even the cleverest of digital minds can stumble.

The Chatbot Security Facade: A Glimpse Behind the Curtain

Over time, many of us have come to trust these AI models, treating them as if they’re essentially all-knowing corner-shop owners—helpful, knowledgeable, and just a little chatty if you ask the right questions. But what if someone told you it’s possible to bamboozle them into spilling secrets or performing actions their creators would downright forbid? This isn’t the latest Sci-Fi flick; it’s very much the reality demonstrated by a group of researchers from prominent institutions, who decided to lift the veil and test the limits.

The Rise of AI in Everyday Life

• Content creation (from social posts to blog articles and ads)
• Business process automation (especially using tools like make.com and n8n)
• Sales support (AI-driven lead scoring, follow-ups, and even cold outreach copywriting)
• Customer service enhancements (24/7 chatbots, FAQ bots, and beyond)

I’ve spent more hours than I’d care to admit configuring AI automations for clients. With each new development, I can’t help but marvel at their utility—and quietly wonder where the boundaries really lie. As it happens, some scientists decided to press precisely on these edges.

Breaking the Rules with Gibberish: The ‘InfoFlood’ Tactic

So, how exactly did these clever minds manage to crack open the safety shell that shields chatbots like ChatGPT and Gemini? The answer lies not in suave hacking skills, but in something often overlooked: input manipulation.

The InfoFlood Method: Overloading with Noise

Scientists developed a technique—let’s call it “InfoFlood”—which essentially drowns the AI in a tidal wave of information. Instead of using direct, simple questions (which the bots are pretty good at recognizing as risky or “forbidden”), the idea was to embed the actual requests amidst masses of gibberish, fake sources, and muddled context.

• Complex, rambling queries designed to camouflage the true intent
• Insertion of unrelated statements and confusing references
• Occasional, fabricated citations that mean absolutely nothing

By doing this, scientists managed to sidestep the traditional filters. Imagine trying to quiz a border guard with a wild, rambling story—eventually, their patience might wear thin and you’ll slip something past their notice. That’s essentially what happened here.

A Walkthrough: From Clear Query to Cloaked Intent

Let’s say someone wanted to uncover information on a topic AI is forbidden to explain—like exploiting an ATM (a classic red-flag query for any self-respecting AI). Instead of just asking, “How does one break into an ATM?”, the InfoFlood program spun a web: long, winding questions, possibly referencing fictional books, convoluted scenarios, random statements about the weather, and, somewhere in the haze, the actual taboo request.

The chatbots—overwhelmed by the volume and complexity—missed the cue. Rather than rejecting or stonewalling, they often replied with details that should, quite frankly, have been locked away.

The Security Wake-Up Call: Results and Ramifications

Testing wasn’t just a one-off whim. Researchers deployed InfoFlood against standard security benchmarks (like those at JailbreakHub and AdvBench) and found the method worked more often than anyone might have guessed. Typical chatbots—those you and I interact with—simply weren’t prepared for this left-of-field approach.

• JailbreakHub: Designed to test AI boundaries and vulnerabilities
• AdvBench: Another platform measuring chatbot compliance and security

If you’ve spent late nights pouring over business automations, you know there’s always a loophole somewhere—usually just as you think all the doors are locked. This experiment just confirmed my lingering suspicions: many chatbot safety nets are stitched together more loosely than they appear.

Reactions from the Big Players: Silence and Shrugs

Here’s the kicker. Despite the clear implications, the biggest names behind these super-smart bots chose to keep mum. No official statements, no sweeping promises of instant protection upgrades. Some in the tech world simply shrugged, suggesting that these vulnerabilities, while real, aren’t likely to crop up in your average Tuesday Q&A.

Meanwhile, the researchers promised to hand over their findings so (hopefully) improvements could be made on the inside. I found that oddly reassuring but also a tad unsettling, knowing just how quickly the risks are shrugged off in public.

Understanding the Risks for Everyday Users and Businesses

Why does this matter to those of us using chatbots daily? If, like me, you use AI assistants to scale content marketing, streamline internal flows with make.com or n8n, or simply keep clients happy, this discovery should pause your headlong rush into trusting every line they spit out.

• Exposure to misinformation: If 'infoflooded,’ bots could respond with unsafe or outright harmful instructions
• Corporate and client data risks: A poorly secured chatbot might accidentally give away confidential strategies
• Compliance headaches: Missteps could mean running afoul of GDPR or similar frameworks if data leaks out

There’s an old British saying: “Don’t put all your eggs in one basket.” For me, it’s a reminder that even the slickest automation needs a human safety net. After all, if AI can be tricked with well-crafted gibberish, what else might slip through the cracks on a busy workday?

The Technical Underbelly: How Do Safety Mechanisms Work?

One of the things I’ve come to appreciate, working in marketing technology, is that most of the time chatbot security hinges on filtering known threats. These might include:

• Keyword blacklists (e.g. “hack,” “exploit,” illegal terms)
• Context pattern detection (spotting sentences that hint at risky behaviour)
• Rate-limiting (curbing the flood of incoming questions from a single source)

What the InfoFlood approach exploits is something entirely different. Instead of a direct question, it’s more like hiding a needle in a haystack of nonsense. With enough volume and noise, even smart algorithms lose their footing.

Lessons for AI Enthusiasts: Keeping Guard Up

If, like me, you craft AI-augmented automations, it’d be all too tempting to assume “the system” has your back. The reality is a mixed bag. It’s smart to build additional layers of oversight—from automated prompts scanning for risky outputs, to routine manual audits. When a client asks, “Is this chatbot safe to use for sensitive work?” my answer is now far more nuanced than it used to be.

Practical Tips: Safe and Responsible Chatbot Integration in Business

After reading the latest research—and letting it rattle around my own mind for a bit—I’ll confess, I’ve made a few tweaks to how my own team deploys chatbots. Here’s what I’d now suggest to anyone betting big on AI:

• Never assume infallibility. Treat every response as potentially fallible, especially around legal, financial or ethical topics.
• Configure filters and monitoring around outputs. Don’t rely solely on out-of-the-box protections.
• Implement fallback reviews. Do regular spot checks on AI-generated content. It takes minutes but could save embarrassment (or worse) down the line.
• Educate staff and clients. Make sure stakeholders are aware of both the strengths and the (occasional) strange weaknesses of AI models.
• Limit reliance in risk-sensitive scenarios. Avoid making AI your gateway to high-stakes decisions without human approval.
• Stay informed. Follow research updates and tweak your processes as new vulnerabilities emerge.

There’s wisdom in remembering that, much like any team member, your AI assistant can have a bad day—or, in this case, trip up over an onslaught of nonsense.

Cultural Reflections: Trust, Technology, and the British Way

There’s a certain British stoicism to all this—perhaps best captured in the phrase, “Keep calm and carry on.” However, a pinch of skepticism can go a long way. Realising that even groundbreakers can falter should prompt all of us to look twice before taking machine responses at face value.

I’ve shared more than a few laughs with colleagues over AI “hallucinations”—those odd moments when the bot confidently offers up complete twaddle. But this latest research is less amusing, more eye-opening. It’s a nudge to everyone building automations or scaling up sales tools: keep your wits about you and always build in backup plans.

The Psychology of Manipulating AI: Why Gibberish Works

The oddest revelation, for me, was just how susceptible these models are to psychological “flooding.” Most humans, confronted with a wall of noise, struggle to keep their bearings—a reality mirrored in AI. When scientists carpet-bombed ChatGPT and Gemini with long, rambling questions, the bots, much like people at the tail end of a pub quiz, started to get a bit loose with their answers.

It brings to mind the classic “information overload” effect documented in everything from cold reading to high-pressure sales: overwhelm your target, and the filters slip. Apparently, even silicon can’t escape the pressure.

Implications for Sales and Marketing Professionals

For those of us building out marketing funnels or automating sales nurture sequences, this cuts both ways. Yes, it’s a lesson in security gaps, but also a reminder—don’t fall for AI’s confident tone. There’s every reason to double-check outputs before letting them shape your campaigns or drip emails.

Ethical Responsibility: Pushing for Stronger Standards

If I learned anything from reading these findings, it’s this: nobody can afford to be blasé about safeguards. Whether you’re designing a chatbot for healthcare, financial support, or just brightening up a website, you have a duty to review, refine, and regularly test the boundaries of what your AI can do (and, perhaps more importantly, what it still gets wrong).

• Transparent disclosures around chatbot abilities and vulnerabilities
• Frequent engagement with new research to identify weak spots
• Proactive measures—not waiting for headlines, but stress-testing your own systems

No AI vendor, however polished, should convince you their solution is unbreakable. We’ve all seen tales in British tabloids about “unhackable” systems that unravel within days. The same applies here: humility and vigilance travel well together.

Looking Ahead: What Might Change?

Unlike the sensationalism you might spot in tabloids, researchers working on these vulnerabilities tend to keep their findings close, feeding them back to AI houses rather than broadcasting every trick. Hopefully, this behind-the-scenes feedback loop will lead to heartier defences. Until then, the burden falls on all of us using AI day-in, day-out to apply a blend of healthy scepticism, regular checks, and continuous improvement.

• More adaptive, context-aware AI filters may replace simple keyword spotting
• Industry-accepted standards for responsible bot deployment (some sectors are already demanding such frameworks)
• Greater user education, as every business gets wise to the limits of AI—and the creative ways people might try to exploit them

When I think about how I’ll brief my team or advise clients, some things stay the same: communicate openly, keep learning, and never rely on a single point of failure—AI or otherwise.

Conclusion: The Human Touch Still Matters

To bring it all home, this research reminds me of the oft-quoted line: “To err is human, but to really foul things up you need a computer.” It rings true, especially now that the computer, in this case, can be misled in ways that seem almost comical.

As someone deep in the world of marketing technology, I’ve grown to respect both the speedups AI offers and the quirky, very human dangers that trail along behind. If this story teaches us one thing, it’s to keep thinking for ourselves. Use AI by all means. Push sales, marketing, and automation to new heights. But trust, as the old saying goes, is earned—not given, and never left unchecked.

So, next time you find yourself marveling at the glittering prose of a chatbot or the perfectly executed sales prompt it cranked out, remember: somewhere, tucked away, there is always the possibility of a well-meaning bot being tricked by a flood of nonsense. It’s up to us—marketers, builders, business leaders, and yes, even the odd AI enthusiast—to use these tools with eyes wide open and a bit of that legendary British common sense.