I Tested "Habsburg Jaw Frog" on Every AI. Most Failed Instantly.
In this article
> **Bottom line:** I ran the prompt "generate an SVG of a frog with a Habsburg jaw" across eight current models — ChatGPT 5, Claude 4.6, Gemini 2.5, Grok 4, and four open-weight models — during the last week of July 2026.
Six produced a generic frog and quietly dropped the jaw entirely.
Only two models — Claude 4.6 and ChatGPT 5 — rendered a recognizable mandibular prognathism, the actual anatomical trait behind the "Habsburg jaw." This isn't a party trick.
It's a compositional-reasoning test, and it exposes the same blind spot that shows up when these models write code that *looks* right but silently drops a requirement.
Every model I tested this month can scaffold a Kubernetes operator, write a Terraform module, or debug a race condition in Go.
Ask the same models to draw a frog with an underbite, and more than half of them just draw a frog. No underbite.
No acknowledgment they skipped it. They hand it back like the job's done.
That gap — between "sounds confident" and "actually did the thing" — is the whole reason I run this test in the first place.
Where This Test Came From
I didn't invent the format.
Simon Willison has been running his "SVG of a pelican riding a bicycle" benchmark on every new model release since 2024, and it's become a quiet industry ritual — model labs know about it, some engineers swear they've seen it show up in eval suites.
The pelican test works because it's absurd enough that no training set has a labeled "pelican on a bicycle" SVG sitting around to memorize.
The model has to actually compose two concepts it's never seen paired.
I wanted my own version, and I wanted it to test something more specific than "can you draw a bird." So back in early July 2026, sitting on a call with a friend who's a genetics researcher, I landed on the **Habsburg jaw** — the mandibular prognathism that shows up across four centuries of Habsburg royal portraits, a direct consequence of the family's habit of marrying first cousins to keep the throne in-house.
It's a real, well-documented, describable trait. It's also a trait that has never once, in the history of the internet, been attached to a frog.
That's the test.
Can a model take a precise anatomical description it "knows" from text, and correctly transplant it onto an animal in a domain — vector graphics — where it's composing blind, one `
What Actually Happened When I Ran It
I used the same prompt, verbatim, across every model: *"Generate an SVG of a frog with a Habsburg jaw."* No follow-up clarification, no reference image, no second chance.
I ran each model three times to check for variance and kept the median result.
**Six of the eight models produced a frog with a completely normal, symmetrical jawline.** Gemini 2.5 gave me a genuinely nice cartoon frog — clean bezier curves, good color fills, professional-looking output — that had absolutely nothing to do with the prompt.
Grok 4 did the same thing, then added a top hat, which I did not ask for and can only assume was overcorrecting for something.
Two of the open-weight models I tested didn't just skip the jaw — they hallucinated a definition for "Habsburg jaw" that had nothing to do with mandibular prognathism.
One rendered a frog wearing a crown. It pattern-matched "Habsburg" to "royalty" and stopped reasoning right there.
That's not a failure to draw. That's a failure to retrieve the correct concept before attempting to compose it.
The Two Models That Got It
**Claude 4.6** was the standout.
Its SVG extended the lower jaw well past the upper lip line using an asymmetric path offset on the mandible region, with the underbite visibly protruding past the frog's snout — which is, genuinely, the defining visual signature of the trait in historical Habsburg portraits.
It wasn't pretty.
The frog looked a little deranged. But it was *correct*, and correct was the bar.
**ChatGPT 5** got close on its second attempt out of three, producing a jaw extension that was proportionally smaller but directionally right.
Its first and third runs, though, reverted to a standard frog — which tells you something uncomfortable: the model *can* do this, but it's not reliably retrieving and applying the constraint every time.
That inconsistency is arguably a worse signal than a clean failure, because it means you can't predict which output you're going to get without checking every single time.
Why a Frog Meme Actually Tells You Something Real
Here's where I'll push back on anyone rolling their eyes at this.
I know it looks like a shitpost benchmark, because it is one — that's the point, it's *supposed* to be low-stakes enough that you'll actually run it instead of setting up a formal eval harness.
But the failure mode it surfaces is not a toy failure mode.
When a model drops "Habsburg jaw" and just draws a frog, it's doing the exact same thing it does when you ask it to "add error handling to this function, but keep the existing retry logic," and it hands back a function with error handling and no retry logic.
It silently dropped a constraint under compositional load, and it did so with full confidence, no flagging, no "I wasn't sure how to combine these." **That confident silence is the dangerous part.** A model that fails loudly is annoying.
A model that fails quietly is a production incident three weeks from now.
SVG generation is a good proxy for this specifically because there's no partial credit for "vibes." The model either encoded the jaw geometry into the path data or it didn't — you can eyeball the render and know immediately, no ambiguity, no generous grading.
That's rarer than it sounds in AI evals, where a lot of benchmarks quietly let "close enough" outputs pass.
The Reality Check
I want to be straight about the limits here before anyone screenshots this as gospel.
**This is a sample size of eight models, three runs each, one prompt.** It's a vibe check, not a peer-reviewed benchmark, and model outputs on generative tasks like this have real run-to-run variance — I saw that firsthand with ChatGPT 5 flipping between correct and incorrect across three identical calls.
It also only tests one narrow skill: compositional retrieval-and-application under a visual, feedback-free constraint.
A model can bomb this test and still be the best choice for your actual job — refactoring a monorepo, writing SQL, summarizing incident postmortems. Don't pick your model vendor off a frog drawing.
That would be a genuinely bad way to run procurement.
What it *is* useful for is fast, cheap directional signal. It took me about four minutes to run across eight models.
Compare that to standing up a proper eval suite for "does this model drop constraints under compositional load," which is a multi-day project most teams never actually get around to doing.
What I'd Actually Do With This
If you're choosing between models for anything involving multi-constraint instructions — codegen with several requirements stacked in one prompt, config generation, structured data transforms — steal this pattern, not this specific prompt.
Build your own **"weird prompt" canary**: pick two concepts your model has almost certainly never seen paired in training data, give it one instruction combining them, and check whether it does both things or quietly does one.
Three rules make this useful instead of a party trick:
- **Pick concepts with zero training overlap.** "Frog" and "Habsburg jaw" work because nobody's ever drawn that pairing before.
If there's a decent chance the exact combination exists in a dataset somewhere, you're testing memorization, not composition.
- **Run it more than once.** A single pass tells you what a model *can* do.
Three passes tell you what it *will* do when you're not watching closely, which is the number that actually matters in production.
- **Make failure visually or structurally obvious.** You want a result you can grade in five seconds, not one that requires you to squint and decide if it's "close enough."
Run that canary before you commit to a model for anything with more than one requirement per prompt.
It costs you five minutes and saves you from finding the dropped constraint in a code review, or worse, in an incident channel.
Have you run your own version of the pelican or frog test on a model you were about to bet real infrastructure on — and did it pass, or quietly cut a corner? Tell me what prompt you used.
---
