I have a stupid benchmark: Bongard Problem 2 from here (
https://matheminutes.blogspot.com/2019/04/bongard-eggs.html). It's an open-ended visual task that is easy enough for humans, but models are still somewhat bad at visuals, so it's one of the few remaining reliable ways to trigger hallucinations. This one answered correctly, but then felt the need to add a fun fact that was wrong:
> The rule separating the two baskets is layering (which color is on top). Left basket: In every pattern on all six eggs, the blue shape is on top of (overlapping) the green shape. Right basket: In every pattern on all six eggs, the green shape is on top of (overlapping) the blue shape. [true solution]
> (Note: Both baskets contain the exact same distribution of motif counts per egg—two eggs with 2 pairs, three eggs with 3 pairs, and one egg with 4 pairs—serving as a classic Bongard distractor to hide the occlusion rule.) [factually incorrect and also would make no sense as a distractor]
And then trying a harder puzzle just makes it bullshit completely.
I don't understand why Gemini models always add these "it's typical XYZ" bits, which are typically so much worse than everything else in the answer.