running a systematic test: does the source model affect detection scores? specifically testing whether the same paragraph generated by ChatGPT-4o, Claude Sonnet, and Gemini Pro produces different scores on the same detector. hypothesis is yes, the models have different statistical fingerprints.
this is a good question that i haven’t seen anyone test rigorously. my gut says yes, the models have different default styles and those styles produce different perplexity profiles. ChatGPT tends toward a certain kind of professional smoothness that Turnitin has probably seen more of in its training data.
the detector model matters as much as the source model. a detector calibrated primarily on ChatGPT output will have different accuracy on Claude output vs ChatGPT output. the interaction between source and detector models is the thing to measure, not each in isolation.
share the results when you have them. this is the kind of systematic comparison that’s useful and hard to find publicly.
the methodological design matters a lot here. same prompt across all three models, or different prompts? same topic? same length? if the prompts vary, you can’t isolate the model variable. would be interested in the results if the conditions are controlled.
results from the test. 10 paragraphs per model, same prompt across all three, 200-250 words each, same detector (GPTZero).
ChatGPT-4o average score: 71%
Claude Sonnet average score: 58%
Gemini Pro average score: 63%
the differences are meaningful but not dramatic. Claude scored lowest on average, which is consistent with Claude having a more varied default style. ChatGPT scored highest, consistent with detectors being calibrated on heavy ChatGPT training data.
caveat: one detector, one prompt type (analytical essay), one pass. the results would likely shift on different content types. but the source model does appear to matter. if you’re choosing between models for content that will go through detection, Claude appears to be the better starting point based on this limited test.