Generative AI has initiated a technological arms race between the creation of hyper-realistic synthetic media and the development of tools to detect it. While much research focuses on automated detection, a critical gap exists in understanding human perception: How do people judge the authenticity of content when the lines between human- and machine-generated text are increasingly blurred?
As identified in our foundational survey, "Blessing or Curse? A Survey on the Impact of Generative AI on Fake News" (arXiv:2404.03021), there is a pressing need for empirical data to understand how these technologies influence public trust and information integrity. Our follow-up research extends this work in two directions:
- "Industrialized Deception: The Collateral Effects of LLM-Generated Misinformation on Digital Ecosystems" (WWW '26 Companion, arXiv:2601.21963) examines the systemic effects of LLM-generated misinformation on digital platforms, trust networks, and information ecosystems.
- "Eroding the Truth-Default: A Causal Analysis of Human Susceptibility to Foundation Model Hallucinations and Disinformation in the Wild" (WWW '26 Companion, arXiv:2601.22871) reports the human perception findings from stimuli generated by this framework.
JudgeGPT is not merely a survey; it is a live research platform designed to systematically collect and analyze human judgments on news authenticity at scale.
JudgeGPT operates as the second stage of a two-part experimental apparatus:
┌─────────────────────────────────────────────────────────────────┐
│ Research Pipeline │
│ │
│ ┌─────────────┐ ┌──────────┐ ┌─────────────┐ │
│ │ RogueGPT │───▶│ MongoDB │───▶│ JudgeGPT │ │
│ │ Stimulus │ │ Fragment │ │ Human │ │
│ │ Generation │ │ Store │ │ Evaluation │ │
│ └─────────────┘ └──────────┘ └──────┬──────┘ │
│ │ │
│ ┌──────▼──────┐ │
│ │ Analysis │ │
│ │ Pipeline │ │
│ └─────────────┘ │
└─────────────────────────────────────────────────────────────────┘
-
Controlled Stimulus Generation (RogueGPT): Fragments are produced with explicit control over model, style, language, format, and seed phrase. All generation parameters are persisted alongside the content. The corpus currently spans 37 model configurations across 10 providers, 4 languages, 3 formats, and 5 journalistic styles per language.
-
Data Storage (MongoDB): Each fragment, along with its full provenance metadata, is stored in a shared MongoDB database, enabling reproducible filtering by any experimental variable.
-
Human Evaluation (JudgeGPT): Participants assess fragments on continuous dual-axis scales (source attribution and authenticity), producing quantitative perception data linked to generation parameters.
-
Analysis: The combined dataset supports investigations into model-specific detectability, cross-linguistic perception differences, and the role of individual differences in judgment accuracy.
Participants evaluate news fragments along three dimensions:
- Source Attribution (Human vs. Machine Generated) on a 7-point scale
- Veracity Assessment (Legitimate vs. Fake News) on a 7-point scale
- Topic Familiarity (self-reported domain knowledge) on a 7-point scale
Each response is timestamped and linked to the participant's demographic profile, enabling multi-factor analysis of perception accuracy.
After each submission, participants receive immediate feedback revealing:
- Ground truth: Whether the fragment was human- or machine-generated
- Model identity: The specific LLM that produced the content (e.g., "GPT-5.2 (OpenAI)")
- Accuracy feedback: Whether the participant's assessment was correct
- Streak tracking: Consecutive correct answers to sustain engagement
This feedback loop serves both educational and research purposes: it functions as an "inoculation" mechanism (as described in the misinformation literature) while maintaining participant motivation for longer sessions, thereby increasing data volume and quality.
Every 5 responses, participants receive a visual score card summarizing their performance:
🔍 JudgeGPT — Can You Spot AI Fakes?
🤖 AI Detection: 🟩🟩🟩🟩🟩🟩🟩🟩⬜⬜ 80%
📰 Fake News: 🟩🟩🟩🟩🟩🟩⬜⬜⬜⬜ 60%
📊 10 fragments evaluated
Can you beat my score? 👇
Participants can share results via X (Twitter), LinkedIn, WhatsApp, and Email with a single click. Each share includes a challenge link (see below), creating a viral recruitment mechanism for the study.
Participants can challenge others to evaluate the exact same set of fragments via a URL-encoded challenge link. The link encodes fragment IDs and the challenger's score, enabling:
- Direct comparison: "Someone scored 85% — can you beat them?"
- Reproducible evaluation: Both participants see identical stimuli, enabling controlled pairwise comparisons
- Organic recruitment: Each challenge link is a self-contained invitation to participate in the study
Challenge links require no additional database infrastructure; fragment identifiers are Base64url-encoded directly in the URL.
| Feature | Research Purpose |
|---|---|
| Accuracy badges (≥70%) | Reward sustained engagement; increase session length |
| Streak counter | Maintain attention quality through intrinsic motivation |
| 5-response milestone cards | Create natural sharing moments; drive organic recruitment |
| Challenge mode | Enable pairwise comparison data; viral participant acquisition |
These mechanisms are not incidental: longer sessions and higher participant counts directly improve the statistical power of the resulting dataset.
| Audience | Primary Goal | Action |
|---|---|---|
| General Public | Test your ability to spot AI-generated news and contribute to our dataset. | Participate in the Survey |
| Researchers | Understand, cite, or collaborate on this research. | Read the Paper / Contact Us / Citation |
| Developers | Contribute code, fix bugs, or suggest features. | Fork the Repo / Open an Issue |
| Educators | Use JudgeGPT in courses on media literacy or AI ethics. | Share challenge links with students for in-class exercises |
Are you an expert in AI, policy, or journalism? We are conducting a follow-up study to gather expert perspectives on AI-driven disinformation risks and mitigation strategies:
All responses are treated confidentially and reported in anonymized, aggregated form.
| Component | Technology | Purpose |
|---|---|---|
| Frontend | Streamlit (Python) | Primary survey interface |
| Backend | Python (app.py) |
Application logic, scoring, challenge encoding |
| Database | MongoDB Atlas | Fragment storage, participant data, response collection |
| Stimulus Engine | RogueGPT | Controlled fragment generation (CLI, MCP, Web UI) |
git clone https://github.com/aloth/JudgeGPT.git
cd JudgeGPT
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txtConfigure MongoDB credentials in .streamlit/secrets.toml:
[mongo]
connection = "mongodb+srv://user:pass@cluster.mongodb.net/..."Run the application:
streamlit run app.py| Parameter | Example | Description |
|---|---|---|
language |
?language=de |
Override language (en, de, fr, es) |
min_age / max_age |
?min_age=18&max_age=30 |
Filter participant age range |
challenge |
?challenge=eyJmIjpb... |
Load specific fragments in challenge mode |
msg |
?msg=0 |
Control announcement banner (0=hide) |
The data_analysis/ directory contains tools for exporting and processing collected data:
- Data Export: Participants, results, and fragments to CSV and JSON
- Automated Processing: Timestamped exports with summary reports
- Research Ready: Designed for independent use by collaborating researchers
See the data_analysis README for detailed usage instructions.
The underlying stimulus corpus and collected perception data are available on Zenodo under restricted access for academic research:
To request access, please provide a brief description of your intended use. All data are shared under terms that require ethical review and prohibit redistribution.
- Multimodal stimuli: Image and video fragments for deepfake perception research
- Cross-model expansion: Systematic coverage of new model releases via RogueGPT's CLI and MCP interfaces
- Content verification layer: Integration with fact-checking services for "inoculation" experiments
- Classroom mode: Session codes with live leaderboards for educational deployment
- Embeddable widget: Single-fragment iframe for blogs, news articles, and lecture slides
- Production deployment: Azure-hosted infrastructure for long-term scalability
The JudgeGPT Human Perception Data (504 participants, 2,438 dual-axis judgments, plus the full RogueGPT stimulus corpus) is available on Zenodo:
Access is restricted to academic research. Submit a request via the Zenodo page.
If you use JudgeGPT or its underlying research in your work, please cite:
@inproceedings{loth2026collateraleffects,
author = {Loth, Alexander and Kappes, Martin and Pahl, Marc-Oliver},
title = {Industrialized Deception: The Collateral Effects of
LLM-Generated Misinformation on Digital Ecosystems},
booktitle = {Companion Proceedings of the ACM Web Conference 2026
(WWW '26 Companion)},
year = {2026},
month = apr,
publisher = {ACM},
address = {New York, NY, USA},
location = {Dubai, United Arab Emirates},
doi = {10.1145/3774905.3795471},
url = {https://arxiv.org/abs/2601.21963},
note = {To appear. Also available as arXiv:2601.21963}
}
@inproceedings{loth2026eroding,
author = {Loth, Alexander and Kappes, Martin and Pahl, Marc-Oliver},
title = {Eroding the Truth-Default: A Causal Analysis of Human
Susceptibility to Foundation Model Hallucinations and
Disinformation in the Wild},
booktitle = {Companion Proceedings of the ACM Web Conference 2026
(WWW '26 Companion)},
year = {2026},
month = apr,
publisher = {ACM},
address = {New York, NY, USA},
location = {Dubai, United Arab Emirates},
doi = {10.1145/3774905.3795832},
url = {https://arxiv.org/abs/2601.22871},
DOI = {10.1145/3774905.3795832},
note = {To appear. Also available as arXiv:2601.22871}
}
@article{loth2024blessing,
author = {Loth, Alexander and Kappes, Martin and Pahl, Marc-Oliver},
title = {Blessing or Curse? A Survey on the Impact of Generative AI
on Fake News},
journal = {arXiv preprint arXiv:2404.03021},
year = {2024},
url = {https://arxiv.org/abs/2404.03021}
}- Fork the repository
- Create a feature branch (
git checkout -b feature/new-feature) - Commit your changes (
git commit -m 'Add new feature') - Push to the branch (
git push origin feature/new-feature) - Open a Pull Request
For substantial changes, please open an issue first.
This project is licensed under the GNU General Public License v3.0. See LICENSE for details.
This research is supported by Frankfurt University of Applied Sciences and IMT Atlantique. We thank the open-source communities behind Streamlit, MongoDB, and the broader AI research ecosystem for the infrastructure that makes this work possible.
JudgeGPT is an independent research project. The use of "GPT" in the project name follows pars pro toto convention, referring to the broader class of generative pre-trained transformer models. This project is not affiliated with or endorsed by OpenAI. All research adheres to established ethical guidelines for AI safety research.