BugLens

Structured Reasoning vs. Zero-Shot Prompting for Automatic Bug Severity Classification: an empirical comparison of XML-tagged prompts and Extended Thinking in Claude against GPT-4o.

Overview

This repository contains the full experiment pipeline for the BugLens paper. It compares four prompting strategies for classifying GitHub issue severity (critical / major / minor) on the Eclipse Bug Repository:

Condition	Model	Strategy
A	GPT-4o (gpt-4o-2024-11-20)	Zero-shot
B	Claude Sonnet (claude-sonnet-4-5)	Zero-shot
C	Claude Sonnet	XML-structured prompt with <thinking> directive
D	Claude Sonnet	XML-structured prompt + Extended Thinking API

Setup

Prerequisites

• mise (manages Python 3.11 and Task runner)
• API keys for OpenAI and Anthropic

Install

git clone https://github.com/anthonyafgx/buglens.git
cd buglens

# Let mise install Python 3.11 and make it available
mise install

# Create virtual environment and install dependencies
task setup

# Copy and fill in your API keys
cp .env.example .env
# Edit .env with your actual keys

Manual setup (without mise)

python3.11 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Reproduce Results

Run all scripts in order:

python src/01_load_data.py
python src/02_condition_a_gpt.py
python src/03_condition_b_claude_zs.py
python src/04_condition_c_claude_xml.py
python src/05_condition_d_claude_extended.py
python src/06_evaluate.py

Or use the Taskfile to run the full pipeline:

task pipeline

Dry Run

To test the pipeline without spending API credits (processes only 5 issues per condition):

python src/02_condition_a_gpt.py --dry-run
python src/03_condition_b_claude_zs.py --dry-run
python src/04_condition_c_claude_xml.py --dry-run
python src/05_condition_d_claude_extended.py --dry-run

Or:

task run-dry

Docker

docker build -t buglens .
docker run -e ANTHROPIC_API_KEY=your_key -e OPENAI_API_KEY=your_key buglens

Taskfile Commands

Command	Description
task setup	Create venv and install dependencies
task data	Download / generate dataset
task run-all	Run all four conditions
task run-dry	Run all conditions in dry-run mode
task evaluate	Compute metrics and generate figures
task pipeline	Full end-to-end run
task docker-build	Build Docker image
task docker-run	Run pipeline in Docker
task mlflow	Launch MLflow UI
task clean	Remove generated artefacts

MLflow Dashboard

After running the experiments, launch the tracking UI:

task mlflow
# or
mlflow ui

Then open http://localhost:5000.