VisPath

Official code and data release for VisPath: Visual-Intent-Guided Path Reasoning for Multimodal Knowledge Graph Question Answering.

VisPath is a training-free framework for multimodal multi-hop knowledge graph question answering (MM-KGQA). Instead of using the image only once for entity grounding, VisPath keeps visual evidence involved throughout the reasoning process:

1. Visual- and Graph-aware Starting Entity Identification — grounds the query image to a starting entity that is both visually consistent and structurally useful for reasoning;
2. Intent-Guided Reasoning Path Discovery — regenerates a hop-specific visual intent at every hop to guide edge expansion;
3. Reasoning-Chain Pruning and Selection — scores and prunes candidate paths as complete reasoning chains;
4. Visual-Grounded Question Answering — summarizes retained paths against the image and verifies answerability before answering.

We also release VisPath-Bench, a 2,000-question benchmark requiring 2–4 hop reasoning over a Wikidata5M-based multimodal KG, where the starting entity cannot be read off the question text and must be grounded from the image.

Repository layout

vispath/               VisPath pipeline (method code)
  pipeline.py          end-to-end pipeline (5 stages)
  topic_link.py        visual- and graph-aware starting entity identification
  search.py            intent-guided path discovery + reasoning-chain pruning
  answer.py            visual-grounded answering and terminal selection
  prompts.py           all prompt templates
  run_eval.py          CLI runner: jsonl in -> predictions + usage logs out
  eval_strict.py       multi-axis evaluator (SBERT@k, EM, alias-EM, QID accuracy)
  vendor/              bundled KG/LLM helpers (KG backend, entity retriever, LLM client)
baselines/embedkgqa/   EmbedKGQA baseline
benchmark/             VisPath-Bench construction pipeline + relation whitelist
scripts/               scoring utilities (uniform LLM-extract + SBERT scorer)
data/
  vispath_bench/       vispath_bench.jsonl (2,000 questions; images via HuggingFace)
  eval_subsets/        processed evaluation subsets (ReasonVQA, MMhops, E-VQA)
docs/REPRODUCE.md      per-table reproduction commands

Installation

pip install -r requirements.txt

Model endpoints are configured through environment variables (OpenAI-compatible APIs):

export TEXT_LLM_BASE_URL=...      # text LLM endpoint (paper: GPT-4o)
export TEXT_LLM_API_KEY=...
export TEXT_LLM_MODEL=gpt-4o
export VLM_BASE_URL=...           # VLM endpoint (paper: Qwen2.5-VL-72B, self-hosted)
export VLM_API_KEY=...
export VLM_MODEL=qwen2.5-vl-72b

Knowledge graph setup

VisPath reasons over Wikidata5M (21.4M triples) with entity images attached from the Wikidata P18 property (1,621,245 image-carrying entities, 33.7% coverage in our build).

1. Download Wikidata5M from its official release (wikidata5m_entity.txt, wikidata5m_relation.txt, wikidata5m_all_triplet.txt).
2. Build the local KG index and the entity-name ANN index:

python benchmark/build_kg_index.py          # adjacency indices (~1.2 GB)
python benchmark/build_entity_embeddings.py # BGE/FAISS entity-name index

3. (Optional) Attach P18 images:

python benchmark/fetch_p18_images.py

Data

• VisPath-Bench: data/vispath_bench/vispath_bench.jsonl (2,000 questions: 500 2-hop / 1,200 3-hop / 300 4-hop; 997 unique topic entities). Images and the full tabular view (image + question + gold path per row) are distributed via the HuggingFace dataset Apenglow/VisPath-Bench:

  from datasets import load_dataset
  ds = load_dataset("Apenglow/VisPath-Bench", split="train")
  ds[0]["image"], ds[0]["question"], ds[0]["gold_path_readable"]

  Three questions whose topic image is no longer served by Wikimedia are marked image_path: null; all reported results use the 1,997 image-available questions.

• Evaluation subsets: data/eval_subsets/ contains the processed ReasonVQA (1,951), MMhops (1,914), and E-VQA (740) files used in the paper, re-grounded onto Wikidata5M. Source images come from the original datasets (Visual Genome / Wikipedia / iNat / GLDv2); see data/README.md.

• Schema and construction details: data/README.md and benchmark/README.md.

Reproduce the paper tables

See docs/REPRODUCE.md for per-table commands. Example (main results on VisPath-Bench):

python vispath/run_eval.py \
  --dataset data/vispath_bench/vispath_bench.jsonl \
  --index-dir <path-to-kg-index> \
  --retriever-emb-dir <path-to-entity-embeddings> \
  --predictions-out predictions/vispath_bench_gpt4o.jsonl

python vispath/eval_strict.py \
  --predictions predictions/vispath_bench_gpt4o.jsonl \
  --gold data/vispath_bench/vispath_bench.jsonl

Availability statement (for the paper)

Code, the VisPath-Bench benchmark, and the processed evaluation subsets are publicly available at https://github.com/W-RMSL/VisPath (code) and https://huggingface.co/datasets/Apenglow/VisPath-Bench (benchmark images and data).

Contact

Open a GitHub issue for questions.

License

Code is released under the MIT License (see LICENSE). VisPath-Bench is released under CC-BY-4.0. Wikidata5M and the source VQA datasets remain under their original licenses; images are redistributed under their Wikimedia/Wikipedia/Visual Genome/iNat/GLDv2 terms.