Benchmark CLI follow-ups: clarify --backend help and include evaluator config in eval cache key

[GopalakrishnanN]

Describe your changes

Follow-ups to #2420 addressing the remaining (non-blocking) review comments.

olive benchmark --backend help text — Reworded to clarify it applies to lm-eval evaluation (not just "ONNX model evaluation"): ort/ortgenai require ONNX input, ortgenai additionally requires GenAI-packaged model assets (e.g. genai_config.json), and auto infers the backend from the model type.

Evaluation cache key now includes the evaluator config — Engine._evaluate_model previously keyed the evaluation cache on model_id + accelerator only, so re-running the same model with a different --backend (i.e. model_class), or different lm-eval tasks/limit/batch_size/metrics, would silently reuse a stale cached result. The key now folds in hash_dict(evaluator_config.to_json()), so any change to the evaluation parameters produces a distinct cache entry. The key is internal to _evaluate_model (only _load_evaluation/_cache_evaluation consume it); footprints still record the raw model_id.

Tests — Added model_class-absent backward-compat assertions to test_benchmark_command_hfmodel/test_benchmark_command_onnxmodel, and a new test_evaluate_model_cache_key_includes_evaluator_config engine test (same config → cache hit; different config → cache miss).

Checklist before requesting a review

• Add unit tests for this change.
• Make sure all tests can pass.
• Update documents if necessary.
• Lint and apply fixes to your code by running lintrunner -a
• Is this a user-facing change? If yes, give a description of this change to be included in the release notes.
  • olive benchmark no longer reuses a cached evaluation result when re-running the same model with a different --backend or different lm-eval parameters; the --backend help text was also clarified.

(Optional) Issue link

Follow-up to #2420.

[Copilot]

Pull request overview

This PR refines the olive benchmark CLI UX and fixes a correctness issue in Engine evaluation caching by ensuring cached evaluation results are keyed by the evaluator configuration (preventing stale cache reuse when evaluation parameters change).

Changes:

• Clarifies olive benchmark --backend help text to describe lm-eval backend selection and requirements.
• Updates Engine._evaluate_model evaluation cache key to incorporate a hash of evaluator_config, preventing incorrect cache hits across differing eval settings.
• Adds/updates tests to assert backward-compatible benchmark config output (no model_class when --backend auto) and to validate cache hit/miss behavior based on evaluator config changes.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
olive/engine/engine.py	Includes evaluator-config hash in evaluation cache key to avoid stale cached evaluation reuse.
olive/cli/benchmark.py	Rewords --backend help text to clarify lm-eval scope and ONNX/ORTGenAI requirements.
test/engine/test_engine.py	Adds a unit test verifying evaluation cache hit/miss depends on evaluator config changes.
test/cli/test_cli.py	Adds assertions ensuring model_class is absent when backend is default/auto for HF and ONNX benchmark configs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[jambayk]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).