Skip to content

.NET: Python: .NET: Make skill resource guidance rule-only - #7664

Draft
Sam (samrusani) wants to merge 1 commit into
microsoft:mainfrom
samrusani:fix/7663-skill-resource-instructions
Draft

.NET: Python: .NET: Make skill resource guidance rule-only#7664
Sam (samrusani) wants to merge 1 commit into
microsoft:mainfrom
samrusani:fix/7663-skill-resource-instructions

Conversation

@samrusani

Copy link
Copy Markdown

Motivation & Context

AgentSkillsProvider currently places fictional resource names in the default
Python and .NET instructions. A model can mistake those examples for resources
that the loaded skill actually provides and make a guaranteed-to-fail
read_skill_resource call.

This change keeps resource access grounded in the loaded skill while preserving
the existing resource discovery and tool APIs.

Fixes #7663

Description & Review Guide

  • What are the major changes?
    • Replace the concrete resource-name examples in both implementations with
      matching rule-only guidance: read only explicitly referenced resources,
      preserve the referenced path exactly, and never infer resource paths.
    • Add focused Python and .NET prompt regressions that reject the fictional
      examples.
    • Add positive controls showing that a real resource path remains advertised
      exactly when the loaded skill explicitly provides it.
  • What is the impact of these changes?
    • Default skill instructions no longer prime models with nonexistent resource
      names. Public APIs and valid resource listings are unchanged.
  • What do you want reviewers to focus on?
    • Confirm that the three resource rules remain equivalent across Python and
      .NET and that the C# assertions match the existing skills-test conventions.

The focused Python regressions fail against the base prompt and pass with this
change. The complete Python test_skills.py file and core syntax checks pass.
The .NET tests were not run locally because the .NET CLI was unavailable; draft
CI is expected to provide that validation.

Related Issue

Fixes #7663

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

Copilot AI balanced review requested due to automatic review settings August 14, 2026 11:07
@samrusani
Sam (samrusani) deployed to github-app-auth August 14, 2026 11:07 — with GitHub Actions Active
@samrusani
Sam (samrusani) deployed to github-app-auth August 14, 2026 11:07 — with GitHub Actions Active
@samrusani
Sam (samrusani) deployed to github-app-auth August 14, 2026 11:07 — with GitHub Actions Active
@agent-framework-automation agent-framework-automation Bot added python Usage: [Issues, PRs], Target: Python .NET Usage: [Issues, PRs], Target: .Net labels Aug 14, 2026
@samrusani
Sam (samrusani) deployed to github-app-auth August 14, 2026 11:08 — with GitHub Actions Active
@github-actions github-actions Bot changed the title Python: .NET: Make skill resource guidance rule-only .NET: Python: .NET: Make skill resource guidance rule-only Aug 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Removes fictional skill resource examples from Python and .NET prompts while preserving explicit resource discovery.

Changes:

  • Replaces examples with equivalent rule-only guidance.
  • Adds prompt regressions and explicit-path positive controls.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
python/packages/core/agent_framework/_skills.py Updates Python resource guidance.
python/packages/core/tests/core/test_skills.py Adds Python regression coverage.
dotnet/src/Microsoft.Agents.AI/Skills/AgentSkillsProvider.cs Updates .NET resource guidance.
dotnet/tests/Microsoft.Agents.AI.UnitTests/AgentSkills/AgentSkillsProviderTests.cs Adds .NET regression coverage.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

@KOTerra

Copy link
Copy Markdown

Thanks for preparing this fix. We also tested a broader, domain-neutral revision of the skills prompt that preserves the rule-only resource guidance while making skill selection and follow-up orchestration more explicit.

Sharing the exact tested prompt for further consideration:

You have access to skills containing domain-specific knowledge and capabilities.
Each skill provides specialized instructions, reference documents, and assets for specific tasks.

<available_skills>
{skills}
</available_skills>

When an available skill matches the user's request:
- Select skills by the outcome the user is asking for and the capability needed to produce it, not merely by surface cues or newly supplied details.
- For explanations of expected behavior, requirements, safety, or procedures, prefer a skill whose purpose is to retrieve authoritative guidance, even when the question arose from a specific situation.
- Treat questions about whether observed behavior is expected or functioning as designed as explanation requests unless answering them requires situation-specific evidence.
- Use a skill whose purpose is to gather or analyze situation-specific evidence only when the requested outcome requires that evidence. The presence of identifiers, errors, or other details does not by itself change the requested outcome.
- When several skills could apply, choose the most specific skill that directly accepts the available inputs and can produce the needed result. Do not load generic entry-point or parallel skills unless they provide a distinct capability required to answer.
- Call `load_skill` for the best matching skill before providing a substantive answer or asking routine clarifying questions.
- If that skill is already loaded in the conversation, do not load it again; follow its instructions.
- On follow-up turns, preserve the user's unresolved requested outcome. New details refine the request; they do not by themselves change the capability needed. Change skills only when the requested outcome or required capability changes, or when the loaded skill directs delegation.
- Never answer from general knowledge instead of loading a matching skill.
- Follow the loaded skill's guidance.
- If the loaded skill requires delegation to another skill or requires tool calls, complete those steps before answering.
- Before making factual claims or recommending changes, distinguish observations about the current situation from generally supported behavior. If the loaded skill or tool results do not establish the claim or recommendation, load a skill that retrieves authoritative guidance before answering.
{resource_instructions}
{script_instructions}
Only load what is needed, when it is needed.

We evaluated this with 60 uncached runs across 12 multi-turn scenarios. Compared with our previous prompt, it preserved overall quality while improving resolution and follow-up routing:

  • Overall mean: 0.598 versus 0.604
  • Valid-transcript mean: 0.652 versus 0.649
  • Resolved: 18/60 versus 15/60
  • Resource calls/errors caused by guessed paths: 0
  • Answers without loading/calling tools: 0
  • Expected-behavior follow-ups selected authoritative guidance in 5/5 runs
  • Another follow-up-sensitive scenario resolved in 10/10 runs across its full and identifier-stripped variants

The remaining failures were tied to application-specific skill content, eval rubrics and safety policy rather than generic skill orchestration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

.NET Usage: [Issues, PRs], Target: .Net python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: .NET: [Bug]: skills prompt causes reads of nonexistent example resources

3 participants