Add dotnet test --list-tests json via SDK-side rendering#55299
Add dotnet test --list-tests json via SDK-side rendering#55299Evangelink wants to merge 4 commits into
dotnet test --list-tests json via SDK-side rendering#55299Conversation
The MTP IPC wire contract already delivers the full discovered-test payload (uid, displayName, filePath, lineNumber, namespace, typeName, methodName, parameterTypeFullNames, traits) over the 'dotnet test' pipe. In --list-tests mode the SDK handshakes as Discover and receives these messages, but previously dropped all but four fields before rendering text. This plumbs an optional text|json value through --list-tests and, for json, renders a versioned, container-grouped JSON document (assembly + TFM + architecture -> tests) directly from the messages the SDK already receives. No testfx change and no protocol break: the bare --list-tests token is still forwarded to the host, and the format stays SDK-local. For json, ANSI/progress/assembly banners are suppressed so stdout contains only the JSON document. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 537abc96-3b4d-450f-883e-643ba843dfd1
|
Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Adds support for dotnet test --list-tests json for the Microsoft.Testing.Platform-based dotnet test flow by rendering JSON in the SDK from the discovered-test payload already received over the existing dotnettestcli IPC protocol (no protocol changes required).
Changes:
- Extends
--list-teststo optionally accepttext|json, and plumbs the selected format throughTestOptions. - Preserves the full discovered-test payload (namespace/type/method/params/traits/location) and renders a versioned, container-grouped JSON document via
Utf8JsonWriter. - Adds/updates unit, parser, snapshot, and end-to-end tests; updates localized strings for the revised help text.
Show a summary per file
| File | Description |
|---|---|
| test/dotnet.Tests/CommandTests/Test/TestProgressStateTests.cs | Updates discovery-state tests for DiscoveredTestInfo (including Uid rename). |
| test/dotnet.Tests/CommandTests/Test/TestCommandParserTests.cs | Adds parser tests for --list-tests accepting `text |
| test/dotnet.Tests/CommandTests/Test/TestApplicationHandlerTests.cs | Updates TestOptions construction to include ListTestsFormat. |
| test/dotnet.Tests/CommandTests/Test/TerminalTestReporterTests.cs | Adds unit test asserting the JSON envelope/container/test shape. |
| test/dotnet.Tests/CommandTests/Test/snapshots/MTPHelpSnapshotTests.VerifyMTPHelpOutput.verified.txt | Updates help snapshot for `--list-tests <text |
| test/dotnet.Tests/CommandTests/Test/GivenDotnetTestBuildsAndDiscoversTests.cs | Adds end-to-end test validating JSON discovery output shape/content. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.zh-Hant.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.zh-Hans.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.tr.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.ru.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.pt-BR.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.pl.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.ko.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.ja.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.it.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.fr.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.es.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.de.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.cs.xlf | Updates localized resource entry for revised --list-tests description. |
| src/Cli/Microsoft.DotNet.Cli.Definitions/Commands/Test/TestCommandDefinition.MicrosoftTestingPlatform.cs | Changes --list-tests arity to ZeroOrOne, constrains values to `text |
| src/Cli/Microsoft.DotNet.Cli.Definitions/CommandDefinitionStrings.resx | Updates CmdListTestsDescription to document optional `text |
| src/Cli/dotnet/Commands/Test/MTP/TestApplicationHandler.cs | Forwards the full discovered-test payload into DiscoveredTestInfo. |
| src/Cli/dotnet/Commands/Test/MTP/Terminal/TestProgressState.cs | Stores rich discovery nodes via DiscoveredTestInfo instead of a reduced tuple. |
| src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporterOptions.cs | Adds ListTestsFormat option to control discovery rendering. |
| src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporter.cs | Adds JSON discovery rendering and routes discovery summary based on ListTestsFormat. |
| src/Cli/dotnet/Commands/Test/MTP/Options.cs | Introduces TestListFormat and extends TestOptions with ListTestsFormat. |
| src/Cli/dotnet/Commands/Test/MTP/MicrosoftTestingPlatformTestCommand.cs | Derives list-tests format from parse result and suppresses non-JSON terminal features during JSON discovery. |
Copilot's findings
- Files reviewed: 27/27 changed files
- Comments generated: 0
CI triage: the 3 red legs are pre-existing infra failures, unrelated to this changeI investigated the failing pipeline (build
None of these touch the These infra legs are owned by the NativeAOT CI work (#55298 and the #54719 runner) and are out of scope for this feature PR. |
Resolve TerminalTestReporter conflicts by preserving both SDK-side JSON discovery rendering and main's output-truncation behavior and tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 537abc96-3b4d-450f-883e-643ba843dfd1
… into dev/amauryleve/improved-tribble
Fixes #49754.
Approach
This implements
dotnet test --list-tests jsonby having the SDK render the JSON itself from the discovered-test data it already receives over thedotnettestcliIPC protocol — rather than forwarding the flag to the test host.This supersedes #54490 (now closed). That PR forwarded
--list-tests jsonto MTP'sTerminalOutputDevice, which early-returns when_isServerModeis true — and the SDK always sets--server dotnettestcli. So the host never emits JSON throughdotnet test, and it would have required a separate testfx fix plus risked stdout interleaving.The key realization: the MTP IPC wire contract (vendored from microsoft/testfx, already on
mainviacd590b6b4b) already carries the full discovered-test payload —Uid,DisplayName,FilePath,LineNumber,Namespace,TypeName,MethodName,ParameterTypeFullNames,Traits. In--list-testsmode the SDK handshakes asDiscoverand receives these messages today; it was simply dropping 5 of the 9 fields before rendering text. So no testfx change and no protocol break are needed — just SDK-side plumbing and rendering.JSON shape
A versioned envelope grouped by test container (assembly + TFM + architecture), preserving every field the wire contract provides:
{ "version": "1.0", "testContainers": [ { "assemblyPath": "...\\TestProject.dll", "targetFramework": "net11.0", "architecture": "x64", "tests": [ { "uid": "...", "displayName": "...", "namespace": "My.Ns", "typeName": "MyClass", "methodName": "MyMethod", "parameterTypeFullNames": ["System.Int32"], "traits": [ { "key": "Category", "value": "Fast" } ], "filePath": "...", "lineNumber": 42 } ] } ] }Grouping by container (not by class) preserves the module/TFM dimension a multi-project
dotnet testspans, keeps each entry truthful to the underlying test node (uniqueUid), and lets consumers group bynamespace+typeNamethemselves. The envelope is versioned so the shape can evolve.Changes
--list-testsarityZero→ZeroOrOne, restricted totext/jsonviaAcceptOnlyFromAmong,HelpName=text|json. UpdatedCmdListTestsDescriptionand regenerated all.xlffiles (/t:UpdateXlf).TestListFormatenum +TestOptions.ListTestsFormat; read from the option value (bare--list-testsstill defaults to text).MicrosoftTestingPlatformTestCommand: forjson, force NoAnsi + no progress + no assembly banners so stdout contains only the JSON document.TestApplicationHandler.OnDiscoveredTestsReceived: forward all 9 fields (was dropping namespace/typeName/methodName/parameterTypeFullNames/traits).TestProgressState:DiscoveredTestNamesnow holds aDiscoveredTestInforecord with all fields.TerminalTestReporter: renders the JSON viaUtf8JsonWriter(trim/AOT-safe, no reflection); text discovery output is unchanged.--list-teststoken is still forwarded to the host unchanged — the format selection stays SDK-local.Testing
TerminalTestReporterTests).text/jsonaccepted, bare--list-tests= null/text, invalid values rejected).--list-tests json).TestProgressStateTests/TestApplicationHandlerTestsfor the widened signatures, and the MTP help snapshot.For frameworks that don't set
TestMethodIdentifierProperty, the rich fields are simplynull/empty (each test still carries itsuid/displayName).