Skip to content

Add dotnet test --list-tests json via SDK-side rendering#55299

Open
Evangelink wants to merge 4 commits into
mainfrom
dev/amauryleve/improved-tribble
Open

Add dotnet test --list-tests json via SDK-side rendering#55299
Evangelink wants to merge 4 commits into
mainfrom
dev/amauryleve/improved-tribble

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Fixes #49754.

Approach

This implements dotnet test --list-tests json by having the SDK render the JSON itself from the discovered-test data it already receives over the dotnettestcli IPC protocol — rather than forwarding the flag to the test host.

This supersedes #54490 (now closed). That PR forwarded --list-tests json to MTP's TerminalOutputDevice, which early-returns when _isServerMode is true — and the SDK always sets --server dotnettestcli. So the host never emits JSON through dotnet test, and it would have required a separate testfx fix plus risked stdout interleaving.

The key realization: the MTP IPC wire contract (vendored from microsoft/testfx, already on main via cd590b6b4b) already carries the full discovered-test payload — Uid, DisplayName, FilePath, LineNumber, Namespace, TypeName, MethodName, ParameterTypeFullNames, Traits. In --list-tests mode the SDK handshakes as Discover and receives these messages today; it was simply dropping 5 of the 9 fields before rendering text. So no testfx change and no protocol break are needed — just SDK-side plumbing and rendering.

JSON shape

A versioned envelope grouped by test container (assembly + TFM + architecture), preserving every field the wire contract provides:

{
  "version": "1.0",
  "testContainers": [
    {
      "assemblyPath": "...\\TestProject.dll",
      "targetFramework": "net11.0",
      "architecture": "x64",
      "tests": [
        {
          "uid": "...",
          "displayName": "...",
          "namespace": "My.Ns",
          "typeName": "MyClass",
          "methodName": "MyMethod",
          "parameterTypeFullNames": ["System.Int32"],
          "traits": [ { "key": "Category", "value": "Fast" } ],
          "filePath": "...",
          "lineNumber": 42
        }
      ]
    }
  ]
}

Grouping by container (not by class) preserves the module/TFM dimension a multi-project dotnet test spans, keeps each entry truthful to the underlying test node (unique Uid), and lets consumers group by namespace+typeName themselves. The envelope is versioned so the shape can evolve.

Changes

  • CLI definition: --list-tests arity ZeroZeroOrOne, restricted to text/json via AcceptOnlyFromAmong, HelpName = text|json. Updated CmdListTestsDescription and regenerated all .xlf files (/t:UpdateXlf).
  • TestListFormat enum + TestOptions.ListTestsFormat; read from the option value (bare --list-tests still defaults to text).
  • MicrosoftTestingPlatformTestCommand: for json, force NoAnsi + no progress + no assembly banners so stdout contains only the JSON document.
  • TestApplicationHandler.OnDiscoveredTestsReceived: forward all 9 fields (was dropping namespace/typeName/methodName/parameterTypeFullNames/traits).
  • TestProgressState: DiscoveredTestNames now holds a DiscoveredTestInfo record with all fields.
  • TerminalTestReporter: renders the JSON via Utf8JsonWriter (trim/AOT-safe, no reflection); text discovery output is unchanged.
  • The bare --list-tests token is still forwarded to the host unchanged — the format selection stays SDK-local.

Testing

  • New unit test asserting the JSON shape (TerminalTestReporterTests).
  • New CLI parser tests (text/json accepted, bare --list-tests = null/text, invalid values rejected).
  • New end-to-end discovery integration test (--list-tests json).
  • Updated TestProgressStateTests / TestApplicationHandlerTests for the widened signatures, and the MTP help snapshot.
  • Verified end-to-end against the built redist SDK: stdout is exactly the JSON document + newline, nothing else.

For frameworks that don't set TestMethodIdentifierProperty, the rich fields are simply null/empty (each test still carries its uid/displayName).

The MTP IPC wire contract already delivers the full discovered-test payload (uid, displayName, filePath, lineNumber, namespace, typeName, methodName, parameterTypeFullNames, traits) over the 'dotnet test' pipe. In --list-tests mode the SDK handshakes as Discover and receives these messages, but previously dropped all but four fields before rendering text.

This plumbs an optional text|json value through --list-tests and, for json, renders a versioned, container-grouped JSON document (assembly + TFM + architecture -> tests) directly from the messages the SDK already receives. No testfx change and no protocol break: the bare --list-tests token is still forwarded to the host, and the format stays SDK-local. For json, ANSI/progress/assembly banners are suppressed so stdout contains only the JSON document.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 537abc96-3b4d-450f-883e-643ba843dfd1
Copilot AI review requested due to automatic review settings July 15, 2026 14:26
@Evangelink
Evangelink requested a review from a team as a code owner July 15, 2026 14:26
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds support for dotnet test --list-tests json for the Microsoft.Testing.Platform-based dotnet test flow by rendering JSON in the SDK from the discovered-test payload already received over the existing dotnettestcli IPC protocol (no protocol changes required).

Changes:

  • Extends --list-tests to optionally accept text|json, and plumbs the selected format through TestOptions.
  • Preserves the full discovered-test payload (namespace/type/method/params/traits/location) and renders a versioned, container-grouped JSON document via Utf8JsonWriter.
  • Adds/updates unit, parser, snapshot, and end-to-end tests; updates localized strings for the revised help text.
Show a summary per file
File Description
test/dotnet.Tests/CommandTests/Test/TestProgressStateTests.cs Updates discovery-state tests for DiscoveredTestInfo (including Uid rename).
test/dotnet.Tests/CommandTests/Test/TestCommandParserTests.cs Adds parser tests for --list-tests accepting `text
test/dotnet.Tests/CommandTests/Test/TestApplicationHandlerTests.cs Updates TestOptions construction to include ListTestsFormat.
test/dotnet.Tests/CommandTests/Test/TerminalTestReporterTests.cs Adds unit test asserting the JSON envelope/container/test shape.
test/dotnet.Tests/CommandTests/Test/snapshots/MTPHelpSnapshotTests.VerifyMTPHelpOutput.verified.txt Updates help snapshot for `--list-tests <text
test/dotnet.Tests/CommandTests/Test/GivenDotnetTestBuildsAndDiscoversTests.cs Adds end-to-end test validating JSON discovery output shape/content.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.zh-Hant.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.zh-Hans.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.tr.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.ru.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.pt-BR.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.pl.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.ko.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.ja.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.it.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.fr.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.es.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.de.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/xlf/CommandDefinitionStrings.cs.xlf Updates localized resource entry for revised --list-tests description.
src/Cli/Microsoft.DotNet.Cli.Definitions/Commands/Test/TestCommandDefinition.MicrosoftTestingPlatform.cs Changes --list-tests arity to ZeroOrOne, constrains values to `text
src/Cli/Microsoft.DotNet.Cli.Definitions/CommandDefinitionStrings.resx Updates CmdListTestsDescription to document optional `text
src/Cli/dotnet/Commands/Test/MTP/TestApplicationHandler.cs Forwards the full discovered-test payload into DiscoveredTestInfo.
src/Cli/dotnet/Commands/Test/MTP/Terminal/TestProgressState.cs Stores rich discovery nodes via DiscoveredTestInfo instead of a reduced tuple.
src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporterOptions.cs Adds ListTestsFormat option to control discovery rendering.
src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporter.cs Adds JSON discovery rendering and routes discovery summary based on ListTestsFormat.
src/Cli/dotnet/Commands/Test/MTP/Options.cs Introduces TestListFormat and extends TestOptions with ListTestsFormat.
src/Cli/dotnet/Commands/Test/MTP/MicrosoftTestingPlatformTestCommand.cs Derives list-tests format from parse result and suppresses non-JSON terminal features during JSON discovery.

Copilot's findings

  • Files reviewed: 27/27 changed files
  • Comments generated: 0

@Evangelink

Copy link
Copy Markdown
Member Author

CI triage: the 3 red legs are pre-existing infra failures, unrelated to this change

I investigated the failing pipeline (build 1510506). All three failing legs reproduce identically on the latest main build (1509939), which this branch is already based on top of:

Failing leg Root cause Pre-existing?
Build AoT: windows (arm64) LNK1322: cannot avoid potential ARM hazard (Cortex-A53 MPCore processor bug #843419) during the NativeAOT win-x64 → win-arm64 cross-link of dotnet-aot.exe. A toolchain/linker (ILC / MSVC arm64 linker) failure, no managed code involved. Yes — tracked by #55298 (cc @dotnet/illink), same failure & duration on main.
Build TestBuild: linux (x64) Run NativeAOT CLI Tests"Zero tests ran" (exit code 8). Yes — the NativeAOT CLI test runner added in #54719 has reported zero tests since it was introduced (green at 7ee70d13 → red starting at 1c430679 which added it). Same failure on main.
Build TestBuild: windows (x64) Same "Zero tests ran" as above. Yes — same as above.

None of these touch the dotnet test --list-tests json code paths in this PR. This change is validated locally: 113 targeted tests pass (JSON unit test, CLI parser tests, discovery integration test, updated help snapshot), and a real end-to-end dotnet test --list-tests json run against the built redist SDK emits exactly the JSON document on stdout.

These infra legs are owned by the NativeAOT CI work (#55298 and the #54719 runner) and are out of scope for this feature PR.

Evangelink and others added 3 commits July 16, 2026 19:31
Resolve TerminalTestReporter conflicts by preserving both SDK-side JSON discovery rendering and main's output-truncation behavior and tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 537abc96-3b4d-450f-883e-643ba843dfd1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow dotnet test --list-tests json

2 participants