Best practice for smoke testing generated code after emitter move to typespec-azure

[JialinHuang803]

Context

When @azure-tools/typespec-ts lived in Azure/autorest.typescript, we had a sibling package typespec-test that served as a smoke test: it ran the emitter against a large suite of TypeSpec specs and committed the full generated TypeScript output into the repo. This gave us several practical benefits during day-to-day development:

• Diffable generated code — any emitter change produced a visible diff in PRs, making regressions and unintended changes obvious at review time.
• Easy local debugging — contributors could open the generated client code directly, set breakpoints, inspect output, and iterate on the emitter quickly.
• End-to-end coverage — caught build/compile/format issues in generated code that unit tests and Spector tests don't surface.

After the emitter moved to Azure/typespec-azure, typespec-test was not moved along with it because of the size of the committed generated code. As a result, we've lost the convenient "see the diff, debug the output" workflow in the new repo.

Discussion

I'd like to hear from other language emitter owners (Python, Java, Go, etc.) on how they handle this:

• Do you keep a smoke-test-style fixture repo with committed generated code? Where does it live?
• If not, how do you review the impact of an emitter change on real-world output before merging?
• How do contributors locally debug generated code against an in-development emitter?
• Are there shared tools or conventions in typespec-azure we could adopt instead of each language reinventing one?

Goal

Agree on a best practice for TypeScript (and potentially other languages) so that:

1. PR reviewers can still see the impact of emitter changes on generated code.
2. Contributors retain a fast inner-loop for debugging generated output.
3. We don't bloat the main repo with megabytes of generated code.

[lirenhe]

It would be good that we could come up with an approach that could apply to different languages, cc @iscai-msft

[msyyc]

Python stores generated code in sdk repo and we developed a github action to see the code diff for emitter PR: https://github.com/Azure/azure-sdk-for-python/blob/19919227d756612e1d3867d8f85fd8fbcad2a476/eng/tools/azure-sdk-tools/emitter/generated/README.md

[timotheeguerin]

I will also reiterate that smoke test have been use way too much as unit test in our emitters. I believe strongly that if you have a failure that happens in a smoke test it does mean you are missing upstream testing.

[jhendrixMSFT]

I will also reiterate that smoke test have been use way too much as unit test in our emitters.

I agree with the sentiment. However, I think this was largely due to the (maybe incorrect) assumption that all emitters must always build the corpus of tests (autorest.testserver/swagger, now all the Spector tsp files) which makes a lot of unit testing largely redundant (although, I do think unit tests could help cover scenarios not present in the corpus to a certain degree).

Of course, unit tests will never answer the questions "does the code compile and is it lint free", both of which are requirements before releasing an updated emitter. So, while I agree that unit tests would be complimentary, they can't replace validation by a language's tool chain.

And finally, there's always the question "does the code work". I mention this one last as, at least for Go/Rust, it largely just does so I don't worry about it as much. The only exceptions (in my language space) are LROs and XML as these tend to be complex (XML with wrapped arrays with elements that have alternate names, ugh).