Skip to content

feat: Add Visual Studio extension for AI edit detection#1535

Merged
svarlamov merged 40 commits into
git-ai-project:mainfrom
sanjog-gururaj:visual-studio-extension
Jun 12, 2026
Merged

feat: Add Visual Studio extension for AI edit detection#1535
svarlamov merged 40 commits into
git-ai-project:mainfrom
sanjog-gururaj:visual-studio-extension

Conversation

@sanjog-gururaj

@sanjog-gururaj sanjog-gururaj commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a Visual Studio (VSIX) extension that detects Copilot-generated code edits and calls git ai checkpoint to record AI vs human authorship, matching the existing IntelliJ and VS Code extensions.

Extension (C#/.NET, agent-support/visualstudio/):

  • Stack trace analysis on ITextBuffer.Changed to detect Copilot Chat edits (CopilotBufferUpdater.ApplyEditsAndSaveAsync)
  • before_edit / after_edit checkpoint pairs for AI edits with 300ms debounce
  • known_human checkpoint on document save with 500ms debounce
  • Binary resolution with version check (≥1.0.23), git repo root detection
  • Unit tests for pure logic (BinaryResolver, GitRepoResolver, CheckpointInput serialization, DocumentSaveListener)

Rust-side (src/mdm/agents/visual_studio.rs):

  • VisualStudioInstaller implementing HookInstaller trait
  • Windows-only VS installation discovery via vswhere.exe
  • Extension presence detection by scanning VSIX manifests
  • Process name detection (devenv)

CI (.github/workflows/test-visualstudio-extension.yml):

  • Build + test on windows-latest, path-filtered to agent-support/visualstudio/**

Detection coverage

Scenario Detected Method
Copilot Chat edits (Apply) Yes (High confidence) Microsoft.VisualStudio.Conversations.UI.Internal.Copilot* namespace
Human typing Correctly excluded No Copilot frames in stack
Inline completions (Tab) Not yet No Copilot-specific frame available (generic VS SuggestionService infrastructure)

Known limitations

  • Inline completion acceptance uses generic VS SuggestionService.AcceptSuggestionCommandHandler — not attributable to Copilot specifically. Will auto-detect if GitHub adds Copilot-specific frames in future updates.
  • CheckpointService.Current singleton has a theoretical startup race (MEF loads TextBufferListener before GitAiPackage.InitializeAsync completes). Low practical impact.
  • Automatic VSIX installation via VSIXInstaller.exe is stubbed — currently provides manual Marketplace install instructions.

Test plan

  • Human typing → no AI detection, known_human on save
  • Copilot Chat edit → agent-v1 before/after checkpoints, correct attribution in git ai log
  • Full stack trace analysis to verify no false positives (CopilotPreemptingCommandFilter is command chain infrastructure, present in ALL edits)
  • Unit tests pass (dotnet test)
  • CI workflow builds on Windows

Open in Devin Review

sanjog-gururaj and others added 21 commits June 5, 2026 15:24
C# VSIX extension that detects GitHub Copilot edits in Visual Studio
via stack trace analysis (same approach as the IntelliJ plugin) and
records attribution via git-ai checkpoint. Includes a Rust-side
VisualStudioInstaller for auto-install via git ai install-hooks.
- Fix before_edit checkpoint to use pre-change snapshot (e.Before.GetText())
  instead of post-change content, matching IntelliJ's beforeDocumentChange
- Fix case-insensitive path comparison in GitRepoResolver for Windows
- Fix vsixmanifest prerequisite version range to [17.0, 19.0)
- Replace placeholder GUIDs with real random values
- Align CheckpointService with IntelliJ's GitAiService: read stderr on
  failure, structured multi-line error logs, named timeout constants
- Add structured logging to BinaryResolver for not-found/version-too-old
- Add [SAVE] sub-prefix to DocumentSaveListener log lines
- Remove dead code: TabCompletionFilter, TrackedAgent, FormatStackTrace,
  _fileContentBeforeEdit (all unused)
- Add unit test project with 23 tests for pure logic functions
- Add CI workflow (build + test on windows-latest)
- Update DESIGN.md to reflect actual implementation and discovered
  Copilot namespace prefixes
- Create GitAiVS.sln solution file

Co-authored-by: Cursor <cursoragent@cursor.com>
…k after detection

Move the CheckpointSvc null check after stack trace analysis so buffer
change events are always logged, even before the package finishes
initializing. Add log line when TextBufferListener attaches to a buffer.

Co-authored-by: Cursor <cursoragent@cursor.com>
The GUID changes caused VS to treat the extension as a completely
different package, leaving the old extension registered alongside
the new one. Revert to original GUIDs so the deployment correctly
updates the existing extension.

Co-authored-by: Cursor <cursoragent@cursor.com>
… burst

Stop clearing _beforeEditTriggered in the debounce callback. The 5s
expiry naturally prevents re-triggering within the same Copilot edit
session. Previously, each debounce cycle would clear the flag, causing
the next character to fire a new before_edit + after_edit pair.

Co-authored-by: Cursor <cursoragent@cursor.com>
Microsoft.VisualStudio.Editor.Implementation.Copilot namespace includes
CopilotPreemptingCommandFilter which sits in the command chain for ALL
keystrokes, not just Copilot-generated text. This caused every human
edit to be attributed to Copilot. Remove it from namespace prefixes.

Remaining prefixes:
- GitHub.Copilot: Copilot extension's own assemblies
- Microsoft.VisualStudio.Copilot: VS Copilot integration
- Microsoft.VisualStudio.Conversations.UI.Internal.Copilot: chat edits

Co-authored-by: Cursor <cursoragent@cursor.com>
Temporary diagnostic to empirically discover which stack frames are
present during Copilot inline completion acceptance vs human typing.
Remove this once the correct namespace prefixes are identified.

Co-authored-by: Cursor <cursoragent@cursor.com>
Stack trace analysis complete. Confirmed:
- Human edits flow through EditorOperations.InsertText (no Copilot frames)
- Copilot Chat edits flow through CopilotBufferUpdater.ApplyEditsAndSaveAsync
- CopilotPreemptingCommandFilter is command chain infrastructure (all edits)

Current namespace prefixes correctly detect Chat edits without
false positives on human typing.

Co-authored-by: Cursor <cursoragent@cursor.com>
Add AcceptSuggestionCommandHandler namespace prefix to detect when
users accept inline ghost text suggestions (Tab key). This frame
only appears when CommitGrayTextAsync fires, never during normal
human typing (which goes through TypeCharCommandHandler instead).

Empirically verified via full stack trace analysis:
- Human typing: EditorOperations.InsertText via TypeCharCommandHandler
- Copilot Chat: CopilotBufferUpdater.ApplyEditsAndSaveAsync
- Inline accept: SuggestionSession.CommitGrayTextAsync via
  AcceptSuggestionCommandHandler.ExecuteCommand

Co-authored-by: Cursor <cursoragent@cursor.com>
SuggestionService.AcceptSuggestionCommandHandler is generic VS
infrastructure for all inline suggestion providers, not
Copilot-specific. Only detect edits we can provably attribute to
Copilot: Chat edits via CopilotBufferUpdater and any future frames
from GitHub.Copilot or Microsoft.VisualStudio.Copilot assemblies.

Co-authored-by: Cursor <cursoragent@cursor.com>
@CLAassistant

CLAassistant commented Jun 10, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

- DESIGN.md: Remove CopilotPreemptingCommandFilter from namespace
  prefixes (fires for ALL keystrokes, not Copilot-specific). Document
  inline completions as not detectable via stack trace.
- visual_studio.rs: Fix VS settings directory path format from
  "{major}_{instanceId}" to "{major}.0_{instanceId}" so extension
  detection actually finds installed extensions.
- visual_studio.rs: Support VS 2025 (18.x) in addition to VS 2022
  (17.x), matching the VSIX manifest target range [17.0, 19.0).
- CheckpointService.cs: Read stdout/stderr asynchronously before
  WaitForExit to prevent pipe buffer deadlock.
- BinaryResolver.cs: Trim input before splitting in ParseVersion
  to handle leading whitespace correctly.

Co-authored-by: Cursor <cursoragent@cursor.com>
devin-ai-integration[bot]

This comment was marked as resolved.

ReadToEnd() blocks until the process exits, making the subsequent
WaitForExit(timeout) dead code. If the spawned process hangs,
ReadToEnd() blocks indefinitely and the timeout/kill logic never fires.

Switch to ReadToEndAsync() before WaitForExit() so the pipe buffers
are consumed concurrently while the timeout remains effective. Same
pattern already used in CheckpointService.

Co-authored-by: Cursor <cursoragent@cursor.com>
devin-ai-integration[bot]

This comment was marked as resolved.

sanjog-gururaj and others added 2 commits June 10, 2026 12:25
…edit tracking

Two fixes to TextBufferListener checkpoint logic:

1. Capture after-edit content via e.After.GetText() at event time
   instead of reading buffer.CurrentSnapshot 300ms later when the
   debounce fires. Prevents human edits during the debounce window
   from being incorrectly attributed to the AI agent.

2. Clean up _beforeEditTriggered entry AFTER the after-edit checkpoint
   fires, allowing a fresh before-edit for subsequent AI edits on
   the same file. Matches IntelliJ's DocumentChangeListener.kt which
   removes beforeEditTriggered in executeAfterEditCheckpoint.

Co-authored-by: Cursor <cursoragent@cursor.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@svarlamov svarlamov merged commit a9ee975 into git-ai-project:main Jun 12, 2026
25 checks passed

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

Open in Devin Review

Comment on lines +229 to +239
.args([
"-all",
"-format",
"json",
"-property",
"installationPath",
"-property",
"installationVersion",
"-property",
"instanceId",
])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 vswhere called with multiple -property flags, but only the last one takes effect

The find_visual_studio_windows() function passes three -property flags (installationPath, installationVersion, instanceId) to vswhere.exe. However, vswhere's -property option is a single-value parameter — each subsequent -property overwrites the previous one. Only the last flag (instanceId) takes effect, so the JSON output contains only instanceId per entry.

The filter_map at lines 260-280 tries to extract all three properties (installationPath, installationVersion, instanceId), but entry.get("installationPath") returns None (because that property wasn't requested), causing every entry to be filtered out via the ? operator. This means find_visual_studio_windows() always returns an empty Vec, making the entire Visual Studio installer non-functional — it will never detect VS installations.

Correct usage of vswhere -property (from the CI workflow in this same PR)

The workflow file at .github/workflows/test-visualstudio-extension.yml:74 correctly uses a single -property flag:

$vsRoot = & $vswhere -latest -requires Microsoft.Component.MSBuild -property installationPath

The fix is to remove all -property flags so that vswhere returns all properties in its JSON output.

Suggested change
.args([
"-all",
"-format",
"json",
"-property",
"installationPath",
"-property",
"installationVersion",
"-property",
"instanceId",
])
"-all",
"-format",
"json",
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +88 to +95
foreach (var absolutePath in paths)
{
try
{
if (!File.Exists(absolutePath)) continue;
var content = File.ReadAllText(absolutePath);
dirtyFiles[absolutePath] = content;
editedPaths.Add(absolutePath);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 DocumentSaveListener sends absolute paths in known_human checkpoint dirty_files keys, inconsistent with agent-v1 relative paths

In DocumentSaveListener.ExecuteCheckpoint, the dirtyFiles dictionary keys and editedPaths list use absolute file paths (e.g. C:\Users\dev\project\src\main.cs). In contrast, TextBufferListener.SendBeforeEditIfNeeded and ScheduleAfterEditCheckpoint use GitRepoResolver.ToRelativePath() to convert to repo-relative paths (e.g. src\main.cs) for the same fields in agent-v1 checkpoints.

While the Rust known_human.rs preset uses resolve_absolute() which handles absolute paths correctly (so this doesn't cause a runtime crash), the path format inconsistency means known_human checkpoint dirty_file keys won't match the POSIX-normalized relative paths used elsewhere in git-ai's working log system (AGENTS.md states: "Paths are POSIX-normalized"). This could cause dirty-file content matching to fail when reconciling known_human checkpoints against agent-v1 checkpoints for the same files.

Suggested change
foreach (var absolutePath in paths)
{
try
{
if (!File.Exists(absolutePath)) continue;
var content = File.ReadAllText(absolutePath);
dirtyFiles[absolutePath] = content;
editedPaths.Add(absolutePath);
foreach (var absolutePath in paths)
{
try
{
if (!File.Exists(absolutePath)) continue;
var content = File.ReadAllText(absolutePath);
var relativePath = GitRepoResolver.ToRelativePath(absolutePath, workspaceRoot);
dirtyFiles[relativePath] = content;
editedPaths.Add(relativePath);
}
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants