MST mode validation unnecessarily enables DSC when link bandwidth is sufficient

[coleleavitt]

Summary

MST mode validation in compoundQueryAttachMST() unconditionally attempts DSC compression when the device supports it, even when the link has sufficient bandwidth for the uncompressed mode. This causes instability through MST hubs and USB-C docks that don't handle DSC negotiation well, manifesting as spurious HPD short pulses, DPCD AUX channel failures, and intermittent display disconnection.

Environment

• Driver Version: 590.48.01 (open kernel modules)
• GPU: NVIDIA RTX PRO 4000 Blackwell Generation Laptop GPU
• Dock: Lenovo ThinkPad USB-C Dock Gen 2 (40AS) - DP 1.4 only
• Monitor: LEN S27q-10 2560x1440@60Hz via DisplayPort through dock
• OS: Gentoo Linux, kernel 6.19.0-rc8
• Compositor: niri (Wayland)

Symptoms

1. Monitor intermittently shows "input signal out of range" and disconnects
2. nvidia-modeset logs show DSC being enabled at 161 bpp (10 bpp compressed) for a mode that only needs ~5.8 Gbps (DP 1.4 HBR3 provides 25.92 Gbps)
3. DPCD register 0x7 write failures
4. Spurious HPD short pulses every few seconds
5. ConnectorImpl2x selected based on GPU capability (DP 2.0), not sink capability (dock is DP 1.4)

Root Cause Analysis

The bug is in src/common/displayport/src/dp_connectorimpl.cpp in compoundQueryAttachMST() at line 1396:

if (compoundQueryAttachMSTIsDscPossible(target, modesetParams, pDscParams))
{
    // Immediately enters DSC path without checking if DSC is NEEDED
    unsigned int forceDscBitsPerPixelX16 = pDscParams->bitsPerPixelX16;
    result = compoundQueryAttachMSTDsc(...);
    ...
}

The SST path (compoundQueryAttachSST) correctly checks bandwidth BEFORE enabling DSC:

(pDscParams->forceDsc == DSC_FORCE_ENABLE) ||    // DD forced DSC
(modesetParams.modesetInfo.mode == DSC_DUAL) ||   // DD wants 2Head1OR
(!this->willLinkSupportModeSST(lc, ...))           // Mode NOT possible without DSC

SST only enables DSC when the mode doesn't fit. MST should do the same.

Additional Observations

1. ConnectorImpl2x is selected at createConnector() (line 8016) based solely on dpParams.dpVersionsSupported (GPU's DP 2.0 capability) BEFORE the dock is even discovered
2. MST DSC BPP is hardcoded to 10 bpp (PREDEFINED_DSC_MST_BPPX16 = 160 at dp_deviceimpl.h:41) regardless of actual bandwidth requirements
3. The 128b/132b signaling path (DP 2.x) is attempted through a DP 1.4 dock, causing negotiation failures

Proposed Fix

Add a PBN pre-check before entering the DSC path in compoundQueryAttachMST(): if the uncompressed mode fits within available local link PBN, skip DSC and proceed directly to compoundQueryAttachMSTGeneric(). This mirrors the SST pattern without introducing state management issues (PBN check is side-effect-free).

I have a working patch that implements this fix - see linked PR.

Related Issues

• Driver 575, 580, 590: second external display missing and NvKmsKapiDisplay errors #879 - Second external display missing via USB-C hub (575/580/590)
• [Regression] 580.105.08: DisplayPort monitor limited to 1920x1080 instead of native 2560x1440 #960 - DisplayPort monitor limited to 1920x1080 instead of native 2560x1440 (Lenovo ThinkVision)
• Blackwell cannot do 7680x2160p@240hz via DP2.1 on Linux #816 - Blackwell cannot do high refresh via DP2.1
• RTX 5090 FE doesn't support 480Hz refresh rate for PG27AQDP with DP 1.4 (DSC) #851 - DSC not enabling properly for high refresh rates via DP 1.4
• [REGRESSION] [535.54.03] The entire screen is frequently flickering #511 - Screen flickering regression

nvidia-modeset Debug Log Excerpt

[  357.639052] nvidia-modeset: NVRM: DPLIB: DP-6: pxlBytes 8294400 | pxlStreams 8294400 | hblankBytes 1263800 | vblankBytes 1184832 | targetBpp 161 | targetBppVrr 0
[  357.639054] nvidia-modeset: NVRM: DPLIB: DP-6: compoundQueryAttach: bEnableDsc is true | bEnableFEC is true

The driver computes targetBpp 161 (10 bpp DSC) for a 2560x1440@60Hz mode that fits trivially in DP 1.4 HBR3 uncompressed.