(release/25.0) security: add integer overflow checks in allocation calculations

[metux]

Backport of #3578 (master)

• fb/fbpixmap.c: check widthbpp and heightpaddedWidth overflows
• hw/xfree86/common/xf86fbman.c: check length+gran-1 overflow, safe h computation, h*w overflow
• hw/xfree86/drivers/modesetting/drmmode_display.c: validate multiplications before allocating shadow buffers
• hw/xquartz/xpr/xprScreen.c: check displayCount * sizeof overflow
• hw/xnest/Color.c: check ncolors * sizeof(uint32_t) overflow

These prevent potential heap overflows due to wrapped sizes.

Signed-off-by: Enrico Weigelt, metux IT consult info@metux.net

[metux]

🤖 Automated review — generated by Starfleet ship Discovery on behalf of @metux. Not a human review.

Thorough security hardening — no objections.

What it does: adds integer overflow checks to allocation size calculations across 5 files/subsystems, preventing heap overflows from wrapped sizes:

• fb/fbpixmap.c: width*bpp, paddedWidth calculation, height*paddedWidth, datasize+adjust — all checked against INT_MAX/SIZE_MAX before the multiply.
• xf86fbman.c: length+gran-1 addition overflow, safe h = ceil(length/pitch) via (length-1)/pitch+1 to avoid (length+pitch-1) overflow, and h*w overflow check.
• drmmode_display.c: displayWidth*virtualY*cpp shadow buffer allocation — size_t intermediates, zero-dimension guard, UINT32_MAX fit check. Duplicated for shadow_enable2 (acceptable for clarity, could be a helper in a follow-up).
• xnest/Color.c: ncolors*sizeof(uint32_t) overflow + missing malloc NULL check (the original had none — bonus fix).
• xprScreen.c: displayCount*sizeof(CGDirectDisplayID) overflow before FatalError.

Patterns are correct: INT_MAX / x division-before-multiply, size_t widening, proper goto fail / return FALSE / FatalError per subsystem context.

ABI impact: none. Internal allocation guards, no struct or export changes.

Backport candidate: yes — heap overflow prevention via integer overflow. Security hardening; all three release lines should carry it. The fixes are pure guards (no behavior change on valid inputs), so cherry-pick risk is minimal.

Minor note: drmmode_display.c duplicates the overflow-check pattern twice. A shared helper could reduce repetition, but this is a style preference and does not affect correctness.

Verdict: passes review.

[metux]

@X11Libre/dev ping