Multiple quickstarts crashing with memory corruption (double free) and networksecurity timeouts

While triaging GCB failures, I found a recurring failure in fedora-latest-cmake-quickstart-production.

The build officially fails because networksecurity_quickstart repeatedly times out after 300 seconds:

CMake Error at /workspace/cmake/quickstart-runner.cmake:90 (message):
  Non-zero exit status (Process terminated due to timeout) running
  /workspace/cmake-out/google/cloud/networksecurity/networksecurity_quickstart

However, the raw logs show a much deeper systemic memory corruption issue. Several other quickstarts are crashing violently but showing as "Passed" because of the --repeat until-pass:3 CTest flag.

Examples from the logs:

backupdr_quickstart & alloydb_quickstart: double free or corruption (fasttop)
video_quickstart: corrupted double-linked list
servicedirectory_quickstart: malloc_consolidate(): unaligned fastbin chunk detected
gkebackup_quickstart & vpcaccess_quickstart: Segmentation fault

This looks like a regression in a core library or a dependency ABI mismatch causing heap corruption.

Failed Build Link: https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/30f1aa53-9157-4939-ba8e-a2790abc278b?project=cloud-cpp-testing-resources

Similar issue: #14411

Update: This same memory corruption issue is also crashing core unit tests.

While triaging further, I found that the gcc-oldest-ci build failed because bigtable_data_connection_test aborted with the exact same error:

[==========] 10 tests from 1 test suite ran. (412 ms total)
[  PASSED  ] 10 tests.
double free or corruption (fasttop)

This confirms the regression is not isolated to the quickstarts or the Fedora environment.

Failed Build Link: https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/6ffb59b7-2898-4f59-a617-43251f06eed0?project=cloud-cpp-testing-resources

fix: isolate bigtable test endpoints and quickstart timeout

fix(networksecurity): use global location in quickstart

PR #16496 only resolved the networksecurity timeout portion of this issue.

The underlying memory corruption (double free and malloc_consolidate) is still present. After merging the timeout fix, the quickstart-production-ci build immediately failed again, this time crashing on servicehealth_quickstart with malloc_consolidate(): unaligned fastbin chunk detected.

Hopefully #16525 resolves this issue; closing for now.

Navigation Menu

Repository navigation

Multiple quickstarts crashing with memory corruption (double free) and networksecurity timeouts #16475

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Multiple quickstarts crashing with memory corruption (double free) and networksecurity timeouts #16475

Description

Activity

scotthart commented on Sep 22, 2026

colinmoy commented on Sep 22, 2026

colinmoy commented on Sep 25, 2026

scotthart commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions