Init DRBG single-threadedly to fix data race#330
Merged
Conversation
smvv
force-pushed
the
init-rand
branch
4 times, most recently
from
1a4b5f9 to
78f5fa4
Compare
Member
|
I think this PR is doing multiple things, logically I think:
are separated PRs / concerns. |
Member
Author
|
ok i split the PR. This is only TSAN and the rest in #331. It's not so easy to separate the runners PR from the main PR because runners requires the main branch for CI tools. |
Member
|
Thanks! |
rustyconover
pushed a commit
to Query-farm-haybarn/haybarn-httpfs
that referenced
this pull request
May 27, 2026
Cherry-picked from duckdb/duckdb-httpfs#330. (cherry picked from commit 8d71226)
krlmlr
pushed a commit
to krlmlr/duckdb
that referenced
this pull request
Jun 21, 2026
I got a threadsan data race [warning](https://github.com/duckdb/duckdb/actions/runs/26271430486/job/77325736063?pr=22832#step:7:36) in another [PR](duckdb#22832): ``` [0/1] (0%): test/sql/copy/encryption/concurrent_encrypted_attach.test [1/1] (100%): test/sql/copy/encryption/concurrent_encrypted_attach.test =============================================================================== All tests passed (10 assertions in 1 test case) WARNING: ThreadSanitizer: data race (pid=92574) Read of size 8 at 0x721c0003c158 by thread T22 (mutexes: write M0): https://redirect.github.com/duckdb/duckdb/pull/0 bcmp <null> (unittest+0x29683e) (BuildId: 4a71c7e25393c04f8fb25aecb86e6508425b3ef1) https://redirect.github.com/duckdb/duckdb/pull/1 ossl_ht_get <null> (libduckdb.so+0x634377b) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/2 duckdb::EncryptCanary(duckdb::MainHeader&, duckdb::shared_ptr<duckdb::EncryptionState, true> const&, unsigned char const*) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:118:21 (libduckdb.so+0x4846ca8) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/3 duckdb::SingleFileBlockManager::StoreEncryptedCanary(duckdb::AttachedDatabase&, duckdb::MainHeader&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:456:2 (libduckdb.so+0x484b4b0) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/4 duckdb::SingleFileBlockManager::CreateNewDatabase(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:597:3 (libduckdb.so+0x4830bad) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/5 duckdb::SingleFileStorageManager::LoadDatabase(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/storage_manager.cpp:474:21 (libduckdb.so+0x482f845) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/6 duckdb::StorageManager::Initialize(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/storage_manager.cpp:356:2 (libduckdb.so+0x482de39) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) ... Previous write of size 8 at 0x721c0003c158 by thread T18 (mutexes: write M1, write M2, write M3): https://redirect.github.com/duckdb/duckdb/pull/0 memcpy <null> (unittest+0x2857b2) (BuildId: 4a71c7e25393c04f8fb25aecb86e6508425b3ef1) https://redirect.github.com/duckdb/duckdb/pull/1 ossl_ht_insert <null> (libduckdb.so+0x6342ab1) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/2 duckdb::EncryptCanary(duckdb::MainHeader&, duckdb::shared_ptr<duckdb::EncryptionState, true> const&, unsigned char const*) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:118:21 (libduckdb.so+0x4846ca8) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/3 duckdb::SingleFileBlockManager::StoreEncryptedCanary(duckdb::AttachedDatabase&, duckdb::MainHeader&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:456:2 (libduckdb.so+0x484b4b0) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/4 duckdb::SingleFileBlockManager::CreateNewDatabase(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:597:3 (libduckdb.so+0x4830bad) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/5 duckdb::SingleFileStorageManager::LoadDatabase(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/storage_manager.cpp:474:21 (libduckdb.so+0x482f845) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/6 duckdb::StorageManager::Initialize(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/storage_manager.cpp:356:2 (libduckdb.so+0x482de39) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/7 duckdb::AttachedDatabase::Initialize(duckdb::optional_ptr<duckdb::ClientContext, true>) /home/runner/work/duckdb/duckdb/src/main/attached_database.cpp:242:12 (libduckdb.so+0x43f099f) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/8 duckdb::DatabaseManager::AttachDatabase(duckdb::ClientContext&, duckdb::AttachInfo&, duckdb::AttachOptions&) /home/runner/work/duckdb/duckdb/src/main/database_manager.cpp:204:16 (libduckdb.so+0x43f099f) ``` TSAN found two threads entering the canary encryption path at the same time (`EncryptCanary` / `DecryptCanary` during parallel ATTACH/open). Inside that path, OpenSSL internals performed concurrent read/write access to the same heap-backed crypto state (reported around `ossl_ht_insert` / `memcpy` / `bcmp`) and that caused the data race warning. ~~Adding a lock to `EncryptCanary` and `DecryptCanary` avoids concurrent read/writes to the internal crypto state.~~ I added Mark/Claude's suggestion to `httpfs` and merged that [PR](duckdb/duckdb-httpfs#330). So, this PR bumps `httpfs` and that should resolve the threadsan data race. Upstream-commit: duckdb@bd75aa4
krlmlr
pushed a commit
to krlmlr/duckdb
that referenced
this pull request
Jun 21, 2026
I got a threadsan data race [warning](https://github.com/duckdb/duckdb/actions/runs/26271430486/job/77325736063?pr=22832#step:7:36) in another [PR](duckdb#22832): ``` [0/1] (0%): test/sql/copy/encryption/concurrent_encrypted_attach.test [1/1] (100%): test/sql/copy/encryption/concurrent_encrypted_attach.test =============================================================================== All tests passed (10 assertions in 1 test case) WARNING: ThreadSanitizer: data race (pid=92574) Read of size 8 at 0x721c0003c158 by thread T22 (mutexes: write M0): https://redirect.github.com/duckdb/duckdb/pull/0 bcmp <null> (unittest+0x29683e) (BuildId: 4a71c7e25393c04f8fb25aecb86e6508425b3ef1) https://redirect.github.com/duckdb/duckdb/pull/1 ossl_ht_get <null> (libduckdb.so+0x634377b) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/2 duckdb::EncryptCanary(duckdb::MainHeader&, duckdb::shared_ptr<duckdb::EncryptionState, true> const&, unsigned char const*) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:118:21 (libduckdb.so+0x4846ca8) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/3 duckdb::SingleFileBlockManager::StoreEncryptedCanary(duckdb::AttachedDatabase&, duckdb::MainHeader&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:456:2 (libduckdb.so+0x484b4b0) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/4 duckdb::SingleFileBlockManager::CreateNewDatabase(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:597:3 (libduckdb.so+0x4830bad) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/5 duckdb::SingleFileStorageManager::LoadDatabase(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/storage_manager.cpp:474:21 (libduckdb.so+0x482f845) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/6 duckdb::StorageManager::Initialize(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/storage_manager.cpp:356:2 (libduckdb.so+0x482de39) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) ... Previous write of size 8 at 0x721c0003c158 by thread T18 (mutexes: write M1, write M2, write M3): https://redirect.github.com/duckdb/duckdb/pull/0 memcpy <null> (unittest+0x2857b2) (BuildId: 4a71c7e25393c04f8fb25aecb86e6508425b3ef1) https://redirect.github.com/duckdb/duckdb/pull/1 ossl_ht_insert <null> (libduckdb.so+0x6342ab1) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/2 duckdb::EncryptCanary(duckdb::MainHeader&, duckdb::shared_ptr<duckdb::EncryptionState, true> const&, unsigned char const*) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:118:21 (libduckdb.so+0x4846ca8) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/3 duckdb::SingleFileBlockManager::StoreEncryptedCanary(duckdb::AttachedDatabase&, duckdb::MainHeader&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:456:2 (libduckdb.so+0x484b4b0) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/4 duckdb::SingleFileBlockManager::CreateNewDatabase(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/single_file_block_manager.cpp:597:3 (libduckdb.so+0x4830bad) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/5 duckdb::SingleFileStorageManager::LoadDatabase(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/storage_manager.cpp:474:21 (libduckdb.so+0x482f845) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/6 duckdb::StorageManager::Initialize(duckdb::QueryContext) /home/runner/work/duckdb/duckdb/src/storage/storage_manager.cpp:356:2 (libduckdb.so+0x482de39) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/7 duckdb::AttachedDatabase::Initialize(duckdb::optional_ptr<duckdb::ClientContext, true>) /home/runner/work/duckdb/duckdb/src/main/attached_database.cpp:242:12 (libduckdb.so+0x43f099f) (BuildId: 1933a57ba8cd5721b7a93b4b0bb4d66d4448190c) https://redirect.github.com/duckdb/duckdb/pull/8 duckdb::DatabaseManager::AttachDatabase(duckdb::ClientContext&, duckdb::AttachInfo&, duckdb::AttachOptions&) /home/runner/work/duckdb/duckdb/src/main/database_manager.cpp:204:16 (libduckdb.so+0x43f099f) ``` TSAN found two threads entering the canary encryption path at the same time (`EncryptCanary` / `DecryptCanary` during parallel ATTACH/open). Inside that path, OpenSSL internals performed concurrent read/write access to the same heap-backed crypto state (reported around `ossl_ht_insert` / `memcpy` / `bcmp`) and that caused the data race warning. ~~Adding a lock to `EncryptCanary` and `DecryptCanary` avoids concurrent read/writes to the internal crypto state.~~ I added Mark/Claude's suggestion to `httpfs` and merged that [PR](duckdb/duckdb-httpfs#330). So, this PR bumps `httpfs` and that should resolve the threadsan data race. Upstream-commit: duckdb@bd75aa4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.