SDSTOR-21981: make shard create/seal log-only, avoid header/footer I/O, add prod repro issue

[Hooper9973]

• Switch create_shard and seal_shard to log-only; remove data channel path
• Stop persisting shard header/footer on disk for create/seal flows
• Add production GC issue reproductions:
  • issue1: concurrent seal_shard, create_shard, and GC
  • issue2: concurrent put_blob and seal_shard can place blob into sealed shard

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 55.12821% with 35 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (dev/refactor_create_seal_shard@e1c23e1). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/lib/homestore_backend/hs_shard_manager.cpp	48.83%	18 Missing and 4 partials ⚠️
src/lib/homestore_backend/heap_chunk_selector.cpp	45.45%	6 Missing ⚠️
src/lib/homestore_backend/hs_blob_manager.cpp	71.42%	2 Missing and 2 partials ⚠️
...ib/homestore_backend/replication_state_machine.cpp	40.00%	3 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@                        Coverage Diff                        @@
##             dev/refactor_create_seal_shard     #434   +/-   ##
=================================================================
  Coverage                                  ?   53.52%           
=================================================================
  Files                                     ?       36           
  Lines                                     ?     5338           
  Branches                                  ?      671           
=================================================================
  Hits                                      ?     2857           
  Misses                                    ?     2188           
  Partials                                  ?      293           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Hooper9973]

Run the Reproduced UT on the stablev4.x branch, the result show the UT reproduced the scenario.

$ ./homestore_test_gc -csv error --executor immediate --config_path ./ --chunks_per_pg 1 --override_config hs_backend_config.enable_gc=true --override_config hs_backend_config.gc_garbage_rate_threshold=0 --override_config hs_backend_config.gc_scan_interval_sec=5 --gtest_filter=HomeObjectFixture.StalePChunkRouteAfterGC:HomeObjectFixture.StaleBlobRouteAfterSealAndGC

Port 4000 is not in use.
Port 4001 is not in use.
Port 4002 is not in use.
Note: Google Test filter = HomeObjectFixture.StalePChunkRouteAfterGC:HomeObjectFixture.StaleBlobRouteAfterSealAndGC
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from HomeObjectFixture
[ RUN      ] HomeObjectFixture.StalePChunkRouteAfterGC

Note: Google Test filter = HomeObjectFixture.StalePChunkRouteAfterGC:HomeObjectFixture.StaleBlobRouteAfterSealAndGC
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from HomeObjectFixture
[ RUN      ] HomeObjectFixture.StalePChunkRouteAfterGC

Note: Google Test filter = HomeObjectFixture.StalePChunkRouteAfterGC:HomeObjectFixture.StaleBlobRouteAfterSealAndGC
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from HomeObjectFixture
[ RUN      ] HomeObjectFixture.StalePChunkRouteAfterGC
/home/ubuntu/WorkSpace/Upstream/HomeObject/src/lib/homestore_backend/tests/hs_gc_tests.cpp:1050: Failure
Expected equality of these values:
  p2.value()
    Which is: 87
  live_pchunk
    Which is: 90
shard2 on replica 2 is routed to pchunk 87 but the live vchunk->pchunk mapping is 90 (stale shard/blob route; see PG 39 / PG 3409)


/home/ubuntu/WorkSpace/Upstream/HomeObject/src/lib/homestore_backend/tests/hs_gc_tests.cpp:1080: Failure
Expected equality of these values:
  p2.value()
    Which is: 87
  live_pchunk_after
    Which is: 90
after putting blobs into shard2 on replica 2, shard2's recorded pchunk 87 diverged from the live mapping 90

[  FAILED  ] HomeObjectFixture.StalePChunkRouteAfterGC (16069 ms)
[       OK ] HomeObjectFixture.StalePChunkRouteAfterGC (17070 ms)
[ RUN      ] HomeObjectFixture.StaleBlobRouteAfterSealAndGC
[ RUN      ] HomeObjectFixture.StaleBlobRouteAfterSealAndGC
[       OK ] HomeObjectFixture.StalePChunkRouteAfterGC (15083 ms)
[ RUN      ] HomeObjectFixture.StaleBlobRouteAfterSealAndGC

/home/ubuntu/WorkSpace/Upstream/HomeObject/src/lib/homestore_backend/tests/hs_gc_tests.cpp:1223: Failure
Value of: blob_rejected
  Actual: false
Expected: true
[StaleBlobRouteAfterSealAndGC-fix] late _put_blob should have been rejected by sealed_lsn guard!

[  FAILED  ] HomeObjectFixture.StaleBlobRouteAfterSealAndGC (10255 ms)
[       OK ] HomeObjectFixture.StaleBlobRouteAfterSealAndGC (10255 ms)
[----------] 2 tests from HomeObjectFixture (25338 ms total)

[----------] 2 tests from HomeObjectFixture (27326 ms total)

[----------] Global test environment tear-down
[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (25338 ms total)
[==========] 2 tests from 1 test suite ran. (27326 ms total)
[  PASSED  ] 1 test.
[  PASSED  ] 2 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] HomeObjectFixture.StaleBlobRouteAfterSealAndGC

 1 FAILED TEST
[       OK ] HomeObjectFixture.StaleBlobRouteAfterSealAndGC (10255 ms)
[----------] 2 tests from HomeObjectFixture (26325 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (26325 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] HomeObjectFixture.StalePChunkRouteAfterGC

[xiaoxichen]

LGTM.

The compatibility of shard_info_superblk/shard_info need to be handle carefully, on metablk as well as the raft log entries (new code can receive/read back log entries from old code)

[JacksonYao287]

I will take a look tomorrow

[Besroy]

@JacksonYao287 Please check whether this logic can be deleted (other error like not leader can also release chunk)

 // we have already added release_chunk logic to thenValue of hoemobject#create_shard in originator, so here
            // we just need to release_chunk for non-originater case since it will bring a bug if a chunk is released
            // for two times. for exampele, as a originator:

            // t1 : chunk1 is released in the rollback of create_shard, the chunk state is marked as available
            // t2 : chunk1 is select by a new create shard (shard1), the chunk state is marked as inuse
            // t3 : chunk1 is released in thenValue of create_shard, the chunk state is marked as available
            // t4 : chunk1 is select by a new create shard (shard2), the chunk state is marked as inuse
            // now, shard1 and shard2 hold the same chunk.
            bool res = release_chunk_based_on_create_shard_message(header);
            if (!res) {
                RELEASE_ASSERT(false,
                               "shardID=0x{:x}, pg={}, shard=0x{:x}, failed to release chunk based on create shard msg",
                               msg_header->shard_id, (msg_header->shard_id >> homeobject::shard_width),
                               (msg_header->shard_id & homeobject::shard_mask));
            }

[JacksonYao287]

since we have commented out "total_blob_occupied_blk_count += 2; /header and footer/"， why comment out these ASSERT_EQ

[Hooper9973]

Since GC still writes the shard header/footer, I’d like to temporarily comment out these blk_count
validations. I will remove the legacy GC shard header/footer code before this dev branch is merged into
stablev4.

[JacksonYao287]

after commentting out "total_blob_occupied_blk_count += 2; /header and footer/", I think all the assert should pass. pls enable them in the next pr

[JacksonYao287]

we need also ctx->promise_.setValue to notify the proposer if create_shard is rolled back?

[Hooper9973]

done

[JacksonYao287]

typo : availble

[Hooper9973]

done