Skip to content

Add non-partitioned table tests for RewriteFiles and OverwriteFiles#4137

Open
Takch02 wants to merge 5 commits intoapache:masterfrom
Takch02:add-non-partitioned-tests
Open

Add non-partitioned table tests for RewriteFiles and OverwriteFiles#4137
Takch02 wants to merge 5 commits intoapache:masterfrom
Takch02:add-non-partitioned-tests

Conversation

@Takch02
Copy link

@Takch02 Takch02 commented Mar 20, 2026

Motivation

Currently, TestRewriteFiles and TestOverwriteFiles only verify the behavior of partitioned tables, expecting a CannotAlterHiveLocationException when partial rewrites or overwrites attempt to reference multiple directories. Non-partitioned tables, however, are not bound by Hive's strict 1-partition-1-folder constraint and should successfully commit these operations. This PR adds the missing coverage for non-partitioned tables.

Key Changes

Added NonPartitioned methods in TestRewriteFiles and TestOverwriteFiles using Assume.assumeFalse(isPartitionedTable()).

Verified successful commits instead of expecting exceptions, as non-partitioned tables gracefully handle files across multiple directories.

Introduced Dynamic Assertions: Replaced hardcoded integer values with dynamic size assertions to prevent fragile tests. This ensures the tests remain robust regardless of table properties (e.g., Primary Key bucketing or target file size limits).

Comprehensive State Verification: Implemented a two-step verification process for each test:

Action Verification: Checked the exact number of files added in the latest transaction using lastedAddedFiles().

State Verification: Checked the total number of live files in the current snapshot using baseStore.newScan().planFiles().

Example of Verification Logic:

Rewrite: Verified afterFiles.size() and totalLiveFiles against addFiles.size().

Overwrite: Verified afterFiles.size() against secondDataFiles.size(), and totalLiveFiles against firstDataFiles.size() + secondDataFiles.size().

@github-actions github-actions bot added the module:mixed-hive Hive moduel for Mixed Format label Mar 20, 2026
@Takch02
Copy link
Author

Takch02 commented Mar 20, 2026

I re-uploaded it after checking the CI and adhering to code conventions.

@codecov-commenter
Copy link

codecov-commenter commented Mar 20, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 29.90%. Comparing base (cc03949) to head (042e849).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##             master    #4137   +/-   ##
=========================================
  Coverage     29.90%   29.90%           
  Complexity     4160     4160           
=========================================
  Files           670      670           
  Lines         53533    53533           
  Branches       6797     6797           
=========================================
+ Hits          16008    16010    +2     
+ Misses        36343    36341    -2     
  Partials       1182     1182           
Flag Coverage Δ
core 29.90% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:mixed-hive Hive moduel for Mixed Format

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants