[VOTE] Add invalidated_fragments to IndexMetadata

[westonpace]

There is a discussion thread here and a PR here.

The basic problem occurs when an existing indexed column is modified (on one or more fragments). Previously we removed those fragments from the index's fragment bitmap. However, the index might still be searched if it contains other valid (unmodified) fragments. The results must be masked out to block the stale results referring to invalidated fragments.

One approach could be to always apply an allow-list for the fragment bitmap of the index. However, this would add considerable cost to the happy path (indexed column is not updated after index is created) because we would need to apply this allow list on every search.

A simpler approach is to apply the block list to fragments removed from the fragment bitmap, only if fragments have actually been removed. To do this we need to know which fragments were removed from the index.

[jackye1995]

+1

[BubbleCal]

+1

[sinianluoye]

+1

[Xuanwo]

+1 binding

[wjones127]

@westonpace What's the status of this? It looks like the vote passed, but #6514 was closed rather than merged.

[westonpace]

In the discussion thread you brought up the possibility of always applying an allow-list (which we can calculate) instead of selectively applying a block-list (which requires a format change to calculate).

I mentioned that I was concerned about the performance in #5453 (reply in thread) but I never did follow up with my benchmark results. I did some benchmarking and did not see a meaningful cost to applying the allow list. We've had the "always use allow list" code deployed for some time and benchmarks have not regressed and we haven't noticed degradation in our production workloads.

As a result, I didn't see the need for the change (and extra complexity of keeping track of invalidated fragments) and abandoned it. It's possible the issue could surface again in the future if we start to fine tune more performance scenarios but until then it isn't really a priority for me to continue.

[wjones127]

Close this vote thread, while it passed, we did not implement this per #6529 (reply in thread)

Can re-open if we have evidence it's worth re-exploring.