RATIS-2166. Update nextChunkIndex when snapshot is ALREADY_INSTALLED

[133tosakarin]

What changes were proposed in this pull request?

Fix legacy problems with #1145

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-2166

[133tosakarin]

@szetszwo
These failed tests seem to meet expectations

[szetszwo]

+1 the change looks good.

[szetszwo]

@133tosakarin , on a second thought, let's change the Preconditions.assertEquals(..) to throw IOException. Preconditions usually is for correctness checking. If Preconditions fails, it should indicate a bug.

@@ -192,8 +193,12 @@ class SnapshotInstallationHandler {
         //TODO: We should only update State with installed snapshot once the request is done.
         state.installSnapshot(request);
 
-        int idx = nextChunkIndex.getAndIncrement();
-        Preconditions.assertEquals(snapshotChunkRequest.getRequestIndex(), idx, "nextChunkIndex");
+        final int expectedChunkIndex = nextChunkIndex.getAndIncrement();
+        if (snapshotChunkRequest.getRequestIndex() != expectedChunkIndex) {
+          throw new IOException("Unexpected request chunk index: " + snapshotChunkRequest.getRequestIndex()
+              + " (the expected index is " + expectedChunkIndex + ")");
+        }
+
         // update the committed index
         // re-load the state machine if this is the last chunk

[OneSizeFitsQuorum]

LGTM

[szetszwo]

@133tosakarin , thanks of the update!

[szetszwo]

@133tosakarin , you are right that the failures does not seem related; see #1156 . Let's merge this without further retries.