JAMES-4166 JMAP search: drop scroll search, use from/size pagination and collapse by messageId

[quantranhong1999]

Also, modify the search API with SearchOptions and push the offset onto the search engine.

[chibenwa]

Nitpicks along the way but IMO it really goes into the right direction

[Arsnael]

Read it, nothing to add.

[chibenwa]

Nice work !

Here are some little comments to polish it further.

I also identified some search related issues but those are to be addressed separately.

[quantranhong1999]

For the record, we did run a james-gatling performance test to compare the scroll search (before) vs the from/size pagination (after).

scroll search

from/size pagination

Email/query seems on bar with the scroll search (https://github.com/linagora/james-project-private/issues/1149), but better MAX response time (595ms vs 894ms).

However, the result is not really representative: in our JMAP performance test scenario, we do Email/query on a random keyword, which should return very few results. Also, the scroll context is closed asynchronously in the current code CF

james-project/backends-common/opensearch/src/main/java/org/apache/james/backends/opensearch/search/ScrolledSearch.java

Line 117 in bfc994c

client.clearScroll(clearScrollRequest).subscribe()).sneakyThrow());

This explains similar results.

We imagine we would need to go > 25 rqs on emailQuery to see significant improvements.

But the gains are obvious, so we won't investigate deeper in the performance test:

• 1 round trip vs multiple round trips
• only getting exactly what we need vs over-fetch (even worse when deep paging)

[Arsnael]

Related or not? https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2928/3/testReport/junit/org.apache.james.mailbox.lucene.search/LuceneMailboxMessageSearchIndexTest/searchAllShouldLimitTheSize/

[quantranhong1999]

https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-2928/4/testReport/junit/org.apache.james.mailets/GatewayRemoteDeliveryIntegrationTest/mailFromShouldBePreservedUponConcurrency_File_/

Failure tests are not related to this work. It seems failing on master (without the build cache). It may be related to the rest-smtp-sink image's outdated schema with recent Docker engine upgrade (https://docs.docker.com/engine/deprecated/?_gl=1*1xwttz4*_gcl_au*MjAzMDk1NDQ0Mi4xNzcwMTA3MzYy*_ga*MjI5OTA0NjQxLjE3NjI5MTQwMzY.*_ga_XJWPQMJYHQ*czE3NzAxMDczNjIkbzExJGcxJHQxNzcwMTA3MzgzJGozOSRsMCRoMA..#pushing-and-pulling-with-image-manifest-v2-schema-1).

I am having a look.

[quantranhong1999]

Failure tests are not related to this work. It seems failing on master (without the build cache).

Actually, these failed on my Mac (notable for not being fully compatible with docker engine), but passed on my Ubuntu PC (docker engine v28.3.3).

I do not understand what is going on. Will try to rerun the CI.

[quantranhong1999]

Green build! I squashed the fixup.