SRCH-6599 Cache OpenSearch document queries in Redis by luisgmetzger · Pull Request #2177 · GSA/search-gov

luisgmetzger · 2026-09-18T19:46:53Z

Summary

SRCH-6599
Caches one page of OpenSearch document-search JSON in the searches Redis (REDIS_CACHE_URL, namespace searches) for 15 minutes so repeat queries skip the cluster.
Updated for the Sep 25 acceptance criteria:
- Per-affiliate Super Admin toggle search_cache_enabled under Enable/disable Settings. Defaults off for every site.
- Global REDIS_CACHE_ON (must be true) and REDIS_CACHE_DURATION (minutes, default 15). Companion TF: searchgov-tf #353.
- Maintainer bypass renamed to disable_search_cache=true|1 on web and API v2. It skips both read and write, and is excluded from the key and impression params. (The old memory param was never permitted on API v2; that is fixed.)
- Cache key is searches:<affiliate_id>:v1:<sha256> over index, offset, size, and the full OpenSearch request body, so every param that changes the request changes the key. Only the user query is case-folded (uppercase AND/OR/NOT kept). Site paths and filters keep their case. Timestamps are floored to the minute so "past hour" stays cacheable.
- Impression diagnostics now logs { module: "SRCH", cached: bool, cache: "hit"|"miss"|"bypass"|"disabled" } for Logstash and CloudWatch.
Maintainer doc: docs/opensearch_query_cache.md (levers, bypass, per-site flush, rollout, rollback, known behavior).

Rollout

Merge after release #2187 lands (it overlaps engine.rb, production.rb, schema.rb); rebase once more before merge.
Apply searchgov-tf SRCH-822 use prefix_length in Elasticsearch phrase suggester #353, then deploy so .env picks up the new SSM params. A Puma restart alone does not.
Migration adds a boolean with default false (instant on MySQL 8). Hosts that boot before db:migrate treat every site as disabled (Affiliate#search_cache_enabled? guard).
Staging: enable one site, verify hit, case-folded hit, bypass, and disabled.
Production: enable justice, watch diagnostics.cache hit rate, OpenSearch query rate, p95 latency, and dgs-prod-searches memory/evictions.

Rollback

Uncheck the site in Super Admin (next request). All sites: Affiliate.update_all(search_cache_enabled: false).
Code path off: set REDIS_CACHE_ON=false in SSM and redeploy.

Test plan

Automated (all passing locally, 592 examples across touched specs):

Gate matrix: REDIS_CACHE_ON (unset/empty/false/yes/1/TRUE), affiliate flag, bypass values, duration (0, negative, non-numeric, 015).
Key edges: case and whitespace, OR vs or, quoted phrases, accented capitals, Turkish dotted I, site path case, missing offset, offsets/sizes/indices/affiliates, every filter option, timestamp flooring.
Param coverage: searches_controller_cache_key_spec.rb and api/v2/searches_controller_cache_key_spec.rb fail if a newly permitted param is not classified as changing the key or not reaching OpenSearch.
Store behavior: TTL expiry, empty results cached, client errors not cached, Marshal round-trip, per-site delete_matched flush against real Redis.
Redis unreachable fails open and logs via error_handler.
Super Admin subgroup, export column, toggle persistence, migration up/down, migration-window guard.
Impression log includes the cache label and omits the bypass param.

Manual (staging):

Same query twice shows cache: hit; different case also hits.
disable_search_cache=true shows bypass and writes no key.
Unchecking the site shows disabled immediately.
Confirm diagnostics.cache is queryable in Logstash.
Repeat once on dev to exercise rediss:// (staging uses plain redis://).

Findings and follow-ups (not in this PR)

tags is permitted on web and API but never forwarded to OpenSearch (DocumentSearchable#facet_filter_hash). Pre-existing; separate ticket.
dgs-prod-searches uses volatile-lru (fine; all keys have TTLs) and has no CloudWatch alarms. Propose Evictions, memory, and CPU alarms alongside searchgov-tf SRCH-779 - update validations to use \A instead of ^ #337.
Propose a WAF rate rule for requests carrying disable_search_cache.
The related-searches govbox query is not cached.

Checklist

Functionality Checks

You have merged the latest changes from the target branch (usually main) into your branch.
Your primary commit message is of the format SRCH-#### <description> matching the associated Jira ticket.
PR title is either of the format SRCH-#### <description> matching the associated Jira ticket (i.e. "SRCH-123 implement feature X"), or Release - SRCH-####, SRCH-####, SRCH-#### matching the Jira ticket numbers in the release.
Automated checks pass. If Code Climate checks do not pass, explain reason for failures:

Process Checks

You have specified at least one "Reviewer".

Repeat searches during traffic spikes should hit the searches cluster instead of OpenSearch, with a maintainer bypass and fail-open Redis timeouts.

Add a per-affiliate Super Admin toggle (search_cache_enabled, default off), gate caching on REDIS_CACHE_ON and REDIS_CACHE_DURATION, and rename the maintainer bypass to disable_search_cache (web and API v2). Key the cache on the full OpenSearch request so every param that changes the request changes the key. Only the user query is case-folded; site paths and filters keep their case, and timestamps are floored to the minute so relative ranges stay cacheable. Log diagnostics cache as hit, miss, bypass, or disabled, and add coverage specs that fail when a new search param is not classified.

jayandra

The changes LGTM and I was able to verify the caching behavior locally 🚀 I had a couple of comments about diagnostics module and default value for search_cache_enabled but both are non-blocking.

Also, should we add REDIS_CACHE_ON, REDIS_CACHE_DURATION and REDIS_CACHE_URL to .env.development, Thoughts?

===== NOTE TO SELF =====
There are three levers to enable/disable caching:

disable_search_cache query params
REDIS_CACHE_ON and REDIS_CACHE_DURATION env variables
search_cache_enabled attribute in affiliates.

Steps taken to verify in localhost:

Updated the cache store for development.rb to be redis_cache_store.
Started sever by running:
REDIS_CACHE_ON=true REDIS_CACHE_DURATION=10 REDIS_CACHE_URL=redis://localhost:6379 bin/dev
Enabled affiliates search_cache_enabled flag in http://localhost:3000/admin/affiliates
Navigated to http://localhost:3000/search?affiliate=va&query=diabetes
Verified that reloading above page got its results from cache by looking for absence of Query: ***** block in rails log.
Verified the presence of SRCH module (e.g. {"cached":true,"cache":"hit","module":"SRCH"})to keep track of cache hits/misses in impressions.log:

jayandra · 2026-09-28T18:43:54Z

@@ -0,0 +1,5 @@
+class AddSearchCacheEnabledToAffiliates < ActiveRecord::Migration[7.1]
+  def change
+    add_column :affiliates, :search_cache_enabled, :boolean, default: false, null: false


Should this default to :true?

Default for search cache should be NOT enabled. We will want to launch the feature with just one affiliate using the cache. Then, as we trust it, add more. At some point, a phase 3 would be enabling cache for all affilates.

jayandra · 2026-09-28T18:50:17Z

+    params = process_array_parameters(build_search_params).merge(indices: search_index)
+    document_search = OpenSearch::DocumentSearch.new(params, affiliate: @affiliate)
+    search_results = document_search.search
+    diagnostics['SRCH'] = { cached: document_search.cache_status == 'hit', cache: document_search.cache_status }


What is the full form for SRCH? Should we put the full form instead?

===NOTE TO SELF===
The diagnostics makes into impressions.log as "diagnostics":[{"cached":true|false,"cache":"hit|miss|disabled|bypass","module":"SRCH"}].

selfdanielj · 2026-09-28T20:38:59Z

                                         :created_since,
                                         :created_until,
                                         :dc,
+                                         :disable_search_cache,


@luisgmetzger i thought we decided that we don't need this query param? right @jayandra @DK1999

selfdanielj · 2026-09-28T20:39:14Z

    utf8
    redesign
    include_facets
+    disable_search_cache


@luisgmetzger same for this

Yeah, we decided not to have a query param for disabling search cache. We're going to do it internally via toggle.

selfdanielj · 2026-09-29T13:52:22Z

Also, should we add REDIS_CACHE_ON, REDIS_CACHE_DURATION and REDIS_CACHE_URL to .env.development, Thoughts?

Yes I think adding those params to .env.Development would be helpful (at least to me!)

selfdanielj · 2026-09-29T14:05:11Z

Updated the cache store for development.rb to be redis_cache_store.

@luisgmetzger @jayandra is it normal to have to go in and edit that file to enable caching when working in development mode? Is there a way to make it respect the environment variable? Or is that not desirable for some reason? What about the test config file? Do we ever use that and should it be updated as well?

jayandra · 2026-10-01T18:00:34Z

Updated the cache store for development.rb to be redis_cache_store.

@luisgmetzger @jayandra is it normal to have to go in and edit that file to enable caching when working in development mode? Is there a way to make it respect the environment variable? Or is that not desirable for some reason? What about the test config file? Do we ever use that and should it be updated as well?

@selfdanielj I do see value in updating the cache store to be redis in development.rb. The only downside would be that the dev environment would now have explicit dependency in redis.

Currently, we should be able to run dev server and see everything working (besides the redis admin page) even while redis is not running. With the cache_store change, the server won't even start without a running redis service to connect to.

selfdanielj · 2026-10-01T19:20:54Z

Updated the cache store for development.rb to be redis_cache_store.

@luisgmetzger @jayandra is it normal to have to go in and edit that file to enable caching when working in development mode? Is there a way to make it respect the environment variable? Or is that not desirable for some reason? What about the test config file? Do we ever use that and should it be updated as well?

@selfdanielj I do see value in updating the cache store to be redis in development.rb. The only downside would be that the dev environment would now have explicit dependency in redis.

Currently, we should be able to run dev server and see everything working (besides the redis admin page) even while redis is not running. With the cache_store change, the server won't even start without a running redis service to connect to.

I'm assuming we already rely on MySQL being present and both of those are started together in search-services by default

luisgmetzger marked this pull request as draft September 21, 2026 01:37

luisgmetzger added 2 commits September 27, 2026 17:47

SRCH-6599 Cache OpenSearch document queries in Redis for 15 minutes

5ec450e

Repeat searches during traffic spikes should hit the searches cluster instead of OpenSearch, with a maintainer bypass and fail-open Redis timeouts.

luisgmetzger force-pushed the SRCH-6599-opensearch-query-cache branch from 0803055 to 98e4c67 Compare September 27, 2026 23:47

luisgmetzger requested review from jayandra and selfdanielj September 27, 2026 23:59

luisgmetzger marked this pull request as ready for review September 27, 2026 23:59

jayandra approved these changes Sep 28, 2026

View reviewed changes

selfdanielj reviewed Sep 28, 2026

View reviewed changes

selfdanielj assigned luisgmetzger Sep 29, 2026

luisgmetzger added this pull request to the merge queue Oct 5, 2026

Merged via the queue into GSA:main with commit ca92141 Oct 5, 2026
5 checks passed

luisgmetzger mentioned this pull request Oct 7, 2026

SRCH-6599: Remove disable_search_cache bypass and add opt-in local Redis cache #2194

Merged

2 of 4 tasks

selfdanielj mentioned this pull request Oct 13, 2026

Staging Release SRCH-6599 SRCH-6777 SRCH-6778 SRCH-6780 SRCH-6783 SRCH-6790 #2200

Merged

jayandra mentioned this pull request Oct 15, 2026

Release - staging to production #2204

Merged

Navigation Menu

Repository navigation

SRCH-6599 Cache OpenSearch document queries in Redis - #2177

SRCH-6599 Cache OpenSearch document queries in Redis#2177
luisgmetzger merged 2 commits into
GSA:mainfrom
luisgmetzger:SRCH-6599-opensearch-query-cache

luisgmetzger commented Sep 18, 2026 •

edited by atlassian-us-gov Bot

Loading

Uh oh!

jayandra left a comment

Uh oh!

jayandra Sep 28, 2026

Uh oh!

DK1999 Sep 28, 2026

Uh oh!

jayandra Sep 28, 2026

Uh oh!

selfdanielj Sep 28, 2026

Uh oh!

selfdanielj Sep 28, 2026

Uh oh!

DK1999 Sep 28, 2026

Uh oh!

selfdanielj commented Sep 29, 2026

Uh oh!

selfdanielj commented Sep 29, 2026

Uh oh!

jayandra commented Oct 1, 2026

Uh oh!

selfdanielj commented Oct 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

luisgmetzger commented Sep 18, 2026 • edited by atlassian-us-gov Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Rollout

Rollback

Test plan

Findings and follow-ups (not in this PR)

Checklist

Functionality Checks

Process Checks

Uh oh!

jayandra left a comment

Choose a reason for hiding this comment

Uh oh!

jayandra Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

DK1999 Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

jayandra Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

selfdanielj Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

selfdanielj Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

DK1999 Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

selfdanielj commented Sep 29, 2026

Uh oh!

selfdanielj commented Sep 29, 2026

Uh oh!

jayandra commented Oct 1, 2026

Uh oh!

selfdanielj commented Oct 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

luisgmetzger commented Sep 18, 2026 •

edited by atlassian-us-gov Bot

Loading