Skip to content

run: Omit '--watch' from startup#321

Merged
tklauser merged 1 commit into
cilium:masterfrom
jrajahalme:run-without-watch
Jul 10, 2026
Merged

run: Omit '--watch' from startup#321
tklauser merged 1 commit into
cilium:masterfrom
jrajahalme:run-without-watch

Conversation

@jrajahalme

Copy link
Copy Markdown
Member

Passing the --watch option causes the HTTP listener restarts under parallel POST requests, making the listener unavailable for brief periods, during which the incoming connections get reset (TCP RST). When Envoy upstream connections get reset like this, HTTP status 503 is reported to the HTTP client at the source pod, as seen in Cilium CI when sending parallel POST requests via Envoy:

  ❌ command "curl --silent --fail --show-error --connect-timeout 2 --max-time 10 -6 --parallel --parallel-immediate -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code}\n --output /dev/null -X POST http://[fd00:10:244:1::c17f]:8080/private http://[fd00:10:244:1::c17f]:8080/private http://[fd00:10:244:1::c17f]:8080/private" failed: command failed (pod=cilium-test-2/client2-78bbbfb455-b47fk, container=client2): command terminated with exit code 22
  ℹ️  curl stdout:
  fd00:10:244:1::cdfa:38402 -> fd00:10:244:1::c17f:8080 = 503
fd00:10:244:1::cdfa:38404 -> fd00:10:244:1::c17f:8080 = 503 fd00:10:244:1::cdfa:38420 -> fd00:10:244:1::c17f:8080 = 503
  ℹ️  curl stderr:
  curl: (22) The requested URL returned error: 503
curl: (22) The requested URL returned error: 503
curl: (22) The requested URL returned error: 503

The --watch option is only needed when the file can be modified by another process, which is not the case when using cilium/json-mock image for Cilium testing, so it should be safe to omit the --watch option. When running without the --watch option, the HTTP listener does not get restarted and parallel POST requests succeed as expected.

Reproduction:

$ docker run --rm -d --name json-mock-watch-repro \
  -p 18080:8080 -e PORT=8080 quay.io/cilium/json-mock:v1.4.0

$ for i in $(seq 1 100); do \
  curl --silent --fail --show-error \
  --connect-timeout 1 --max-time 3 \
  --parallel --parallel-immediate \
  -X POST \
  http://127.0.0.1:18080/public \
  http://127.0.0.1:18080/public \
  http://127.0.0.1:18080/public \
  >/dev/null || echo "failed iteration $i"; done

This quickly reproduces these failures, and the docker container logs show repeated messages like:

/default.json has changed, reloading...

  Loading /default.json
  Loading /middleware.js
  Done

When running without the --watch option the curl loop produces no failures.

Passing the '--watch' option causes the HTTP listener restarts under
parallel POST requests, making the listener unavailable for brief
periods, during which the incoming connections get reset (TCP RST). When
Envoy upstream connections get reset like this, HTTP status 503 is
reported to the HTTP client at the source pod, as seen in Cilium CI when
sending parallel POST requests via Envoy:

  ❌ command "curl --silent --fail --show-error --connect-timeout 2 --max-time 10 -6 --parallel --parallel-immediate -w %{local_ip}:%{local_port} -> %{remote_ip}:%{remote_port} = %{response_code}\n --output /dev/null -X POST http://[fd00:10:244:1::c17f]:8080/private http://[fd00:10:244:1::c17f]:8080/private http://[fd00:10:244:1::c17f]:8080/private" failed: command failed (pod=cilium-test-2/client2-78bbbfb455-b47fk, container=client2): command terminated with exit code 22
  ℹ️  curl stdout:
  fd00:10:244:1::cdfa:38402 -> fd00:10:244:1::c17f:8080 = 503
fd00:10:244:1::cdfa:38404 -> fd00:10:244:1::c17f:8080 = 503
fd00:10:244:1::cdfa:38420 -> fd00:10:244:1::c17f:8080 = 503
  ℹ️  curl stderr:
  curl: (22) The requested URL returned error: 503
curl: (22) The requested URL returned error: 503
curl: (22) The requested URL returned error: 503

The '--watch' option is only needed when the file can be modified by
another process, which is not the case when using cilium/json-mock image
for Cilium testing, so it should be safe to omit the '--watch'
option. When running without the '--watch' option, the HTTP listener does
not get restarted and parallel POST requests succeed as expected.

Reproduction:

$ docker run --rm -d --name json-mock-watch-repro \
  -p 18080:8080 -e PORT=8080 quay.io/cilium/json-mock:v1.4.0

$ for i in $(seq 1 100); do \
  curl --silent --fail --show-error \
  --connect-timeout 1 --max-time 3 \
  --parallel --parallel-immediate \
  -X POST \
  http://127.0.0.1:18080/public \
  http://127.0.0.1:18080/public \
  http://127.0.0.1:18080/public \
  >/dev/null || echo "failed iteration $i"; done

This quickly reproduces these failures, and the docker container logs
show repeated messages like:

/default.json has changed, reloading...

  Loading /default.json
  Loading /middleware.js
  Done

When running without the '--watch' option the curl loop produces no
failures.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>
@jrajahalme jrajahalme added the bug Something isn't working label Jul 9, 2026
@jrajahalme

Copy link
Copy Markdown
Member Author

Example Cilium CI workflow run with this flake: https://github.com/cilium/cilium/actions/runs/29041892696/job/86200869896

@tklauser tklauser merged commit 05b9cf1 into cilium:master Jul 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants