Add App Logs Collection feature #10156

jpbempel · 2025-12-10T13:16:47Z

What Does This Do

Add logs capture based on log framework instrumentation that hooks when log event are generated to be written. Log events are sent to Logs backend through Datadog agent's Event Platform (Evp) proxy. We are reusing a previous infrastructure used for uploading logs for CI Visibility (see #7082).
We are adding support for Logback.
This feature is opt-in through DD_APP_LOGS_COLLECTION_ENABLED config. Logs are sent as structured log layout and allow to add any additional attributes. For now we are adding directly trace correlation ids (traceId and spanId) and give the logs injection feature for free and without any setup on the log framework side.
Smoke tests are based on Logs Injection tests.

Motivation

On-boarding Logs easily through tracer that are already installed through SSI

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-12-10T14:42:07Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/logs-capture
git_commit_date	1766060680	1766071501
git_commit_sha	`7442d01`	`ba23ef3`
release_version	1.58.0-SNAPSHOT~7442d01dcc	1.58.0-SNAPSHOT~ba23ef3c76

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1766073483	1766073483
ci_job_id	1306528640	1306528640
ci_pipeline_id	87576746	87576746
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-1e2pvyrg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-1e2pvyrg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 7 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.58.0-SNAPSHOT~ba23ef3c76, baseline=1.58.0-SNAPSHOT~7442d01dcc

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.084 s) : 0, 1083853
Total [baseline] (10.799 s) : 0, 10798510
Agent [candidate] (1.079 s) : 0, 1079211
Total [candidate] (10.728 s) : 0, 10728461
section appsec
Agent [baseline] (1.269 s) : 0, 1269217
Total [baseline] (10.997 s) : 0, 10997010
Agent [candidate] (1.262 s) : 0, 1261780
Total [candidate] (10.975 s) : 0, 10974860
section iast
Agent [baseline] (1.225 s) : 0, 1225420
Total [baseline] (11.25 s) : 0, 11250132
Agent [candidate] (1.223 s) : 0, 1222846
Total [candidate] (11.246 s) : 0, 11245881
section profiling
Agent [baseline] (1.203 s) : 0, 1202754
Total [baseline] (10.876 s) : 0, 10876074
Agent [candidate] (1.205 s) : 0, 1204801
Total [candidate] (10.911 s) : 0, 10910882

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.084 s	-
Agent	appsec	1.269 s	185.364 ms (17.1%)
Agent	iast	1.225 s	141.567 ms (13.1%)
Agent	profiling	1.203 s	118.901 ms (11.0%)
Total	tracing	10.799 s	-
Total	appsec	10.997 s	198.499 ms (1.8%)
Total	iast	11.25 s	451.622 ms (4.2%)
Total	profiling	10.876 s	77.564 ms (0.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.079 s	-
Agent	appsec	1.262 s	182.569 ms (16.9%)
Agent	iast	1.223 s	143.635 ms (13.3%)
Agent	profiling	1.205 s	125.59 ms (11.6%)
Total	tracing	10.728 s	-
Total	appsec	10.975 s	246.399 ms (2.3%)
Total	iast	11.246 s	517.42 ms (4.8%)
Total	profiling	10.911 s	182.421 ms (1.7%)

gantt
    title petclinic - break down per module: candidate=1.58.0-SNAPSHOT~ba23ef3c76, baseline=1.58.0-SNAPSHOT~7442d01dcc

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.19 ms) : 0, 1190
crashtracking [candidate] (1.178 ms) : 0, 1178
BytebuddyAgent [baseline] (649.883 ms) : 0, 649883
BytebuddyAgent [candidate] (647.903 ms) : 0, 647903
GlobalTracer [baseline] (283.086 ms) : 0, 283086
GlobalTracer [candidate] (281.265 ms) : 0, 281265
AppSec [baseline] (32.298 ms) : 0, 32298
AppSec [candidate] (32.08 ms) : 0, 32080
Debugger [baseline] (68.166 ms) : 0, 68166
Debugger [candidate] (68.302 ms) : 0, 68302
Remote Config [baseline] (632.7 µs) : 0, 633
Remote Config [candidate] (595.778 µs) : 0, 596
Telemetry [baseline] (9.127 ms) : 0, 9127
Telemetry [candidate] (8.838 ms) : 0, 8838
Flare Poller [baseline] (3.806 ms) : 0, 3806
Flare Poller [candidate] (3.611 ms) : 0, 3611
section appsec
crashtracking [baseline] (1.19 ms) : 0, 1190
crashtracking [candidate] (1.202 ms) : 0, 1202
BytebuddyAgent [baseline] (691.713 ms) : 0, 691713
BytebuddyAgent [candidate] (688.568 ms) : 0, 688568
GlobalTracer [baseline] (260.412 ms) : 0, 260412
GlobalTracer [candidate] (258.019 ms) : 0, 258019
AppSec [baseline] (173.197 ms) : 0, 173197
AppSec [candidate] (173.316 ms) : 0, 173316
Debugger [baseline] (68.592 ms) : 0, 68592
Debugger [candidate] (67.382 ms) : 0, 67382
Remote Config [baseline] (708.505 µs) : 0, 709
Remote Config [candidate] (700.593 µs) : 0, 701
Telemetry [baseline] (9.09 ms) : 0, 9090
Telemetry [candidate] (8.921 ms) : 0, 8921
Flare Poller [baseline] (3.942 ms) : 0, 3942
Flare Poller [candidate] (3.864 ms) : 0, 3864
IAST [baseline] (24.774 ms) : 0, 24774
IAST [candidate] (24.319 ms) : 0, 24319
section iast
crashtracking [baseline] (1.197 ms) : 0, 1197
crashtracking [candidate] (1.198 ms) : 0, 1198
BytebuddyAgent [baseline] (792.398 ms) : 0, 792398
BytebuddyAgent [candidate] (790.582 ms) : 0, 790582
GlobalTracer [baseline] (256.424 ms) : 0, 256424
GlobalTracer [candidate] (255.362 ms) : 0, 255362
AppSec [baseline] (35.011 ms) : 0, 35011
AppSec [candidate] (33.528 ms) : 0, 33528
Debugger [baseline] (65.05 ms) : 0, 65050
Debugger [candidate] (67.249 ms) : 0, 67249
Remote Config [baseline] (573.797 µs) : 0, 574
Remote Config [candidate] (606.778 µs) : 0, 607
Telemetry [baseline] (8.497 ms) : 0, 8497
Telemetry [candidate] (8.533 ms) : 0, 8533
Flare Poller [baseline] (3.528 ms) : 0, 3528
Flare Poller [candidate] (3.519 ms) : 0, 3519
IAST [baseline] (27.153 ms) : 0, 27153
IAST [candidate] (26.894 ms) : 0, 26894
section profiling
ProfilingAgent [baseline] (96.957 ms) : 0, 96957
ProfilingAgent [candidate] (97.435 ms) : 0, 97435
crashtracking [baseline] (1.21 ms) : 0, 1210
crashtracking [candidate] (1.245 ms) : 0, 1245
BytebuddyAgent [baseline] (700.75 ms) : 0, 700750
BytebuddyAgent [candidate] (701.891 ms) : 0, 701891
GlobalTracer [baseline] (220.753 ms) : 0, 220753
GlobalTracer [candidate] (220.895 ms) : 0, 220895
AppSec [baseline] (32.195 ms) : 0, 32195
AppSec [candidate] (32.176 ms) : 0, 32176
Debugger [baseline] (68.06 ms) : 0, 68060
Debugger [candidate] (68.135 ms) : 0, 68135
Remote Config [baseline] (607.511 µs) : 0, 608
Remote Config [candidate] (618.772 µs) : 0, 619
Telemetry [baseline] (8.818 ms) : 0, 8818
Telemetry [candidate] (8.879 ms) : 0, 8879
Flare Poller [baseline] (3.729 ms) : 0, 3729
Flare Poller [candidate] (3.754 ms) : 0, 3754
Profiling [baseline] (97.541 ms) : 0, 97541
Profiling [candidate] (98.007 ms) : 0, 98007

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.58.0-SNAPSHOT~ba23ef3c76, baseline=1.58.0-SNAPSHOT~7442d01dcc

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.083 s) : 0, 1083276
Total [baseline] (8.766 s) : 0, 8765748
Agent [candidate] (1.082 s) : 0, 1081965
Total [candidate] (8.773 s) : 0, 8773171
section iast
Agent [baseline] (1.219 s) : 0, 1218905
Total [baseline] (9.35 s) : 0, 9350365
Agent [candidate] (1.22 s) : 0, 1219769
Total [candidate] (9.374 s) : 0, 9373732

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.083 s	-
Agent	iast	1.219 s	135.628 ms (12.5%)
Total	tracing	8.766 s	-
Total	iast	9.35 s	584.616 ms (6.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.082 s	-
Agent	iast	1.22 s	137.803 ms (12.7%)
Total	tracing	8.773 s	-
Total	iast	9.374 s	600.56 ms (6.8%)

gantt
    title insecure-bank - break down per module: candidate=1.58.0-SNAPSHOT~ba23ef3c76, baseline=1.58.0-SNAPSHOT~7442d01dcc

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.199 ms) : 0, 1199
crashtracking [candidate] (1.193 ms) : 0, 1193
BytebuddyAgent [baseline] (649.654 ms) : 0, 649654
BytebuddyAgent [candidate] (649.568 ms) : 0, 649568
GlobalTracer [baseline] (282.887 ms) : 0, 282887
GlobalTracer [candidate] (282.526 ms) : 0, 282526
AppSec [baseline] (32.377 ms) : 0, 32377
AppSec [candidate] (32.149 ms) : 0, 32149
Debugger [baseline] (68.024 ms) : 0, 68024
Debugger [candidate] (67.602 ms) : 0, 67602
Remote Config [baseline] (639.35 µs) : 0, 639
Remote Config [candidate] (607.538 µs) : 0, 608
Telemetry [baseline] (9.073 ms) : 0, 9073
Telemetry [candidate] (9.026 ms) : 0, 9026
Flare Poller [baseline] (3.781 ms) : 0, 3781
Flare Poller [candidate] (3.687 ms) : 0, 3687
section iast
crashtracking [baseline] (1.194 ms) : 0, 1194
crashtracking [candidate] (1.186 ms) : 0, 1186
BytebuddyAgent [baseline] (789.039 ms) : 0, 789039
BytebuddyAgent [candidate] (789.379 ms) : 0, 789379
GlobalTracer [baseline] (254.753 ms) : 0, 254753
GlobalTracer [candidate] (254.874 ms) : 0, 254874
AppSec [baseline] (31.809 ms) : 0, 31809
AppSec [candidate] (33.718 ms) : 0, 33718
Debugger [baseline] (67.261 ms) : 0, 67261
Debugger [candidate] (65.833 ms) : 0, 65833
Remote Config [baseline] (605.67 µs) : 0, 606
Remote Config [candidate] (609.076 µs) : 0, 609
Telemetry [baseline] (8.417 ms) : 0, 8417
Telemetry [candidate] (8.454 ms) : 0, 8454
Flare Poller [baseline] (3.506 ms) : 0, 3506
Flare Poller [candidate] (3.498 ms) : 0, 3498
IAST [baseline] (26.783 ms) : 0, 26783
IAST [candidate] (26.88 ms) : 0, 26880

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/logs-capture
git_commit_date	1766060680	1766071501
git_commit_sha	`7442d01`	`ba23ef3`
release_version	1.58.0-SNAPSHOT~7442d01dcc	1.58.0-SNAPSHOT~ba23ef3c76

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1766073985	1766073985
ci_job_id	1306528641	1306528641
ci_pipeline_id	87576746	87576746
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-3zt74hoh 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-3zt74hoh 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 4 performance regressions! Performance is the same for 15 metrics, 15 unstable metrics.

scenario	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p95	Δ mean throughput	candidate mean agg_http_req_duration_p50	candidate mean agg_http_req_duration_p95	candidate mean throughput	baseline mean agg_http_req_duration_p50	baseline mean agg_http_req_duration_p95	baseline mean throughput
scenario:load:insecure-bank:iast_FULL:high_load	worse [+192.787µs; +491.162µs] or [+3.903%; +9.944%]	worse [+336.141µs; +874.508µs] or [+2.787%; +7.250%]	unstable [-131.600op/s; +32.600op/s] or [-15.982%; +3.959%]	5.281ms	12.667ms	773.906op/s	4.939ms	12.062ms	823.406op/s
scenario:load:petclinic:code_origins:high_load	worse [+570.211µs; +1106.310µs] or [+3.366%; +6.531%]	worse [+0.813ms; +2.025ms] or [+2.937%; +7.318%]	unstable [-39.903op/s; +18.216op/s] or [-14.853%; +6.780%]	17.777ms	29.092ms	257.812op/s	16.939ms	27.673ms	268.656op/s
scenario:load:petclinic:profiling:high_load	better [-2.341ms; -1.549ms] or [-11.501%; -7.607%]	better [-2.775ms; -1.298ms] or [-8.682%; -4.062%]	unstable [-2.098op/s; +50.223op/s] or [-0.932%; +22.321%]	18.412ms	29.927ms	249.062op/s	20.357ms	31.964ms	225.000op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.58.0-SNAPSHOT~ba23ef3c76, baseline=1.58.0-SNAPSHOT~7442d01dcc
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.188 ms) : 1177, 1200
.   : milestone, 1188,
iast (3.192 ms) : 3150, 3235
.   : milestone, 3192,
iast_FULL (5.613 ms) : 5557, 5669
.   : milestone, 5613,
iast_GLOBAL (3.566 ms) : 3507, 3625
.   : milestone, 3566,
profiling (1.902 ms) : 1887, 1918
.   : milestone, 1902,
tracing (1.841 ms) : 1826, 1856
.   : milestone, 1841,
section candidate
no_agent (1.188 ms) : 1176, 1199
.   : milestone, 1188,
iast (3.21 ms) : 3165, 3256
.   : milestone, 3210,
iast_FULL (5.976 ms) : 5915, 6037
.   : milestone, 5976,
iast_GLOBAL (3.594 ms) : 3541, 3646
.   : milestone, 3594,
profiling (1.974 ms) : 1956, 1992
.   : milestone, 1974,
tracing (1.858 ms) : 1841, 1874
.   : milestone, 1858,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.188 ms [1.177 ms, 1.2 ms]	-
iast	3.192 ms [3.15 ms, 3.235 ms]	2.004 ms (168.7%)
iast_FULL	5.613 ms [5.557 ms, 5.669 ms]	4.425 ms (372.4%)
iast_GLOBAL	3.566 ms [3.507 ms, 3.625 ms]	2.378 ms (200.1%)
profiling	1.902 ms [1.887 ms, 1.918 ms]	714.363 µs (60.1%)
tracing	1.841 ms [1.826 ms, 1.856 ms]	652.84 µs (54.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.188 ms [1.176 ms, 1.199 ms]	-
iast	3.21 ms [3.165 ms, 3.256 ms]	2.022 ms (170.2%)
iast_FULL	5.976 ms [5.915 ms, 6.037 ms]	4.788 ms (403.1%)
iast_GLOBAL	3.594 ms [3.541 ms, 3.646 ms]	2.406 ms (202.5%)
profiling	1.974 ms [1.956 ms, 1.992 ms]	785.776 µs (66.1%)
tracing	1.858 ms [1.841 ms, 1.874 ms]	669.593 µs (56.4%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.58.0-SNAPSHOT~ba23ef3c76, baseline=1.58.0-SNAPSHOT~7442d01dcc
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.115 ms) : 18918, 19313
.   : milestone, 19115,
appsec (18.523 ms) : 18335, 18711
.   : milestone, 18523,
code_origins (17.367 ms) : 17196, 17537
.   : milestone, 17367,
iast (17.429 ms) : 17258, 17599
.   : milestone, 17429,
profiling (20.755 ms) : 20549, 20961
.   : milestone, 20755,
tracing (17.797 ms) : 17622, 17972
.   : milestone, 17797,
section candidate
no_agent (19.052 ms) : 18858, 19246
.   : milestone, 19052,
appsec (19.046 ms) : 18853, 19240
.   : milestone, 19046,
code_origins (18.101 ms) : 17922, 18281
.   : milestone, 18101,
iast (17.675 ms) : 17498, 17852
.   : milestone, 17675,
profiling (18.74 ms) : 18554, 18926
.   : milestone, 18740,
tracing (18.283 ms) : 18100, 18465
.   : milestone, 18283,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.115 ms [18.918 ms, 19.313 ms]	-
appsec	18.523 ms [18.335 ms, 18.711 ms]	-592.164 µs (-3.1%)
code_origins	17.367 ms [17.196 ms, 17.537 ms]	-1.749 ms (-9.1%)
iast	17.429 ms [17.258 ms, 17.599 ms]	-1.687 ms (-8.8%)
profiling	20.755 ms [20.549 ms, 20.961 ms]	1.64 ms (8.6%)
tracing	17.797 ms [17.622 ms, 17.972 ms]	-1.318 ms (-6.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.052 ms [18.858 ms, 19.246 ms]	-
appsec	19.046 ms [18.853 ms, 19.24 ms]	-5.782 µs (-0.0%)
code_origins	18.101 ms [17.922 ms, 18.281 ms]	-951.011 µs (-5.0%)
iast	17.675 ms [17.498 ms, 17.852 ms]	-1.377 ms (-7.2%)
profiling	18.74 ms [18.554 ms, 18.926 ms]	-312.225 µs (-1.6%)
tracing	18.283 ms [18.1 ms, 18.465 ms]	-769.313 µs (-4.0%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/logs-capture
git_commit_date	1766060680	1766071501
git_commit_sha	`7442d01`	`ba23ef3`
release_version	1.58.0-SNAPSHOT~7442d01dcc	1.58.0-SNAPSHOT~ba23ef3c76

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1766073703	1766073703
ci_job_id	1306528642	1306528642
ci_pipeline_id	87576746	87576746
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-974l3i33 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-974l3i33 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.58.0-SNAPSHOT~ba23ef3c76, baseline=1.58.0-SNAPSHOT~7442d01dcc
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.56 s) : 15560000, 15560000
.   : milestone, 15560000,
appsec (14.541 s) : 14541000, 14541000
.   : milestone, 14541000,
iast (18.172 s) : 18172000, 18172000
.   : milestone, 18172000,
iast_GLOBAL (17.687 s) : 17687000, 17687000
.   : milestone, 17687000,
profiling (15.075 s) : 15075000, 15075000
.   : milestone, 15075000,
tracing (14.504 s) : 14504000, 14504000
.   : milestone, 14504000,
section candidate
no_agent (15.279 s) : 15279000, 15279000
.   : milestone, 15279000,
appsec (14.777 s) : 14777000, 14777000
.   : milestone, 14777000,
iast (18.052 s) : 18052000, 18052000
.   : milestone, 18052000,
iast_GLOBAL (17.989 s) : 17989000, 17989000
.   : milestone, 17989000,
profiling (14.718 s) : 14718000, 14718000
.   : milestone, 14718000,
tracing (14.473 s) : 14473000, 14473000
.   : milestone, 14473000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.56 s [15.56 s, 15.56 s]	-
appsec	14.541 s [14.541 s, 14.541 s]	-1.019 s (-6.5%)
iast	18.172 s [18.172 s, 18.172 s]	2.612 s (16.8%)
iast_GLOBAL	17.687 s [17.687 s, 17.687 s]	2.127 s (13.7%)
profiling	15.075 s [15.075 s, 15.075 s]	-485.0 ms (-3.1%)
tracing	14.504 s [14.504 s, 14.504 s]	-1.056 s (-6.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.279 s [15.279 s, 15.279 s]	-
appsec	14.777 s [14.777 s, 14.777 s]	-502.0 ms (-3.3%)
iast	18.052 s [18.052 s, 18.052 s]	2.773 s (18.1%)
iast_GLOBAL	17.989 s [17.989 s, 17.989 s]	2.71 s (17.7%)
profiling	14.718 s [14.718 s, 14.718 s]	-561.0 ms (-3.7%)
tracing	14.473 s [14.473 s, 14.473 s]	-806.0 ms (-5.3%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.58.0-SNAPSHOT~ba23ef3c76, baseline=1.58.0-SNAPSHOT~7442d01dcc
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
.   : milestone, 1474,
appsec (3.687 ms) : 3470, 3905
.   : milestone, 3687,
iast (2.21 ms) : 2146, 2274
.   : milestone, 2210,
iast_GLOBAL (2.252 ms) : 2187, 2317
.   : milestone, 2252,
profiling (2.055 ms) : 2003, 2108
.   : milestone, 2055,
tracing (2.042 ms) : 1991, 2093
.   : milestone, 2042,
section candidate
no_agent (1.474 ms) : 1463, 1486
.   : milestone, 1474,
appsec (3.708 ms) : 3489, 3926
.   : milestone, 3708,
iast (2.212 ms) : 2147, 2276
.   : milestone, 2212,
iast_GLOBAL (2.248 ms) : 2183, 2313
.   : milestone, 2248,
profiling (2.064 ms) : 2012, 2117
.   : milestone, 2064,
tracing (2.036 ms) : 1985, 2086
.   : milestone, 2036,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.474 ms [1.463 ms, 1.486 ms]	-
appsec	3.687 ms [3.47 ms, 3.905 ms]	2.213 ms (150.1%)
iast	2.21 ms [2.146 ms, 2.274 ms]	735.815 µs (49.9%)
iast_GLOBAL	2.252 ms [2.187 ms, 2.317 ms]	777.404 µs (52.7%)
profiling	2.055 ms [2.003 ms, 2.108 ms]	581.229 µs (39.4%)
tracing	2.042 ms [1.991 ms, 2.093 ms]	567.718 µs (38.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.474 ms [1.463 ms, 1.486 ms]	-
appsec	3.708 ms [3.489 ms, 3.926 ms]	2.233 ms (151.5%)
iast	2.212 ms [2.147 ms, 2.276 ms]	737.42 µs (50.0%)
iast_GLOBAL	2.248 ms [2.183 ms, 2.313 ms]	774.137 µs (52.5%)
profiling	2.064 ms [2.012 ms, 2.117 ms]	590.048 µs (40.0%)
tracing	2.036 ms [1.985 ms, 2.086 ms]	561.402 µs (38.1%)

Add application logs collection based on log framework instrumentation that hooks when log event are generated to be written. Log events are sent to Logs backend through Datadog agent's Event Platform (Evp) proxy. We are reusing a previous infrastructure used for uploading logs for CI Visibility (see #7082). We are adding support for Logback. This feature is opt-in through DD_APP_LOGS_COLLECTION_ENABLED config. Logs are sent as structured log layout and allow to add any additional attributes. For now we are adding directly trace correlation ids (traceId and spanId) and give the logs injection feature for free and without any setup on the log framework side. Smoke tests are based on Logs Injection tests. # Conflicts: # metadata/supported-configurations.json

amarziali · 2025-12-18T13:23:48Z

...ck-1.0/src/main/java/datadog/trace/instrumentation/logback/LogbackLoggerInstrumentation.java

  }
+
+  public static class CallAppendersAdvice2 {
+    @Advice.OnMethodEnter


can you suppress all the Throwables here ?

amarziali · 2025-12-18T13:26:44Z

...og4j-2.0/src/main/java/datadog/trace/instrumentation/log4j2/LoggerConfigInstrumentation.java

-        && InstrumenterConfig.get().isAgentlessLogSubmissionEnabled();
+    return (super.isApplicable(enabledSystems)
+            && InstrumenterConfig.get().isAgentlessLogSubmissionEnabled())
+        || Config.get().isAppLogsCollectionEnabled();


we tend to use InstrumenterConfig here since is more lightweight to build comparing to Config (since this is called during the premain). However it's true that other instrumentations are using Config.get(). If possible, moving that to InstrumenterConfig will help keeping that design idea

amarziali

Thanks for that cool addition. It looks OK to me

jpbempel temporarily deployed to ci-build December 10, 2025 13:16 — with GitHub Actions Inactive

jpbempel added comp: logging Tracer internal logging type: feature request labels Dec 10, 2025

jpbempel force-pushed the jpbempel/logs-capture branch from 0a5090f to e60662e Compare December 10, 2025 13:54

jpbempel temporarily deployed to ci-build December 10, 2025 13:54 — with GitHub Actions Inactive

jpbempel force-pushed the jpbempel/logs-capture branch from e60662e to 9470cec Compare December 10, 2025 15:52

jpbempel temporarily deployed to ci-build December 10, 2025 15:53 — with GitHub Actions Inactive

jpbempel force-pushed the jpbempel/logs-capture branch from 9470cec to 6ef9061 Compare December 10, 2025 16:51

jpbempel temporarily deployed to ci-build December 10, 2025 16:51 — with GitHub Actions Inactive

jpbempel force-pushed the jpbempel/logs-capture branch from 6ef9061 to 3d4d2ca Compare December 11, 2025 09:13

jpbempel temporarily deployed to ci-build December 11, 2025 09:13 — with GitHub Actions Inactive

jpbempel force-pushed the jpbempel/logs-capture branch 2 times, most recently from 0d5b1e5 to 20f1e33 Compare December 17, 2025 18:13

jpbempel changed the title ~~Add Logs Capture feature~~ Add App Logs Collection feature Dec 18, 2025

jpbempel force-pushed the jpbempel/logs-capture branch from 20f1e33 to d32101e Compare December 18, 2025 13:10

jpbempel temporarily deployed to ci-build December 18, 2025 13:10 — with GitHub Actions Inactive

jpbempel marked this pull request as ready for review December 18, 2025 13:10

jpbempel requested review from a team as code owners December 18, 2025 13:10

jpbempel requested review from dougqh and removed request for a team December 18, 2025 13:10

amarziali reviewed Dec 18, 2025

View reviewed changes

Address comments

ba23ef3

jpbempel temporarily deployed to ci-build December 18, 2025 15:25 — with GitHub Actions Inactive

amarziali approved these changes Dec 18, 2025

View reviewed changes

jpbempel merged commit 3a70b4c into master Dec 18, 2025
737 of 744 checks passed

jpbempel deleted the jpbempel/logs-capture branch December 18, 2025 17:32

github-actions bot added this to the 1.58.0 milestone Dec 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add App Logs Collection feature #10156

Add App Logs Collection feature #10156

Uh oh!

jpbempel commented Dec 10, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Dec 10, 2025 •

edited

Loading

Uh oh!

amarziali Dec 18, 2025

Uh oh!

jpbempel Dec 18, 2025

Uh oh!

amarziali Dec 18, 2025

Uh oh!

jpbempel Dec 18, 2025

Uh oh!

amarziali left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add App Logs Collection feature #10156

Add App Logs Collection feature #10156

Uh oh!

Conversation

jpbempel commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

amarziali Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

jpbempel Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

amarziali Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

jpbempel Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

amarziali left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jpbempel commented Dec 10, 2025 •

edited

Loading

pr-commenter bot commented Dec 10, 2025 •

edited

Loading