add X-Trace-Id and traceparent headers to all HTTP calls

[adamkorynta]

this will allow for log correlation from CDA for all logs emitted from a single request

See: USACE/cwms-data-api#1682

[MikeNeilson]

While not nothing, the value should be per session, not per request. Or at least within a Span.

example desire:

1. User clicked Extract (in HEC-RTS)
2. Generate initial x-trace-id/traceparent
3. All HTTP requests get those headers (and like ALL HTTP REQUESTS is valid here, but we can focus on CDAC)
4. Additionally local logs would use the same x-trace-id/traceparent
5. ...Profit. (actually somewhat serious there, should take far less time to correlate log entries and figure what the heck is going on for any given situation)

[adamkorynta]

I think if we're wanting a session id, we should add in a X-Client-Identifier header that is constant for the full session. For the case of extract, I'll then pull in the opentelemetry api on the client side for propagating real traceparent identifiers.

[adamkorynta]

It looks like OpenTelemetry uses baggage: https://opentelemetry.io/docs/concepts/signals/baggage for the client id rather than a custom header.

[MikeNeilson]

NOTE: CDA was updated to use Otel itself to handle the W3C trace parent propagation. However, I'm not sure if handles baggage by default or if that's another Otel "Propagator"

[MikeNeilson]

NOTE: CDA was updated to use Otel itself to handle the W3C trace parent propagation. However, I'm not sure if handles baggage by default or if that's another Otel "Propagator"

It does not, A Baggage propagator would need to be added. Whether we want to do so or not is a separate discussion.

[MikeNeilson]

I think one change that should be made, don't manually create the TraceIDs this way. Operate as though the old system, upstream and down from this, are using OTEL and appropriately create/nest Spans and retrieve the required TraceParent/Span ID from the span context.