[monitoring] Add OpenTelemetry option

[the-glu]

This is a set of 3 PRs, proposition to add OpenTelemetry to generate optional traces.
You can check the base PR, the one for handlers and the one for datastores.

The idea is to contribute to #1257 by providing an option to enable traces.

This first PR provide base support, including HTTP & SQL traces.

Traces looks like that (in that PR):

As OpenTelemetry is a framework, this allow processing traces with various compatibles services. Self-hostable examples include jaeger, openobserve, Grafana Tempo, SigNoz. Multiples SaaS solutions exists (some previous tools provide them). As long as there is OpenTelemetry support it can be used, and that also mean it could be integrated into the existing stack of an organization.

As an example, same traces (of last PR) in SigNoz:

.

Idea behind thoses PR is to use it only for tracing, not metrics nor logging, however it also possible to do so in the future.

Goal is to have it optional behind a flag, and only enabled if needed, mostly for debugging. Configuration is done via OpenTelemetry's own flags.

Documentation is not done yet, but this can be tested with a local jaeger (port 16686) with the following changes:

diff --git a/build/dev/docker-compose_dss.yaml b/build/dev/docker-compose_dss.yaml
index e66d2ba4..d204104b 100644
--- a/build/dev/docker-compose_dss.yaml
+++ b/build/dev/docker-compose_dss.yaml
@@ -133,6 +133,7 @@ services:
       COMPOSE_PROFILES: ${COMPOSE_PROFILES}
       # Note: requires the Dockerfile to have been built with "-cover" in the EXTRA_GO_INSTALL_FLAGS var
       GOCOVERDIR: "/startup/coverdata"
+      OTEL_EXPORTER_OTLP_ENDPOINT: "http://jaeger:4317"
     command: /startup/core_service.sh ${DEBUG_ON:-0}
     ports:
       - "4000:4000"
@@ -178,6 +179,16 @@ services:
       start_period: 30s
       start_interval: 5s
 
+  jaeger:
+    image: jaegertracing/jaeger:2.14.0
+    ports:
+      - "16686:16686"
+      - "4317:4317"
+      - "4318:4318"
+      - "5778:5778"
+      - "9411:9411"
+    networks:
+      - dss_sandbox_default_network
 networks:
   dss_sandbox_default_network:
     name: dss_sandbox-default
diff --git a/build/dev/startup/core_service.sh b/build/dev/startup/core_service.sh
index f09bda59..8c96cf3e 100755
--- a/build/dev/startup/core_service.sh
+++ b/build/dev/startup/core_service.sh
@@ -44,5 +44,6 @@ else
   -enable_scd \
   -allow_http_base_urls \
   -locality local_dev \
-  -public_endpoint http://127.0.0.1:8082
+  -public_endpoint http://127.0.0.1:8082 \
+  -enable_opentelemetry
 fi

[mickmis]

LGTM modulo comments.

[mickmis]

LGTM modulo comments

[mickmis]

I see you changed that, and indeed for the better, however checking the semantic conventions, the second part should be the route template and not the actual path (e.g. /uss/v1/operational_intents/{entityid}/telemetry and not /uss/v1/operational_intents/4654654/telemetry).

[the-glu]

Yes I wanted to do that in the openapi-server generator (since here we don't have it), but I started some cleanup in it first.

I suggest we merge it like this for know and improve it in a follow up PRs. (And ev. switch back to HTTP here if that too problematic).

[the-glu]

(See #1391)