Trace ID does not match lambda trace ID

[mathom]

For some reason the trace header coming into a WSGI app is not matching the trace from api gateway -> lambda. Needs investigation.

[mbeacom]

@mathom Could this bug be related to the handling here? https://github.com/mathom/xrayvision/blob/master/xrayvision/ext/wsgi/middleware.py#L25

[mathom]

In my testing it was getting a different trace ID than the lambda had, so it appears as a second, separate trace in there. The lower is just for the key, not the trace ID value itself.

[mathom]

Turns out the lambda trace ID (in the os.environ) is different than what cloudfront/api gateway gives you (in HTTP headers). Since the lambda trace ID has sampling and a parent ID I've decided to just prefer this over the CF one.

We might make this configurable if somebody wants to use the CF ID instead of the one from lambda.

[mathom]

Anyway here's the complete trace attached to everything:

[mathom]

I take it back - I'm going to default it to use the HTTP header ID. Here's why:

With the lambda parent ID, the UI doesn't display the downstream HTTP calls.

With the HTTP parent ID, it does.

Anyway, I've made this configurable on the WSGI wrapper. You can toggle it like this:

wsgi_app = XRayMiddleware(wsgi_app, 'myapp', prefer_lambda_trace=False)

If you set the prefer_lambda_trace to True (default is False, currently) it will pull the lambda ID and use that as the parent, resulting in the strange trace UI above.

This definitely needs some more investigation but for now we have some options to play with.

[mbeacom]

@mathom Sounds good. Oddly enough, using master with default settings, I am seeing - as URL in x-ray: wsgi_app = XRayMiddleware(wsgi_app, 'myapp').

[mathom]

The - URL should always show up as a result of non-URL tracing in the project. Are you seeing your project in the service map?