K8s: annotate the events originating from kubernetes with pod metadata

[MOZGIII]

Annotate the events originating from the k8s with pod metadata, such as:

• pod namespace
• pod name
• pod uid
• container name
• pod labels
• etc...

Part of #2222.

Focus os in-source implementation, but if possible, implement generically, so the implementation can be reused within a transform that annotates the metadata.

[rrichardson]

Fantastic work. I have this running in our stage k8s cluster.

I am wondering what the final shape of the pod labelling will take. I am targetting Loki, so I'd like to be able to specify labels by which the events will be indexed. Unfortunately, I don't think that there is a standard way to group pods by their deployment without using labels, which means there is no standard kubernetes naming convention.

This means that I'd need to be able to inform the kubernetes_logs source about what pod metadata I want turned into what event-fields. Is this part of the design?

[MOZGIII]

kubernetes_logs passes all the labels as an key-value map field. We'll probably have to make loki sink work with map fields to allow passing pod labels as loki labels.

See also: #2710 (for others checking this out)

[rrichardson]

I just noticed that the labels follow the same convention as all of the other event fields in loki, so this actually works:

mydomain_service = "{{ kubernetes.pod_labels.service.mydomain.com }}"

[MOZGIII]

Oh, yeah! If you want just one, known upfront, label - just specifying it will work.
Anothe example would be k8s_pod_label_app = "{{ kubernetes.pod_labels.app.kubernetes.io/name }}"