Prometheus Metric Description

Service Metrics #

Service metrics are generated from the server-side events, which are used to show the quality of service.

Metrics List #

Labels List #

Notes #

Note 1: The label namespace holds a value NOT_FOUND_INTERNAL when the container_id and the IP can’t be found in the current Kubernetes cluster, in which case the entity isn’t maintained by the current Kubernetes.

Note 2: The labels request_content and response_content hold different values when protocol is different.

• When protocol is http:

• When protocol is dns:

• When protocol is mysql:

• When protocol is kafka:

• When protocol is dubbo:

• For other cases, the request_content and response_content are both empty.

Note 3: The histogram metric kindling_entity_request_average_duration_nanoseconds_* is disabled by default as it could be high-cardinality. If this metric is needed, please add a new line to the exporters.otelexporter.metric_aggregation_map section of the configuration file.

exporters:
  otelexporter:
    metric_aggregation_map:
      # add the following line
      kindling_entity_request_average_duration_nanoseconds: histogram 

Topology Metrics #

Topology metrics are typically generated from the client-side events, which are used to show the service dependencies map, so the metrics are called “topology”. Some timeseries may be generated from the server-side events, which contain a non-empty label dst_container_id. These timeseries are generated only when the source IP is not the pod’s IP inside the Kubernetes cluster, which are useful when there is no agent installed on the client-side.

Metrics List #

Labels List #

Notes #

Note 1: We define two custom terms for the label src_namespace and dst_namespace, which are NOT_FOUND_INTERNAL and NOT_FOUND_EXTERNAL. The meanings are described as follows. These terms also apply to other metrics in this doc.

These two terms are composed of two parts.

1. NOT_FOUND: NOT_FOUND means the IP is neither a pod’s one nor a service’s one in the current Kubernetes cluster. The IP could belong to a host or an external service.
2. INTERNAL or EXTERNAL: There are two cases in which INTERNAL will be set. The first case is when the IP belongs to a node that resides in the current Kubernetes cluster. The second case is when the source or destination is running on the same host with the kindling agent, which is generally applicable for non-Kubernetes clusters. EXTERNAL is set for other cases if the IP is NOT_FOUND. Note another Kubernetes cluster is also considered “external”.

Note 2: The field “status_code” holds different values when “protocol” is different.

• HTTP: ‘Status Code’ of HTTP response.
• DNS: rcode of DNS response.
• MySQL: Error code of the error response.
• DUBBO: ‘Error Code’ of Dubbo request.
• others: empty temporarily

Note 3: The histogram metric kindling_topology_request_average_duration_nanoseconds_* is disabled by default as it could be high-cardinality. If this metric is needed, please add a new line to the exporters.otelexporter.metric_aggregation_map section of the configuration file.

exporters:
  otelexporter:
    metric_aggregation_map:
      # add the following line
      kindling_topology_request_average_duration_nanoseconds: histogram 

Trace As Metric #

We made some rules for considering whether a request is abnormal. For the abnormal request, the detail request information is considered as useful for debugging or profiling. We name this kind of data “trace”. It is not a good practice to store such data in Prometheus as some labels are high-cardinality, so we picked up some labels from the original ones to generate a new kind of metric, which is called “Trace As Metric”. The following table shows what labels this metric contains.

Metrics List #

Labels List #

TCP Status Metrics #

Metrics List #

Labels List #

TCP Socket Connects Metrics #

Metrics List #

Labels List #

Notes #

Note 1: The field success for kindling_tcp_connect_duration_nanoseconds_total is always true.

Note 2: The field errno is not 0 only if the TCP socket is blocking and there is an error happened. There are multiple possible values it could contain. See the ERRORS section of the connect(2) manual for more details.

Note 3: The field pid and comm will not exist if you set need_process_info to false (default is false), that will reduce the pressure of Prometheus.

PromQL Example #

Here are some examples of how to use these metrics in Prometheus, which can help you understand them faster.