常见问题

• Error: Precompiled module at /opt/.kindling/ is not found
• Error: Prometheus cannot match error
• kindling probe init err: xxxxxx: Permission denied

Error: Precompiled module at /opt/.kindling/ is not found #

Problem description and logs #

If the kindling pod is not running normally, you can use the following command to check the logs.

kubectl logs --tail=100 -f kindling-agent-xxxx  -c kindling-agent -n kindling

* Mounting debugfs
* Failure to find a BPF probe, try to load kernel probe
* Unloading kindling-falcolib-probe, if present
* Trying to find precompiled kindling-falcolib-probe for 4.19.1-1.el7.elrepo.x86_64
* Error: Precompiled module at /opt/.kindling/ is not found, and the agent will not work as expected
Start kindling probe...
KINDLING_PROBE_VERSION: v0.1-2021-1221
Unable to load the driver
kindling probe init err: error opening device /host/dev/kindling-falcolib0. Make sure you have root credentials and that the kindling-falcolib

Solution #

1. You must install the kernel headers manually first.

For Debian-style distributions, run the command:

apt-get -y install linux-headers-$(uname -r)

For RHEL-style distributions, run the command:

yum -y install kernel-devel-$(uname -r)

Of course, you can also download RPM files manually to install the header files.

2. Rebuild the Kindling Agent image.

bash -c "$(curl -fsSL https://k8s-bpf-probes-public.oss-cn-hangzhou.aliyuncs.com/recompile-module.sh)"

The product of this step is an image. Please ensure that the image can be accessed by the k8s cluster node. Maybe you can upload the image to a private harbor, or compress it and distribute it to each node of the k8s cluster. (You should set the imagePullPolicy of kindling-agent to ifNotPresent when compress and distribute image.)

3. Finally, modify the image name of Kindling Agent.

kubectl set image ds/kindling-agent kindling-agent=kindlingproject/kindling-probe:bymyself -n kindling

Error: Prometheus cannot match error #

Problem description and logs #

After executing the script install.sh. The following error occurred:

error: unable to recognize "kindling-prometheus-servicemonitor.yml":
no matches for kind "ServiceMonitor" in version"monitoring.coreos.com/v1"

Solution #

If you have not installed Prometheus, please visit Prometheus-Operator to install it.

If you have installed Prometheus without operator, please ignore this error and manually configure the exposed metrics endpoint of Kindling Agent in Prometheus. The URL is http://localhost:9500/metrics.

kindling probe init err: xxxxxx: Permission denied #

Problem description and logs #

kindling probe init err: can't create map: Permission denied

or

insmod error:cloud not insert xxx.ko:Permission denied

Solution #

• Confirm that the kindling agent has enabled the privileged mode.
• Confirm that SELinux is turned off on the node where the kindling agent is located.
• Confirm that CRI is running with root privileges.