Modifications of async-profiler

Based on async-profiler , Kindling support the following features:

• [Modify] Real-time collected CPU Events
• [Modify] Real-time collected lock Events
• [Add] Periodic collected thread names
• [Add] Copy the agent from host to the container and attach it by jattach.
• [Modify] Make release

Real-time collected CPU Events #

./profiler.sh -e cpu -d 60 $pid

The original behaviors #

• Sample Event: Receive call stacks generated by perf_events, aggerate them into CallTraceStorage
• Aggregation Output: When stop profiling or after a specified period of time, like 60 seconds, the flame graph will be aggregated and output.

What we changed #

• Feature: Output the call stack into /dev/null in time.

  • Solution 1: Sample events and print the call stack into /dev/null
    • Problem - The operation of symbol parsing is not signal-safe which could make Java applications crash.
  • Solution 2: Sample Event just stores the call stack, and CallStack Output will parse the symbols and print by period.

  Sample Event: Add new API printSample()and printExternalSample() to replacerecordXXX() which store call stacks into FrameEventCache

  CallStack Output: Take call stacks from FrameEventCache every 10ms, parse symbols and print into /dev/null

• Feature: Do not crash the java application after attaching/detaching async-profiler N times.
  • FrameName is used for parsing symbols. When stopping the async-profiler, it will be destroyed and crash when free locale.
  • The local is added to fix Issue 94: Corrupted SVG due to Locale settings which is used to export SVG file. It’s not available in our case.

  Remove saved_local in FrameName

Real-time collected lock Events #

./profiler.sh -e lock -t $pid

Feature: Obtain the time-consuming caused by “waiting” in the Java program with the corresponding thread, the waiting object, and the code stack that executes the waiting procedure.

To implement this feature, the following JVM Tool Interface events are additionally enabled:

• JVMTI_EVENT_MONITOR_WAIT
• JVMTI_EVENT_MONITOR_WAITED

We added the LockRecorder class to record this information, created a Map to store waiting objects and their threads, and cleaned up invalid objects periodically. Once the waiting time for an object is obtained, the data will be output to /dev/null for the Kindling agent to collect.

Periodic collected thread names #

A new thread is started, which is used to output the name of each execution thread in the current process. The thread name list is printed to /dev/null. This data is collected by the Kindling agent and used to update the full thread name of processes.

The new thread’s name is “Async-profiler Threads Dump” and it prints data every 5 seconds.

Enhance Jattach #

For java application in container, jattach should access libasyncprofile.so by the same absolute path as on the host.

• Feature: Copy the agent into containers
  • Auto copy agent into /tmp/kindling/ of container.

• Feature: Allow users to attach agents with different versions
  • Store different agent in path /tmp/kindling/<version>/

Build Binary Package #

Update make release, which now packages Kindling Java Agent into a tarball.

async-profiler-2.8.3-linux-x64
├── agent
│   ├── 1.0.0
│   │   ├── agent-core.jar
│   │   └── plugin-traceid-sw.jar
│   ├── agent-boot.jar
│   └── version
├── build
│   ├── async-profiler.jar
│   ├── converter.jar
│   ├── fdtransfer
│   ├── jattach
│   └── libasyncProfiler.so
├── CHANGELOG.md
├── LICENSE
├── profiler.sh
└── README.md