The modern way to report profile statistics about an application (any application, not just Spark or Java application) is generating a single .SVG file called "flame graph". Since this is a regular vector graphic format, the file can be opened in any browser. Moreover, you can navigate between different stack frames by clicking on them, and even search for a symbol name by clicking on "Search" link.
This is how sample flame graph looks like:
In order to generate flame graphs, there are two mandatory processes usually:
- Capture stack traces from a running process, and dump them to disk.
- Parse these stack traces, and generate .SVG file.
In OpenJDK this feature is not available, but luckily there are other options. Once of them is using statsd JVM profiler library from Etsy. This library integrates as agent into JVM, gathers statistics like CPU or memory usage, and send it to statsd server in real time. Apparently, this library supports reporting to InfluxDB as well .
Keeping the above in mind, the whole process will look like this:
- Have InfluxDB running on random port.
- Start Spark with the statsd profiler Jar in its classpath and with the configuration that tells it to report statistics back to the InfluxDB instance.
- After running Spark application, query all the reported metrics from the InfluxDB instance.
- Run a script that generates the target report.
- Stop the InfluxDB instance.
- Store generated .SVG file somewhere, or send it to someone.
For the sake of justice, it should be noted that the following utilities must present on your system prior to running the script: perl, python2.7 and pip. Otherwise, the script was used in Amazon EMR environment without any issues. Just use the script instead of usual spark-submit command, and it will profile your application. and create a report:
[hadoop@ip-10-121-4-244 tmp]$ ./spark-submit-flamegraph --name 'etlite' --jars file://$(pwd)/probe-events-1.0.jar etlite_2.11-0.1.0.jar s3://mobility-artifacts/airflow/latest/config/etlite.conf [2017-06-05T12:34:05] Installing dependencies [2017-06-05T12:34:09] Starting InfluxDB [2017-06-05T12:34:10] Executing: spark-submit --jars /home/hadoop/.spark-flamegraph/statsd-jvm-profiler.jar,file:///tmp/probe-events-1.0.jar --conf spark.executor.extraJavaOptions=-javaagent:statsd-jvm-profiler.jar=server=10.121.4.244,port=48081,reporter=InfluxDBReporter,database=profiler,username=profiler,password=profiler,prefix=sparkapp,tagMapping=spark --name etlite etlite_2.11-0.1.0.jar s3://mobility-artifacts/airflow/latest/config/etlite.conf 17/06/05 12:34:11 INFO Main$: Configuration file = 's3://mobility-artifacts/airflow/latest/config/etlite.conf' 17/06/05 12:34:14 INFO S3NativeFileSystem: Opening 's3://mobility-artifacts/airflow/latest/config/etlite.conf' for reading 17/06/05 12:34:15 INFO SparkContext: Running Spark version 2.1.0 ... running Spark application ... 17/06/05 12:35:17 INFO SparkContext: Successfully stopped SparkContext 17/06/05 12:35:17 INFO ShutdownHookManager: Shutdown hook called 17/06/05 12:35:17 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-fa12133c-b605-4a73-814a-2dfd4ed6fdde ... generating .svg file ... [2017-06-05T12:35:25] Created flamegraph: /tmp/flamegraph.svg
Integrating this script into Airflow Spark operator is straightforward, especially if your Spark operator is derived from BashOperator. Just make sure the script is available on all Spark Airflow workers, then do the replacement of spark-submit command depending on whether profile=True is passed as the operator argument.
Post your weird flame graphs in comments! :)
