SVBench: Scientific Visualization Benchmarking Suite

The SVBench benchmark suite is designed to examine visualization processes on a diverse set of system configurations and has the flexibility to evaluate and steer the evolution of system configurations. The front-end to the benchmark suite is a set of scripts that generate batch job submission scripts, parsing scripts, and graph generation scripts. The benchmark suite is straightforward to update as it consists of a set of bash and python scripts which interact with python modules that either directly use system timing information or application logs to extract the pertinent information. Paraview was selected as the primary benchmark applications as it is able to analyze extremely large datasets using distributed memory computing resources across scientific disciplines. Some scientific disciplines use specialized software for visualization and the benchmark suite can be extended to also include these applications where appropriate. For example, molecular science research account for a quarter of TACC research projects, but Paravew and VisIt do not have tools to represent protein secondary structure, and are therefore not typically used for molecular visualization of proteins. To meet the needs of this research community, the benchmark suite also includes the most commonly used application for molecular science visualization, VMD.

Motivation

High performance computing systems are designed to allow users to request resources (nodes, processors, memory, accelerators, etc) to match the demands of their workload. In order to design a generalized benchmark suite that will be reproducible and scalable to a variety of target platforms, it is necessary to reduce the parameter space (both hardware and software options) to a manageable subset of hardware configurations. The overall visualization pipeline includes the visualization algorithm computation (isosurfacing, streamline computation), rendering algorithm computation (ray tracing, rasterization), image generation and data I/O. While each step of the pipeline can significantly effect the overall performance, the focus of this benchmark suite will be on the rendering performance (geometry to pixels) and determining the bottlenecks in rendering performance. The goal of this benchmark suite is to capture "typical" visualization use-cases across a variety of disciplines, and also to design a series of performance tests that allow useful metrics to be extracted from each visualization workflow. To narrow down the wide-variety of visualization tasks that might be used for a particular field of science, a survey of TACC usage statistics was used to prioritize research areas and select workflows that are representative of the research field as a whole.