The Graphalytics benchmark is an industrial-grade benchmark for graph analysis platforms such as Giraph, Spark GraphX, and GraphBLAS. It consists of six core algorithms, standard datasets, and reference outputs, enabling the objective comparison of graph analysis platforms.

The benchmark harness consists of a core component, which is extendable by a driver for each different platform implementation. The benchmark includes the following algorithms:

1. breadth-first search (BFS)
2. PageRank (PR)
3. weakly connected components (WCC)
4. community detection using label propagation (CDLP)
5. local clustering coefficient (LCC)
6. single-source shortest paths (SSSP)

The choice of these algorithms was carefully motivated, using the LDBC TUC and extensive literature surveys to ensure good coverage of scenarios. The standard datasets include both real and synthetic datasets, which are classified into intuitive “T-shirt” sizes (S, M, L, etc.).

Each experiment set in Graphalytics consists of multiple platform runs (a platform executes an algorithm on a dataset), and diverse set of experiments are carried out to evaluate different performance characteristics of a system-under-test.

All completed benchmarks must go through a strict validation process to ensure the integrity of the performance results.

The development of Graphalytics is supported by many active vendors in the field of large-scale graph analytics. Currently, Graphalytics already facilitates benchmarks for a large number of graph analytics platforms, such as GraphBLAS, Giraph, GraphX, and PGX.D, allowing comparison of the state-of-the-art system performance of both community-driven and industrial-driven platforms. To get started, the details of the Graphalyics documentation and its software components are described below.

Documents and repositories

• Benchmark specification. The source code is stored in the ldbc_graphalytics_docs repository
• VLDB paper
• ldbc_graphalytics: Generic driver
• ldbc_graphalytics_platforms_umbra: Umbra implementation
• ldbc_graphalytics_platforms_graphblas: GraphBLAS implementation

Graphalytics competition 2023

In 2023, we will hold a new round of the Graphalytics competition. See the LDBC Graphalytics Benchmark presentation for an introduction to the benchmark framework and the competition’s rules.

Artifacts:

• benchmark framework
• reference implementations
• datasets (datasets and expected results) are available on GitHub

Rules

• Participation is free.
• There are no monetary prizes.
• Single-node and distributed implementations are allowed.
• Partial implementations (e.g. just small to mid-sized datasets and only a few algorithms) are allowed.
• Submissions should execute each algorithm-dataset combination three times. From these, the arithmetic mean of the processing times is used for ranking.
• The results of the competition will be published on the LDBC website in the form of leaderboards, which rank them based on performance and price-performance (adjusted for the system price).
• There is a global leaderboard that includes all algorithms and scale factors. Additionally, there is a separate leaderboard for each scale (S, M, L, XL, 2XL+), algorithm and system category (CPU-based/GPU-based, single-node vs. distributed) to for fine-grained comparison.
• Submissions are subject to code review and reproducibility attempts from the organizers.
• System prices should be reported following the TPC Pricing specification.

Recommendations for submissions

• Submissions using modern hardware are welcome (GPUs, FPGAs, etc.).
• We encourage the use of cloud compute instances for running the benchmark (if possible).

Data sets

The Graphalytics datasets are compressed using zstd. The total size of the compressed archives is approx. 350 GB. When decompressed, the datasets require approximately 1.5 TB of disk space.

For detailed information on the datasets, see the table with their statistics.

The datasets are available in a public Cloudflare R2 bucket.

• Shell script to download the datasets from Cloudflare R2
• Download scripts for individual sizes: test graphs, sizes up to S, size M, size L, size XL, sizes 2XL+
• CWI/SURFsara data repository

Note that some of the Graphalytics datasets were fixed in March 2023. If you downloaded the datasets prior to this point, some datasets had missing/incorrect reference outputs for certain algorithms. Therefore, we recommend to download the datasets again.