Home

Information about the overall goals of the Linked Data Benchmark Council

Home » Public » Why Benchmarking

Benchmarking is a concept introduced by economists and business administrators to compare one's business processes to industry bests or best practices from other industries. Dimensions typically measured are quality, time and cost. This concept has also successfully been applied to information technology to compare computer hardware and software. The most relevant two examples for LDBC of industry benchmarking consortia in information technology are SPEC and the TPC.

The SPEC (Systems Performance Evaluation Cooperation) is a non-profit organization of major computer hardware vendors formed to establish, maintain and endorse a standard set of relevant benchmarks that can be applied to the newest generation of high-performance computers. It has developed multiple benchmarks. One example is SPECcpu, which was introduced in 1992, and is one of the tools people use to compare the performance new computer processors, such as Intel Xeon, AMD Opteron and IBM Power. The benchmark has gone to various updates of its workloads, and offers performance metrics such as SPECint2006, SPECfp2006 for integer and floating-point performance and SPECint_rate2006 and SPECfp_rate2006 for comparing performance scalability with multiple cores. SPEC benchmarks are typically not audited, and the benchmark software is available at a small fee.

The TPC (Transaction Processing Council) has had strong success in standardizing benchmarks for relational database systems, and played an important role in maturing relational database technology, transforming and simplifying IT infrastructures worldwide. The TPC was founded in 1988 by eight companies to create clear rules to compare results of Jim Gray's DebitCredit transaction benchmark, which later evolved into the TPC-C benchmark. TPC-C simulates a company selling products to customers and measures throughput (transactions completed per second), while imposing quality of service requirements and requiring data sizes to scale with the performance level achieved. This effectively forces vendors to run systems on large data sizes that do not fit into RAM, and leads to test systems with extremely large storage infrastructures, making posting a new benchmark high score an expensive affair. TPC also maintains a successful benchmark for analytical workloads, called TPC-H, that tests relational data warehousing performance both when the system is used in isolation (the Power metric) as well as under concurrent workload (the Throughput metric), which also features bulk inserts and deletes. TPC introduced audited benchmarks, where an independent consultant authorized by TPC performs an on-site visit to check if the tested system complies with all requirements and the benchmark experiments were carried out fairly. This imposes auditing cost in creating an official result. The TPC benchmark software is freely available, though not open source.

Both SPEC and TPC are examples for LDBC, which is also an industry consortium, but with company members from the emerging graph and RDF database industry. With the the growing interest in graph data management, LDBC is looking to accelerate the maturing of this new software market and is ready to cooperate with existing organizations in achieving its goal of creating useful benchmarks for graph data management workloads. LDBC benchmarks are designed to provide meaningful comparisons even when run on diverse platforms (from single computers to clusters) by including a metric that normalizes performance to monetary system cost. LDBC's result auditing will be more lightweight than TPC auditing as it can be performed using remote access, and all its benchmark software is released in open-source and available on github.