Home

Benchmark Specifications and Results

Home » Benchmarks » Semantic Publishing Benchmark
The Semantic Publishing Benchmark (SPB) is an LDBC benchmark for testing the performance of RDF engines inspired by the Media/Publishing industry. In particular, LDBC worked with British Broadcasting Corporation (BBC) to define this benchmark, for which BBC donated workloads, ontologies and data. The publishing industry is an area where significant adoption of RDF is taking place.

There have been many academic benchmarks for RDF but none of these are truly industrial-grade. The SPB  combines a set of complex queries under inference with continuous updates and special failover tests for systems implementing replication.

SPB performance is measured by producing a workload of CRUD (Create, Read, Update, Delete) operations which are executed simultaneously. The benchmark offers a data generator that uses real reference data to produce datasets of various sizes and tests the scalability aspect of RDF systems. The benchmark workload consists of (a) editorial operations that add new data, alter or delete existing (b) aggregation operations that retrieve content according to various criteria. The benchmark also tests conformance for various rules inside the OWL2-RL rule-set.

The SPB specification contains the description of the benchmark and the data generator and all information about its software components can be found on the SPB developer page.

Semantic Publishing Benchmark (SPB) Audited Results for Scale Factors SF1 - 64M, SF3 - 256M and SF5 - 1G triples are shown below. 

Scale Factor

Interactive (Q/s)

 Updates (ops/sec)
Analytical
Cost
Software
Hardware
Test Sponsor
 Date
1  100.85 10.19 n.a. €37,504 GraphDB EE6.2 Xeon1650v3 6-core 3.5Ghz 96GB RAM ONTOTEXT AD 2015Apr26
1 142.7588 10.6725 n.a €35,323  GraphDB SE 6.3 alpha CPU Intel Xeon E5-1650 v3 3.5Ghz,15MB L3 cache, s2011 ONTOTEXT AD 2015Jun10
3 29.90 9.50 n.a. €37,504 GraphDB EE6.2 Xeon1650v3 6-core 3.5GHz 96GB RAM ONTOTEXT AD 2015Apr26
3

54.6364

9.4967

 n.a. €35,323  GraphDB SE 6.3 alpha CPU Intel Xeon E5-1650 v3 3.5Ghz,15MB L3 cache, s2011 ONTOTEXT AD 2015Jun10
1 149.0385 156.8325 n.a. $20,213 (€17,801 rate of 21/06/2015)

Virtuoso Opensource Version 7.50.3213

 Intel Xeon E5-2630, 6x 2.30GHz, Sockel 2011, boxed, 192 GB RAM OpenLink Software 2015Jun09
3 80.6158 92.7072 n.a $20,213 (€17,801 rate 21/06/2015) Virtuoso Opensource Version 7.50.3213 Intel Xeon E5-2630, 
6x 2.30GHz, Sockel 2011, boxed, 192 GB RAM		 OpenLink Software​ 2015Jun09
3 115.3838 109.8517 n.a $24,528 (€21,601 rate 21/06/2015) Virtuoso Opensource Version 7.50.3213 Amazon EC2, r3.8xlarge OpenLink Software 2015Jun10
5 32.2789 72.7192 n.a. $20,213 (€17,801 rate 21/06/2015) Virtuoso Opensource Version 7.50.3213 Intel Xeon E5-2630, 
6x 2.30GHz, Sockel 2011, boxed, 192 GB RAM		 OpenLink Software 2015Jun09
5 45.8101 55.4467 n.a $24,528 (€21,601 rate 21/06/2015) Virtuoso Opensource Version 7.50.3213 Amazon EC2, r3.8xlarge OpenLink Software 2015Jun10

The query substitution parameters for the scale factors SF1 and SF3 are given in the table below:

SF Query Substitution Parameters
1 generated.64.7z
3 generated.256.7z