LDBC SPB (Semantic Publishing Benchmark) is based on the BBC linked data
platform use case. Thus the data modelling and transaction mix reflects
the BBC’s actual utilization of RDF. But a benchmark is not only a
condensation of current best practices. The BBC linked data platform is
an Ontotext Graph DB deployment. Graph DB was formerly known as Owlim.
So, in SPB we wanted to address substantially more complex queries than
the lookups that the BBC linked data platform primarily serves. Diverse
dataset summaries, timelines and faceted search qualified by keywords
and/or geography are examples of online user experience that SPB needs
to cover.
SPB is not per se an analytical workload but we still find that the
queries fall broadly in two categories:
- Some queries are centred on a particular search or entity. The data
touched by the query size does not grow at the same rate as the dataset. - Some queries cover whole cross sections of the dataset, e.g. find the
most popular tags across the whole database.
These different classes of questions need to be separated in a metric,
otherwise the short lookup dominates at small scales and the large query
at large scales.
Another guiding factor of SPB was the BBC’s and others' express wish to
cover operational aspects such as online backups, replication and
fail-over in a benchmark. True, most online installations have to deal
with these things, which are yet as good as absent from present
benchmark practice. We will look at these aspects in a different
article, for now, I will just discuss the matter of workload mix and
metric.
Normally the lookup and analytics workloads are divided into different
benchmarks. Here we will try something different. There are three things
the benchmark does:
- Updates - These sometimes insert a graph, sometimes delete and
re-insert the same graph, sometimes just delete a graph. These are
logarithmic to data size. - Short queries - These are lookups that most often touch on recent data
and can drive page impressions. These are roughly logarithmic to data
scale. - Analytics - These cover a large fraction of the dataset and are
roughly linear to data size.
A test sponsor can decide on the query mix within certain bounds. A
qualifying run must sustain a minimum, scale-dependent update throughput
and must execute a scale-dependent number of analytical query mixes or
run for a scale-dependent duration. The minimum update rate, the minimum
number of analytics mixes and the minimum duration all grow
logarithmically to data size. Within these limits, the test sponsor can
decide how to mix the workloads. Publishing several results, emphasizing
different aspects is also possible. A given system may be specially good
at one aspect, leading the test sponsor to accentuate this.
The benchmark has been developed and tested at small scales, between 50
and 150M triples. Next we need to see how it actually scales. There we
expect to see how the two query sets behave differently. One effect that
we see right away when loading data is that creating the full text index
on the literals is in fact the longest running part. For a SF 32 ( 1.6
billion triples) SPB database we have the following space consumption
figures:
- 46886 MB of RDF literal text
- 23924 MB of full text index for RDF literals
- 23598 MB of URI strings
- 21981 MB of quads, stored column-wise with default index scheme
Clearly, applying column-wise compression to the strings is the best
move for increasing scalability. The literals are individually short, so
literal per literal compression will do little or nothing but applying
this by the column is known to get a 2x size reduction with Google
Snappy. The full text index does not get much from column store
techniques, as it already consists of words followed by space efficient
lists of word positions. The above numbers are measured with Virtuoso
column store, with quads column wise and the rest row-wise. Each number
includes the table(s) and any extra indices associated to them.
Let’s now look at a full run at unit scale, i.e. 50M triples.
The run rules stipulate a minimum of 7 updates per second. The updates
are comparatively fast, so we set the update rate to 70 updates per
second. This is seen not to take too much CPU. We run 2 threads of
updates, 20 of short queries and 2 of long queries. The minimum run time
for the unit scale is 10 minutes, so we do 10 analytical mixes, as this
is expected to take 10 a little over 10 minutes. The run stops by itself
when the last of the analytical mixes finishes.
The interactive driver reports:
Seconds run : 2144
Editorial:
2 agents
68164 inserts (avg : 46 ms, min : 5 ms, max : 3002 ms)
8440 updates (avg : 72 ms, min : 15 ms, max : 2471 ms)
8539 deletes (avg : 37 ms, min : 4 ms, max : 2531 ms)
85143 operations (68164 CW Inserts (98 errors), 8440 CW Updates (0 errors), 8539 CW Deletions (0 errors))
39.7122 average operations per second
Aggregation:
20 agents
4120 Q1 queries (avg : 789 ms, min : 197 ms, max : 6767 ms, 0 errors)
4121 Q2 queries (avg : 85 ms, min : 26 ms, max : 3058 ms, 0 errors)
4124 Q3 queries (avg : 67 ms, min : 5 ms, max : 3031 ms, 0 errors)
4118 Q5 queries (avg : 354 ms, min : 3 ms, max : 8172 ms, 0 errors)
4117 Q8 queries (avg : 975 ms, min : 25 ms, max : 7368 ms, 0 errors)
4119 Q11 queries (avg : 221 ms, min : 75 ms, max : 3129 ms, 0 errors)
4122 Q12 queries (avg : 131 ms, min : 45 ms, max : 1130 ms, 0 errors)
4115 Q17 queries (avg : 5321 ms, min : 35 ms, max : 13144 ms, 0 errors)
4119 Q18 queries (avg : 987 ms, min : 138 ms, max : 6738 ms, 0 errors)
4121 Q24 queries (avg : 917 ms, min : 33 ms, max : 3653 ms, 0 errors)
4122 Q25 queries (avg : 451 ms, min : 70 ms, max : 3695 ms, 0 errors)
22.5239 average queries per second. Pool 0, queries [ Q1 Q2 Q3 Q5 Q8 Q11 Q12 Q17 Q18 Q24 Q25 ]
45318 total retrieval queries (0 timed-out)
22.5239 average queries per second
The analytical driver reports:
Aggregation:
2 agents
14 Q4 queries (avg : 9984 ms, min : 4832 ms, max : 17957 ms, 0 errors)
12 Q6 queries (avg : 4173 ms, min : 46 ms, max : 7843 ms, 0 errors)
13 Q7 queries (avg : 1855 ms, min : 1295 ms, max : 2415 ms, 0 errors)
13 Q9 queries (avg : 561 ms, min : 446 ms, max : 662 ms, 0 errors)
14 Q10 queries (avg : 2641 ms, min : 1652 ms, max : 4238 ms, 0 errors)
12 Q13 queries (avg : 595 ms, min : 373 ms, max : 1167 ms, 0 errors)
12 Q14 queries (avg : 65362 ms, min : 6127 ms, max : 136346 ms, 2 errors)
13 Q15 queries (avg : 45737 ms, min : 12698 ms, max : 59935 ms, 0 errors)
13 Q16 queries (avg : 30939 ms, min : 10224 ms, max : 38161 ms, 0 errors)
13 Q19 queries (avg : 310 ms, min : 26 ms, max : 1733 ms, 0 errors)
12 Q20 queries (avg : 13821 ms, min : 11092 ms, max : 15435 ms, 0 errors)
13 Q21 queries (avg : 36611 ms, min : 14164 ms, max : 70954 ms, 0 errors)
13 Q22 queries (avg : 42048 ms, min : 7106 ms, max : 74296 ms, 0 errors)
13 Q23 queries (avg : 48474 ms, min : 18574 ms, max : 93656 ms, 0 errors)
0.0862 average queries per second. Pool 0, queries [ Q4 Q6 Q7 Q9 Q10 Q13 Q14 Q15 Q16 Q19 Q20 Q21 Q22 Q23 ]
180 total retrieval queries (2 timed-out)
0.0862 average queries per second
The metric would be 22.52 qi/s, 310 qa/h, 39.7 u/s @ 50Mt (SF 1)
The SUT is dual Xeon E5-2630, all in memory. The platform utilization is
steadily above 2000% CPU (over 20/24 hardware threads busy on the DBMS).
The DBMS is Virtuoso open source,
(v7fasttrack at github.com,
feature/analytics).
The minimum update rate of 7/s was sustained but fell short of the
target of 70./s. In this run, most demand was put on the interactive
queries. Different thread allocations would give different ratios of the
metric components. The analytics mix is for example about 3x faster
without other concurrent activity.
Is this good or bad? I would say that this is possible but better can
certainly be accomplished.
The initial observation is that Q17 is the worst of the interactive lot.
3x better is easily accomplished by avoiding a basic stupidity. The
query does the evil deed of checking for a substring in a URI. This is
done in the wrong place and accounts for most of the time. The query is
meant to test geo retrieval but ends up doing something quite different.
Optimizing this right would almost double the interactive score. There
are some timeouts in the analytical run, which as such disqualifies the
run. This is not a fully compliant result but is close enough to give an
idea of the dynamics. So we see that the experiment is definitely
feasible, is reasonably defined and that the dynamics seen make sense.
As an initial comment of the workload mix, I’d say that interactive
should have a few more very short point lookups to stress compilation
times and give a higher absolute score of queries per second.
Adjustments to the mix will depend on what we find out about scaling. As
with SNB, it is likely that the workload will shift a little, so this
result might not be comparable with future ones.
In the next SPB article, we will look closer at performance dynamics and
choke points and will have an initial impression on scaling the
workload.