Jignesh redid MonetDB’s benchmark

January 13, 2009 at 8:33 am (development, geek) (, )

Knowing that MonetDB updated TPC-H benchmark result and indicating that postgresql performance is troublesome, he redid the benchmark on his own.

I redid a quick test with the same workload on one of my systems with SF
10 which is about 10GB
(I hope it comes out properly displayed)

Jignesh From Monet (8.3T/8.2.9)

..details skipped. Follow the link to see …

All time is in seconds (sub seconds where availabe)
Ratio > 1 means 8.3.3 is slower and <1 means 8.3.3 is faster

My take on the results:

* I had to tweak the statement of Q1 in order to execute it.
(TPC-H kit does not directly support POSTGRESQL statements)

* Timings with 8.3.3 and bit of tuning gives much better time overall
This was expected (Some queries finish in 7% of the time than what
MonetDB reported. From the queries that worked only Q7 & Q21 seem to
have regressed)

* However Empty rows results is occuring consistently
(Infact Q11 also returned empty for me while it worked in their test)
Queries: 4,5,6,10,11,12,14,15
(ACTION ITEM: I will start separate threads for each of those queries in
HACKERS alias to figure out the problem since it looks like Functional
problem to me and should be interesting to hackers alias)

* Two queries 17,20 looks like will not finish (I let Q17 to run for 18 hrs and yet it had not completed. As for Q20 I killed it as it was approaching
an hour.)
(ACTION ITEM: Not sure whether debugging for these queries will go in
hackers or perform alias but I will start a thread on them too.)

* Looks like bit of tuning is required for Q1, Q7, Q9, Q21 to improve their
overall time. Specially understanding if PostgreSQL is missing a more
efficient plan for them.
(ACTION ITEM: I will start separate threads on performance alias to
dig into those queries)

I hope to start separate threads for each queries so we can track them
easier. I hope to provide explain analyze outputs for each one of them
and lets see if there are any problems.

Feedback welcome on what you want to see for each threads.

Regards,
Jignesh

2 Comments

  1. Anonymous said,

    January 14, 2009 at 2:01 pm

    Try running a benchmark where the database does not fit in memory. In those cases, PostgreSQL will easily beat MonetDB, MonetDB cannot handle it (at least last time I checked). MonetDB will run for hours or days swapping like crazy.

    Also, try running with the latest PostgreSQL 8.4 beta, a couple of the troublesome queries are now *much* faster.

    Reply

  2. René Leonhardt said,

    March 26, 2009 at 9:32 pm

    With PostgreSQL 8.4 Beta 1 near at hand it would be interesting to know what the current TPC-H status is.

    Reply

Post a Comment