Opened 7 years ago

Closed 7 years ago

#2143 closed task (fixed)

Run benchmarks and write report

Reported by: tomek Owned by: tomek
Priority: medium Milestone: Sprint-DHCP-20120917
Component: documentation Version:
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: DHCP Feature Depending on Ticket:
Estimated Difficulty: 0 Add Hours to Ticket: 0
Total Hours: 0 Internal?: no

Description

Once benchmarks (tickets #2040, #2041, #2042) are implemented, they should be used to measure actual backend performance.

The results should be written down and a report of some kind should be created. It seems reasonable to use this opportunity to write first section of planned DHCP Performance Guide document.

Measurements should be taken for both synchronous and asynchronous mode.

Subtickets

Change History (10)

comment:1 Changed 7 years ago by tomek

  • Status changed from new to reviewing

Initial proposal for DHCP Performance Guide with performance measurement results was written. It is available on trac2040 branch. It can be generated by:

cd tests/tools/dhcp-ubench
make doc

Generated version is also available here:
file:///home/thomson/devel/bind10/tests/tools/dhcp-ubench/dhcp-perf-guide.html

comment:2 Changed 7 years ago by tomek

Benchmarks are to be rerun, this time with compiled statements enabled.

comment:3 Changed 7 years ago by tomek

The tests have been rerun with optimizations (-Ofast and recently implemented support for precompiled statements).

The ISC DHCP Performance Guide has been updated.

Note: there are several minor fixes in the benchmarking code.

Please review.

comment:4 Changed 7 years ago by shane

I'll also have a look. :)

comment:5 Changed 7 years ago by stephen

  • Owner changed from UnAssigned to stephen

comment:6 Changed 7 years ago by stephen

  • Owner changed from stephen to tomek

Very good idea to make this report part of the documentation - as well as giving us a framework and a start on the performance documentation, it means that we don't throw away any work.

I've made some editorial changes to the English and formatting and have pushed them to the repository.

Regarding content:

  • I think that "synchronous" and "asynchronous" should be explained somewhere (i.e. "synchronous" involves a disk write after each create/update/delete operation.) This clarifies what they mean in this context.
  • I'm not certain what was meant by the last sentence in the introduction to 3.4 ("This approach takes advantage of the fact that simple append is faster than edition with potential whole file relocation.")
  • The numbers in the tables can be restricted to three significant figures to improve readability without loss of any significant information.

comment:7 Changed 7 years ago by tomek

  • Owner changed from tomek to shane

Updated as requested.

Reassigning to Shane.

comment:8 Changed 7 years ago by shane

I also like that this is part of the documentation, however it has a very strange feel to it because of that. There are quite a few references to how temporary this document is, and what we are going to do with it, and so on. Maybe we can change the following:

Athough trivial now, they are expected to evolve into useful tools that will allow users to measure performance in their specific environment.

To be:

These benchmarks are expected to evolve into useful tools that will allow users to measure performance in their specific environment.

(Note typo for "Although" there, too.)

And perhaps:

It will be eventually merged with the main BIND10 makefile system, but that is a low priority for now.

Should just be:

It will be eventually merged with the main BIND10 build system.


A minor issue is that the MySQL "tweaks" section is a different format than both the SQLite and memfile benchmarks.


Also like Stephen mentioned, it is not clear from the text what synchronous vs. asynchronous is, except for the memfile case. Does asynchronous mean not using COMMIT, or doing it periodically, or...?

comment:9 Changed 7 years ago by shane

  • Owner changed from shane to tomek

I think it's okay to ship to Comcast, with or without addressing the questions in my review, BTW.

comment:10 Changed 7 years ago by tomek

  • Resolution set to fixed
  • Status changed from reviewing to closed

All things pointed out by Shane were fixed, and the changes were merged to master. Stephen will send out the report today.

Thanks for the review.

Note: See TracTickets for help on using tickets.