Opened 8 years ago

Closed 8 years ago

#1264 closed task (complete)

Design document for DHCP benchmarking utility

Reported by: stephen Owned by: johnd
Priority: medium Milestone: Sprint-DHCP-20111026
Component: dhcp Version:
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: DHCP Feature Depending on Ticket:
Estimated Difficulty: 0 Add Hours to Ticket: 0
Total Hours: 0 Internal?: no

Description

A short, high-level document describing the benchmarking utility. In particular, basic structure (probably very simple) plus a description of how the measurements are done (e.g. whether there is a synchronous exchange of packets or whether packets are sent to the system asynchronously).

Subtickets

Change History (14)

comment:1 Changed 8 years ago by stephen

  • Owner set to johnd
  • Status changed from new to reviewing

Design document can be found on the same page as the requirements - DhcpBenchmarking

Some questions and comments:

Question: what is the significance of All_DHCP_Relay_Agents_and_Servers in the description of the IPv6 operation?

The IDs in requests sent to the server are made unique as necessary to track responses. In order to calculate response latency, either a table of transmitted packet IDs with transmit-times or some means of mapping ID to transmit time is used.

Question: does this imply a need to add a command option to set a base ID number so that we can use multiple instances of perfdhcp from different machines without confusion?

Comment: a table might be the simplest - allocate an array "num-request" elements in length and index into it using "packet-id - base-packet-id" (which implies that packets are sent with increasing IDs). If we set the default value of "-n" to one million, even with an 64-bit time field only 8MB of space would be used.

Last edited 8 years ago by stephen (previous) (diff)

comment:2 Changed 8 years ago by johnd

In DHCPv6, All_DHCP_Relay_Agents_and_Servers is the multicast address to which SOLICIT requests are transmitted. Servers listen at this address (FF02::1:2).

It should be possible to create IDs that are machine-specific (e.g. including a MAC address) while also including a sequence number. However, it now occurs to me that a fixed mapping of sequence number to transmit time would in any case only be sufficient for the first round of timing (D-O or S-A); a table will be needed to track both latencies of a 4-way exchange. It may still be useful to have options to control sequence starting points, etc.

comment:3 Changed 8 years ago by johnd

  • Owner changed from johnd to stephen

comment:4 follow-up: Changed 8 years ago by stephen

  • Owner changed from stephen to johnd

In DHCPv6, All_DHCP_Relay_Agents_and_Servers is the multicast address to which SOLICIT requests are transmitted. Servers listen at this address (FF02::1:2).

As the tool will be distributed with BIND 10 DHCP, the code may be used by people not familiar with the details of DHCP for IPV6, so you may want to modify the description, e.g.

...to
All_DHCP_Relay_Agents_and_Servers (the broadcast address FF02::1:2) via this
interface.

It should be possible to create IDs that are machine-specific (e.g. including a MAC address) while also including a sequence number. However, it now occurs to me that a fixed mapping of sequence number to transmit time would in any case only be sufficient for the first round of timing (D-O or S-A); a table will be needed to track both latencies of a 4-way exchange.

How about using even-numbered IDs for the initial packet exchange and (initial exchange ID + 1) as the ID for the second packet exchange? If you use a four-column table (start/end time of first exchange, start/end time of second exchange) the ID will uniquely identify the packet exchange and the latency. When you've decided on this point, a note to that effect should be added to the design.

On a related point, with the -n switch we might want to add a -o<file> option to output table in the form of a CSV file. That would allow further analysis by external tools

comment:5 Changed 8 years ago by johnd

I've modified the description & design document as suggested. A -o option sounds useful, to be firmed up once free-formatted output is settled.

comment:6 Changed 8 years ago by johnd

  • Owner changed from johnd to stephen

comment:7 follow-up: Changed 8 years ago by stephen

  • Owner changed from stephen to johnd

Additional comments can be found in comment 2 and comment 3 of #1263.

Replying to comment 3 johnd

I'm thinking, for the initial version, from these suggestions adding:
:

  • Explicit description of what the exit status means. Perhaps just status 1 if any dropped packets, unless other pass criteria options are added.

For now, I think that would be sufficient.

Should any of the other suggested options be in the initial version?

See below.

Replying to johnd

once free-formatted output is settled.

My initial thought was date/time but given the difficulty of parsing various formats, I think that for each event, a simple time (in seconds) since the start of the test expressed as a floating-point number would be sufficient. So the output format would be:

perfdhcp version
Command line options
Date/time test started
send_time,receive_time[,send_time_packet_2,receive_time_packet_2]
send_time,receive_time[,send_time_packet_2,receive_time_packet_2]
:

(with the date in yyyy-mm-dd format to avoid confusion between dd/mm/yy and mm/dd/yy.)

The last two columns are optional, being absent if only the initial packet exchange is measured. And if a packet is lost, put -1.0 in the receive field (and in the other two fields if a full 4-way packet exchange is being measured).

I've included the version and other data in the output as the first few lines; Tomek's point about missing information when reproducing a problem is very pertinent.

Replying to tomek

I have couple of comments.

Hmm... this is a definition of "couple" of which I was previously unaware :-)

In DHCPv6 we need 3 transmission modes:

That sounds reasonable. Re-reading the perfdhcp command line, the use of the command-line argument is inconsistent: for IPv4, it is the address to which packets are sent, in IPv6 it is the interface from which packets are sent.

I suggest that the interface be specified with the -l option. This already sets the local hostname/address for a IPv4 packet exchange - it could specify the local interface for an IPv6 exchange. The target to which packets are should be the argument to the command line. To simplify things for V6 use, as well as allowing an IPv6 address, the program should also recognise the strings "all" (for All_DHCP_Relay_Agents_and_Servers) and "servers" (for All_DHCP_Servers)

There is also rapid-commit option that, when supported by both server and client, will cause SOLICIT to be answered immediately with REPLY. That is not needed in first version, but it is something that we should plan to implement later.

Agreed. Ticket #1334 has been raised for it and put on the general backlog.

Regarding the -r option, it is useful, but it is not enough.
:
To meet those usages, -time (or -t) option should be added that specifies duration...

Agreed. Since r * t = n, the command parser should accept any two and calculate the third (objecting if all three are given). I would suggest that the default for r be something like 10/second; if neither t nor n is specified, assume a value of n equal to a 232 - 1, i.e. essentially unlimited.

(As an aside, allowing very large values for n complicates the mapping of packet ID to information about the exchange as a simple pre-allocated array cannot be used. However some form of double-buffer - where a buffer can be reused once a the time equal to (time last packet using this buffer was sent + packet drop time) has passed - should work.)

Other things that we should consider at a later date is turning this into stress testing. Let's call it --torture or similar. It starts sending data at some rate and increses it slowly until server starts dropping. That is the maximum rate the server can handle.

Agreed. Ticket #1335 has been raised for it and put on the general backlog.

There should be option to conclude (fail) the test if there is a single drop. We don't want to wait 12 hours to see that 5 seconds after test started something broke. Not sure how to implement this in the most convenient way. Maybe --drop-threshold that specified acceptable amount of dropped traffic? It seems useful to have it specified in both percentage and absolute numbers.

I suggest that it be specified as simple packets for now with something like "-t<lost-packets>". If not specified, there is no limit to the number of dropped packets.

Besides of using dhcperf as manual tool, it will also be used as automated test. In that case it should have clearly state if specified pass criteria are met or not. Something that could be easily parsed by automated environments.

Make sure that return code will specify status.

This sounds useful, although I'm not clear what you mean here. In any case, I think it is something that can be added later. Could you raise a ticket for it?

for now, as suggested above, the return code should be 1 if any packets were dropped.

For automated test tools it is very convenient to print out command-line parameters. That's a practical experience. I received many logs that were useless because it was not possible to reproduce the problem due to missing information about used parameters.

See above when the "-o" option is specified. Do you think there is a need to echo them if used interactively, at a terminal?

There is no --version parameter. Tool should also print out its version when started. See above comment about reproduction concerns.

Agreed. I would make the "-v" switch do this.

As to verbose option, I suggest this be merged with the debug option. In a small program such as this it is probably more different areas of the code you want debugging information for than different levels of debug information. If, instead of a "debug-level", the argument to the -x switch were a "debug-mask", different pieces of debug information could be output by setting bits in the mask value. Displaying the packet contents and communication would then require using -x with a value that has the appropriate bits set.

Another feature that could make this tool much more powerful is the ability to specify additional options. While it would be great to have custom option definition framework, for now we can do something much simpler. A command-line option that specifies extra payload that is appended to the message. A proper warning "it is user's responsibility to take care a proper format". For example, to specify that I want to send option type 100 with length 2 containg 0xabcd, I could do: --extra-data 00:64:00:02:ab:cd. This seems simple enough (parse command-line + a single memcpy will do the trick)

Another useful thing would to be to specify which options client should request. That is also not too difficult. This is just adding 8 bit(v4) or 16bit (v6) integers to PRL or ORO, respectively. Usage could be simple: --option 45 --option 5.

Added #1336 to the backlog.

It would be useful to elaborate on reply verification. V4 server responding with NACK is ok or not? What about v6 server sending REPLY with status-code=no-addrs-avail? That is another thing we could eventually add as a feature. In some scenarios negative response as considered a proper one (test passed) and in others it is not (test failed). Make sure that the verification could be tuneable. For now it can be simple, but it will be more complex later.

Added #1337 to the backlog.

The options suggested for immediate inclusion do not seem too much work, and tickets have been raised for the more complicated stuff. I suggest the design document is updated with these suggestions then we close the ticket and start work on implementation. Given our time constraints for the work, we need to do this ASAP.

comment:8 in reply to: ↑ 4 Changed 8 years ago by tomek

Replying to stephen:

In DHCPv6, All_DHCP_Relay_Agents_and_Servers is the multicast address to which SOLICIT requests are transmitted. Servers listen at this address (FF02::1:2).

As the tool will be distributed with BIND 10 DHCP, the code may be used by people not familiar with the details of DHCP for IPV6, so you may want to modify the description, e.g.

...to
All_DHCP_Relay_Agents_and_Servers (the broadcast address FF02::1:2) via this
interface.

Not true! There are no broadcast addresses in IPv6. That is multicast address.

It should be possible to create IDs that are machine-specific (e.g. including a MAC address) while also including a sequence number. However, it now occurs to me that a fixed mapping of sequence number to transmit time would in any case only be sufficient for the first round of timing (D-O or S-A); a table will be needed to track both latencies of a 4-way exchange.

How about using even-numbered IDs for the initial packet exchange and (initial exchange ID + 1) as the ID for the second packet exchange? If you use a four-column table (start/end time of first exchange, start/end time of second exchange) the ID will uniquely identify the packet exchange and the latency. When you've decided on this point, a note to that effect should be added to the design.

Four-column table seems reasonable. Note that first exchange (DO or SA) is usually quick as there is no actual lease assignment. It is expected that second part (RA or RR) will be much longer. Both measurements are useful. Also total sum of DO+RA and SA+RR should be printed as well.

On a related point, with the -n switch we might want to add a -o<file> option to output table in the form of a CSV file. That would allow further analysis by external tools

Good idea. Format should be documented somewhere. As a convenience for user, it can be useful to print first line commented with names of columns + units used (seconds, ms, bytes etc.). That's just a trivial printf() in the code, but it sometimes helps a lot if user doesn't know where to find documentation.

comment:9 in reply to: ↑ 7 Changed 8 years ago by tomek

Replying to stephen:

Replying to tomek

I have couple of comments.

Hmm... this is a definition of "couple" of which I was previously unaware :-)

I planed this comment to be short, but it kind of got carried away.

Regarding the -r option, it is useful, but it is not enough.
:
To meet those usages, -time (or -t) option should be added that specifies duration...

Agreed. Since r * t = n, the command parser should accept any two and calculate the third (objecting if all three are given). I would suggest that the default for r be something

No necessarily object. In stability or stress testing, these can be considered exit criteria. Tool should conclude test whichever comes first. "Try to get 1 million leases, but I don't want this test to last longer than 1 hour" is a valid test scenario.

like 10/second; if neither t nor n is specified, assume a value of n equal to a 232 - 1, i.e. essentially unlimited.

(As an aside, allowing very large values for n complicates the mapping of packet ID to information about the exchange as a simple pre-allocated array cannot be used. However some form of double-buffer - where a buffer can be reused once a the time equal to (time last packet using this buffer was sent + packet drop time) has passed - should work.)

We may ignore this aspect for our first release, but this tool should have reasonable memory requirements. When used in stability test (e.g. hammer our server over a weekend), it should not take much more memory than during relatively short test. Some form of sliding window should be used.

Besides of using dhcperf as manual tool, it will also be used as automated test. In that case it should have clearly state if specified pass criteria are met or not. Something that could be easily parsed by automated environments.

Make sure that return code will specify status.

This sounds useful, although I'm not clear what you mean here. In any case, I think it is something that can be added later. Could you raise a ticket for it?

I believe there is no ticket necessary. That is really simple. dhcperf should print out TEST PASSED or TEST FAILED as one of the last line. That will be easy to parse by a script or execution environment of some sort.

Note that just giving PASS/FAIL status without explanation is not enough. If test fails, it should explain why. For example "Maximum allowed dropped packets:1 actually dropped: 5". That is another suggestion from experience. In time, we will use this tool to run many different test scenarios in automated environment somewhat similar to our build bot. When we spot that test failed, the first question will always be "why?". Having the ability to just scroll down the log file and get first rough idea about what went wrong is very useful.

for now, as suggested above, the return code should be 1 if any packets were dropped.

Sounds good.

See above when the "-o" option is specified. Do you think there is a need to echo them if used interactively, at a terminal?

Yes. People often redirect terminal output to a file. Or even copy-paste from terminal if something breaks. Printing out a single line does not make the log much bigger, and it will spare us troubles.

comment:10 Changed 8 years ago by tomek

Came up with 2 more things that could be useful:

  1. Optional ability to write traffic to a file (ticket #1338)
  1. perfdhcp should warn if there are any DHCP clients running (ticket #1339)

Ok. That's it. I have more ideas how to expand this tool, but we have to be realistic about our goals.

comment:11 Changed 8 years ago by johnd

I've modified the design document with the previously discussed changes.

I've kept rate/num/period separate, rather than trying to calculate between them, because -r really represents an upper limit; resource limitations may prevent it from being reached, in which case we want to be able to exit if either num or period are reached.

I've merged verbose/debug mask style as suggested.

I had made the non-option argument mean different things for v4 and v6 because the usual
parameter to specify is different for these two. However, for the sake of consistency
I've changed the server argument and -l as suggested.

comment:12 Changed 8 years ago by johnd

  • Owner changed from johnd to stephen

comment:13 Changed 8 years ago by stephen

  • Owner changed from stephen to johnd

OK, that (version 6 of DhcpBenchmarking) will do for now. Close this ticket and write it!

Last edited 8 years ago by stephen (previous) (diff)

comment:14 Changed 8 years ago by stephen

  • Resolution set to complete
  • Status changed from reviewing to closed
Note: See TracTickets for help on using tickets.