Opened 8 years ago

Closed 8 years ago

#1263 closed task (complete)

Finish current experimental DHCP benchmarking work

Reported by: stephen Owned by: johnd
Priority: medium Milestone: Sprint-DHCP-20111026
Component: dhcp Version:
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: DHCP Feature Depending on Ticket:
Estimated Difficulty: 0 Add Hours to Ticket: 0
Total Hours: 0 Internal?: no

Description

Finish current benchmark experiments in sending and receiving DHCP packets. No code will be committed to the repository as a result of this, the ticket being more for tracking purposes than anything else.

Subtickets

Change History (7)

comment:1 Changed 8 years ago by johnd

  • Owner set to johnd
  • Status changed from new to accepted

comment:2 Changed 8 years ago by tomek

I have couple of comments. In DHCPv6 we need 3 transmission modes:

  • sent to All_DHCP_Relay_Agents_and_Servers link-local multicast address. This is the default mode for clients and so it should be the default for this tool.
  • sent to All_DHCP_Servers. This is the address that relays forward packets to. Server that supports relays will also listen on this address. This should be the second (in order of importance) address.
  • sent to unicast address. Servers may allow unicast configuration.

There is also rapid-commit option that, when supported by both server and client, will cause SOLICIT to be answered immediately with REPLY. That is not needed in first version, but it is something that we should plan to implement later.

There's one significant concern about how -r option will work. In particular that it will require interruption to finish operation. That will make any automated tests infeasible. That is ok if we also implement method to run test automatically (i.e. execute and conclude without any intervention, so it could be used from a script or fancier test environment)

Regarding the -r option, it is useful, but it is not enough. Users will want to answer several different questions. This option will answer only "Can my server handle X requests per second?". The other often asked question is "How long will it take to do X?". This may or may not be achieved with -n option, depending on the interpretation of its description.

I think the following usage scenarios are reasonable:

  • I want to send X exchanges per second and see if server can handle that for Y seconds/minutes/hours
  • I want to send X exchanges and see how long it will take server to handle them

To meet those usages, -time (or -t) option should be added that specifies duration. -r, -n and -t should be allowed to be used together (or in different combinations).

Other things that we should consider at a later date is turning this into stress testing. Let's call it --torture or similar. It starts sending data at some rate and increses it slowly until server starts dropping. That is the maximum rate the server can handle.

There should be option to conclude (fail) the test if there is a single drop. We don't want to wait 12 hours to see that 5 seconds after test started something broke. Not sure how to implement this in the most convenient way. Maybe --drop-threshold that specified acceptable amount of dropped traffic? It seems useful to have it specified in both percentage and absolute numbers.

Besides of using dhcperf as manual tool, it will also be used as automated test. In that case it should have clearly state if specified pass criteria are met or not. Something that could be easily parsed by automated environements.

Make sure that return code will specify status.

For automated test tools it is very convenient to print out command-line parameters. That's a practical experience. I received many logs that were useless because it was not possible to reproduce the problem due to missing information about used parameters.

There is no --version parameter. Tool should also print out its version when started. See above comment about reproduction concerns.

Another feature that could make this tool much more powerful is the ability to specify additional options. While it would be great to have custom option definition framework, for now we can do something much simpler. A command-line option that specifies extra payload that is appended to the message. A proper warning "it is user's responsibility to take care a proper format". For example, to specify that I want to send option type 100 with length 2 contaiing 0xabcd, I could do: --extra-data 00:64:00:02:ab:cd. This seems simple enough (parse command-line + a single memcpy will do the trick)

Another useful thing would to be to specify which options client should request. That is also not too difficult. This is just adding 8 bit(v4) or 16bit (v6) integers to PRL or ORO, respectively. Usage could be simple: --option 45 --option 5.

It would be useful to elaborate on reply verification. V4 server responding with NACK is ok or not? What about v6 server sending REPLY with status-code=no-addrs-avail? That is another thing we could eventually add as a feature. In some scenarios negative response as considered a proper one (test passed) and in others it is not (test failed). Make sure that the verification could be tuneable. For now it can be simple, but it will be more complex later.

comment:3 follow-up: Changed 8 years ago by johnd

These look like useful options.

Testing how long it takes for some operations could be handled by using both -r and -n, with a -d value large enough that the tool will wait a long time for any backlog that results to clear. The statistics would then tell how long it took for the backlog to clear.

What pass criteria would be be useful to set up, to allow for an unambiguous pass result?

I did have a torture-like option in mind for the future.

I'm thinking, for the initial version, from these suggestions adding:

  • The drop-threshold-abort
  • A version option
  • Include in the output the version and the parameters used
  • Explicit description of what the exit status means. Perhaps just status 1 if any dropped packets, unless other pass criteria options are added.
  • A test-period option, which seems to me an alternative to -n.

Should any of the other suggested options be in the initial version?

comment:4 Changed 8 years ago by johnd

  • Owner changed from johnd to tomek
  • Status changed from accepted to reviewing

comment:5 Changed 8 years ago by stephen

  • Owner changed from tomek to johnd

I suggest that any further comments be placed on ticket #1264 (which is related to the design of the benchmarking utility). This ticket was just a placeholder so that John's work on getting acquainted with DHCP was within the sprint system.

If work on the experiments has finished, I suggest we close this ticket.

comment:6 in reply to: ↑ 3 Changed 8 years ago by tomek

Replying to johnd:

These look like useful options.

Testing how long it takes for some operations could be handled by using both -r and -n, with a -d value large enough that the tool will wait a long time for any backlog that results to clear. The statistics would then tell how long it took for the backlog to clear.

What pass criteria would be be useful to set up, to allow for an unambiguous pass result?

See comments to ticket #1337. I put some ideas there. And don't worry. These are not needed for this milestone. That is something we could develop eventually.

I'm thinking, for the initial version, from these suggestions adding:

  • The drop-threshold-abort
  • A version option
  • Include in the output the version and the parameters used
  • Explicit description of what the exit status means. Perhaps just status 1 if any dropped packets, unless other pass criteria options are added.
  • A test-period option, which seems to me an alternative to -n.

Should any of the other suggested options be in the initial version?

That sounds fine for this release. It is already ambitious goal as we have around a month. Let's not make this even harder.

comment:7 Changed 8 years ago by johnd

  • Resolution set to complete
  • Status changed from reviewing to closed
Note: See TracTickets for help on using tickets.