Opened 8 years ago

Closed 8 years ago

#1290 closed task (complete)

initial setup of system tests with cucumber

Reported by: jelte Owned by: jelte
Priority: medium Milestone: Sprint-20111108
Component: Unclassified Version:
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: DNS Feature Depending on Ticket:
Estimated Difficulty: 9 Add Hours to Ticket: 0
Total Hours: 0 Internal?: no

Description

In order to better be able to manage high-level system tests, we want an extensible systems. So far only Cucumber has been mentioned as a tool for this, but if you know anything much better you are free to suggest it.

For now the idea is to use one of the python implementations of cucumber, and make a simple setup that configures and runs a system, and add a few simple tests, so that we can extend it with other tests (the first subject will be #1211 probably).

The long-term goal is to make this a separate 'project', with its own repository, and test other software with it too. For now we can include it here and just use it for BIND 10

Subtickets

Change History (13)

comment:1 Changed 8 years ago by jelte

ticket #611 has some implementation suggestions

comment:2 Changed 8 years ago by jelte

  • Estimated Difficulty changed from 0 to 9

comment:3 Changed 8 years ago by jelte

  • Owner set to jelte
  • Status changed from new to assigned

comment:4 Changed 8 years ago by jreed

hanging here:

                   # features/querying.py:110
    A query for www.example.org type A class IN to 127.0.0.1:47806 should have r    A query for www.example.org type A class IN to 127.0.0.1:47806 should have rcode NOERROR       # features/querying.py:110

(I will research later.)

comment:5 Changed 8 years ago by jreed

    The SOA serial for example.org should be 1234                                                  # features/querying.py:125
    Traceback (most recent call last):
      File "/usr/pkg/lib/python2.6/site-packages/lettuce/core.py", line 117, in __call__
        ret = self.function(self.step, *args, **kw)
      File "/home/reed/work/isc/bind10/git/bind10/tests/lettuce/features/querying.py", line 128, in query_soa
        "Got " + query_result.rcode + ", expected NOERROR"
    AssertionError: Got NO_ANSWER, expected NOERROR
    A query for doesnotexist.example.org should have rcode NXDOMAIN

comment:6 Changed 8 years ago by jreed

It would be nice if there could be an option to run the lettuce tests with the installed version of bind10 (even as a possible to have tests/lettuce directory and no other source).

comment:7 Changed 8 years ago by jelte

  • Milestone changed from Sprint-20111025 to Sprint-20111108

comment:8 Changed 8 years ago by jelte

  • Owner changed from jelte to UnAssigned
  • Status changed from assigned to reviewing

Apart from the addition of the directory tree tests/lettuce and its
contents, I've had to make a couple of improvements to some of the
rest of the system;

  • bind10_src.py: pass on -v flag to b10-cmdtl
  • b10-cmdctl: log instead of print startup message, and do it after it is listening (so test system knows when it can run bindctl)
  • bindctl: exit with non-zero value if it was unable to connect to cmdctl

The change in sockcreator is a partial fix to get it working good
enough, you can ignore it (this has been completely fixed in the
now-merged 1246 and i will revert this change upon merge)

(there are also a lot of whitespace fixes in the files i touched, if
possible you might want to use a diff tool that can ignore those ;)

Last edited 8 years ago by jelte (previous) (diff)

comment:9 Changed 8 years ago by stephen

  • Owner changed from UnAssigned to stephen

comment:10 follow-up: Changed 8 years ago by stephen

  • Owner changed from stephen to jelte

Reviewed commit 46adf014f18c6b3f9a685b8f0fdd0775a583a7c5.

First off, well done. I'm impressed by the amount that's been covered in this ticket and think it a very good first step in getting a comprehensive system test suite together.

src/bin/bindctl/bindcmd.py
run(): the first "if" check includes a "return" without a value although all other return statements return integers.

tests/lettuce/README

The bind10 main script, bindctl script, and dig must all be in the default search path of your environment, and BIND 10 must not be running if you use the installed version when you run the tests.

What is the "bind10 main script"?

If you want to test an installed version of bind 10, just run 'lettuce' in this directory.

But where is BIND 10 installed? Perhaps we should mandate some environment variables, e.g.

  • BIND10_TOP - the directory tree holding the installed version of BIND 10. If not defined, the version used is the one in the tree in which the tests are located.
  • BIND9_TOP - holds the (build) source tree of BIND 9. (I add this as for the moment, some of the interoperability tests (i.e. IXFR) require use of another nameserver.) If defined, "dig" is obtained from here, otherwise it is sought from the standard search path.

We have set up the tests to assume that lettuce is run from this directory, so even if you specify a specific feature file, you should do it from this directory.

Comment: At some time in the future we may want to be able to run the tests from another directory.

These directories are currently not divided further; we may want to consider this as the set grows. Due to a (current?) limitation of Lettuce, for feature files this is currently not possible; the python files containing steps and terrain must be below or at the same level of the feature files.

Git allows symbolic links to be stored in the repository, so we could consider some form of hierarchy with symbolic links from the location expected by Lettuce.

as that way we might run into a 'symmetric bug'

I assume that you mean that to do the test we will be relying on code that we are testing?

tests/lettuce/README.tutorial
The content is there although I think it could be phrased better in a few cases. However, I think it is fine for now; as we get more experience with Lettuce we may need to revisit this documentation. A few points:

When explaining the starting of BIND 10 (the paragraph that first includes the phrase "python decorator"), the various arguments are mentioned. It would be helpful to illustrate this with an example statement staring BIND 10 and specify a different port and process name.

Also worth noting that when defining a step, Lettuce is searching each line in the scenario for the defined phrase and that the extra words are just for readability (e.g in the last example, "Given" could be replaced by "After" with no effect on the test).

tests/lettuce/configurations/example.org.config.orig
tests/lettuce/configurations/example2.org.config
tests/lettuce/configurations/no_db_file.config
These would be easier to read if the configuration were split across several lines.

tests/lettuce/features/example.feature
Typo - "existance" should be spelt "existence".

tests/lettuce/features/terrain/bind10_control.py
This is Python code and even though it includes Lettuce decorators, it should still be properly commented - including ISC header, function headers etc. In particular, the parameters expected for each step should be listed (e.g. "set bind10 configuration (\S+) to (.*)" - is the first parameter the optional name of which bind10 I am referring to?)

check_lines() is more really "find_line". Also, the function should end "return None" to make it clearer what is returned if the line is not found.

How exact is the match of the phrase? In particular, I am thinking of initial capital letters: for example, do both

Then set bind10 configuration to x
Set bind10 configuration to x

... match the phrase required to execute set_config_command(). If not, is is possible to be flexible and let each step start with either a capital or non-capital letter?

We probably need some documentation as to what phrases are available.

tests/lettuce/features/terrain/querying.py
Needs ISC header.

Functions/methods need headers - examples would be useful.

Not clear from the documentation how you check a combination of flags in a query response. Must the order you give them in your question be the same as they appear in the "dig" output?

tests/lettuce/features/terrain/steps.py
tests/lettuce/features/terrain/terrain.py
Needs ISC header.

Functions/methods need headers

comment:11 in reply to: ↑ 10 Changed 8 years ago by jelte

  • Owner changed from jelte to stephen

Replying to stephen:

Reviewed commit 46adf014f18c6b3f9a685b8f0fdd0775a583a7c5.

First off, well done. I'm impressed by the amount that's been covered in this ticket and think it a very good first step in getting a comprehensive system test suite together.

Thanks :)

src/bin/bindctl/bindcmd.py
run(): the first "if" check includes a "return" without a value although all other return statements return integers.

ack, fixed

tests/lettuce/README

The bind10 main script, bindctl script, and dig must all be in the default search path of your environment, and BIND 10 must not be running if you use the installed version when you run the tests.

What is the "bind10 main script"?

'<prefix>/bin/bind10'. Change it to 'executable'?

If you want to test an installed version of bind 10, just run 'lettuce' in this directory.

But where is BIND 10 installed? Perhaps we should mandate some environment variables, e.g.

  • BIND10_TOP - the directory tree holding the installed version of BIND 10. If not defined, the version used is the one in the tree in which the tests are located.
  • BIND9_TOP - holds the (build) source tree of BIND 9. (I add this as for the moment, some of the interoperability tests (i.e. IXFR) require use of another nameserver.) If defined, "dig" is obtained from here, otherwise it is sought from the standard search path.

I must say I don't really see why. I know we have something like this
for current systests, but why not 'just' take whatever the environment
is already setup to use? (e.g. 'just take the bind10 from the users
path settings. If user wants different bind10, let him change his
settings', also implying 'test the system by default exactly how it
would be called outside of this environment'. And of course provide
some helper scripts to set it up for some special occasions, like the
current setup script to run from source). But maybe I'm just too
used to running different versions software from non-standard paths,
and switching between them. But see below.

We have set up the tests to assume that lettuce is run from this directory, so even if you specify a specific feature file, you should do it from this directory.

Comment: At some time in the future we may want to be able to run the tests from another directory.

we can put in a run script here as well that just changes cwd to this
directory, maybe that could also translate those _TOP thingies. But
i'm not sure whether this would not be more confusing, really.

These directories are currently not divided further; we may want to consider this as the set grows. Due to a (current?) limitation of Lettuce, for feature files this is currently not possible; the python files containing steps and terrain must be below or at the same level of the feature files.

Git allows symbolic links to be stored in the repository, so we could consider some form of hierarchy with symbolic links from the location expected by Lettuce.

ack. Not that I have taken any action in that directory, but I prefer
to improve lettuce to support a base environment for (all or specific)
tests :)

as that way we might run into a 'symmetric bug'

I assume that you mean that to do the test we will be relying on code that we are testing?

yes

tests/lettuce/README.tutorial
The content is there although I think it could be phrased better in a few cases. However, I think it is fine for now; as we get more experience with Lettuce we may need to revisit this documentation. A few points:

When explaining the starting of BIND 10 (the paragraph that
first includes the phrase "python decorator"), the various arguments
are mentioned. It would be helpful to illustrate this with an
example statement staring BIND 10 and specify a different port and
process name.

ok, added an example

Also worth noting that when defining a step, Lettuce is searching
each line in the scenario for the defined phrase and that the extra
words are just for readability (e.g in the last example, "Given"
could be replaced by "After" with no effect on the test).

added

tests/lettuce/configurations/example.org.config.orig
tests/lettuce/configurations/example2.org.config
tests/lettuce/configurations/no_db_file.config
These would be easier to read if the configuration were split across several lines.

done

tests/lettuce/features/example.feature
Typo - "existance" should be spelt "existence".

oopsie, fixed

tests/lettuce/features/terrain/bind10_control.py
This is Python code and even though it includes Lettuce decorators,
it should still be properly commented - including ISC header,
function headers etc. In particular, the parameters expected for
each step should be listed (e.g. "set bind10 configuration (\S+) to
(.*)" - is the first parameter the optional name of which bind10 I am
referring to?)

I've documented them all as function parameters, where if it is a
step, the step 'part' is it is taken from is described as well, and
whether it is optional. I'm not sure whether this is the best approach
in the end but it is all i could come up with right now.

check_lines() is more really "find_line". Also, the function should end "return None" to make it clearer what is returned if the line is not found.

actually, it wasn't even used anymore, so I removed it completely.

How exact is the match of the phrase? In particular, I am thinking of initial capital letters: for example, do both

Then set bind10 configuration to x
Set bind10 configuration to x

... match the phrase required to execute set_config_command(). If not, is is possible to be flexible and let each step start with either a capital or non-capital letter?

It is completely case-insensitive when matching, but any part we
pull out and use as arguments are taken as-is (e.g. for
'with configuration <file>', the file will be case-sensitive).

Added something about this to the readme.

We probably need some documentation as to what phrases are available.

Hmm, yes, probably. Not sure how to do it though, I don't really want
to retype the entire function description yet again (which will go
stale).

Not clear from the documentation how you check a combination of flags in a query response. Must the order you give them in your question be the same as they appear in the "dig" output?

Added a line to the example feature, and to the initial comments for
queryresult class.

comment:12 Changed 8 years ago by stephen

  • Owner changed from stephen to jelte

Reviewed commit b03d29677700c1dd2a527dafe9987defb7556e97

What is the "bind10 main script"?

'<prefix>/bin/bind10'. Change it to 'executable'?

Or program. I just thought that "script" was a bit confusing.

BIND9_TOP/BIND10_TOP

just take the bind10 from the users path settings.

I think it unlikely that most people will have bind10 in their current path. However, this is something that can be altered later - we don't need to change it now.

tests/lettuce/README.tutorial
One minor thing: the text talks of "of my bind10", but the example that follows includes "as b10_second_instance". The two should match.

We probably need some documentation as to what phrases are available.
Hmm, yes, probably. Not sure how to do it though, I don't really want to retype the entire function description yet again (which will go stale).

Taking the description from the "@step" decorator doesn't really work since, as it includes a regular expression, it can be fairly cryptic.

I would suggest adding lines giving the lettuce step syntax to the start of each of the function descriptions, e.g.

@step('start bind10(?: with configuration (\S+))?' +\
      '(?: with cmdctl port (\d+))?(?: as (\S+))?')
def start_bind10(step, config_file, cmdctl_port, process_name):
    """
    Start bind10 [with configuration <config_file>]
                 [with cmdctl port <cmdctl_port>]
                 [as <process_name]

    Start BIND 10 with the given optional config file, cmdctl port, and
    store the running process in world with the given process name.
     :

Then write a Python script that goes through the relevant Python files, searches for a line beginning "@step", then extracts the all lines from the following function header (as delineated by the triple quotes tokens) into a separate file.

Just one minor change to README.tutorial then please merge. I don't need to see it again.

comment:13 Changed 8 years ago by jelte

  • Resolution set to complete
  • Status changed from reviewing to closed

OK, thanks, done, merged, closing.

Note: See TracTickets for help on using tickets.