Opened 8 years ago

Closed 8 years ago

#1878 closed defect (complete)

lettuce tests should be more fail safe about runtime environment

Reported by: jinmei Owned by: jelte
Priority: medium Milestone: Sprint-20120515
Component: build system Version:
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: Core Feature Depending on Ticket:
Estimated Difficulty: 4 Add Hours to Ticket: 0
Total Hours: 0 Internal?: no

Description

After experiencing 100 times trying to figure out why my lettuce tests
fail strangely just to remember I need to install the entire system or
incorporate the setup_intree_bind10 script, I'd now argue this setup
is so error prone.

I don't like to include the environments of the setup script in my
usual shell initialization because it has side effects for non lettuce
environments.

My suggestion is to introduce another shell script to run the lettuce
test like this:

source setup_intree_bind10.sh
export LETTUCE_SETUP_COMPLETED=1
lettuce

and somewhere in features/terrain check the LETTUCE_SETUP_COMPLETED
variable, and if it's undefined abort the test with a warning.

This way we don't have to pollute the non-test environment, don't have
to install the system beforehand, and can know what's wrong if we
don't run the test from the script.

Subtickets

Change History (10)

comment:1 Changed 8 years ago by vorner

And it would also be nice to have "make lettuce" that'd do everything, as we have make cppcheck or make systest.

comment:2 Changed 8 years ago by jelte

this suggestion would break it for running the lettuce tests with an installed bind10.

But we could add a run-script and/or make target without the mandatory check from within the tests themselves.

comment:3 Changed 8 years ago by jinmei

okay, so my revised suggestion is to introduce a "run" script:

% run-lettuce.sh [-I] [feature...]

If -I is specified, it means the installed environment; otherwise
the in-tree environment should be used and the necessary environment
variables are set internally. In either case it sets the special
LETTUCE_SETUP_COMPLETED environment variable. some terrain script
checks whether this variable is set or not. So simply running
'lettuce' will fail with a warning.

feature... is an optional space-separated list of features. If
explicitly specified, only the specified tests will be run; by default
all features will be run.

'make lettuce' executes this script without any options.

comment:4 Changed 8 years ago by jelte

  • Milestone changed from Next-Sprint-Proposed to Sprint-20120501

comment:5 Changed 8 years ago by jinmei

Note: don't forget updating README (and tutorial also, maybe).

comment:6 Changed 8 years ago by jelte

  • Owner changed from UnAssigned to jelte
  • Status changed from new to assigned

comment:7 Changed 8 years ago by jelte

  • Owner changed from jelte to UnAssigned
  • Status changed from assigned to reviewing

Ready for review, the run script itself is quite simple (it uses the existing setup script if -I is not given), and the main terrain script prints a hopefully useful error message if lettuce is used directly.

comment:8 Changed 8 years ago by muks

  • Owner changed from UnAssigned to muks

Picking for review.

comment:9 Changed 8 years ago by muks

  • Owner changed from muks to jelte

The solution looks OK, but I'd advise a small change: Move the check whether LETTUCE_SETUP_COMPLETED is set from terrain.py to run_lettuce.sh (after ./setup_intree_bind10.sh is sourced). This makes vanilla lettuce invocations work too. With the code in the branch, it'll only work if run through run_lettuce.sh.

As this is a matter of preference, you can merge the branch if you don't want this. It looks good otherwise.

comment:10 Changed 8 years ago by jelte

  • Resolution set to complete
  • Status changed from reviewing to closed

Thanks :)

But unless I misunderstood the original request, the idea is that vanilla invocations do not work (or at least not without the user very explicitely stating that it wants to use it). So I'm gonna leave this as is.

Oh I made one last change, instead of 'source' I changed it to '.'.

Assuming that is trivial enough, I merged the branch. Closing ticket.

Note: See TracTickets for help on using tickets.