Opened 7 years ago

Closed 7 years ago

#2968 closed defect (fixed)

Database upgrade tests fail with sqlite3 sqlite-3.7.17

Reported by: vorner Owned by: muks
Priority: medium Milestone: Sprint-20130611
Component: Unclassified Version:
Keywords: Cc:
CVSS Scoring: Parent Tickets:
Sensitive: no Defect Severity: N/A
Sub-Project: DNS Feature Depending on Ticket:
Estimated Difficulty: 4 Add Hours to Ticket: 0
Total Hours: 0.38 Internal?: no

Description

After upgrading my sqlite3 to sqlite-3.7.17, the tests for dbutil started to fail for me (in unrelated branches that passed short time before). My guess is the shell magic around is brittle and there's some small difference in output, but I didn't verify that.

A (long) output of the tests is attached.

Subtickets

Attachments (2)

dbutil.log (33.3 KB) - added by vorner 7 years ago.
debug.diff (492 bytes) - added by muks 7 years ago.
Patch to print schema output when there is a failure

Download all attachments as: .zip

Change History (14)

Changed 7 years ago by vorner

comment:1 Changed 7 years ago by muks

  • Estimated Difficulty changed from 0 to 4

comment:2 Changed 7 years ago by muks

  • Milestone changed from Next-Sprint-Proposed to Sprint-20130611

comment:3 Changed 7 years ago by muks

  • Owner set to muks
  • Status changed from new to assigned

Picking

comment:4 Changed 7 years ago by muks

  • Owner changed from muks to UnAssigned
  • Status changed from assigned to reviewing

Up for review.

The bug was that with the new version of SQLite, some schema statements are split into multiple lines, and also the statements are in a different order now. So the change:

  • Converts them into single line SQL statements
  • Sorts the statements to enforce an order

The sed expressions may not be compatible on all platforms. I've asked Jeremy to put this branch to the builders.

Michal: You reported this bug from your system, so please check if this branch fixes the bug for you. I have verified it with a local install of SQLite 3.7.17 here.

comment:5 Changed 7 years ago by muks

This branch seems to have passed tests on the BSDs, Solaris and Linuxes. MacOS builders didn't pick up for some reason, but I don't expect its sed to be different from that of FreeBSD.

comment:6 follow-up: Changed 7 years ago by vorner

It still does not pass here :-(.

2013-06-10 09:37:21.649 INFO  [b10-dbutil.dbutil/23905] DBUTIL_FILE Database file: /tmp/bind10/bind10-3/bind10-20130529/_build/src/bin/dbutil/tests/dbutil_test_tempfile_23861

2013-06-10 09:37:21.649 INFO  [b10-dbutil.dbutil/23905] DBUTIL_BACKUP created backup of /tmp/bind10/bind10-3/bind10-20130529/_build/src/bin/dbutil/tests/dbutil_test_tempfile_23861 in /tmp/bind10/bind10-3/bind10-20130529/_build/src/bin/dbutil/tests/dbutil_test_tempfile_23861.backup

2013-06-10 09:37:21.650 INFO  [b10-dbutil.dbutil/23905] DBUTIL_UPGRADING upgrading database from V1.0 to V2.0

2013-06-10 09:37:21.653 INFO  [b10-dbutil.dbutil/23905] DBUTIL_UPGRADING upgrading database from V2.0 to V2.1

2013-06-10 09:37:21.653 INFO  [b10-dbutil.dbutil/23905] DBUTIL_UPGRADING upgrading database from V2.1 to V2.2

2013-06-10 09:37:21.653 INFO  [b10-dbutil.dbutil/23905] DBUTIL_UPGRADE_SUCCESFUL database upgrade successfully completed
ERROR: upgraded schema not as expected
*** FAIL

2013-06-10 09:37:21.736 INFO  [b10-dbutil.dbutil/23925] DBUTIL_FILE Database file: /tmp/bind10/bind10-3/bind10-20130529/_build/src/bin/dbutil/tests/dbutil_test_verfile_23861

2013-06-10 09:37:21.736 INFO  [b10-dbutil.dbutil/23925] DBUTIL_VERSION_CURRENT database version V2.2
2013-06-10 09:37:21.736 INFO  [b10-dbutil.dbutil/23925] DBUTIL_CHECK_OK this is the latest version of the database schema. No upgrade is required
--- PASS
6.1. Database is new V1 database - check

2013-06-10 09:37:21.879 INFO  [b10-dbutil.dbutil/23934] DBUTIL_FILE Database file: /tmp/bind10/bind10-3/bind10-20130529/_build/src/bin/dbutil/tests/dbutil_test_verfile_23861

2013-06-10 09:37:21.880 INFO  [b10-dbutil.dbutil/23934] DBUTIL_VERSION_LOW database version V1.0, latest version is V2.2.

2013-06-10 09:37:21.880 INFO  [b10-dbutil.dbutil/23934] DBUTIL_CHECK_UPGRADE_NEEDED re-run this program with the --upgrade switch to upgrade
--- PASS
6.2. Database is a new V1 database - upgrade

2013-06-10 09:37:22.010 INFO  [b10-dbutil.dbutil/23941] DBUTIL_FILE Database file: /tmp/bind10/bind10-3/bind10-20130529/_build/src/bin/dbutil/tests/dbutil_test_tempfile_23861

2013-06-10 09:37:22.011 INFO  [b10-dbutil.dbutil/23941] DBUTIL_BACKUP created backup of /tmp/bind10/bind10-3/bind10-20130529/_build/src/bin/dbutil/tests/dbutil_test_tempfile_23861 in /tmp/bind10/bind10-3/bind10-20130529/_build/src/bin/dbutil/tests/dbutil_test_tempfile_23861.backup

2013-06-10 09:37:22.011 INFO  [b10-dbutil.dbutil/23941] DBUTIL_UPGRADING upgrading database from V1.0 to V2.0

2013-06-10 09:37:22.023 INFO  [b10-dbutil.dbutil/23941] DBUTIL_UPGRADING upgrading database from V2.0 to V2.1

2013-06-10 09:37:22.023 INFO  [b10-dbutil.dbutil/23941] DBUTIL_UPGRADING upgrading database from V2.1 to V2.2

2013-06-10 09:37:22.025 INFO  [b10-dbutil.dbutil/23941] DBUTIL_UPGRADE_SUCCESFUL database upgrade successfully completed
ERROR: upgraded schema not as expected
*** FAIL

IMO the bigger problem is using (unportable) shell and unportable output of the sqlite3 command line utility.

comment:7 Changed 7 years ago by vorner

  • Owner changed from UnAssigned to vorner

I have time to take full review now anyway.

comment:8 in reply to: ↑ 6 Changed 7 years ago by muks

Replying to vorner:

IMO the bigger problem is using (unportable) shell and unportable output of the sqlite3 command line utility.

You faced this issue on Gentoo, correct? So we both use GNU/Linux and have similar shells, sed, etc.

The only problem I found was with the sqlite3 .schema output. This output is now updated so that a "canonical" form is generated by making each SQL statement go into a single line, and then sorting the statements.

If it still doesn't pass for you, can you apply the attached patch and provide output?

Changed 7 years ago by muks

Patch to print schema output when there is a failure

comment:9 Changed 7 years ago by vorner

  • Owner changed from vorner to muks
  • Total Hours changed from 0 to 0.38

Sorry for the confusion, it was my fault. I started full build & test on two branches in parallel and the other failed with the DB error. I should have guessed it, or at least remember which branch I have in which window.

I run several times now and it works.

May I ask to add an explaining comment to the sed command? It took me quite some time to decipher it and I was teaching sed before, so I have some experience. Otherwise, it should be OK to merge.

Once again, sorry for creating more work for you :-|.

comment:10 Changed 7 years ago by muks

  • Owner changed from muks to vorner

Hi vorner

I have added a comment describing the sed command. Please review it.

comment:11 Changed 7 years ago by vorner

  • Owner changed from vorner to muks

Yes, looks good. Please merge.

comment:12 Changed 7 years ago by muks

  • Resolution set to fixed
  • Status changed from reviewing to closed

Merged to master branch in commit 7212536c15e7374828bb74657d6f6d30cac4160d:

* 8b82620 [2968] Document the sed command
* 6f11ce8 [2968] Split into multiple lines
* 99d9be3 [2968] Fix canonical schema generation to work with SQLite 3.7.17 output

Resolving as fixed. Thank you for the reviews Michal.

Note: See TracTickets for help on using tickets.