Grails test-app fails on CI server, but not on developer machine

I have two machines, one Continuus integration server and one developer machine.

Both machines run the same version of java, maven, and grails and both are running Ubuntu. The only difference that i can think of is that the CI is in a virtualized environment.

Some proof:

CI java -version

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) 

DEV java -version

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode) 

CI mvn -version

Apache Maven 2.2.1 (rdebian-1)
Java version: 1.6.0_24
Java home: /usr/lib/jvm/java-6-sun-1.6.0.24/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "lin开发者_开发技巧ux" version: "2.6.32-31-server" arch: "amd64" Family: "unix" 

DEV mvn -version

Apache Maven 2.2.1 (rdebian-1)
Java version: 1.6.0_24
Java home: /usr/lib/jvm/java-6-sun-1.6.0.24/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux" version: "2.6.32-31-generic" arch: "amd64" Family: "unix" 

CI grails

Welcome to Grails 1.3.7 - http://grails.org/
Licensed under Apache Standard License 2.0
Grails home is set to: /usr/local/lib/grails-1.3.7 

DEV grails

Welcome to Grails 1.3.7 - http://grails.org/
Licensed under Apache Standard License 2.0
Grails home is set to: /home/netbrain/dev/apps/grails-1.3.7 

I did a clean checkout from version control on both machines and deleted the folders ~/.m2 and ~/.ivy2 in order to avoid any indescrepencies.

When i run grails test-app on my CI server 6 tests seem to fail. When they on my DEV system all passes in the same case.

CI grails test-app

Tests Completed in 28213ms ...
-------------------------------------------------------
Tests passed: 14
Tests failed: 6
------------------------------------------------------- 

DEV grails test-app

Tests Completed in 25889ms ...
-------------------------------------------------------
Tests passed: 20
Tests failed: 0
------------------------------------------------------- 

When i look at the error messages im getting from the test output i can see things like:

junit.framework.AssertionFailedError: expected:<1> but was:<0>
not-null property references a null or transient value

As far as i can see, the tests are logically correct, and they shouldn' get these errors, especially when several tests that get the "transient value" error doesn't store anything to the database!

For me it seems like the problem lies somewhere in the combination of hibernate/memory-database/test phase.

There HAS to be some difference between the environments that is causing this issue. Does anyone have any tips on how i can proceed in order to debug the problem further?

Thanks again!

EDIT

Ok, so here is what i tried:

1. delete the folders ~/.m2, ~/.ivy2, ~/.grails
2. run grails clean
3. run grails test-app

But still im seeing failed tests on my CI environment, but not on dev machine.

EDIT

Some new developments..

I tried running only the tests that failed on the CI, and it seems that when running grails test-app :integration path.to.failing.tests they all PASS! while running grails test-app causes those tests to FAIL! :(

Im totally baffled and clueless of whats going on here...

---

I would almost guarantee that this is a test pollution issue and the different servers are running the tests in different order.

The biggest clue of test pollution is always when you can run the test successfully by itself, but when you run other tests first, it fails.

I've had this happen multiple times where I've messed with the metaClass of a Class or an injected spring singleton (like a service bean) in one test, forgot to clean it up, and had other tests impacted by it. Watch the output of which tests are getting run and compare them to make sure they're running in exactly the same order on both boxes. If they aren't it's a test ordering issue.

The "easiest" way to find out what tests are interacting badly is to run grails test-app and pass in each test name in the same order that they are happening on the failing box (I normally use a text editor to slice out the test names from a previous run and concat them together so I know I have them in the same order).

Another potential issue is if one machine is using mysql and you've misconfigured to have "myisam" as the default storage engine rather than "innodb" (or another engine that supports transactions). You can type "show engines;" on the mysql instance to see what engine is default. Sounds like you're using hsqldb or another in memory db for your tests though, so this might not be your problem.