How do you handle testing with Selenium when you are running AB tests?

We just started using Selenium to test our 开发者_开发百科site and its working really well, except that it breaks 1/2 of the time when we introduce an AB test. How do you guys handle Ab tests when testing with Selenium?

Thanks!

---

This really depends on exactly what you're looking for. I would suggest one of three options:

1. Disable A/B testing for Selenium. One of the keys to testing is determination. With randomized return, you won't be able to be as confident in your results. To accomplish this, I would pass in a parameter, e.g. http://my.website.com/?ab=0. This could always select one specific path which will be tested.

2. Build a parameter for Selenium to "choose" an ab test-case. http://my.website.com/?ab[show-panel]=1. Again, this will give you a deterministic result, and allow you to actually test the ab test cases you have. This can also be accomplished through creative the use of cookies in your setup.

3. Hack- don't "check" the a-b tested aspects of your site, and build selenium to navigate around them. This depends on exactly what you are testing, but if it's images or text, this shouldn't be an issue. Non-deterministic workflows should not be tested.

Hope these ideas help. The take-away here is to make sure to keep things deterministic for your tests. Random will only ever lead you astray.