How to Write an Experiment Framework

I'm an intermediate C#/ASP.net coder working on a fun MVC project. I'd like to implement an experiment framework that will allow objects to "jiggle" functionality in order to find better ways of doing things. Where can I read more about best practices or tutorials on this kind of thing?

e.开发者_如何学Gog

// A helper class that might return different flavors of "mean".
public class MeanHelper
{
  ...
  Mean(IEnumerable<double> input) { <shuffle between the 3 below.> }

  OrdinaryMean(IEnumerable<double> input) { ... }
  GeometricMean(IEnumerable<double> input) { ... }
  HarmonicMean(IEnumerable<double> input) { ... }
  ...
}

Thanks so much!

---

What you describe sounds like a strategy pattern; you could for instance have an interface IMean, with a method Calculate, and 3 implementations.

---

I use my own experimentation framework with the following properties:

• experiments are first-class entities
• the application runs only as long as there are ongoing experiments
• each experiment determines when it has collected enough data
• the record is stored in a database

The application undergoing experiments exposes named properties for the experimentation framework to manipulate and monitor. The main application method is an event generator. Each experiment lists a property to manipulate and other properties to monitor. The experimentation framework iterates through the main application method, feeding the events to the ongoing experiments. Each experiment filters the events it recieves, updating variance statistics and manipulating its associated application property. When the experiment signals that it has processed an event, the event, as well as the values of the properties it manipulates and monitors are stored in the experiment database.

Experiment databases are analysized through inspection or with Sho or R. Results of an experiment can be used to study effects of individual property values, as well as correlation and independence of different application properties.

I chose to use the generator pattern, instead of signals or inversion of control, because the experimentation framework should be in charge of when the application runs and stops. The compiler does all the hard work of creating a state machine in the main application method based on the yield command.

I am still refining the framework and rethinking parts of it. I intend to make it an open-source project at some point in the future.

Google's "Overlapping Experiment Infrastructure" paper is an excellent read on the subject.

---

I think by "jiggle functionality" you mean substitute different functionality based on configuration so that you can experiment things.

If this is the case, then you should look into polymorphism (most likely via interfaces) and perhaps you'd also frameworks and design patterns associated with inversion of control.