Python milk library: object weights issue

I'm trying to use one_vs_one composition of decision trees for multiclass classification. The problem is, when I pass different object weights to a classifier, the result stays the same.

Do I misunderstand something with weights, or do they just work incorrectly?

Thanks for your replies!

Here is my code:

class AdaLearner(object):
    def __init__(self, in_base_type, in_multi_type):
        self.base_type = in_base_type
        self.multi_type = in_multi_type

    def train(self, in_features, in_labels):
        model = AdaBoost(self.base_type, self.multi_type)
        model.learn(in_features, in_labels)

        return model

class AdaBoost(object):
    CLASSIFIERS_NUM = 100
    def __init__(self, in_base_type, in_multi_type):
        self.base_type = in_base_type
        self.multi_type = in_multi_type
        self.classifiers = []
        self.weights = []

    def learn(self, in_features, in_labels):
        labels_number = len(set(in_labels))
        self.weights = self.get_initial_weights(in_labels)

        for iteration in xrange(AdaBoost.CLASSIFIERS_NUM):
            classifier = self.multi_type(self.base_type())
            self.classifiers.append(classifier.train(in_features,
                                                     in_labels,
                                                     weights=self.weights))
            answers = []
            for obj in in_features:
                answers.append(self.classifiers[-1].apply(obj))
            err = self.compute_weighted_error(in_labels, answers)
            print err
            if abs(err - 0.) < 1e-6:
            break

            alpha = 0.5 * log((1 - err)/err)

            self.update_weights(in_labels, answers, alpha)
            self.normalize_weights()

    def apply(self, in_features):
        answers = {}
        for classifier in self.classifiers:
            answer = classifier.apply(in_features)
            if answer in answers:
                answers[answer] += 1
            else:
                answers[answer] = 1
        ranked_answers = sorted(answers.iteritems(),
                                key=lambda (k,v): (v,k),
                                reverse=True)
        return ranked_answers[0][0]

    def compute_weighted_error(self, in_labels, in_answers):
        error = 0.
        w_sum = sum(self.weights)
        for ind in xr开发者_StackOverflow中文版ange(len(in_labels)):
            error += (in_answers[ind] != in_labels[ind]) * self.weights[ind] / w_sum
        return error

    def update_weights(self, in_labels, in_answers, in_alpha):
        for ind in xrange(len(in_labels)):
            self.weights[ind] *= exp(in_alpha * (in_answers[ind] != in_labels[ind]))

    def normalize_weights(self):
        w_sum = sum(self.weights)
        for ind in xrange(len(self.weights)):
            self.weights[ind] /= w_sum

    def get_initial_weights(self, in_labels):
        weight = 1 / float(len(in_labels))
        result = []
        for i in xrange(len(in_labels)):
            result.append(weight)
        return result

As you can see, it is just a simple AdaBoost (I instantiated it with in_base_type = tree_learner, in_multi_type = one_against_one) and it worked the same way no matter how many base classifiers were engaged. It just acted as one multiclass decision tree. Then I've made a hack. I chose a random sample of objects on the each iteration with respect to their weights and trained classifiers with a random subset of objects without any weights. And that worked as it was supposed to.

---

The default tree criterion, namely information gain, does not take the weights into account. If you know of a formula which would do it, I'll implement it.

In the meanwhile, using neg_z1_loss will do it correctly. By the way, there was a slight bug in that implementation, so you will need to use the most current github master.