A better way to loop through multiple lists in a django template

I have some big data sets that I am looping through to display a table of data. The trouble is the looping takes a ton of time, which is okay at the moment as this is an internal tool but I would like to improve it.

The model:

class Metric_Data(models.Model):
  metric = models.ForeignKey(Metric)
  count = models.IntegerField()
  start_date = models.DateField()

I am displaying a table where the first column is the dates then each following column is a metric listing the count for that date. Like so:

Dates Metric Metric Metric ...
10/11     10     11     12
11/11     22    100   1000
...      ...    ...    ...

I tried looping over the data in the view and creating the table out of lists and passing this to the template for rendering but with several metrics and thousands of data points per metric this was rather slow. I have since switched to a template tag:

def getIndex(parser, token):
  try:
    tag_name, a_list, index = token.split_contents()
  except ValueError:
    raise template.TemplateSyntaxError, "%r tag requires exactly two arguments" % token.contents.split()[0]
  return GetIndexNode(a_list, index)

class GetIndexNode(template.Node):
  def __init__(self, a_list, index):
    self.the_list = template.Variable(a_list)
    self.index = template.Variable(index)

  def render(self, context):
    try:
      the_list = self.the_list.resolve(context)
      i = self.index.resolve(context)
      return the_list[i]
    except template.VariableDoesNotExist:
      return ''

Which is still rather slow, which just could because it is my first time writing a template tag and I've done something wrong.

EDIT: I am fetching the data in the view like so:

def show_all(request):
  metrics = Metric.objects.all()
  dates = Metric_Data.objects.all().values_list('start_date',flat=True).distinct().order_by('start_date')

  data = []

  for metric in metrics:
    data.append(Metric_Data.objects.filter(metric=metric).order_by('start_date').values_list('count', flat=True))

  return render_to_response('metric/show_all.html', {'dates': dates,
                                                   'metrics': metrics,
                                                   'data': data})

Edit: And the template

<table id="theTable" class="paginate-5">
<thead> 
<tr>
<th>Dates</th>
    {% for metric in metrics %}
        <th>{{ metric.name }}</th>
    {% endfor %}
</tr>
</thead>
<tbody> 
{% for date in dates %}
    <tr>
    <td>{{date}}</td>
    {% for metric in data %}
        <td>{% get_index metric forloop.parentloop.counter0 %}</td>
    {% endfor %}
    </tr>
{% endfor %}
</tbody>

I am thinking the best place to fix this problem might be in the model but I'm not sure how to go about it. Create 开发者_如何学Ca table for the dates perhaps and do the query on that table?

Ideas much appreciated thanks!

---

I think you're just badly grouping your data, so that you end up looping multiple times over the same items, yielding very poor complexity. Try to structure your data very closely to the way it will be used in the template.

For instance:

def metric_count_on ( metric, date ):
    return Metric_Data.objects.filter(metric=metric,start_date=date).values_list('count',flat=True)
def show_all(request):
    metrics = Metric.objects.all()
    dates = Metric_Data.objects.all().values_list('start_date',flat=True).distinct().order_by('start_date')
    # generate full report.  now, template only has to loop.
    data = [{'date':date, 'metrics':metric_count_on(date, metric)}
       for (date,metric) in itertools.product(dates,metrics)]
    # ...

Then, in the template, you can basically just loop as:

{% for row in data %}
    <tr>
    <td>{{ row.date }}</td>
    {% for count in row.metrics %}
      <td>{{ count }}</td>
    {% endfor %}
    </tr>
{% endfor %}

---

If you find your view to be slow, the problem is often in the database. Are you sure you know what queries are going to the database? It's possible you can make a small change that would greatly reduce the db traffic.

---

I found this blog post which seems to hint at similar problems.

http://www.xorad.com/blog/?p=1497

Using "|safe" on all my variables cut my load time in half which is at least something...

Posting in case anyone else stumbles on this problem.