Repeatedly querying xml using python_问答_开发者

开发者 https://www.devze.com 2022-12-25 00:28 出处：网络

I have some xml documents I need to run queries on. I\'ve created some python scripts (using ElementTree) to do this, since I\'m vaguely familiar with using it.

I have some xml documents I need to run queries on. I've created some python scripts (using ElementTree) to do this, since I'm vaguely familiar with using it.

The way it works is I run the scripts several times with different arguments, depending on开发者_如何学Go what I want to find out.

These files can be relatively large (10MB+) and so it takes rather a long time to parse them. On my system, just running:

tree = ElementTree.parse(document)

takes around 30 seconds, with a subsequent findall query only adding around a second to that.

Seeing as the way I'm doing this requires me to repeatedly parse the file, I was wondering if there was some sort of caching mechanism I can use so that the ElementTree.parse computation can be reduced on subsequent queries.

I realise the smart thing to do here may be to try and batch as many queries as possible together in the python script, but I was hoping there might be another way.

Thanks.

While I second the suggestion to use lxml, you can get a huge performance boost by using the builtin cElementTree.

from xml.etree import cElementTree as ElementTree

Seconding the lxml recommendation, look at this article for how to improve performance by using an iterative (SAX-like) parsing method. It can be a pain at first since it can turn really procedural and messy, but it makes things faster. As you can see from these benchmarks, lxml is most likely your best bet for performance.

First off, consider using the lxml implementation of ElementTree:
http://lxml.de/ This is a wrapper for libxml2, which I find performs well.

Run python interactively, make multiple queries against the same etree object. ipython is an enhanced interactive python interpreter with easy access to introspection and convenience-syntax.

E.g. interactively examining note.xml with ipython using lxml.etree.

$ ipython
Python 2.5.1 (r251:54863, Jul 10 2008, 17:24:48)
Type "copyright", "credits" or "license" for more information.

IPython 0.8.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: from lxml import etree

In [2]: doc = etree.parse(open("note.xml"))

In [3]: etree.dump(doc.getroot())
<note>
        <to>Tove</to>
        <from>Jani</from>
        <heading>Reminder</heading>
        <body>Don't forget me this weekend!</body>
</note>
In [4]: doc.xpath('/note/*')
Out[4]:
[<Element to at 89cf02c>,
 <Element from at 89cf054>,
 <Element heading at 89cf07c>,
 <Element body at 89cf0a4>]