What is the best way in Python to read XML from a file, build a tree, do some rewriting in the tree and generate Python code? (I'm slightly confused as in Python there seem to be multiple options such as expat, ElementTree, XSLT, ...).
You should check out lxml. It is easy to use and does what you want in a few steps. If you want to stick to the stdlib, then you check out ElementTree (Python 2.5+) . The type of XML processing you want depends on your needs and for high performance XML parsing, read this.
EDIT: My answer is for XML parsing with Python and does not answer your last question: "generate Python code", cause that makes no sense :)
I use Beautiful Soup. Its relatively easy to use and powerful.
Check it out here
Related
I need to parse XML file and put results on HTML form, but I am new to Python. Does Python 2.7 have something like LINQ to XML from C# or any good library for XML can suggest me ?
Check out Pynq – Python Language Integrated Query https://github.com/heynemann/pynq/wiki
I'm sure though if Pynq will be sufficient for you, although it does implement exprenssion trees in Python just like LINQ does for C#.
For an easy way to access XML in Python, you could check out BeautifulSoup (http://www.crummy.com/software/BeautifulSoup/documentation.html). Note: for XML processing: use BeautifulStoneSoup for XML processing.
A simple example: "find first three a tags"
soup.findAll('p', limit=3)
For a more comprehensive selection of XML libraries for Python, please see "PythonXml" in PythonInfo Wiki.
Look at lxml, more specifically the combination of:
the ElementTree interface: an easier xml model/api than DOM, much like the XDocument and XElement classes (attributes and text are even simpler because they are not separate "nodes", which may look strange at first if you're working with "mixed content" models a lot and you're used to DOM interfaces)
The E-factory of lxml.builder (like "functional construction" in LINQ to XML, but even better ;-))
python's built-in list comprehensions and generator expressions, which can get you very close to the LINQ query syntax (without looking like SQL however ;-))
... will get you a very similar experience. (having played with LINQ to XML in .NET as well: I much prefer working python and lxml)
Another remark: lxml has some good support for html (even ill-formed) as well, including facilities for filling out html forms (I'm not sure if that is what you meant with "put results on html form" however)
The most featured XML library in python is lxml http://lxml.de but it doesn't support LINQ interface, as I know.
You can use ElementTree interface http://lxml.de/tutorial.html#the-elementtree-class
or XPAth selectors http://lxml.de/xpathxslt.html#xpath
or CSS selectors http://lxml.de/cssselect.html
for data extraction from XML.
I am currently writing a non-web program using mostly python and have gotten to the point where I need to create a structure for save and settings files.
I decided to use xml over creating my own format but have come into a bit of a barrier when trying to figure out what to use to actually process the xml. I want to know if I can get some of the pros and cons about packages that are available for python since there are a lot and I'm not entirely sure which ones seem best from just looking at the documentation.
I basically want to know if there is a package or library that will let me write and read data from an xml file with relative ease by just knowing what the tag name I'm looking for is.
P.S. My app is mostly geared to be used on Linux if it makes any difference.
If your data is only for the use of your Python programs, pickle might be an easier solution.
Don't use xml -- use yaml or json -- some would argue xml has outlived it's useful life...
That being said, I like ElementTree:
import xml.etree.cElementTree as ET
fname = 'foo.xml'
root = ET.parse(open(fname))
... yada yada...
A search for "python" and "xml" returns a variety of libraries for combining the two.
This list probably faulty:
xml.dom
xml.etree
xml.sax
xml.parsers.expat
PyXML
beautifulsoup?
HTMLParser
htmllib
sgmllib
Be nice if someone can offer a quick summary of when to use which, and why.
The DOM/SAX divide is a basic one. It applies not just to python since DOM and SAX are cross-language.
DOM: read the whole document into memory and manipulate it.
Good for:
complex relationships across tags in the markup
small intricate XML documents
Cautions:
Easy to use excessive memory
SAX: parse the document while you read it. Good for:
Long documents or open ended streams
places where memory is a constraint
Cautions:
You'll need to code a stateful parser, which can be tricky
beautifulsoup:
Great for HTML or not-quite-well-formed markup. Easy to use and fast. Good for screen scraping, etc. It can work with markup where the XML based ones would just through an error saying the markup is incorrect.
Most of the rest I haven't used, but I don't think there's hard and fast rules about when to use which. Just your standard considerations: who is going to maintain the code, which APIs do you find most easy to use, how well do they work, etc.
In general, for basic needs, it's nice to use the standard library modules since they are "standard" and thus available and well known. However, if you need to dig deep into something, almost always there are newer nonstandard modules with superior functionality outside of the standard library.
I find xml.etree essentially sufficient for everything, except for BeautifulSoup if I ever need to parse broken XML (not a common problem, differently from broken HTML, which BeautifulSoup also helps with and is everywhere): it has reasonable support for reading entire XML docs in memory, navigating them, creating them, incrementally-parsing large docs. lxml supports the same interface, and is generally faster -- useful to push performance when you can afford to install third party Python extensions (e.g. on App Engine you can't -- but xml.etree is still there, so you can run exactly the same code). lxml also has more features, and offers BeautifulSoup too.
The other libs you mention mimic APIs designed for very different languages, and in general I see no reason to contort Python into those gyrations. If you have very specific needs such as support for xslt, various kinds of validations, etc, it may be worth looking around for other libraries yet, but I haven't had such needs in a long time so I'm not current wrt the offerings for them.
For many problems you can get by with the xml. It has the major advantage of being part of the standard library. This means that it is pre-installed on almost every system and that the interface will be static. It is not the best, or the fastest, but it is there.
For everything else there is lxml. Specically, lxml is best for parsing broken HTML, xHTML, or suspect feeds. It uses libxml2 and libxslt to handle XPath, XSLT, and EXSLT. The tutorial is clear and the interface is simplistically straight-forward. The rest of the libraries mentioned exist because lxml was not available in its current form.
This is my opinion.
I don't do much with XML, but when I've needed to, lxml has been a joy to work with and is apparently quite fast. The element tree API is very nice in an object oriented setting.
I know that any language is capable of parsing XML; I'm really just looking for advantages or drawbacks that you may have come across in your own experiences. Perl would be my standard go to here, but I'm open to suggestions.
Thanks!
UPDATE: I ended up going with XML::Simple which did a nice job, but I have one piece of advice if you plan to use it--research the forcearray option first. I had to rewrite a bunch of statements after learning that it is usually best practice to set forcearray. This page had the clearest explanation that I could find. Frankly, I'm surprised this isn't the default behavior.
If you are using Perl then I would recommend XML::Simple:
As more and more Web sites begin using
XML for their content, it's
increasingly important for Web
developers to know how to parse XML
data and convert it into different
formats. That's where the Perl module
called XML::Simple comes in. It takes
away the drudgery of parsing XML data,
making the process easier than you
ever thought possible.
XML::Twig is very nice, especially because it’s not as awfully verbose as some of the other options.
For pure XML parsing, I wouldn't use Java, C#, C++, C, etc. They tend to overcomplicate things, as in you want a banana and get the gorilla with it as well.
Higher-level and interpreted languages such as Perl, PHP, Python, Groovy are more suitable. Perl is included in virtually every Linux distro, as is PHP for the most part.
I've used Groovy recently for especially this and found it very easy. Mind you though that a C parser will be orders of magnitude faster than Groovy for instance.
It's all going to be in the libraries.
Python has great libraries for XML. My preference is lxml. It uses libxml/libxslt so it's fast, but the Python binding make it really easy to use. Perl may very well have equally awesome OO libraries.
I saw that people recommend XML::Simple if you decide on Perl.
While XML::Simple is, indeed, very simple to use and great, is a DOM parser. As such, it is, sadly, completely unsuitable to processing large XML files as your process would run out of memory (it's a common problem for any DOM parser, not limited to XML::Simple or Perl).
So, for large files, you must pick a SAX parser in whichever language you choose (there are many XML SAX parsers in Perl, or use another stream parser like XML::Twig that is even better than standard SAX parser. Can't speak for other languages).
Not exactly a scripting language, but you could also consider Scala. You can start from here.
Scala's XML support is rather good, especially as XML can just be typed directly into Scala programs.
Microsoft also did some cool integrated stuff with their LINQ for XML
But I really like Elementtree and just that package alone is a good reason to use Python instead of Perl ;)
Here's an example:
import elementtree.ElementTree as ET
# build a tree structure
root = ET.Element("html")
head = ET.SubElement(root, "head")
title = ET.SubElement(head, "title")
title.text = "Page Title"
body = ET.SubElement(root, "body")
body.set("bgcolor", "#ffffff")
body.text = "Hello, World!"
# wrap it in an ElementTree instance, and save as XML
tree = ET.ElementTree(root)
tree.write("page.xhtml")
It's not a scripting language, but Scala is great for working with XML natively. Also, see this book (draft) by Burak.
Python has some pretty good support for XML. From the standard library DOM packages to much more 'pythonic' libraries that parse XML directly into more usable object structures.
There isn't really a 'right' language though... there are good XML packages for most languages nowadays.
If you're going to use Ruby to do it then you're going to want to take a look at Nokogiri or Hpricot. Both have their strengths and weaknesses. The language and package selection really comes down to what you want to do with the data after you've parsed it.
Reading Data out of XML files is dead easy with C# and LINQ to XML!
Somehow, although I really love python, I found it hard to parse XML with the standard libraries.
I would say it depends like everything else. VB.NET 2008 uses XML literals, has IntelliSense for LINQ to XML, and a few power toys that help turn XML into XSD. So personally, if you are working in a .NET environment I think this is the best choice.
I'm writing a small Python app for distribution. I need to include simple XML validation (it's a debugging tool), but I want to avoid any dependencies on compiled C libraries such as lxml or pyxml as those will make the resulting app much harder to distribute. I can't find anything that seems to fit the bill - for DTDs, Relax NG or XML Schema. Any suggestions?
Do you mean something like MiniXsv? I have never used it, but from the website, we can read that
minixsv is a lightweight XML schema
validator package written in pure
Python (at least Python 2.4 is
required).
so, it should work for you.
I believe that ElementTree could also be used for that goal, but I am not 100% sure.
Why don't you try invoking an online XML validator and parsing the results?
I couldn't find any free REST or SOAP based services but it would be easy enough to use a normal HTML form based one such as this one or this one. You just need to construct the correct request and parse the results (httplib may be of help here if you don't want to use a third party library such as mechanize to easy the pain).
I haven't looked at it in a while so it might have changed, but Fourthought's 4Suite (community edition) could still be pure Python.
Edit: just browsed through the docs and it's mostly Python which probably isn't enough for you.
The beautifulsoup module is pure Python (and a single .py file), but I don't know if it will fulfill your validation needs, I've only used it for quickly extracting some fields on large XMLs.
http://www.crummy.com/software/BeautifulSoup/