We Keep Coding

Extract xlink:href attribute from XML [duplicate]

I have the following XML which I want to parse using Python's ElementTree:
<rdf:RDF xml:base="http://dbpedia.org/ontology/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:owl="http://www.w3.org/2002/07/owl#"
xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
xmlns="http://dbpedia.org/ontology/">
<owl:Class rdf:about="http://dbpedia.org/ontology/BasketballLeague">
<rdfs:label xml:lang="en">basketball league</rdfs:label>
<rdfs:comment xml:lang="en">
a group of sports teams that compete against each other
in Basketball
</rdfs:comment>
</owl:Class>
</rdf:RDF>
I want to find all owl:Class tags and then extract the value of all rdfs:label instances inside them. I am using the following code:
tree = ET.parse("filename")
root = tree.getroot()
root.findall('owl:Class')
Because of the namespace, I am getting the following error.
SyntaxError: prefix 'owl' not found in prefix map
I tried reading the document at http://effbot.org/zone/element-namespaces.htm but I am still not able to get this working since the above XML has multiple nested namespaces.
Kindly let me know how to change the code to find all the owl:Class tags.

You need to give the .find(), findall() and iterfind() methods an explicit namespace dictionary:
namespaces = {'owl': 'http://www.w3.org/2002/07/owl#'} # add more as needed
root.findall('owl:Class', namespaces)
Prefixes are only looked up in the namespaces parameter you pass in. This means you can use any namespace prefix you like; the API splits off the owl: part, looks up the corresponding namespace URL in the namespaces dictionary, then changes the search to look for the XPath expression {http://www.w3.org/2002/07/owl}Class instead. You can use the same syntax yourself too of course:
root.findall('{http://www.w3.org/2002/07/owl#}Class')
Also see the Parsing XML with Namespaces section of the ElementTree documentation.
If you can switch to the lxml library things are better; that library supports the same ElementTree API, but collects namespaces for you in .nsmap attribute on elements and generally has superior namespaces support.

Here's how to do this with lxml without having to hard-code the namespaces or scan the text for them (as Martijn Pieters mentions):
from lxml import etree
tree = etree.parse("filename")
root = tree.getroot()
root.findall('owl:Class', root.nsmap)
UPDATE:
5 years later I'm still running into variations of this issue. lxml helps as I showed above, but not in every case. The commenters may have a valid point regarding this technique when it comes merging documents, but I think most people are having difficulty simply searching documents.
Here's another case and how I handled it:
<?xml version="1.0" ?><Tag1 xmlns="http://www.mynamespace.com/prefix">
<Tag2>content</Tag2></Tag1>
xmlns without a prefix means that unprefixed tags get this default namespace. This means when you search for Tag2, you need to include the namespace to find it. However, lxml creates an nsmap entry with None as the key, and I couldn't find a way to search for it. So, I created a new namespace dictionary like this
namespaces = {}
# response uses a default namespace, and tags don't mention it
# create a new ns map using an identifier of our choice
for k,v in root.nsmap.iteritems():
if not k:
namespaces['myprefix'] = v
e = root.find('myprefix:Tag2', namespaces)

Note: This is an answer useful for Python's ElementTree standard library without using hardcoded namespaces.
To extract namespace's prefixes and URI from XML data you can use ElementTree.iterparse function, parsing only namespace start events (start-ns):
>>> from io import StringIO
>>> from xml.etree import ElementTree
>>> my_schema = u'''<rdf:RDF xml:base="http://dbpedia.org/ontology/"
... xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
... xmlns:owl="http://www.w3.org/2002/07/owl#"
... xmlns:xsd="http://www.w3.org/2001/XMLSchema#"
... xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
... xmlns="http://dbpedia.org/ontology/">
...
... <owl:Class rdf:about="http://dbpedia.org/ontology/BasketballLeague">
... <rdfs:label xml:lang="en">basketball league</rdfs:label>
... <rdfs:comment xml:lang="en">
... a group of sports teams that compete against each other
... in Basketball
... </rdfs:comment>
... </owl:Class>
...
... </rdf:RDF>'''
>>> my_namespaces = dict([
... node for _, node in ElementTree.iterparse(
... StringIO(my_schema), events=['start-ns']
... )
... ])
>>> from pprint import pprint
>>> pprint(my_namespaces)
{'': 'http://dbpedia.org/ontology/',
'owl': 'http://www.w3.org/2002/07/owl#',
'rdf': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#',
'rdfs': 'http://www.w3.org/2000/01/rdf-schema#',
'xsd': 'http://www.w3.org/2001/XMLSchema#'}
Then the dictionary can be passed as argument to the search functions:
root.findall('owl:Class', my_namespaces)

I've been using similar code to this and have found it's always worth reading the documentation... as usual!
findall() will only find elements which are direct children of the current tag. So, not really ALL.
It might be worth your while trying to get your code working with the following, especially if you're dealing with big and complex xml files so that that sub-sub-elements (etc.) are also included.
If you know yourself where elements are in your xml, then I suppose it'll be fine! Just thought this was worth remembering.
root.iter()
ref: https://docs.python.org/3/library/xml.etree.elementtree.html#finding-interesting-elements
"Element.findall() finds only elements with a tag which are direct children of the current element. Element.find() finds the first child with a particular tag, and Element.text accesses the element’s text content. Element.get() accesses the element’s attributes:"

To get the namespace in its namespace format, e.g. {myNameSpace}, you can do the following:
root = tree.getroot()
ns = re.match(r'{.*}', root.tag).group(0)
This way, you can use it later on in your code to find nodes, e.g using string interpolation (Python 3).
link = root.find(f"{ns}link")

This is basically Davide Brunato's answer however I found out that his answer had serious problems the default namespace being the empty string, at least on my python 3.6 installation. The function I distilled from his code and that worked for me is the following:
from io import StringIO
from xml.etree import ElementTree
def get_namespaces(xml_string):
namespaces = dict([
node for _, node in ElementTree.iterparse(
StringIO(xml_string), events=['start-ns']
)
])
namespaces["ns0"] = namespaces[""]
return namespaces
where ns0 is just a placeholder for the empty namespace and you can replace it by any random string you like.
If I then do:
my_namespaces = get_namespaces(my_schema)
root.findall('ns0:SomeTagWithDefaultNamespace', my_namespaces)
It also produces the correct answer for tags using the default namespace as well.

My solution is based on #Martijn Pieters' comment:
register_namespace only influences serialisation, not search.
So the trick here is to use different dictionaries for serialization and for searching.
namespaces = {
'': 'http://www.example.com/default-schema',
'spec': 'http://www.example.com/specialized-schema',
}
Now, register all namespaces for parsing and writing:
for name, value in namespaces.iteritems():
ET.register_namespace(name, value)
For searching (find(), findall(), iterfind()) we need a non-empty prefix. Pass these functions a modified dictionary (here I modify the original dictionary, but this must be made only after the namespaces are registered).
self.namespaces['default'] = self.namespaces['']
Now, the functions from the find() family can be used with the default prefix:
print root.find('default:myelem', namespaces)
but
tree.write(destination)
does not use any prefixes for elements in the default namespace.

XML parser that contains debug information

I'm looking for an XML parser for python that includes some debug information in each node, for instance the line number and column where the node began. Ideally, it would be a parser that is compatible with xml.etree.ElementTree.XMLParser, i.e., one that I can pass to xml.etree.ElementTree.parse.
I know these parsers don't actually produce the elements, so I'm not sure how this would really work, but it seems like such a useful thing, I'll be surprised if no-body has one. Syntax errors in the XML are one thing, but semantic errors in the resulting structure can be difficult to debug if you can't point to a specific location in the source file/string.

Point to an element by xpath (lxml - getpath)
lxml offers finding an xpath for an element in document.
Having test document:
>>> from lxml import etree
>>> xmlstr = """<root><rec id="01"><subrec>a</subrec><subrec>b</subrec></rec>
... <rec id="02"><para>graph</para></rec>
... </root>"""
...
>>> doc = etree.fromstring(xmlstr)
>>> doc
<Element root at 0x7f61040fd5f0>
We pick an element <para>graph</para>:
>>> para = doc.xpath("//para")[0]
>>> para
<Element para at 0x7f61040fd488>
XPath has a meaning, if we have clear context, in this case it is root of the XML document:
>>> root = doc.getroottree()
>>> root
<lxml.etree._ElementTree at 0x7f610410f758>
And now we can ask, what xpath leads from the root to the element of our interest:
>>> root.getpath(para)
'/root/rec[2]/para'

Python: ElementTree, get the namespace string of an Element

This XML file is named example.xml:
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>14.0.0</modelVersion>
<groupId>.com.foobar.flubber</groupId>
<artifactId>uberportalconf</artifactId>
<version>13-SNAPSHOT</version>
<packaging>pom</packaging>
<name>Environment for UberPortalConf</name>
<description>This is the description</description>
<properties>
<birduberportal.version>11</birduberportal.version>
<promotiondevice.version>9</promotiondevice.version>
<foobarportal.version>6</foobarportal.version>
<eventuberdevice.version>2</eventuberdevice.version>
</properties>
<!-- A lot more here, but as it is irrelevant for the problem I have removed it -->
</project>
If I load example.xml and parse it with ElementTree I can see its namespace is http://maven.apache.org/POM/4.0.0.
>>> from xml.etree import ElementTree
>>> tree = ElementTree.parse('example.xml')
>>> print tree.getroot()
<Element '{http://maven.apache.org/POM/4.0.0}project' at 0x26ee0f0>
I have not found a method to call to get just the namespace from an Element without resorting to parsing the str(an_element) of an Element. It seems like there got to be a better way.

This is a perfect task for a regular expression.
import re
def namespace(element):
m = re.match(r'\{.*\}', element.tag)
return m.group(0) if m else ''

The namespace should be in Element.tag right before the "actual" tag:
>>> root = tree.getroot()
>>> root.tag
'{http://maven.apache.org/POM/4.0.0}project'
To know more about namespaces, take a look at ElementTree: Working with Namespaces and Qualified Names.

I am not sure if this is possible with xml.etree, but here is how you could do it with lxml.etree:
>>> from lxml import etree
>>> tree = etree.parse('example.xml')
>>> tree.xpath('namespace-uri(.)')
'http://maven.apache.org/POM/4.0.0'

Without using regular expressions:
>>> root
<Element '{http://www.google.com/schemas/sitemap/0.84}urlset' at 0x2f7cc10>
>>> root.tag.split('}')[0].strip('{')
'http://www.google.com/schemas/sitemap/0.84'

The lxml.xtree library's element has a dictionary called nsmap, which shows all the namespace that are in use in the current tag scope.
>>> item = tree.getroot().iter().next()
>>> item.nsmap
{'md': 'urn:oasis:names:tc:SAML:2.0:metadata'}

I think it will be easier to take a look at the attributes:
>>> root.attrib
{'{http://www.w3.org/2001/XMLSchema-instance}schemaLocation':
'http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd'}

The short answer is:
ElementTree._namspace_map[ElementTree._namspace_map.values().index('')]
but only if you have been calling
ElementTree.register_namespace(prefix,uri)
in response to every event=="start-ns" received while iterating through the result of
ET.iterparse(...)
and you registered for "start-ns"
The answer the question "what is the default namespace?", it is necessary to clarify two points:
(1) XML specifications say that the default namespace is not necessarily global throughout the tree, rather the default namespace can be re-declared at any element under root, and inherits downwards until meeting another default namespace re-declaration.
(2) The ElementTree module can (de facto) handle XML-like documents which have no root default namespace, -if- they have no namespace use anywhere in the document. (* there may be less strict conditions, e.g., that is "if" and not necessarily "iff").
It's probably also worth considering "what do you want it for?" Consider that XML files can be semantically equivalent, but syntactically very different. E.g., the following three files are semantically equivalent, but A.xml has one default namespace declaration, B.xml has three, and C.xml has none.
A.xml:
<a xlmns="http://A" xlmns:nsB0="http://B0" xlmns:nsB1="http://B1">
<nsB0:b/>
<nsB1:b/>
</a>
B.xml:
<a xlmns="http://A">
<b xlmns="http://B0"/>
<b xlmns="http://B1"/>
</a>
C.xml:
<{http://A}a>
<{http://B0}b/>
<{http://B1}b/>
</a>
The file C.xml is the canonical expanded syntactical representation presented to the ElementTree search functions.
If you are certain a priori that there will be no namespace collisions, you can modify the element tags while parsing as discussed here: Python ElementTree module: How to ignore the namespace of XML files to locate matching element when using the method "find", "findall"

combining some of the answers above, I think the shortest code is
theroot = tree.getroot()
theroot.attrib[theroot.keys()[0]]

Here is my solution on ElementTree 3.9+,
def get_element_namespaces(filename, element):
namespace = []
for key, value in ET.iterparse(filename, events=['start', 'start-ns']):
print(key, value)
if key == 'start-ns':
namespace.append(value)
else:
if ET.tostring(element) == ET.tostring(value):
return namespace
namespace = []
return namespaces
This would return an array of [prefix:URL] tuples like this:
[('android', 'http://schemas.android.com/apk/res/android'), ('tools', 'http://schemas.android.com/tools')]

Python: Read and write namespaced XML using ElementTree

This XML file is named example.xml:
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>14.0.0</modelVersion>
<groupId>.com.foobar.flubber</groupId>
<artifactId>uberportalconf</artifactId>
<version>13-SNAPSHOT</version>
<packaging>pom</packaging>
<name>Environment for UberPortalConf</name>
<description>This is the description</description>
<properties>
<birduberportal.version>11</birduberportal.version>
<promotiondevice.version>9</promotiondevice.version>
<foobarportal.version>6</foobarportal.version>
<eventuberdevice.version>2</eventuberdevice.version>
</properties>
<!-- A lot more here, but as it is irrelevant for the problem I have removed it -->
</project>
If I load the example.xml file above using ElementTree and print the root node:
>>> from xml.etree import ElementTree
>>> tree = ElementTree.parse('example.xml')
>>> print tree.getroot()
<Element '{http://maven.apache.org/POM/4.0.0}project' at 0x26ee0f0>
I see that Element also contains the namespace http://maven.apache.org/POM/4.0.0.
How do I:
Get the foobarportal.version text, increase it by one and write the XML file back while keeping the namespace the document had when loaded and also not change the overall XML layout.
Get it to load using any namespace, not just http://maven.apache.org/POM/4.0.0. I still don´t want to strip the namespace, as I want the XML to stay the same except for changing foobarportal.version as in 1 above.
The current way is not aware of XML but fulfills 1 and 2 above:
Grep for <foobarportal.version>(.*)</foobarportal.version>
Take the contents of the match group and i increase it by one
Write it back.
It would be nice to have an XML aware solution, as it would be more robust. The XML namespace handling of ElementTree is making it more complicated.

If your question is simply: "how do I search by a namespaced element name", then the answer is that lxml understands {namespace} syntax, so you can do:
tree.getroot().find('{http://maven.apache.org/POM/4.0.0}project')

Python XML question

I have an XML document as a str. Now, in the XSD <foo> is unbounded, and while most of the time there is only 1, there COULD be more. I'm trying to use ElementTree, but am running into an issue:
>>> from xml.etree.ElementTree import fromstring
>>>
>>> xml_str = """<?xml version="1.0"?>
... <foo>
... <bar>
... <baz>Spam</baz>
... <qux>Eggs</qux>
... </bar>
... </foo>"""
>>> # Try to get the document
>>> el = fromstring(xml_str)
>>> el.findall('foo')
[]
>>> el.findall('bar')
[<Element 'bar' at 0x1004acb90>]
Clearly, I need to loop through the <foo>s, but because <foo> is at the root, I can't. Obviously, I could create an element called <root> and put el inside of it, but is there a more correct way of doing this?

Each XML document is supposed to have exactly one root element. You will need to adjust your XML if you want to support multiple foo elements.

Alas, wrapping the element in an ElementTree with tree = ElementTree(el) and trying tree.findall('//foo') doesn't seem to work either (it seems you can only search "beneath" an element, and even if the search is done from the full tree, it searches "beneath" the root). As ElementTree doesn't claim to really implement xpath, it's difficult to say whether this is intended or a bug.
Solution: without using lxml with full xpath support (el.xpath('//foo') for example), the easiest solution would be to use the Element.iter() method.
for foo in el.iter(tag='foo'):
print foo
or if you want the results in a list:
list(el.iter(tag='foo'))
Note that you can't use complex paths in this way, just find all elements with a certain tagname, starting from (and including) the element.