Python SOAP to MS WebService(SharePoint)(GetListItems) - python

I am attempting to pull list Items from a sharepoint server(via python/suds) and am having some difficulty with making queries to GetListItems. If i provide no other parameters to GetListItems other than the list Name, it will return all the items in the default view for that List. I want to query a specific set of items(ID=15) to be returned and a specific set of fields(Date and Description) to be returned for those items. Here is my SOAP packet:
<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns0="http://schemas.xmlsoap.org/soap/envelope "xmlns:ns1="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<ns0:Body>
<ns1:GetListItems>
<ns1:listName>MyCalendar</ns1:listName>
<query>
<Where>
<Eq>
<FieldRef Name="_ows_ID">15</FieldRef>
</Eq>
</Where>
</query>
<viewFields>
<FieldRef Name="Description"/>
<FieldRef Name="EventDate"/>
</viewFields>
</ns1:GetListItems>
</ns0:Body>
</SOAP-ENV:Envelope>
This appears to conform to the WSDL, however, when making the query i get back a full list of items with all the fields(as if i did not pass any query XML at all).
Any suggestions for a SOAP noob?
Also, here is my python code that generated this XML:
query = Element('query')
where = Element('Where')
eq = Element('Eq')
eq.append(Element('FieldRef').append(Attribute('Name', '_ows_ID')).setText('15'))
where.append(eq)
query.append(where)
viewFields = Element('viewFields')
viewFields.append(Element('FieldRef').append(Attribute('Name','Description')))
viewFields.append(Element('FieldRef').append(Attribute('Name','EventDate')))
results = c_lists.service.GetListItems('MyCalendar', None, query, viewFields, None, None)
Any help would be greatly appreciated!

I figured this one out. The problem was the CAML query.
1. You need to wrap the CAML 'Query' in a 'query' element.
2. You need to set the proper 'Value Type' Element and attributes.
See me other posting titled 'Sharepoint Filter for List Items(GetListItems)' for more info and code.
Thanks!
Nick

Related

How to iteratively parse and save XML responses that come in as one string?

I am making API calls, that is retrieving IDs, each call represents 10000 IDs and I can only retrieve 10000 at a time. My goal is to save each XML call into a list to count how many people are in the platform automatically.
The problem I running into is two fold.
Each call comes as response object, the response object when I append to a list appends as a single string, so I can not count total number of IDs
To get the next 10000 list of IDs I have to use another API call to get information about each ID, and retrieve a piece of information called website ID and use that to call the next 10000 from the API in #1
I also want to prevent any duplicate IDs in the list but I feel like this is the easiest task.
Here is my code:
1
Call profile IDs (each call brings back 10000)
Append response object 'r' into list 'lst'
import requests
import xml.etree.ElementTree as et
import pandas as pd
from lxml import etree
import time
lst = []
xml = '''
<?xml version="1.0" encoding="utf-8" ?>
<YourMembership>
<Version>2.25</Version>
<ApiKey>*****</ApiKey>
<CallID>009</CallID>
<SaPasscode>*****</SaPasscode>
<Call Method="Sa.People.All.GetIDs">
<Timestamp></Timestamp>
<WebsiteID></WebsiteID>
<Groups>
<Code></Code>
<Name></Name>
</Groups>
</Call>
</YourMembership>
'''
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
r = requests.post('https://api.yourmembership.com', data=xml, headers=headers)
lst.append(r.text)
API Call result
<YourMembership_Response>
<Sa.People.All.GetIDs>
<People>
<ID>1234567</ID>
</People>
</Sa.People.All.GetIDs>
</YourMembership_Response>
2
I take the last ID from API call in #1 and manually input the value
into the API call below in the 'ID' tags.
xml_2 = '''
<?xml version="1.0" encoding="utf-8" ?>
<YourMembership>
<Version>2.25</Version>
<ApiKey>****</ApiKey>
<CallID>001</CallID>
<SaPasscode>****</SaPasscode>
<Call Method="Sa.People.Profile.Get">
<ID>1234567</ID>
</Call>
</YourMembership>
'''
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
r_2 = requests.post('https://api.yourmembership.com', data=xml_2, headers=headers)
print (r_2.text)
API call result:
<YourMembership_Response>
<ErrCode>0</ErrCode>
<ExtendedErrorInfo></ExtendedErrorInfo>
<Sa.People.Profile.Get>
<ID>1234567</ID>
<WebsiteID>7654321</WebsiteID>
</YourMembership_Response>
I take the website ID and rerun this in API Call from #1 (example) with website ID tag filled, get the next 10000 until no more results come back:
xml = '''
<?xml version="1.0" encoding="utf-8" ?>
<YourMembership>
<Version>2.25</Version>
<ApiKey>*****</ApiKey>
<CallID>009</CallID>
<SaPasscode>*****</SaPasscode>
<Call Method="Sa.People.All.GetIDs">
<Timestamp></Timestamp>
<WebsiteID>7654321</WebsiteID>
<Groups>
<Code></Code>
<Name></Name>
</Groups>
</Call>
</YourMembership>
'''
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
r = requests.post('https://api.yourmembership.com', data=xml, headers=headers)
lst.append(r.text)
Hope my question makes sense, and thank you in advance.
I once started building something to crawl over an API which sounds similar to what you are aiming to achieve. One difference in my case though was the response came as json instead of xml but shouldn't be a big deal.
Can't see in your question evidence that you are really using the power of the xml parser. Have a look at the docs. For example you can easily get the id number out of those items you are appending to the list like this:
xml_sample = """
<YourMembership_Response>
<Sa.People.All.GetIDs>
<People>
<ID>1234567</ID>
</People>
</Sa.People.All.GetIDs>
</YourMembership_Response>
"""
import xml.etree.ElementTree as ET
root = ET.fromstring(xml_sample)
print (root[0][0][0].text)
>>> '1234567'
Experiment, apply it in a loop to each element in the list or maybe you will be lucky and the whole response object will parse without needing to look through things.
You should now be able to programmatically instead of manually enter that number in the next bit of code.
Your XML for the next section for the website ID seems to have an invalid line in it <Sa.People.Profile.Get> Once I take it out it can be parsed:
xml_sample2 = """
<YourMembership_Response>
<ErrCode>0</ErrCode>
<ExtendedErrorInfo></ExtendedErrorInfo>
<ID>1234567</ID>
<WebsiteID>7654321</WebsiteID>
</YourMembership_Response>
"""
root2 = ET.fromstring(xml_sample2)
print (root2[3].text)
>>> '7654321'
So not sure if there is always an invalid line there or if you forgot to paste something, maybe remove that line with regex or something before applying xtree.
Would recommend you try sqlite to help you with the interactions between 1 and 2. I think it's good up to half a million rows otherwise you would need to hook to a proper database. It saves a file in your directory and has a bit less setup time and fuss as with a proper database. Perhaps, test the concept with sqlite and if necessary migrate to postgresql.
You can store whichever useful elements from this parsed xml you like user ID, website ID into a table and pull it out again to use in a different section. Is also not hard to go back and forth from sqlite to pandas dataframes if you need it with pandas.read_sql and pandas.DataFrame.to_sql Hope this helps..

SOAP - Must specify a type attribute value for the element

Using Python zeep, I'm interacting with Salesforce's SOAP (specifically, Metadata) API.
Trying to createMetadata I get this error:
Fault: Must specify a {http://www.w3.org/2001/XMLSchema-instance}type attribute value for the {http://soap.sforce.com/2006/04/metadata}metadata element
I've gathered that this is not about parameters passed to the method (the way createMetadata requires a metadata argument, which itself is an object with a fullName field), but rather about a missing xsi:type attribute somewhere.
This is my zeep call:
resp = service['createMetadata'](_soapheaders=soap_headers,
metadata=[{'fullName': 'SomeCustomObject'}])
This is the generated XML:
<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">
<soap-env:Body>
<ns0:createMetadata xmlns:ns0="http://soap.sforce.com/2006/04/metadata">
<ns0:metadata>
<ns0:fullName>SomeCustomObject</ns0:fullName>
</ns0:metadata>
</ns0:createMetadata>
</soap-env:Body>
</soap-env:Envelope>
My question is: how can I set that xsi:type on whatever it needs to be set on (that ns0:metadata guy?) using zeep?
UPDATE:
Instead of using a dictionary to represent the metadata object, I replaced it with this:
metadata_type = client.get_type('{http://soap.sforce.com/2006/04/metadata}Metadata')
metadata = metadata_type(fullName='SomeCustomObject')
resp = service['createMetadata'](_soapheaders=soap_headers, metadata=[metadata])
The new generated XML is:
<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">
<soap-env:Body>
<ns0:createMetadata xmlns:ns0="http://soap.sforce.com/2006/04/metadata">
<ns0:metadata xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns0:Metadata">
<ns0:fullName>SomeCustomObject</ns0:fullName>
</ns0:metadata>
</ns0:createMetadata>
</soap-env:Body>
</soap-env:Envelope>
which has the xsi:type attribute on the ns0:metadata tag, but I get the same error as before. So I guess it wasn't about a missing xsi:type. Any ideas on what it is?
Here you can see that xsi is defined in metadata.
<metadata xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CustomField">
</metadata>
The answer is that xsi:type should use 'CustomObject' (or another appropriate type) instead of 'Metadata', which is the "parent" type, I believe. Which also then requires passing more than just fullName.
In zeep that means instead of
metadata_type = client.get_type('{http://soap.sforce.com/2006/04/metadata}Metadata')
metadata = metadata_type(fullName='SomeCustomObject')
I used
custom_object_type = client.get_type('{http://soap.sforce.com/2006/04/metadata}CustomObject')
custom_object = custom_object_type(fullName='SomeCustomObject__c',
label='SomeCustomObject',
pluralLabel='SomeCustomObjects',
nameField={'label': 'name', 'type': 'Text'},
deploymentStatus='Deployed',
sharingModel='ReadWrite')
and then finally:
resp = service['createMetadata'](_soapheaders=soap_headers,
metadata=[custom_object])

Parsing soap/XML response in Python

I am trying to parse the below xml using the python. I do not understand which type of xml this is as I never worked on this kind of xml.I just got it from a api response form Microsoft.
Now my question is how to parse and get the value of BinarySecurityToken in my python code.
I refer this question Parse XML SOAP response with Python
But look like this has also some xmlns to get the text .However in my xml I can't see any nearby xmlns value through I can get the value.
Please let me know how to get the value of a specific filed using python from below xml.
<?xml version="1.0" encoding="utf-8" ?>
<S:Envelope xmlns:S="http://www.w3.org/2003/05/soap-envelope" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:wsa="http://www.w3.org/2005/08/addressing">
<S:Header>
<wsa:Action xmlns:S="http://www.w3.org/2003/05/soap-envelope" xmlns:wsa="http://www.w3.org/2005/08/addressing" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" wsu:Id="Action" S:mustUnderstand="1">http://schemas.xmlsoap.org/ws/2005/02/trust/RSTR/Issue</wsa:Action>
<wsa:To xmlns:S="http://www.w3.org/2003/05/soap-envelope" xmlns:wsa="http://www.w3.org/2005/08/addressing" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" wsu:Id="To" S:mustUnderstand="1">http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:To>
<wsse:Security S:mustUnderstand="1">
<wsu:Timestamp xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" wsu:Id="TS">
<wsu:Created>2017-06-12T10:23:01Z</wsu:Created>
<wsu:Expires>2017-06-12T10:28:01Z</wsu:Expires>
</wsu:Timestamp>
</wsse:Security>
</S:Header>
<S:Body>
<wst:RequestSecurityTokenResponse xmlns:S="http://www.w3.org/2003/05/soap-envelope" xmlns:wst="http://schemas.xmlsoap.org/ws/2005/02/trust" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion" xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy" xmlns:psf="http://schemas.microsoft.com/Passport/SoapServices/SOAPFault">
<wst:TokenType>urn:passport:compact</wst:TokenType>
<wsp:AppliesTo xmlns:wsa="http://www.w3.org/2005/08/addressing">
<wsa:EndpointReference>
<wsa:Address>https://something.something.something.com</wsa:Address>
</wsa:EndpointReference>
</wsp:AppliesTo>
<wst:Lifetime>
<wsu:Created>2017-06-12T10:23:01Z</wsu:Created>
<wsu:Expires>2017-06-13T10:23:01Z</wsu:Expires>
</wst:Lifetime>
<wst:RequestedSecurityToken>
<wsse:BinarySecurityToken Id="Compact0">my token</wsse:BinarySecurityToken>
</wst:RequestedSecurityToken>
<wst:RequestedAttachedReference>
<wsse:SecurityTokenReference>
<wsse:Reference URI="wwwww=">
</wsse:Reference>
</wsse:SecurityTokenReference>
</wst:RequestedAttachedReference>
<wst:RequestedUnattachedReference>
<wsse:SecurityTokenReference>
<wsse:Reference URI="swsw=">
</wsse:Reference>
</wsse:SecurityTokenReference>
</wst:RequestedUnattachedReference>
</wst:RequestSecurityTokenResponse>
</S:Body>
</S:Envelope>
This declaration is part of the start tag of the root element:
xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"
It means that elements with the wsse prefix (such as BinarySecurityToken) are in the http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd namespace.
The solution is basically the same as in the answer to the linked question. It's just another namespace:
import xml.etree.ElementTree as ET
tree = ET.parse('soap.xml')
print tree.find('.//{http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd}BinarySecurityToken').text
Here is another way of doing it:
import xml.etree.ElementTree as ET
ns = {"wsse": "http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"}
tree = ET.parse('soap.xml')
print tree.find('.//wsse:BinarySecurityToken', ns).text
The output in both cases is my token.
See https://docs.python.org/2.7/library/xml.etree.elementtree.html#parsing-xml-with-namespaces.
Creating a namespace dict helped me. Thank you #mzjn for linking that article.
In my SOAP response, I found that I was having to use the full path to the element to extract the text.
For example, I am working with FEDEX API, and one element that I needed to find was TrackDetails. My initial .find() looked like .find('{http://fedex.com/ws/track/v16}TrackDetails')
I was able to simplify this to the following:
ns = {'TrackDetails': 'http://fedex.com/ws/track/v16'}
tree.find('TrackDetails:TrackDetails',ns)
You see TrackDetails twice because I named the key TrackDetails in the dict, but you could name this anything you want. Just helped me to remember what I was working on in my project, but the TrackDetails after the : is the actual element in the SOAP response that I need.
Hope this helps someone!

Python - parse xml with lxml trouble

I've found a lot of questions on this issue but nothing I saw fits mine. I'm new to lxml so need some help.
my users.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<root>
<user>
<login>elena</login>
<password>elena</password>
<group>1</group>
</user>
<user>
<login>anele</login>
<password>anele</password>
<group>2</group>
</user>
</root>
the trouble function:
def analize_data(login):
doc = etree.parse("/myapp/users.xml")
for elem in doc.iter(tag='login'):
if elem.text == login:
parent = elem.getparent()
group = etree.SubElement(parent, 'group')
return group.text
What I need:
to find a user tag with login passed to function and get the text of group subelement of this user. But this function returns None when testing. What am I doing wrong and how to fix it?
I'm new to all these things, so need help. Thanks in advance!
Try using:
group = parent.iterchildren(tag="group").next()
etree.SubElement does something completely different:
This function creates an element instance, and appends it to an existing element.
Which is clearly not what you want.

Building a service for a given request

I am relatively new to SOAP frameworks and have been reading through Spynes docs and trying to figure out to build a service that accepts the following request:
<?xml version='1.0' encoding='UTF-8'?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://..." xmlns:xsi=http:/..." xmlns:xsd="http://...">
<SOAP-ENV:Body>
<modifyRequest returnData="everything" xmlns="urn:...">
<attr ID="..."/>
<data>
</data>
</modifyRequest>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>
I have read through the docs but just have not seen enough more complex examples to figure out how to put together something to handle this. The <attr ID="..."/> tag must be processed for the ID attribute and the <data> tags contain some varying collection of additional xml. I understand its better to formally define the service but for now I was hoping to use anyXML (?) to accept whatever is in the tags. I need to accept and process the ID attribute along with its xml payload contained within the data tags.
I'd be grateful for any guidance,
Thanks.
Here's how you'd do it:
NS = 'urn:...'
class Attr(ComplexModel):
__namespace__ = NS
_type_info = [
('ID', XmlAttribute(UnsignedInteger32)),
]
class ModifyRequest(ComplexModel):
__namespace__ = NS
_type_info = [
('returnData', XmlAttribute(Unicode(values=['everything', 'something', 'anything', 'etc']))),
('attr', Attr),
('data', AnyXml),
]
class SomeService(ServiceBase):
#rpc(ModifyRequest, _body_style='bare')
def modifyRequest(ctx, request):
pass
This requires Spyne 2.11 though, _body_style='bare' in problematic in 2.10 and older.

Categories