We Keep Coding

How can i copy data from an endpoint which sends response as an xml to a SQL server using ADF?

I have written a code in python in Azure Functions using requests -
import requests
from xml.etree import ElementTree
url = "https:..."
payload = ".."
headers = {
'Content-Type': 'application/xml',
'Accept': 'application/xml'
}
def sageSessionId():
response = requests.request("POST", url, headers=headers, data=payload, stream= True)
tree = ElementTree.fromstring(response.content)
xmlR = ElementTree.dump(tree)
return(xmlR)
The xmlR is a 'NoneType' and the Azure function on invoking only returns 200 but no content (i have called the sageSessionId in the main func in the init file).
I dont know if this is the right way to do or if there is any other way in ADF to do all these.
What i want is - to copy data from an api endpoint(which returns xml body) to sql server. I thought to write a python script as I was failing to pass the required xml body through ADF Web activity. Is it possible to do this using ADF itself invoking the api from it and getting the response also in xml format then to sql server?

You can use HTTP linked service to copy data from an api endpoint (xml response) to sql server. Below are the steps,
Create a HTTP linked service with Base URL and required authentication.
Create source dataset in HTTP with xml format and give the relative URL
Create the sink dataset (for SQL server).
Use the copy activity with the source and sink datasets created in previous steps.
In Mapping tab of copy activity, import schema, then give the appropriate collection reference to flatten the XML response and switch to advanced editor.
Thus, with HTTP linked service with xml format as dataset, we can copy api endpoint data.
Sink Data - sql DB

How to send POST body when creating a python Zeep client

The API I am working with requires that I add a POST body when requesting the wsdl client. I'm using Zeep so it will look like this:
client = Client(wsdl=url, settings=s)
How would I add a post body at this point? I checked the settings documentation and it doesn't include a way for me to that. Is there an alternative way?
If that's not possible by the library, is it possible for me to make the request using the standard python request library and use the Zeep client to process the response?
UPDATE
Here is a sample payload I need to include. Unfortunately, I can't shared the exact URL I using.
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:int="http://acme.com/integration/">
<soap:Header>
<int:AcmeNetOptions>
<int:ImposeConcurrencyId>true</int:ImposeConcurrencyId>
<int:UpdateLastModified>true</int:UpdateLastModified>
<int:CanDeleteMissingEntity>true</int:CanDeleteMissingEntity>
<int:LockOnDataRetrieval>Default</int:LockOnDataRetrieval>
</int:AcmeNetOptions>
</soap:Header>
<soap:Body>
<int:Login>
<int:username>username</int:username>
<int:password>password</int:password>
<int:companyname>company</int:companyname>
</int:Login>
</soap:Body>

Using Spyne, trying to condense the multiple namespaces generated to a single namespace in a SOAP request

I have the below SOAP request generated by SOAPUI using ComplexModel approach of Spyne request.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:ava="Namespace-1"
xmlns:book="spyne.example.django.models">
<soapenv:Header/>
<soapenv:Body>
<ava:GetChangesRequest>
<book:RequestIds>String1</book:RequestIds>
<book:From>2019-09-22</book:From>
<!--Optional:-->
<book:StartIndex>0</book:StartIndex>
</ava:GetChangesRequest>
</soapenv:Body>
</soapenv:Envelope>
And ComplexModel of request is below.
class GetChangesRequest(ComplexModel):
RequestIds = Unicode.customize(min_occurs=1)
From = Date.customize(min_occurs=1)
StartIndex = Integer.customize(default=0, min_occurs=0)
And #rpc definition is
#rpc(
GetChangesRequest,
_returns=GetChangesResponse,
_in_message_name='GetChangesRequest',
_out_message_name='GetChangesResponse',
_body_style='bare',
)
Now I am looking to avoid these multiple namespaces in the Request.
Followed this Stackoverflow post Remove the namespace from Spyne response variables
and was able to manage customize the response the way I wanted. But could not find any such approach for the SOAP Request thru Spyne.
Any pointers here will help.

You need to have the tns argument you pass to the Application to be the same as the GetChangesRequest namespace value.
ie
class GetChangesRequest(ComplexModel):
__namespace__ = 'Namespace-1'
RequestIds = Unicode.customize(min_occurs=1)
From = Date.customize(min_occurs=1)
StartIndex = Integer.customize(default=0, min_occurs=0)

Sending SOAP request using Python Requests

Is it possible to use Python's requests library to send a SOAP request?

It is indeed possible.
Here is an example calling the Weather SOAP Service using plain requests lib:
import requests
url="http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL"
#headers = {'content-type': 'application/soap+xml'}
headers = {'content-type': 'text/xml'}
body = """<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:ns0="http://ws.cdyne.com/WeatherWS/" xmlns:ns1="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
<SOAP-ENV:Header/>
<ns1:Body><ns0:GetWeatherInformation/></ns1:Body>
</SOAP-ENV:Envelope>"""
response = requests.post(url,data=body,headers=headers)
print response.content
Some notes:
The headers are important. Most SOAP requests will not work without the correct headers. application/soap+xml is probably the more correct header to use (but the weatherservice prefers text/xml
This will return the response as a string of xml - you would then need to parse that xml.
For simplicity I have included the request as plain text. But best practise would be to store this as a template, then you can load it using jinja2 (for example) - and also pass in variables.
For example:
from jinja2 import Environment, PackageLoader
env = Environment(loader=PackageLoader('myapp', 'templates'))
template = env.get_template('soaprequests/WeatherSericeRequest.xml')
body = template.render()
Some people have mentioned the suds library. Suds is probably the more correct way to be interacting with SOAP, but I often find that it panics a little when you have WDSLs that are badly formed (which, TBH, is more likely than not when you're dealing with an institution that still uses SOAP ;) ).
You can do the above with suds like so:
from suds.client import Client
url="http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL"
client = Client(url)
print client ## shows the details of this service
result = client.service.GetWeatherInformation()
print result
Note: when using suds, you will almost always end up needing to use the doctor!
Finally, a little bonus for debugging SOAP; TCPdump is your friend. On Mac, you can run TCPdump like so:
sudo tcpdump -As 0
This can be helpful for inspecting the requests that actually go over the wire.
The above two code snippets are also available as gists:
SOAP Request with requests
SOAP Request with suds

Adding up to the last answer, make sure you add to the headers the following attributes:
headers={"Authorization": f"bearer {token}", "SOAPAction": "http://..."}
The authorization is meant when you need some token to access the SOAP API,
Otherwise, the SOAPAction is the action you are going to perform with the data you are sending in,
So if you don't need Authorization, then you could pop that out of the headers,
That worked pretty fine for me,

sending data to xml api of webservice

Im trying to write a python script that basically interacts with a webservice that uses an xml api. The request method is POST.
Usually I would write a request of the form request(url, data, headers) - however, in the case of an xml api it would not work. Also something like data.encode('utf-8') or urllib.urlencode(data) would not work as the data is not a dict.
In this case, data is xml so how am i supposed to sent it over?
[EDIT]
When I send a string of XML I get a urllib2.HTTPError: HTTP Error 415: Unsupported Media Type Exception. Is there any other way I'm supposed to send the data?
Also, the API I am using the Google Contacts API. I'm trying to write a script that adds a contact to my gmail account.

You probably need to set proper Content-Type header, for XML it would probably be:
application/xml
So something like this should get you going:
request = urllib2.Request( 'xml_api.example.com' )
request.add_header('Content-Type', 'application/xml')
response = urllib2.urlopen(request, xml_data_string)
Hope that helps :)