I'm trying to call a SOAP webservice from the Dutch land register (WSDL here). I first tried doing that using the pysimplesoap library. Although I do get relevant xml back, pysimplesoap gives a TypeError: Tag: IMKAD_Perceel invalid (type not found) (I created a SO question about that here). Since I suspect this to be a bug in pysimplesoap I'm now trying to use the suds library.
In pysimplesoap the following returned correct xml (but as I said pysimplesoap gave a TypeError):
from pysimplesoap.client import SoapClient
client = SoapClient(wsdl='http://www1.kadaster.nl/1/schemas/kik-inzage/20141101/verzoekTotInformatie-2.1.wsdl', username=xxx, password=xxx, trace=True)
response = client.VerzoekTotInformatie(
Aanvraag={
'berichtversie': '4.7', # Refers to the schema version: http://www.kadaster.nl/web/show?id=150593&op=/1/schemas/homepage.html
'klantReferentie': 'MyReference1', # Refers to something we can set ourselves.
'productAanduiding': '1185', # a four-digit code referring to whether the response should be in "XML" (1185), "PDF" (1191) or "XML and PDF" (1057).
'Ingang': {
'Object': {
'IMKAD_KadastraleAanduiding': {
'gemeente': 'ARNHEM',
'sectie': 'AC',
'perceelnummer': '1234'
}
}
}
}
)
This produced the xml below:
<soap:Body>
<VerzoekTotInformatieRequest xmlns="http://www.kadaster.nl/schemas/kik-inzage/20141101">
<Aanvraag xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">
<berichtversie xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">4.7</berichtversie>
<klantReferentie xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">ARNHEM-AC-1234</klantReferentie>
<productAanduiding xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">1185</productAanduiding>
<Ingang xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">
<Object xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">
<IMKAD_KadastraleAanduiding xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">
<gemeente xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">ARNHEM AC</gemeente>
<sectie xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">AC</sectie>
<perceelnummer xmlns="http://www.kadaster.nl/schemas/kik-inzage/ip-aanvraag/v20141101">5569</perceelnummer>
</IMKAD_KadastraleAanduiding>
</Object>
</Ingang>
</Aanvraag>
</VerzoekTotInformatieRequest>
</soap:Body>
So now I tried changing this code to use suds instead. So far I came up with this:
from suds.client import Client
client = Client(url='http://www1.kadaster.nl/1/schemas/kik-inzage/20141101/verzoekTotInformatie-2.1.wsdl', username='xxx', password='xxx')
Aanvraag = client.factory.create('ns3:Aanvraag')
Aanvraag.berichtversie = '4.7'
Aanvraag.klantReferentie = 'MyReference1'
Aanvraag.productAanduiding = '1185'
IMKAD_KadastraleAanduiding = client.factory.create('ns3:IMKAD_KadastraleAanduiding')
IMKAD_KadastraleAanduiding.gemeente = 'ARNHEM'
IMKAD_KadastraleAanduiding.sectie = 'AC'
IMKAD_KadastraleAanduiding.perceelnummer = '1234'
Object = client.factory.create('ns3:Object')
Object.IMKAD_KadastraleAanduiding = IMKAD_KadastraleAanduiding
Ingang = client.factory.create('ns3:Ingang')
Ingang.Object = Object
Aanvraag.Ingang = Ingang
result = client.service.VerzoekTotInformatie(Aanvraag)
which produces the following xml:
<ns2:Body>
<ns0:VerzoekTotInformatieRequest>
<ns0:Aanvraag>
<ns1:berichtversie>4.7</ns1:berichtversie>
<ns1:klantReferentie>MyReference1</ns1:klantReferentie>
<ns1:productAanduiding>1185</ns1:productAanduiding>
<ns1:Ingang>
<ns1:Object>
<ns1:IMKAD_KadastraleAanduiding>
<ns1:gemeente>ARNHEM</ns1:gemeente>
<ns1:sectie>AC</ns1:sectie>
<ns1:perceelnummer>1234</ns1:perceelnummer>
</ns1:IMKAD_KadastraleAanduiding>
</ns1:Object>
</ns1:Ingang>
</ns0:Aanvraag>
</ns0:VerzoekTotInformatieRequest>
</ns2:Body>
Unfortunately, this results in the server giving back a Nullpointer:
Traceback (most recent call last):
File "<input>", line 1, in <module>
result = client.service.VerzoekTotInformatie(Aanvraag)
File "/Library/Python/2.7/site-packages/suds/client.py", line 542, in __call__
return client.invoke(args, kwargs)
File "/Library/Python/2.7/site-packages/suds/client.py", line 602, in invoke
result = self.send(soapenv)
File "/Library/Python/2.7/site-packages/suds/client.py", line 649, in send
result = self.failed(binding, e)
File "/Library/Python/2.7/site-packages/suds/client.py", line 702, in failed
r, p = binding.get_fault(reply)
File "/Library/Python/2.7/site-packages/suds/bindings/binding.py", line 265, in get_fault
raise WebFault(p, faultroot)
WebFault: Server raised fault: 'java.lang.NullPointerException'
This error is of course terribly unhelpful. The error gives no hint whatsoever on what causes the NullPointer.
If I look at the differences between the xml which pysimplesoap and suds send over the wire, the xml by suds is missing a lot of xmlns definitions (although I don't know whether they are needed) and the names of the tags include prefixes with for example ns0:. I don't know if these differences are relevant, and I also don't know how I would make suds create the same xml as pysimplesoap.
Although the wsdl file of the service is public, the service itself is paid (€60 yearly + €3 for every successful request). So I guess it is hard/impossible for people reading this to reproduce the issue, and I can't really give out my user credentials here.
But since I'm really stuck on this issue, maybe someone can give me some tips on how to debug this? For example; how can I make suds create the same xml as pysimplesoap? Or how I can get more information on the nullpointer?
Any help is welcome!
This is not so much an answer, but an advice from prior experience with Python and SOAP.
Find some good (established, reference for SOAP) Java tool for making SOAP queries given WSDL.
Make some typical queries, interesting to you, and record what is being sent / received as templates
Forget Python SOAP libraries and just use template to query SOAP endpoint (there are many templating languages for Python).
If the step 2. fails with the prominent Java tool, contact techsupport of the service you are paying for.
Have you checked whether all those nice XSDs are really downloaded by Python SOAP clients?
Related
Using pysolr trying to add document to solr with Python3.9 and getting below error 400 even with only 1 or 2 fields.
The fields that I'm using here are dynamic fields.
No issue with connecting to solr.
#/usr/bin/python
import pysolr
solr = pysolr.Solr('http://localhost:8080/solr/', always_commit=True)
if solr.ping():
print('connection successful')
docs = [{'id':'123c', 's_chan_name': 'TV-201'}]
solr.add(docs)
res = solr.search('123c')
print(res)
Getting below error:
connection successful
Traceback (most recent call last):
File "/tmp/test-solr.py", line 8, in <module>
solr.add(docs)
File "/usr/local/lib/python3.9/site-packages/pysolr.py", line 1042, in add
return self._update( File "/usr/local/lib/python3.9/site-packages/pysolr.py", line 568, in
_update
return self._send_request( File "/usr/local/lib/python3.9/site-packages/pysolr.py", line 463, in
_send_request
raise SolrError(error_message % (resp.status_code, solr_message))
pysolr.SolrError: Solr responded with an error (HTTP 400): [Reason: None]
<html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 400 Unexpected character '[' (code 91) in prolog;
expected '<' at [row,col {unknown-source}]: [1,1]</title>
</head><body><h2>HTTP ERROR 400</h2><p>Problem accessing /solr/update/.
Reason:<pre> Unexpected character '[' (code 91) in prolog;
expected '<' at [row,col {unknown-source}]: [1,1]</pre>
</p><hr><a href="http://eclipse.org/jetty">
Powered by Jetty://9.4.15.v20190215</a><hr/></body></html>
I’d recommend looking at the debug logs (which might require you to use logging.basicConfig() to enable them) and testing the URLs it’s using yourself. Depending on your Solr configuration, it might be the case that your Solr URL needs to have the core at the end (e.g. http://localhost:8983/solr/mycore).
In general, most reports like this which we get turn out to be oddities in how someone configured Solr. In addition to the debug logging, I highly recommend looking through the test suite since that does everything needed to run Solr and can be a handy point for comparison:
https://github.com/django-haystack/pysolr
solr.add method supports only one attribute
docs = {'id':'123c', 's_chan_name': 'TV-201'}
solr.add(docs)
it is not suitable for iterable.
docs = [{'id':'123c', 's_chan_name': 'TV-201'}]
solr.add_many(docs)
it will works
I am updating data on a Neo4j server using Python (2.7.6) and Py2Neo (1.6.4). My load function is:
from py2neo import neo4j,node, rel, cypher
session = cypher.Session('http://my_neo4j_server.com.mine:7474')
def load_data():
tx = session.create_transaction()
for row in dataframe.iterrows(): #dataframe is a pandas dataframe
name = row[1].name
id = row[1].id
merge_query = "MERGE (a:label {name:'%s', name_var:'%s'}) " % (id, name)
tx.append(merge_query)
tx.commit()
When I execute this from Spyder in Windows it works great. All the data from the dataframe is committed to neo4j and visible in the graph. However, when I run this from a linux server (different from the neo4j server) I get the following error at tx.commit(). Note that I have the same version of python and py2neo.
INFO:py2neo.packages.httpstream.http:>>> POST http://neo4j1.qs:7474/db/data/transaction/commit [1360120]
INFO:py2neo.packages.httpstream.http:<<< 200 OK [chunked]
ERROR:__main__:some part of process failed
Traceback (most recent call last):
File "my_file.py", line 132, in load_data
tx.commit()
File "/usr/local/lib/python2.7/site-packages/py2neo/cypher.py", line 242, in commit
return self._post(self._commit or self._begin_commit)
File "/usr/local/lib/python2.7/site-packages/py2neo/cypher.py", line 208, in _post
j = rs.json
File "/usr/local/lib/python2.7/site-packages/py2neo/packages/httpstream/http.py", line 563, in json
return json.loads(self.read().decode(self.encoding))
File "/usr/local/lib/python2.7/site-packages/py2neo/packages/httpstream/http.py", line 634, in read
data = self._response.read()
File "/usr/local/lib/python2.7/httplib.py", line 543, in read
return self._read_chunked(amt)
File "/usr/local/lib/python2.7/httplib.py", line 597, in _read_chunked
raise IncompleteRead(''.join(value))
IncompleteRead: IncompleteRead(128135 bytes read)
This post (IncompleteRead using httplib) suggests that is an httplib error. I am not sure how to handle since I am not calling httplib directly.
Any suggestions for getting this load to work on Linux or what the IncompleteRead error message means?
UPDATE :
The IncompleteRead error is being caused by a Neo4j error being returned. The line returned in _read_chunked that is causing the error is:
pe}"}]}],"errors":[{"code":"Neo.TransientError.Network.UnknownFailure"
Neo4j docs say this is an unknown network error.
Although I can't say for sure, this implies some kind of local network issue between client and server rather than a bug within the library. Py2neo wraps httplib (which is pretty solid itself) and, from the stack trace, it looks as though the client is expecting more chunks from a chunked response.
To diagnose further, you could make some curl calls from your Linux application server to your database server and see what succeeds and what doesn't. If that works, try writing a quick and dirty python script to make the same calls with httplib directly.
UPDATE 1: Given the update above and the fact that the server streams its responses, I'm thinking that the chunk size might represent the intended payload but the error cuts the response short. Recreating the issue with curl certainly seems like the best next step to help determine whether it is a fault in the driver, the server or something else.
UPDATE 2: Looking again this morning, I notice that you're using Python substitution for the properties within the MERGE statement. As good practice, you should use parameter substitution at the Cypher level:
merge_query = "MERGE (a:label {name:{name}, name_var:{name_var}})"
merge_params = {"name": id, "name_var": name}
tx.append(merge_query, merge_params)
Just started learning solr. I am trying to use solrpy as a client. My python code is:
import solr
# create a connection to a solr server
s = solr.SolrConnection('http://localhost:8983/solr')
# add a document to the index
doc = dict(
id='testid123',
title='Lucene in Action',
author=['Erik Hatcher', 'Otis Gospodneti'],
)
s.add(doc, commit=True)
# do a search
response = s.query('title:lucene')
for hit in response.results:
print hit['title']
This is from the example given here
My solr schema.xml is the default schema that comes with solr distribution. I have not made any changes to that. It has a uniqueKey field as "id".
<uniqueKey>id</uniqueKey>
And it is of string type
<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
Still when I run my code, on my client side I get error:
Traceback (most recent call last):
File "/Users/user1/Documents/workspace/PyDelight/src/Test.py", line 12, in <module>
s.add(doc, commit=True)
File "/Library/Python/2.7/site-packages/solrpy-0.9.5-py2.7.egg/solr/core.py", line 678, in add
return Solr.add_many(self, [fields], commit=_commit)
File "/Library/Python/2.7/site-packages/solrpy-0.9.5-py2.7.egg/solr/core.py", line 326, in wrapper
return self._update(content, query)
File "/Library/Python/2.7/site-packages/solrpy-0.9.5-py2.7.egg/solr/core.py", line 550, in _update
rsp = self._post(selector, request, self.xmlheaders)
File "/Library/Python/2.7/site-packages/solrpy-0.9.5-py2.7.egg/solr/core.py", line 639, in _post
return check_response_status(self.conn.getresponse())
File "/Library/Python/2.7/site-packages/solrpy-0.9.5-py2.7.egg/solr/core.py", line 1097, in check_response_status
raise ex
solr.core.SolrException: HTTP code=400, reason=Bad Request
And on the solr trace side I get error:
843169 [qtp1151734776-20] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp=/solr path=/update params={commit=true} {} 0 0
843170 [qtp1151734776-20] ERROR org.apache.solr.core.SolrCore ? org.apache.solr.common.SolrException: Document is missing mandatory uniqueKey field: id
schema.xml file is in solr-4.4.0/example/solr/collection1/conf
And I am running solr by simply running start.jar in example directory.
Any idea where I am going wrong?
i have not used solrpy much (and haven't installed it yet) but from the initial example, it looks like it wants to be called with attribute=value pairs instead of a dictionary. (i know the example you posted is right from the online 0.9.2 documentation! but the current source on github has this in the comments):
add(**params)
Add a document. Pass in all document fields as
keyword parameters:
add(id='foo', notes='bar')
You must "commit" for the addition to be saved.
So try this:
s.add(commit=True, **doc)
and it will probably work. You may need to pull out the commit and do it separately, i don't know.
i am not a solr expert, and just played around with it a little bit, but also i had better luck using sunburnt than solrpy. worth a shot, maybe.
edit: github pointer to that file is here: http://code.google.com/p/solrpy/source/browse/solr/core.py
I haven't used Solr so I could be totally wrong, but in the example you link to the id is an int. Try making yours an int as well, changing your id from 'testid123' to something else like 123 and see what happens.
Trying to work with the echosign SOAP API.
The wsdl is here: https://secure.echosign.com/services/EchoSignDocumentService14?wsdl
When I try to create certain objects, it appears to not be able to find the type, even after listing it in print client
import suds
url = "https://secure.echosign.com/services/EchoSignDocumentService14?wsdl"
client = suds.client.Client(url)
print client
Service ( EchoSignDocumentService14 ) tns="http://api.echosign"
Prefixes (10)
ns0 = "http://api.echosign"
ns1 = "http://dto.api.echosign"
ns2 = "http://dto10.api.echosign"
ns3 = "http://dto11.api.echosign"
ns4 = "http://dto12.api.echosign"
ns5 = "http://dto13.api.echosign"
ns15 = "http://dto14.api.echosign"
ns16 = "http://dto7.api.echosign"
ns17 = "http://dto8.api.echosign"
ns18 = "http://dto9.api.echosign"
Ports (1):
(EchoSignDocumentService14HttpPort)
Methods (45):
...
Types (146):
ns1:CallbackInfo
ns17:WidgetCreationInfo
Trimmed for brevity, but showing the namespaces and the 2 types I'm concerned with right now.
Trying to run WCI = client.factory.create("ns17:WidgetCreationInfo") generates this error:
client.factory.create("ns17:WidgetCreationInfo")
Traceback (most recent call last):
File "", line 1, in
File "build/bdist.macosx-10.7-intel/egg/suds/client.py", line 244, in create
suds.BuildError:
An error occured while building a instance of (ns17:WidgetCreationInfo). As a result
the object you requested could not be constructed. It is recommended
that you construct the type manually using a Suds object.
Please open a ticket with a description of this error.
Reason: Type not found: '(CallbackInfo, http://dto.api.echosign, )'
So it doesn't appear to be able to find the CallbackInfo type. Maybe its because its missing the ns there?
Again, figured it out 15 min after posting here.
suds has an option to cross-pollinate all the namespaces so they all import each others schemas. autoblend can be set in the constructor or using the set_options method.
suds.client.Client(url, autoblend=True)
Take a look in the WSDL, it seems lots of definitions in http://*.api.echosign that suds cannot fetch.
Either update your /etc/hosts to make these not well-formed domains can be reached, or save the wsdl locally, modify it, then use Client('file://...', ...) to create your suds client.
Hey guys, I am a little lost on how to get the auth token. Here is the code I am using on the return from authorizing my app:
client = gdata.service.GDataService()
gdata.alt.appengine.run_on_appengine(client)
sessionToken = gdata.auth.extract_auth_sub_token_from_url(self.request.uri)
client.UpgradeToSessionToken(sessionToken)
logging.info(client.GetAuthSubToken())
what gets logged is "None" so that does seem right :-(
if I use this:
temp = client.upgrade_to_session_token(sessionToken)
logging.info(dump(temp))
I get this:
{'scopes': ['http://www.google.com/calendar/feeds/'], 'auth_header': 'AuthSub token=CNKe7drpFRDzp8uVARjD-s-wAg'}
so I can see that I am getting a AuthSub Token and I guess I could just parse that and grab the token but that doesn't seem like the way things should work.
If I try to use AuthSubTokenInfo I get this:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/__init__.py", line 507, in __call__
handler.get(*groups)
File "controllers/indexController.py", line 47, in get
logging.info(client.AuthSubTokenInfo())
File "/Users/matthusby/Dropbox/appengine/projects/FBCal/gdata/service.py", line 938, in AuthSubTokenInfo
token = self.token_store.find_token(scopes[0])
TypeError: 'NoneType' object is unsubscriptable
so it looks like my token_store is not getting filled in correctly, is that something I should be doing?
Also I am using gdata 2.0.9
Thanks
Matt
To answer my own question:
When you get the Token just call:
client.token_store.add_token(sessionToken)
and App Engine will store it in a new entity type for you. Then when making calls to the calendar service just dont set the authsubtoken as it will take care of that for you also.