I was trying to use http://www.jongsma.org/gc/scripts/ofx-ba.py to grab my bank account information from wachovia. Having no luck, I decided that I would just try to manually construct some request data using this example
So, I have this file that I want to use as the request data. Let's call it req.ofxsgml:
FXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
<OFX>
<SIGNONMSGSRQV1>
<SONRQ>
<DTCLIENT>20071015021529.000[-8:PST]
<USERID>TheNameIuseForOnlineBanking
<USERPASS>MySecretPassword
<LANGUAGE>ENG
<FI>
<ORG>Wachovia
<FID>4309
</FI>
<APPID>Money
<APPVER>1700
</SONRQ>
</SIGNONMSGSRQV1>
<BANKMSGSRQV1>
<STMTTRNRQ>
<TRNUID>438BD6F4-2106-4C88-8DE5-7625915A2FC0
<STMTRQ>
<BANKACCTFROM>
<BANKID>061000227
<ACCTID>101555555555
<ACCTTYPE>CHECKING
</BANKACCTFROM>
<INCTRAN>
<INCLUDE>Y
</INCTRAN>
</STMTRQ>
</STMTTRNRQ>
</BANKMSGSRQV1>
</OFX>
Then, in python, I try:
>>> import urllib2
>>> query = open('req.ofxsgml').read()
>>> request = urllib2.Request('https://pfmpw.wachovia.com/cgi-forte/fortecgi?servicename=ofx&pagename=PFM',
query,
{ "Content-type": "application/x-ofx",
"Accept": "*/*, application/x-ofx"
})
>>> f = urllib2.urlopen(request)
This command gives me a 500 and this traceback. I wonder what is wrong with my request.
Visiting the url with no data and no concern for headers,
>>> f = urllib2.urlopen('https://pfmpw.wachovia.com/cgi-forte/fortecgi?servicename=ofx&pagename=PFM')
yields the same thing as visiting that url directly,
HTTPError: HTTP Error 403: <BODY><H1>Request not allowed</H1></BODY>.
This is pretty obvious but just an observation. Everything on the subject seems to be pretty outdated. Hoping to write a simple python ofx module to open source. Maybe there is already something developed that I have not managed to find?
EDIT -
If I make a flat mapping of the above information:
d = {'ACCTID': '10555555',
'ACCTTYPE': 'CHECKING',
'APPID': 'Money',
'APPVER': '1700',
'BANKID': '061000227',
'DTCLIENT': '20071015021529.000[-8:PST]',
'FID': '4309',
'INCLUDE': 'Y',
'LANGUAGE': 'ENG',
'ORG': 'Wachovia',
'TRNUID': 'I18BD6F4-2006-4C88-8DE5-7625915A2FC0',
'USERID': 'm48m40',
'USERPASS': '12397'}
and then urlencode it and make the request with that as the data
query=urllib.urlencode(d)
request = urllib2.Request('https://pfmpw.wachovia.com/cgi-forte/fortecgi?servicename=ofx&pagename=PFM',
query,
{ "Content-type": "application/x-ofx",
"Accept": "*/*, application/x-ofx"
})
f = urllib2.urlopen(request)
HTTP Error 403: <BODY><H1>Request not allowed</H1></BODY>
The problem was that you were previously passing in the data from your file directly as the data parameter to the Request. The file you were reading in contains both the headers and the data that you should be sending. You needed to supply the headers and the data separately as you have now done.
HTTP error 403 means the request was correct but the server is refusing to respond to it. Have you already signed up and arranged permission to use the web service you are trying to access? If so is there some authentication that you need to do before making the request?
could just be authentication? (or lack therof?)
Related
If I make the request
api-key = 'asdfklhsdfkjahsdlgkjahlkdjahfsa'
url = 'https://www.website.com'
headers = {'api-key': api-key,
'Content-Type': 'application/json'}
request_data = {'foo': 'bar', 'egg': 'spam'}
result = requests.post(url, headers=headers, data=request_data)
The server is contacted. Suppose that instead I want to do something like
request_string = requests.foobar(url, headers=headers, data=request_data)
import os
os.system('curl ' + request_string)
So that I can look to see what the request is doing without bothering the server (possibly to the point that I could c&p it into curl), what would foobar be? Or in general, what is a way to inspect the contents of the request without making it?
Here's another post that implies that you can use Request().prepare() to observe the request without actually sending the request.
Furthermore the official documentation reads "In some cases you may wish to do some extra work to the body or headers (or anything else really) before sending a request. The simple recipe for this is the following" and then it illustrates Request.prepare()
I am planning to send request to server with the following code. I have spent more than 1 day to resolve this but without any progress. And please forgive me I have to hide the real URL address becuase of Company Security Policy.
import requests
get_ci = requests.session()
get_ci_url = 'https://this_is_a_fake_URL_to_paste_in_stackoverflow.JSON'
get_ci_param_dict = {"Username": "fake","Password": "fakefakefake","CIType": "system","CIID": "sampleid","CIName": "","AttrFilter": "","SubObjFilter": ""}
get_ci_param_str = str(get_ci_param_dict)
print(get_ci_param_dict)
print(get_ci_param_str)
get_ci_result = get_ci.request('POST', url=get_ci_url, params=get_ci_param_str, verify=False)
print(get_ci_result.status_code)
print(get_ci_result.text)
And what I get in the Run result is,
C:\Python34\python.exe C:/Users/this/is/the/fake/path/Test_02.py
{'CIID': 'sampleid', 'CIType': 'system', 'AttrFilter': '', 'Password': 'fake', 'CIName': '', 'Username': 'fake', 'SubObjFilter': ''}
{'CIID': 'sampleid', 'CIType': 'system', 'AttrFilter': '', 'Password': 'fake', 'CIName': '', 'Username': 'fake', 'SubObjFilter': ''}
C:\Python34\lib\requests\packages\urllib3\connectionpool.py:843: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
500
<ns1:XMLFault xmlns:ns1="http://cxf.apache.org/bindings/xformat"><ns1:faultstring xmlns:ns1="http://cxf.apache.org/bindings/xformat">*org.codehaus.jettison.json.JSONException: A JSONObject text must begin with '{' at character 0 of* </ns1:faultstring></ns1:XMLFault>
Process finished with exit code 0
More tips,
I have contact the Server Code Developer - They only need is a
string in a JSON format sent in a "Parameter" way. Which means it
is correct to use params in request().
I have tried with dumps.json(get_ci_param_dict) => The same result.
It has returned the 200 code when I only request the server's root,
which proves me the url is ok.
Additional Logs when update the params to data.
C:\Python34\lib\requests\packages\urllib3\connectionpool.py:843: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
500
<html><head><title>Apache Tomcat/7.0.61 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - 1</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>1</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>java.lang.ArrayIndexOutOfBoundsException: 1
com.fake.security.XSSHttpReuquestWrapper.GeneralParameters(XSSHttpReuquestWrapper.java:158)
com.fake.security.XSSHttpReuquestWrapper.checkParameter(XSSHttpReuquestWrapper.java:101)
com.fake.security.XSSHttpReuquestWrapper.validateParameter(XSSHttpReuquestWrapper.java:142)
com.fake.security.XSSSecurityFilter.doFilter(XSSSecurityFilter.java:35)
com.fake.webservice.interceptor.GetContextFilter.doFilter(GetContextFilter.java:24)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.61 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.61</h3></body></html>
Process finished with exit code 0
FINAL SOLUTION FOR THIS QUESTION
#e4c5 your suggestion is helpful to figure out the final solution. The param should be sent to server via data, as the data should be sent out as dict or byte as defined in the offical document, so need use param as the dict key to send. Please see the code as below,
import requests
import json
get_ci_url = 'https://sample.fake.com:0000/sample/fake/fakeagain.JSON'
get_ci_param_dict = {"Username": "fake","Password": "fakefake".......}
get_ci_param_json = json.dumps(get_ci_param_dict)
params = {'param': get_ci_param_json}
get_ci_result = requests.request('POST', url=get_ci_url, data=params, verify=False)
print(get_ci_result.status_code)
print(get_ci_result.text)
ROOT CAUSE:param should be sent via data parameter. Official Document has clearly state that => :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:Request.
Thanks My colleauge - Mr.J and #e4c5's great help.
if what your server expects is json, you should use the json parameter to python requests
get_ci_result = get_ci.request('POST', url=get_ci_url,
json=get_ci_param_dict, verify=False)
also note that the params parameter is usually used with get (and is used to format the query string of a URL), with post and form data it should be data and a dictionary again.
For additional information please refer to : http://docs.python-requests.org/en/master/api/
Your dictionary of data will automatically be form-encoded when the request is made. Use json parameter when the server accepts JSON-Encoded POST/PATCH data instead of form-encoded data.
response = requests.post(url=url, headers=headers, json=data)
Using the json parameter in the request will change the Content-Type in the header to application/json.
Visit https://2.python-requests.org/en/master/user/quickstart/#More-complicated-POST-requests
I also had a similar problem.
Even though my data was already a dictionary I needed to json.dumps(data) again:
response = requests.post(url = url ,headers=head,data = json.dumps(data))
I just wanted to share this in case somebody has a similar problem.
I'm trying to create a super-simplistic Virtual In / Out Board using wx/Python. I've got the following code in place for one of my requests to the server where I'll be storing the data:
data = urllib.urlencode({'q': 'Status'})
u = urllib2.urlopen('http://myserver/inout-tracker', data)
for line in u.readlines():
print line
Nothing special going on there. The problem I'm having is that, based on how I read the docs, this should perform a Post Request because I've provided the data parameter and that's not happening. I have this code in the index for that url:
if (!isset($_POST['q'])) { die ('No action specified'); }
echo $_POST['q'];
And every time I run my Python App I get the 'No action specified' text printed to my console. I'm going to try to implement it using the Request Objects as I've seen a few demos that include those, but I'm wondering if anyone can help me explain why I don't get a Post Request with this code. Thanks!
-- EDITED --
This code does work and Posts to my web page properly:
data = urllib.urlencode({'q': 'Status'})
h = httplib.HTTPConnection('myserver:8080')
headers = {"Content-type": "application/x-www-form-urlencoded",
"Accept": "text/plain"}
h.request('POST', '/inout-tracker/index.php', data, headers)
r = h.getresponse()
print r.read()
I am still unsure why the urllib2 library doesn't Post when I provide the data parameter - to me the docs indicate that it should.
u = urllib2.urlopen('http://myserver/inout-tracker', data)
h.request('POST', '/inout-tracker/index.php', data, headers)
Using the path /inout-tracker without a trailing / doesn't fetch index.php. Instead the server will issue a 302 redirect to the version with the trailing /.
Doing a 302 will typically cause clients to convert a POST to a GET request.
I have some python tools that I would like to have send updates to a hipchat room. I do this elsewhere with shell scripts, so I know it works in our environment, but I can't seem to get the token pushed to the hipchat API. Gotta be something simple.
First, this authenticates properly and delivers a message:
curl -d "room_id=xxx&from=DummyFrom&message=ThisIsATest&color=green" https://api.hipchat.com/v1/rooms/message?auth_token=yyy
But when I try to use the python "requests" module, I am getting stuck.
import requests
room_id_real="xxx"
auth_token_real="yyy"
payload={"room_id":room_id_real,"from":"DummyFrom","message":"ThisIsATest","color":"green"}
headerdata={"auth_token":auth_token_real,"format":"json"}
r=requests.post("https://api.hipchat.com/v1/rooms/message", params=payload, headers=headerdata)
print r.ok, r.status_code, r.text
Here is my error information:
False 401 {"error":{"code":401,"type":"Unauthorized","message":"Auth token not found. Please see: https:\/\/www.hipchat.com\/docs\/api\/auth"}}
Basically I don't seem to be passing the authentication token in properly. How can I get this working?
In case it helps, here's a working V2 API example. I did find the V2 API to be a bit more sensitive about getting the form of the request exactly right. But, it might be more forward-looking to conform to the V2 API (though the original question seemed to pertain to V1).
#!/usr/bin/env python
import json
from urllib2 import Request, urlopen
V2TOKEN = '--V2 API token goes here--'
ROOMID = --room-id-nr-goes-here--
# API V2, send message to room:
url = 'https://api.hipchat.com/v2/room/%d/notification' % ROOMID
message = "It's a<br><em>trap!</em>"
headers = {
"content-type": "application/json",
"authorization": "Bearer %s" % V2TOKEN}
datastr = json.dumps({
'message': message,
'color': 'yellow',
'message_format': 'html',
'notify': False})
request = Request(url, headers=headers, data=datastr)
uo = urlopen(request)
rawresponse = ''.join(uo)
uo.close()
assert uo.code == 204
Another basic example using requests:
import requests, json
amessage = 'Hello World!'
room = 'https://api.hipchat.com/v2/room/18REPLACE35/notification'
headers = {'Authorization':'Bearer UGetYourOwnAuthKey', 'Content-type':'application/json'}
requests.post(url = room, data = json.dumps({'message':amessage}), headers = headers)
As Ianzz said, try including it in the URL query string. Although clunky (you probably want to hash it!), it definitely works.
The other strange quirk is the tokens that you get through Hipchat; I had no end of problems earlier this evening using my own personal token; it seemed to correspond to v2 beta of the API. If you go in through Group Admin and get a token from there, it may help.
Old question is old.
Here's an official list of libs which use the HipChat API v2 interface
https://www.hipchat.com/docs/apiv2/libraries
I'm trying to make a RESTful api call in python/django with requests.post
I can get requests.get(url=url, auth=auth) to work. Similar call in the same api family for this company
I'm trying to do:
data = {'start': 13388, 'end': 133885, 'name': 'marcus0.5'}
r = requests.post(url=url, auth=auth, headers={'Accept': 'application/json'}, data=data)
and I get the following error:
>>> r.text
u'{"status":"error","errorCode":"COMMON_UNSUPPORTED_MEDIA_TYPE","incidentId":"czEtbXNyZXBvcnRzMDQuc3RhZ2V4dHJhbmV0LmFrYW1haS5jb20jMTM3NTgxMzc3MTk4NQ==","errorMessage":"The server is refusing to service the request because the entity of the request is in a format not supported by the requested resource for the requested method.. Content type \'application/x-www-form-urlencoded;charset=UTF-8\' not supported."}'
I think it has something to do with the json, but I'm not sure what and I'm not sure how to fix it. Any ideas?
Extra info [not sure if it applies]:
I imported
import requests, django
I know the the auth is correct and I tested it with the get method
You want to set the Content-Type parameter of your request to 'application/json', not the Accept parameter.
Taken from w3.org:
The Accept request-header field can be used to specify certain media types which are acceptable for the response.
Try this instead:
import json
data = {'start': 13388, 'end': 133885, 'name': 'marcus0.5'}
r = requests.post(url=url, auth=auth, data=json.dumps(data),
headers={'content-type': 'application/json'})
EDIT:
There is a little bit of confusion (for me as well) about when to send data as a dict or a json encoded string (ie. the result of json.dumps). There is an excellent post here that explains the problem. For a brief summary send a dict when the API requires form-encoded data, and a json encoded string when it requires json-encoded data.