Hello!
My problem is the python sending to a controller which works with a prepared html sending script. Here the problem is that upload does not succeed (doesn't even start uploading) although the script runs through. The file is a binary container file. The problem should be in the code, because other way the upload could be completed
Here the output:
09:54:40:11 ...STEP: Upload Firmware
09:54:49:63 ...Upload was successful!
09:54:49:64 ...POST resource
09:54:50:60 ...Response: {"uploadFirmwareAck":0}
So the upload "was done" it says within 9 sec but it should take about 5 minutes. With debugger I monitored that it did not start just jump over it and give the "Upload was successful" message. I have no clue why. Any ideas?
The code:
import pycurl
from cStringIO import StringIO
import urllib2
import simplejson as json
url = 'http://eData/pvi?rName=FirmwareUpload'
req = urllib2.Request(url)
req.add_header('Content-Type','application/json')
c = pycurl.Curl()
c.setopt(c.POST, 1)
c.setopt(c.URL, url)
c.setopt(c.CONNECTTIMEOUT,0)
c.setopt(c.TIMEOUT, 0)
c.setopt(pycurl.FOLLOWLOCATION, 1)
c.setopt(pycurl.MAXREDIRS, 5)
c.setopt(pycurl.NOSIGNAL, 1)
c.setopt(c.HTTPPOST, [("file1", (c.FORM_FILE, "c:\\Users\\dem2bp\\Desktop\\HMI_Firmware update materials\\output_38.efc"))])
c.perform()
print "Upload was successful!"
print "Tx JSON:"
print "POST resource"
res = urllib2.urlopen(req)
print "Response:"
str_0 = res.read()
print str_0
c.close()
From the documentation:
PycURL is targeted at an advanced developer - if you need dozens of
concurrent, fast and reliable connections or any of the sophisticated
features listed above then PycURL is for you.
I would give http://www.python-requests.org/en/latest/ a try. For me, it's always the first choice when doing some http stuff. Usually it just does what it' supposed to do in a few lines of code.
Thank you, but as I see this request library does not run with an old version of Python like 2.6. I think it would be too risky to upgrade. Do you have other idea?
When I import requests library at some point of library which requires later versions throw me synthax errors.
Related
I'm currently trying to send a HTTP POST via requests library in Python to a remote server to validate a serial number. I've already accomplished this with the following script:
validate-request.py
#!/usr/bin/env python
import requests
print "Content-type: text/html\r\n"
print "\r\n"
r = requests.post("https://xx.xx.xx.xx/cgi-bin/verify.py", data={'serial':'XXXXXXX')}, verify=True)
This already works great. When the serial number is validated, I would really like it to respond by offering a download and/or possibly text accompanied with it. I'm currently only able to validate the serial (the actual script is much longer as it fetches it from MySQL, so this is a simpler version.)
validate.py
#!/usr/bin/python
import cgi
form = cgi.FieldStorage()
serial = form.getvalue('serial')
print "Content-type:text/html\r\n\r\n"
print "<b>SERIAL OK</b></br>"
if serial == "XXXXXXX":
print "Content-type:text/html\r\n\r\n"
print "SERIAL <b>OK</b></br>"
else:
print "Content-type:text/html\r\n\r\n"
print "SERIAL <b>NOT OK</b></br>"
What I would also like to include is returning some extra information (text, mainly) and possibly offering a file to download.
This means, on the validate-request.py, I would have to play with the r.text it returns. Is there perhaps a way to scan the return text to find what I'm looking for? Also, how would I go about offering the file from validate.py and how would I go about accepting it from validate-request.py?
I am trying to use Python to get a JSON file from the Web. If I open the URL in my browser (Mozilla or Chromium) I do see the JSON. But when I do the following with the Python:
response = urllib2.urlopen(url)
data = json.loads(response.read())
I get an error message that tells me the following (after translation in English): Errno 10060, a connection troughs an error, since the server after a certain time period did not react, or the connection was erroneous, or the host did not react.
ADDED
It looks like there are many people who faced the described problem. There are also some answers to the similar (or the same) question. For example here we can see the following solution:
import requests
r = requests.get("http://www.google.com", proxies={"http": "http://61.233.25.166:80"})
print(r.text)
It is already a step forward for me (I think that it is very likely that the proxy is the reason of the problem). However, I still did not get it done since I do not know URL of my proxy and I probably will need user name and password. Howe can I find them? How did it happen that my browsers have them I do not?
ADDED 2
I think I am now one step further. I have used this site to find out what my proxy is: http://www.whatismyproxy.com/
Then I have used the following code:
proxies = {'http':'my_proxy.blabla.com/'}
r = requests.get(url, proxies = proxies)
print r
As a result I get
<Response [404]>
Looks not so good, but at least I think that my proxy is correct, because when I randomly change the address of the proxy I get another error:
Cannot connect to proxy
So, I can connect to proxy but something is not found.
I think there might be something wrong, when you're trying to get the json from the online source(URL). Just to make things clear, here is a small code snippet
#!/usr/bin/env python
try:
# For Python 3+
from urllib.request import urlopen
except ImportError:
# For Python 2
from urllib2 import urlopen
import json
def get_jsonparsed_data(url):
response = urlopen(url)
data = str(response.read())
return json.loads(data)
If you still get a connection error, You can try a couple of steps:
Try to urlopen() a random site from the Interpreter (Interactive Mode). If you are able to grab the source code you're good. If not check internet conditions or try the request module. Check here
Check and see if the json in the URL is in the correct syntax. For sample json syntax check here
Try the simplejson module.
Edit 1:
if you want to access websites using a system wide proxy you will have to use a proxy handler to use loopback(local host) to connect to that proxy.. A sample code is shown below.
proxy = urllib2.ProxyHandler({
'http': '127.0.0.1',
'https': '127.0.0.1'
})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
# this way you can send both http and https request using proxies
urllib2.urlopen('http://www.google.com')
urllib2.urlopen('https://www.google.com')
I have not not worked a lot with ProxyHandler. I just know the theory and code. I am sure there are better ways to access websites through proxies; One which does not involve installing the opener everytime you run the program. But hopefully it will point you in the right direction.
I need some idea to test the server from a link. I do not know where to start
Would be:
site = 'example.com'
if(site === Apache)
print '[ok] Apache - Version:'
else
print '[No] Is not apache'
I prefer using requests since it's simple and well documented. And it doesn't return an error like urllib
import requests
request = requests.get("http://stackoverflow.com/")
if "Apache" in request.headers['server']:
print "Apache Server found"
else:
print "This is no Apache Server"
Also see : http://www.python-requests.org/en/latest/ for more information
In python 3:
import urllib.request
response = urllib.request.urlopen('http://www.google.com')
print(response.headers['Server'])
would be the simplest way to get the server header in some cases.
Some sites (like stackoverflow), however, will return 403 error code.
So every second I am making a bunch of requests to website X every second, as of now with the standard urllib packages like so (the requestreturns a json):
import urllib.request
import threading, time
def makerequests():
request = urllib.request.Request('http://www.X.com/Y')
while True:
time.sleep(0.2)
response = urllib.request.urlopen(request)
data = json.loads(response.read().decode('utf-8'))
for i in range(4):
t = threading.Thread(target=makerequests)
t.start()
However because I'm making so much requests after about 500 requests the website returns HTTPError 429: Too manyrequests. I was thinking it might help if I re-use the initial TCP connection, however I noticed it was not possible to do this with the urllib packages.
So I did some googling and discovered that the following packages might help:
Requests
http.client
socket ?
So I have a question: which one is best suited for my situation and can someone show an example of either one of them (for Python 3)?
requests handles keep alive automatically if you use a session. This might not actually help you if the server is rate limiting requests, however, requests also handles parsing JSON so that's a good reason to use it. Here's an example:
import requests
s = requests.Session()
while True:
time.sleep(0.2)
response = s.get('http://www.X.com/y')
data = response.json()
I need a tool, which can download some part of data from web server, and after that i want connection not be closed. Therfore, i thought about a script in python, which can do:
1) send request
2) read some part of response
3) will freeze - server should think that connection exist, and should not close it
is it possilbe to do it in python ? here is my code:
conn = HTTPConnection("myhost", 10000)
conn.request("GET", "/?file=myfile")
r1 = conn.getresponse()
print r1.status, r1.reason
data = r1.read(2000000)
print len(data)
When im running it, all data is received, and after that server closes connection.
thx in advance for any help
httplib doesn't support that. Use another library, like httplib2. Here's example.