Python 3 Get HTTP page - python

How can I get python to get the contents of an HTTP page? So far all I have is the request and I have imported http.client.


Using urllib.request is probably the easiest way to do this:
import urllib.request
f = urllib.request.urlopen("http://stackoverflow.com")
print(f.read())


Usage built-in module "http.client"
import http.client
connection = http.client.HTTPSConnection("api.bitbucket.org", timeout=2)
connection.request('GET', '/2.0/repositories')
response = connection.getresponse()
print('{} {} - a response on a GET request by using "http.client"'.format(response.status, response.reason))
content = response.read().decode('utf-8')
print(content[:100], '...')
Result:
200 OK - a response on a GET request by using "http.client"
{"pagelen": 10, "values": [{"scm": "hg", "website": "", "has_wiki":
true, "name": "tweakmsg", "links ...
Usage third-party library "requests"
response = requests.get("https://api.bitbucket.org/2.0/repositories")
print('{} {} - a response on a GET request by using "requests"'.format(response.status_code, response.reason))
content = response.content.decode('utf-8')
print(content[:100], '...')
Result:
200 OK - a response on a GET request by using "requests"
{"pagelen": 10, "values": [{"scm": "hg", "website": "", "has_wiki":
true, "name": "tweakmsg", "links ...
Usage built-in module "urllib.request"
response = urllib.request.urlopen("https://api.bitbucket.org/2.0/repositories")
print('{} {} - a response on a GET request by using "urllib.request"'.format(response.status, response.reason))
content = response.read().decode('utf-8')
print(content[:100], '...')
Result:
200 OK - a response on a GET request by using "urllib.request"
{"pagelen": 10, "values": [{"scm": "hg", "website": "", "has_wiki":
true, "name": "tweakmsg", "links ...
Notes:
Python 3.4
Result from the responses most likely will be differ only content


You can also use the requests library. I found this particularly useful because it was easier to retrieve and display the HTTP header.
import requests
source = 'http://www.pythonlearn.com/code/intro-short.txt'
r = requests.get(source)
print('Display actual page\n')
for line in r:
print (line.strip())
print('\nDisplay all headers\n')
print(r.headers)


pip install requests
import requests
r = requests.get('https://api.spotify.com/v1/search?type=artist&q=beyonce')
r.json()


Add this code which can format data for human reading:
text = f.read().decode('utf-8')


https://stackoverflow.com/a/41862742/8501970
Check this out instead. Its about the same issue you have and this one is very simple and very few lines of codes.
This sure helped me when i realized python3 cannot use simply get_page.
This is a fine alternative.
(hope this helps, cheers!)

Related

Using Flask in python 2.7 to resolve the CORS issue, I need to display the response in ext.js

This is my implementation,
full_url = url + '?' + params
req = urllib2.Request(full_url, params)
req.add_header('Content-Type', 'application/javascript')
req.add_header('Access-Control-Allow-Origin','*')
req.add_header('Accept', 'application/javascript')
req.add_header('x-lang', 'en')
req.add_header('x-app-version', '1.2.3')
req.add_header('x-consumer-key', 'abc')
req.add_header('x-source', 'web')
req.add_header('x-device-id', 'abc-2501015753736970469271537365900144030')
req.add_header('x-signature', signature)
req.add_header('X-Content-Type-Options', 'nosniff')
req.add_header('x-request-id', request)
req.add_header('x-timestamp', timeStamp)
response = urllib2.urlopen(req)
result = response.read()
result = result.decode('latin-1')
respjson = json.loads(result)
return respjson
Reading the output in ext.js
var script = document.createElement("script");
script.setAttribute("src", url);
script.setAttribute("type", "text/javascript");
script.setAttribute("id", trans.scriptId);
document.getElementsByTagName("head")[0].appendChild(script);
Thank you in advance.
Error shows in the browser "was loaded even though its MIME type (“application/json”) is not a valid JavaScript MIME"
[Error in the browser][1]

Short Answer:
Use JSONP instead of JSON in ExtJS.
Tons of additional information
better ways to retrieve a JSON from ExtJS
But this was not the question I assume.
Let's say you have your JSON and you are trying to populate a store.
Instead of injecting the JSON into the DOM, you could use:
Ext.StoreManager.get('storeId').load(JSON);
// or for the store of a grid
Ext.ComponentQuery.query('grid')[0].getStore().load(JSON);
// or for xtype main
Ext.first('main').getViewModel().set('initData', JSON);
All these methods would get the data into an ExtJS App
Retrieving Data
Getting Data into ExtJS typically would be done using
Ext.Ajax.request({
url: full_url,
header: ...
success: success_fn,
scope: this
})
Or directly in your model/store

Artifactory AQL Query Using Python 3 'Requests'

Would anybody be able to help me identify where I am going wrong with a very basic AQL I am trying to execute using Python 3 and the 'Requests' library?
No matter what I do I cant seem to get past a 400: Bad Request. It is clearly something to do with the formatting of the data I am trying to Post but just can't see it. I'm assuming I want to pass it as a string as when posting sql queries they have to be in plain text.
Just as an aside, I can execute this query absolutely fine using postman etc.
import requests
from requests.auth import HTTPBasicAuth
import json
HEADERS = {'Content-type': 'text'}
DATA = 'items.find({"type":"folder","repo":{"$eq":"BOS-Release-Builds"}}).sort({"$desc":["created"]}).limit(1)'
response = requests.post('https://local-Artifactory/artifactory/api/search/aql', headers=HEADERS, auth=HTTPBasicAuth('user', 'password'), data = DATA)
print (response.status_code)
##print (response.json())
def jprint(obj):
text = json.dumps(obj, sort_keys=True, indent=4)
print (text)
jprint(response.json())
print(response.url)
print(response.encoding)

Ok, after sitting staring at the code for another few minutes I spotted my mistake.
I shouldnt have defined the Header content type in the request and instead let the 'Requests' library deal with this for you.
So the code should look like:
import requests
from requests.auth import HTTPBasicAuth
import json
DATA = 'items.find({"type":"folder","repo":{"$eq":"BOS-Release-Builds"}}).sort({"$desc":["created"]}).limit(1)'
response = requests.post('https://uk-artifactory.flowbird.group/artifactory/api/search/aql', auth=HTTPBasicAuth('user', 'password!'), data = DATA)
print (response.status_code)
##print (response.json())
def jprint(obj):
text = json.dumps(obj, sort_keys=True, indent=4)
print (text)
jprint(response.json())
print(response.url)
print(response.encoding)

Understanding and Interacting with API using Python

I have a basic API installed as my localhost server that does functions such as add camera, star camera, list cameras, snapshot of camera frame, etc.
My problem is after following the documentation I still can't seem to interact with it well and get the response I need. Here is the code I use to log in and get validation token:
import requests
import urllib.request
import json
base_url = "http://localhostip:8080/api/user/login?"
parameters = {
"username": username,
"password": password
}
auth_tok = requests.post(base_url + urllib.parse.urlencode(parameters)).json()
print(auth_tok)
I get the correct documented response with a token, so following the documentation to add camera I need 2 parameters, URL and Name, so I did:
base_url = "http://localhostip:8080/api/camera/add?"
parameters = {
"url": 'rtsp://192.168.1.23/1',
#or video file
"url" : '/home/video/sample.mov'
"name" : 'cam1'
}
r = requests.post(base_url + urllib.parse.urlencode(parameters),headers={'Authorization': auth_tok})
when I print the response:
-print (r)
-print (r.url)
-print(r.status_code)
-print(r.json())
I get this:
<Response [500]>
http://192.168.0.162:8080/service/api/camera/add?url=rtsp%3A%2F%2Frtsp%3A%2F%2F192.168.1.23&name=cam1
500
{'code': -111, 'message': None}
According to documentation the correct url should be like this:
http://192.168.0.6:8080/service/api/camera/add?url=rtsp://192.168.1.23&name=cam1
and the response should be:
Response:
{"status":"ok"}
So why and how to make the URL POST in the correct format, because I suspect this is the issue, the URL has these encoding symbols that may be messing up the request?
When I use the web browser GUI of this API I can add the camera or even a video file to play but I'm trying to do the same with Python so I can do further processing in future.

Your problem is when you encode the ' / / ' symbol, so, in order to fix that, you need to use another function from urllib, urllib.parse.unquote(), and use as parameter your encoding function urllib.parse.urlencode(parameters):
import urllib
parameters = {
"url": 'rtsp://192.168.1.23/1',
"name" : 'cam1'
}
The results are :
print(urllib.parse.urlencode(parameters))
'url=rtsp%3A%2F%2F192.168.1.23%2F1&name=cam1'
print(urllib.parse.unquote(urllib.parse.urlencode(parameters)))
'url=rtsp://192.168.1.23/1&name=cam1'
Source https://docs.python.org/3.0/library/urllib.parse.html#urllib.parse.unquote

Getting Request method 'GET' not supported - Python

So I've been trying to request an API using the following endpoint:
http://viatorapi.viator.com/service/search/products?destId=684&apiKey=98765687*****
Using the following python code:
import requests
import json
resp_1 = requests.get("http://viatorapi.viator.com/service/search/products?destId=684&apiKey=98765687*****")
res = resp_1.json()
print(res)
But I keep getting a Request method 'GET' not supported error even when I try the request directly from the browser.
I've been looking at the documentation for a while now and it's says that It's supposed to be a POST request.
Here: https://docs.viator.com/partner-api/affiliate/technical/#tag/Product-services/paths/~1search~1products/post
Any ideas on why this is happening and how to fix this?
UPDATE
Here's the new code I'm about to try:
import requests
import json
j="""{"destId": 684,"seoId": null,"catId": 3,"subCatId": 5318,"startDate": "2018-10-21","endDate": "2019-10-21","dealsOnly": false,"currencyCode": "EUR","topX": "1-3","sortOrder": "TOP_SELLERS"}"""
resp_1 = requests.post("http://viatorapi.viator.com/service/search/products?apiKey=98765687*****", data=json.loads(j))
res = resp_1.json()
print(res)

According to the documentation you linked,
it is clear that it only takes POST requests for /search/products. Generate a json (like the sample json from the documentation) and do a post request.
import requests
import json
j="""{
"destId": 684,
"seoId": null,
"catId": 3,
"subCatId": 5318,
"startDate": "2018-10-21",
"endDate": "2019-10-21",
"dealsOnly": false,
"currencyCode": "EUR",
"topX": "1-3",
"sortOrder": "TOP_SELLERS"
}"""
headers={'Content-type':'application/json', 'Accept':'application/json'}
resp_1 = requests.post("http://viatorapi.viator.com/service/search/products?destId=684&apiKey=98765687*****", data=j, headers=headers)
print(resp_1.json())

How to trigger authenticated Jenkins job with file parameter using standard Python library

We are currently triggering Jenkins jobs from a Python script with the help of PycURL. We would like, however, to get rid of the PycURL dependency, but have had little success so far. What makes our scenario more complicated is that we need to post a file as a parameter. Our current PycURL logic for posting the request looks as follows:
url = "https://myjenkins/job/myjob/build"
with contextlib.closing(pycurl.Curl()) as curl:
curl.setopt(pycurl.URL, url)
curl.setopt(pycurl.USERPWD, "myuser:mypassword")
curl.setopt(pycurl.SSL_VERIFYPEER, False)
curl.setopt(pycurl.SSL_VERIFYHOST, False)
curl.setopt(pycurl.FAILONERROR, True)
data = [
("name", "integration.xml"),
("file0", (pycurl.FORM_FILE, "integration.xml")),
("json", "{'parameter': [{'name': 'integration.xml', 'file': 'file0'}]}"),
("Submit", "Build"),
]
curl.setopt(pycurl.HTTPPOST, data)
try:
curl.perform()
except pycurl.error, err:
raise JenkinsTriggerError(curl.errstr())
How can we replace this with facilities from the standard Python library?
We've tried before, but had to give up as we could not see how to upload files successfully, as you can see from my question on that issue.

We can do it with the help of requests library only.
import requests
payload = ( ('file0', open("FILE_LOCATION_ON_LOCAL_MACHINE", "rb")),
('json', '{ "parameter": [ {
"name":"FILE_LOCATION_AS_SET_IN_JENKINS",
"file":"file0" }]}' ))
resp = requests.post("JENKINS_URL/job/JOB_NAME/build",
auth=('username','password'),
headers={"Jenkins-Crumb":"9e1cf46405223fb634323442a55f4412"},
files=payload )
Jekins-Crumb if required can be obtained using:
requests.get('http://username:password#JENKINS_URL/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')

I found a solution, using the requests and urllib3 libraries. Not entirely standard, but more lightweight than the PycURL dependency. It should be possible to do this directly with requests (avoiding the urllib3 part), but I ran into a bug.
import urllib3, requests, json
url = "https://myjenkins.com/job/myjob"
params = {"parameter": [
{"name": "integration.xml", "file": "file0"},
]}
with open("integration.xml", "rb") as f:
file_data = f.read()
data, content_type = urllib3.encode_multipart_formdata([
("file0", (f.name, file_data)),
("json", json.dumps(params)),
("Submit", "Build"),
])
resp = requests.post(url, auth=("myuser", "mypassword"), data=data,
headers={"content-type": content_type}, verify=False)
resp.raise_for_status()

If you are familiar with python, then you can use the jenkins REST APT python wrapper provided by the official site. see this link.
Trigger a build is unbelievably easy by using this python wrapper. Here is my example:
#!/usr/bin/python
import jenkins
if __name == "main":
j = jenkins.Jenkins(jenkins_server_url, username="youruserid", password="yourtoken")
j.build_job(yourjobname,{'param1': 'test value 1', 'param2': 'test value 2'},
{'token': "yourtoken"})
For those who don't know where to find the token, here is how:
login to jenkins -> click your userid from the top of the webpage ->
Configure ->Show API Token...
Enjoy it.

Probably it can look something like this:
url = "https://myjenkins/job/myjob/build"
req = urllib2.Request(url)
auth = 'Basic ' + base64.urlsafe_b64encode("myuser:mypassword")
req.add_header('Authorization', auth)
with open("integration.xml", "r") as f:
file0 = f.read()
data = {
"name": "integration.xml",
"file0": file0,
"json": "{'parameter': [{'name': 'integration.xml', 'file': 'file0'}]}",
"Submit": "Build"
}
req.add_data(urllib.urlencode(data))
urllib2.urlopen(req)
Sorry, I don't have installed Jenkins around to test it out.

Another Alternative that I used :
import requests
import json
url = "https://myjenkins/job/myjob/build"
payload = {'key1': 'value1', 'key2': 'value2'}
resp = requests.post(url, params=payload, auth=("username", "password"),verify=False)
json_data = json.loads(resp.text)
For more details you can refer :Make a Request

Here is a similar version to aknuds1 answer where test_result is the xml string:
j_string = "{'parameter': [{'name': 'integration_tests.xml', 'file': 'someFileKey0'}]}"
data = {
"name": "integration_tests.xml",
"json": j_string,
}
for xml_string in tests.values():
post_form_result = requests.post('http://jenkins/job/deployment_tests/build',
data=data,
files={'someFileKey0': xml_string})
print(post_form_result.status_code)
Taking a guess, additional parameters would be passed in as part of the json string array, or additional files, etc. Let me know if this is the case, also, if I find out, i'll update this answer. This solution worked nicely to trigger JUnit tests.
Version:
master* $ pip show requests [21:45:05]
Name: requests
Version: 2.12.3
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: me#kennethreitz.com
License: Apache 2.0
Location: /usr/local/lib/python2.7/site-packages

Categories