I've a trouble with the HTTP headers that the module Requests returns.
I'm using the following code :
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
response = requests.get("http://www.google.co.il",proxies={'http': '','https':''})
data = response.text
# response.text returns the appropriate html code
# (<!doctype html><html dir="rtl" itemscope=""....)
if response.status_code == requests.codes.ok:
# How do I send those headers to the conn (browser)
print "HEADERS: " + str(response.headers)
conn.send(data)
I'm trying to send a GET request to www.google.co.il, and send the response to the browser (on the example I called it "conn"). The problem is that the browser won't show the received HTML code and instead I'm receiving ERR_EMPTY_RESPONSE.
The headers in the response are :
HEADERS: {'Content-Length': '5451', 'X-XSS-Protection': '1; mode=block', 'Content-Encoding': 'gzip', 'Set-Cookie': 'NID=103=RJzu4RTCNxkh-75dvKBHx-_jen9M8iPes_AdOIQqzBVZ0VPTz1PlQaAVLpwYOmxZlTKmcogiDb1VoY__Es0HqSNwlkmHl3SuBZC8_8XUfqh1PzdWTjrXRnB4S738M1lm; expires=Wed, 08-Nov-2017 10:05:46 GMT; path=/; domain=.google.co.il; HttpOnly', 'Expires': '-1', 'Server': 'gws', 'Cache-Control': 'private, max-age=0', 'Date': 'Tue, 09 May 2017 10:05:46 GMT', 'P3P': 'CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."', 'Content-Type': 'text/html; charset=windows-1255', 'X-Frame-Options': 'SAMEORIGIN'}
Someone told me that the problem is that I'm not sending any header to the browser. Is this really the problem ? Any other suggestions ? and if it is the problem, how do I send the appropriate headers to the browser ?
Edit: I forgot to mention that the connection is
through a Proxy server.
Any help would be great!
Thanks alot, Yahli.
I couldn't find anything about geting the raw http response ( not response.raw ) in requests documentation so i wrote a function :
def http_response(response):
return 'HTTP/1.1 {} {}\r\n{}\r\n\r\n{}'.format(
response.status_code, response.reason ,
'\r\n'.join(k + ': ' + v for k, v in response.headers.items()),
response.content
)
I tested it by setting Firefox HTTP proxy to localhost:port ( with a listening socket on port ) , and it works fine .
Alternatively you can get the host from conn.recv , open a new socket to that host, and send the data . Example :
data = conn.recv(1024)
host = [ l.split(':')[1].strip() for l in data.splitlines() if l.startswith('Host:') ]
if len(host) :
cli = socket.socket()
cli.connect((host[0], 80))
cli.send(data)
response = ''
while True :
data = cli.recv(1024)
if not data.strip() :
break
response += data
conn.send(response)
cli.close()
Where conn is the connection to the web browser . This is just a quick example , assuming you have only HTTP requests ( port 80 ) . There is room for much optimization
Related
When I send a message to a queue I want the response returned into an object which I can then include in my log or not. However for some reason when I execute the following code:
from azure.storage.queue import QueueClient, TextBase64EncodePolicy
# ... some code running ##########################
queue = QueueClient.from_connection_string(conn_str=conn_queue, queue_name="items",
message_encode_policy=TextBase64EncodePolicy())
# ... some message generated #####################
response=queue.send_message(json.dumps(item_dict))
a complete message is printed to my log. It looks for example like this:
Request URL: 'https://{some_storage_account}.queue.core.windows.net/items/messages'
Request method: 'POST'
Request headers: 'Accept': 'application/xml'
'Content-Type': 'application/xml; charset=utf-8'
'x-ms-version': 'REDACTED'
'Content-Length': '1295'
'x-ms-date': 'REDACTED'
'x-ms-client-request-id': '3452464c-06b2-11eb-9f96-00155d6ebdc5'
'User-Agent': 'azsdk-python-storage-queue/12.1.3 Python/3.8.5 (Linux-4.19.104-microsoft-standard-x86_64-with-glibc2.2.5)'
'Authorization': 'REDACTED'
A body is sent with the request
Response status: 201
Response headers:
'Transfer-Encoding': 'chunked'
'Content-Type': 'application/xml'
'Server': 'Windows-Azure-Queue/1.0 Microsoft-HTTPAPI/2.0'
'x-ms-request-id': '1720a5da-c003-002b-18be-9ab4d4000000'
'x-ms-version': 'REDACTED'
'Date': 'Mon, 05 Oct 2020 02:26:41 GMT'
How can I prevent this gulp of information to be printed to my log?
I have spent at least half an hour on the docs from microsoft. But I can't find where I can turn this behaviour off.
logger = logging.getLogger("azure.core.pipeline.policies.http_logging_policy")
logger.setLevel(logging.WARNING)
This Worked For Me - Add this to your code
I tried with the logging-levels - it does not do the trick for me. However when I define a new logger for this connection then it works:
queue_logger = logging.getLogger("logger_name")
# queue_logger.disabled = True
queue_logger.setLevel(logging.DEBUG)
queue = QueueClient.from_connection_string(conn_str=conn_queue, queue_name="debug",message_encode_policy=TextBase64EncodePolicy(), logger=queue_logger)
Ultimately I feel like I should use the REST API instead of the Python Azure SDK for this. REST API allows me to log-output based on the response status. For some weird reason the SDK does not offer me this possibility.
The answer should be here, in the README of azure-storage-python in GitHub:
Here is how we use the logging levels, it is recommended to use INFO:
DEBUG: log strings to sign
INFO: log outgoing requests and responses, as well as retry attempts # <--
WARNING: not used
ERROR: log calls that still failed after all the retries
This should do it, as recommended by Kalies LAMIRI (I didn't try myself):
logger = logging.getLogger("azure.storage")
logger.setLevel(logging.ERROR)
I'm following these instructions to use the layout form recognizer service from Azure
Which have the following code:
########### Python Form Recognizer Async Layout #############
import json
import time
from requests import get, post
# Endpoint URL
endpoint = r"<Endpoint>"
apim_key = "<Subscription Key>"
post_url = endpoint + "/formrecognizer/v2.0-preview/Layout/analyze"
source = r"<path to your form>"
headers = {
# Request headers
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': apim_key,
}
with open(source, "rb") as f:
data_bytes = f.read()
try:
resp = post(url = post_url, data = data_bytes, headers = headers)
if resp.status_code != 202:
print("POST analyze failed:\n%s" % resp.text)
quit()
print("POST analyze succeeded:\n%s" % resp.headers)
get_url = resp.headers["operation-location"]
except Exception as e:
print("POST analyze failed:\n%s" % str(e))
quit()
I tried the code I got the following error:
POST analyze failed:
{"error":{"code":"FailedToDownloadImage","message":"Failed to download image from input URL."}}
POST analyze succeeded:
{'Transfer-Encoding': 'chunked', 'Content-Type': 'application/json; charset=utf-8', 'x-envoy-upstream-service-time': '4', 'apim-request-id': '515e93ee-4db8-4174-92b1-63e5c415c056', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'x-content-type-options': 'nosniff', 'Date': 'Sat, 06 Jun 2020 20:47:28 GMT'}
POST analyze failed:
'operation-location'
The code I'm using is:
import json
import time
from requests import get, post
I'm reading the pdf file before making the request and verifying it loaded into the variable
source = r"data/Invoice_7.pdf"
with open(source, "rb") as f:
data_bytes = f.read()
print (data_bytes[0:10])
Then the request details:
endpoint = r"https://xxxx.cognitiveservices.azure.com/"
apim_key = "xxxx"
post_url = endpoint + "/formrecognizer/v2.0-preview/Layout/analyze"
headers = {
# Request headers
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': apim_key,
}
And finally making the request:
try:
resp = post(url = post_url, data = data_bytes, headers = headers)
print (1)
if resp.status_code != 202:
print("POST analyze failed:\n%s" % resp.text)
#quit()
print (2)
print("POST analyze succeeded:\n%s" % resp.headers)
print (3)
get_url = resp.headers["operation-location"]
print (4)
except Exception as e:
print("POST analyze failed:\n%s" % str(e))
#quit()
I'm printing a number at each step because I find very weird that I get both fail and successful requests responses. This is the result:
1
POST analyze failed:
{"error":{"code":"FailedToDownloadImage","message":"Failed to download image from input URL."}}
2
POST analyze succeeded:
{'Transfer-Encoding': 'chunked', 'Content-Type': 'application/json; charset=utf-8', 'x-envoy-upstream-service-time': '1', 'apim-request-id': '93a2a162-d14f-496f-ba8a-077bcfd5d3c7', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'x-content-type-options': 'nosniff', 'Date': 'Sat, 06 Jun 2020 21:00:20 GMT'}
3
POST analyze failed:
'operation-location'
So the code fails at this line:
get_url = resp.headers["operation-location"]
the text in the response variable is:
'{"error":{"code":"FailedToDownloadImage","message":"Failed to download image from input URL."}}'
As defined in the REST API documentation, you need to specify the Content-Type. When you set your Content-Type to application/json, you need to provide a public accessible source via JSON. In your case, you need to set the Content-Type to application/pdf. When you want to make this dynamic, you could make use of the PyPi package filetype.
By the way, did you know that there is a (beta) Python SDK for Form Recognizer, which you can use for your use-case.
I achieve to list and create products through Prestashop API. I want to automate a little bit the product update process in my website.
But i have an issue trying to upload the images both in create new product with image and in upload an image to a product that i create through the webservice.
I don´t see any error in my code, so i want to know if I made wrong using the Prestashop API.
My code:
def addNewImage(product_id):
file = 'foto.png'
fd = io.open(file, "rb")
data = fd.read()
r = requests.post(urlimg + product_id, data=data,auth=(key,""), headers={'Content-Type': 'multipart/form-data'})
print(r.status_code)
print(r.headers)
print(r.content)
Prints:
500
{'Server': 'nginx', 'Date': 'Fri, 31 May 2019 09:18:27 GMT', 'Content-Type': 'text/xml;charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Access-Time': '1559294307', 'X-Powered-By': 'PrestaShop Webservice', 'PSWS-Version': '1.7.5.2', 'Execution-Time': '0.003', 'Set-Cookie': 'PrestaShop-30ff13c7718a401c862ad41ea4c0505f=def50200b7a8c608f3053d32136569a34c897c09cea1230b5f8a0aee977e6caac3e22bea39c63c30bfc955fe344d2cbabf640dc75039c63b33c88c5f33e6b01f2b282047bfb0e05c8f8eb7af08f2cc5b0c906d2060f92fea65f73ce063bf6d87bd8ac4d03d1f9fc0d7b6bf56b1eb152575ef559d95f89fc4f0090124630ae292633b4e08cfee38cee533eb8abe151a7d9c47ed84366a5dd0e241242b809300f84b9bb2; expires=Thu, 20-Jun-2019 09:18:27 GMT; Max-Age=1728000; path=/; domain=example.com; secure; HttpOnly', 'Vary': 'Authorization', 'MS-Author-Via': 'DAV'}
b'<?xml version="1.0" encoding="UTF-8"?>
\n<prestashop xmlns:xlink="http://www.w3.org/1999/xlink">
\n<errors>
\n<error>
\n<code><![CDATA[66]]></code>
\n<message><![CDATA[Unable to save this image]]></message>
\n</error>
\n</errors>
\n</prestashop>\n'
I probe to use the logging library of python but only tell me this:
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): midominio:443
DEBUG:urllib3.connectionpool:https://midominio:443 "POST /api/images/products/20 HTTP/1.1" 500 None
Also I probe to change the file config/defines.inc.php, that i read in the forum of prestashop to active debug mode but any difference.
Also I probe the library prestapyt( and prestapyt3) but don´t work with python 3 and I read that are not compatible with presta 1.7
Edit:
Display_errors, and log_errors are activated in my Plesk Panel:
But when i go to var/www/vhosts/midominio/logs/error_log
I can´t see any error referenced to php or 500 error in any line.
Thanks in advance for any suggestion...
Edit: I probe the suggestion in response, but return same error.
I think the problem is in the post command if all else if working fine on the backend. data is used to send form data and other text data. To upload a file, you should do it like this:
files = {'media': open('test.jpg', 'rb')}
requests.post(url, files=files)
So your code translates to:
def addNewImage(product_id):
file = 'foto.png'
fd = io.open(file, "rb")
r = requests.post(urlimg + product_id, auth=(key,""), headers={'Content-Type': 'multipart/form-data'}, files={ 'media' : fd })
print(r.status_code)
print(r.headers)
print(r.content)
I'm trying to log some connection info in the event that an error occurs. Using httplib's HTTPConnection I can print out the request headers by setting the debug level to 1:
connection = httplib.HTTPConnection('www.example.com')
connection.set_debuglevel(1)
However, this just seems to print directly to the shell without conditions. I need to be able to get this info as a string or something to store in a variable such that I only print it out when an Exception is thrown.
The specific info that I want is the request headers that the library is generating.
I would use requests HTTP library.
To get response headers, you just need this little piece of code:
import requests
try:
r = requests.get("http://www.example.com")
# Raise an exception in case of "bad"
# status code (non-200 response)
r.raise_for_status()
print r.headers
except Exception as e:
print e.message
Output:
{'connection': 'close',
'content-encoding': 'gzip',
'content-length': '1162',
'content-type': 'text/html; charset=UTF-8',
'date': 'Sun, 12 Aug 2012 12:49:44 GMT',
'last-modified': 'Wed, 09 Feb 2011 17:13:15 GMT',
'server': 'Apache/2.2.3 (CentOS)',
'vary': 'Accept-Encoding'}
Turns out the trick is to redirect sys.stdout to a StringIO object, the contents of which can then either be written to file or what ever you like as you can get to the String. Check out StringIO.StringIO.
I made a request to authenticate server side into Facebook for my Django application.
def authenticateViaFacebook(request):
'''
Redirects users to a page that allows for Facebook login.
'''
consumer = oauth2.Consumer(
key = settings.FACEBOOK_APP_ID,
secret = settings.FACEBOOK_APP_SECRET)
# Request token URL for Facebook.
request_token_url = "https://www.facebook.com/dialog/oauth/"
# Create client.
client = oauth2.Client(consumer)
# Other arguments to Facebook.
otherArguments = 'client_id=' + settings.FACEBOOK_APP_ID
otherArguments += '&scope=email'
otherArguments += '&redirect_uri=' + settings.SITE_URL
otherArguments += 'authenticationRedirect/facebook'
otherArguments += '&secret=' + settings.FACEBOOK_APP_SECRET
# The OAuth Client request works just like httplib2 for the most part.
res, content = client.request(request_token_url, "GET", otherArguments)
# Examine the response.
return HttpResponse(str(res))
I got this response from Facebook.
{'status': '200', 'content-length': '15753', 'x-xss-protection': '0',
'content-location': u'https://www.facebook.com/dialog/oauth/?oauth_body_hash=1YZMc0vtMcUa5nO81ZkuwoCSct4%3D&oauth_nonce=14826580&oauth_timestamp=1342669603&oauth_consumer_key=117889941688718&oauth_signature_method=HMAC-SHA1&oauth_version=1.0&oauth_signature=t2cIAmQYs4r5IqcsoODqorFIHbs%3D',
'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked',
'expires': 'Sat, 01 Jan 2000 00:00:00 GMT', 'connection': 'keep-alive',
'-content-encoding': 'gzip', 'pragma': 'no-cache',
'cache-control': 'private, no-cache, no-store, must-revalidate',
'date': 'Thu, 19 Jul 2012 03:46:44 GMT', 'x-frame-options': 'DENY',
'content-type': 'text/html; charset=utf-8',
'x-fb-debug': 'cXSDiq5jL9ZffjalabM6QKEgr50QOhPJsNlHW2MpSSQ='}
I am a bit confused because I do not see a variable called "access token" I even see weird terms like auth_signature_method and oauth_signature, but I have no idea what they mean.
How do I retrieve the access token from Facebook? Furthermore, how can I use this access token to retrieve basic information such as "name" from the user? The access token is just a string, right?
I don't know with your code. But if you use Facebook graph api it would be simple.
def authorize(self):
warnings.filterwarnings('ignore', category=DeprecationWarning)
savout = os.dup(1)
os.close(1)
os.open(os.devnull, os.O_RDWR)
try:
webbrowser.open(FBOAuth.FACEBOOK_GRAPH_URL+'/oauth/authorize?'+urllib.urlencode(
{'client_id':FBOAuth.CLIENT_ID,
'redirect_uri':FBOAuth.REDIRECT_URI,
'scope':'read_stream, publish_stream'}))
finally:
os.dup2(savout, 1)
FBOAuth.SECRET_CODE = raw_input("Secret Code: ")
self.save_secret_code(FBOAuth.SECRET_CODE)
return FBOAuth.SECRET_CODE
def access_token(self):
if not FBOAuth.SECRET_CODE:
FBOAuth.SECRET_CODE = self.authorize()
args = {'redirect_uri': FBOAuth.REDIRECT_URI,
'client_id' : FBOAuth.CLIENT_ID,
'client_secret':FBOAuth.CLIENT_SECRET,
'code':FBOAuth.SECRET_CODE,}
access_token = urllib.urlopen(FBOAuth.FACEBOOK_GRAPH_URL + "/oauth/access_token?" + urllib.urlencode(args)).read()
access_token = urlparse.parse_qs(access_token)
FBOAuth.ACCESS_TOKEN = access_token['access_token'][0]
self.save_access_token(FBOAuth.ACCESS_TOKEN)
return FBOAuth.ACCESS_TOKEN
And by these 2 functions you can get access tokens.
Here is the link for tutorial i used.
Facebook graph api documents for reference.
Just like most auth processes, you pass in your app ID, and get back a "code." In a second "GET", you send that in for the access_token. You need to decrypt the result, which may be in HMAC. Use the SDK to manage these functions, as the other person recommended, and it will be easier.
"auth_signature_method" means the way in which the encrypted signature was signed. It's not the final access_token you can use to query the user's data on Facebook.
Your code is wrong in several ways, mainly in forgetting that you need to have a user participate in the transaction.
See https://developers.facebook.com/docs/authentication/server-side/ for the the steps to follow, these are actually pretty simple and you can do them all by hand using curl.