We Keep Coding

Suppress printing of response from Azure Queue Storage

When I send a message to a queue I want the response returned into an object which I can then include in my log or not. However for some reason when I execute the following code:
from azure.storage.queue import QueueClient, TextBase64EncodePolicy
# ... some code running ##########################
queue = QueueClient.from_connection_string(conn_str=conn_queue, queue_name="items",
message_encode_policy=TextBase64EncodePolicy())
# ... some message generated #####################
response=queue.send_message(json.dumps(item_dict))
a complete message is printed to my log. It looks for example like this:
Request URL: 'https://{some_storage_account}.queue.core.windows.net/items/messages'
Request method: 'POST'
Request headers: 'Accept': 'application/xml'
'Content-Type': 'application/xml; charset=utf-8'
'x-ms-version': 'REDACTED'
'Content-Length': '1295'
'x-ms-date': 'REDACTED'
'x-ms-client-request-id': '3452464c-06b2-11eb-9f96-00155d6ebdc5'
'User-Agent': 'azsdk-python-storage-queue/12.1.3 Python/3.8.5 (Linux-4.19.104-microsoft-standard-x86_64-with-glibc2.2.5)'
'Authorization': 'REDACTED'
A body is sent with the request
Response status: 201
Response headers:
'Transfer-Encoding': 'chunked'
'Content-Type': 'application/xml'
'Server': 'Windows-Azure-Queue/1.0 Microsoft-HTTPAPI/2.0'
'x-ms-request-id': '1720a5da-c003-002b-18be-9ab4d4000000'
'x-ms-version': 'REDACTED'
'Date': 'Mon, 05 Oct 2020 02:26:41 GMT'
How can I prevent this gulp of information to be printed to my log?
I have spent at least half an hour on the docs from microsoft. But I can't find where I can turn this behaviour off.

logger = logging.getLogger("azure.core.pipeline.policies.http_logging_policy")
logger.setLevel(logging.WARNING)
This Worked For Me - Add this to your code

I tried with the logging-levels - it does not do the trick for me. However when I define a new logger for this connection then it works:
queue_logger = logging.getLogger("logger_name")
# queue_logger.disabled = True
queue_logger.setLevel(logging.DEBUG)
queue = QueueClient.from_connection_string(conn_str=conn_queue, queue_name="debug",message_encode_policy=TextBase64EncodePolicy(), logger=queue_logger)
Ultimately I feel like I should use the REST API instead of the Python Azure SDK for this. REST API allows me to log-output based on the response status. For some weird reason the SDK does not offer me this possibility.

The answer should be here, in the README of azure-storage-python in GitHub:
Here is how we use the logging levels, it is recommended to use INFO:
DEBUG: log strings to sign
INFO: log outgoing requests and responses, as well as retry attempts # <--
WARNING: not used
ERROR: log calls that still failed after all the retries
This should do it, as recommended by Kalies LAMIRI (I didn't try myself):
logger = logging.getLogger("azure.storage")
logger.setLevel(logging.ERROR)

Azure form recognizer error: "Failed to download image from input URL."

I'm following these instructions to use the layout form recognizer service from Azure
Which have the following code:
########### Python Form Recognizer Async Layout #############
import json
import time
from requests import get, post
# Endpoint URL
endpoint = r"<Endpoint>"
apim_key = "<Subscription Key>"
post_url = endpoint + "/formrecognizer/v2.0-preview/Layout/analyze"
source = r"<path to your form>"
headers = {
# Request headers
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': apim_key,
}
with open(source, "rb") as f:
data_bytes = f.read()
try:
resp = post(url = post_url, data = data_bytes, headers = headers)
if resp.status_code != 202:
print("POST analyze failed:\n%s" % resp.text)
quit()
print("POST analyze succeeded:\n%s" % resp.headers)
get_url = resp.headers["operation-location"]
except Exception as e:
print("POST analyze failed:\n%s" % str(e))
quit()
I tried the code I got the following error:
POST analyze failed:
{"error":{"code":"FailedToDownloadImage","message":"Failed to download image from input URL."}}
POST analyze succeeded:
{'Transfer-Encoding': 'chunked', 'Content-Type': 'application/json; charset=utf-8', 'x-envoy-upstream-service-time': '4', 'apim-request-id': '515e93ee-4db8-4174-92b1-63e5c415c056', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'x-content-type-options': 'nosniff', 'Date': 'Sat, 06 Jun 2020 20:47:28 GMT'}
POST analyze failed:
'operation-location'
The code I'm using is:
import json
import time
from requests import get, post
I'm reading the pdf file before making the request and verifying it loaded into the variable
source = r"data/Invoice_7.pdf"
with open(source, "rb") as f:
data_bytes = f.read()
print (data_bytes[0:10])
Then the request details:
endpoint = r"https://xxxx.cognitiveservices.azure.com/"
apim_key = "xxxx"
post_url = endpoint + "/formrecognizer/v2.0-preview/Layout/analyze"
headers = {
# Request headers
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': apim_key,
}
And finally making the request:
try:
resp = post(url = post_url, data = data_bytes, headers = headers)
print (1)
if resp.status_code != 202:
print("POST analyze failed:\n%s" % resp.text)
#quit()
print (2)
print("POST analyze succeeded:\n%s" % resp.headers)
print (3)
get_url = resp.headers["operation-location"]
print (4)
except Exception as e:
print("POST analyze failed:\n%s" % str(e))
#quit()
I'm printing a number at each step because I find very weird that I get both fail and successful requests responses. This is the result:
1
POST analyze failed:
{"error":{"code":"FailedToDownloadImage","message":"Failed to download image from input URL."}}
2
POST analyze succeeded:
{'Transfer-Encoding': 'chunked', 'Content-Type': 'application/json; charset=utf-8', 'x-envoy-upstream-service-time': '1', 'apim-request-id': '93a2a162-d14f-496f-ba8a-077bcfd5d3c7', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'x-content-type-options': 'nosniff', 'Date': 'Sat, 06 Jun 2020 21:00:20 GMT'}
3
POST analyze failed:
'operation-location'
So the code fails at this line:
get_url = resp.headers["operation-location"]
the text in the response variable is:
'{"error":{"code":"FailedToDownloadImage","message":"Failed to download image from input URL."}}'

As defined in the REST API documentation, you need to specify the Content-Type. When you set your Content-Type to application/json, you need to provide a public accessible source via JSON. In your case, you need to set the Content-Type to application/pdf. When you want to make this dynamic, you could make use of the PyPi package filetype.
By the way, did you know that there is a (beta) Python SDK for Form Recognizer, which you can use for your use-case.

Flask fails to add custom headers

I'm trying to return custom headers like this:
#app.route("/generate", methods=['POST'])
def generate():
# data is generated here
resp = Response(jsonify(data))
resp.headers['Access-Control-Allow-Origin'] = '*'
resp.headers['Access-Control-Allow-Credentials'] = True
return resp
But they don't seem to be present in the output.
Changing status code to 201 as mentioned here didn't help either (I did this by adding status=201 to Responce construction).
Any bit of help would be appreciated.
EDIT 1:
Added a bit to a code.
Solutions described here didn't work for me either. Tried both Response and make_response, added #app.after_request as suggested - no effect.
The only headers present are:
'Content-Type': 'application/json',
'Content-Length': '113',
'Server': 'Werkzeug/1.0.1 Python/3.7.7',
'Date': 'Fri, 08 May 2020 17:05:38 GMT'

I think the problem is that, although you are adding custom headers, you are still returning a JSON object where you write resp = Response(jsonify(data)). It is possible that flask assingns the response headers automatically based on the response's body's format.

Requests HTTP headers

I've a trouble with the HTTP headers that the module Requests returns.
I'm using the following code :
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests
response = requests.get("http://www.google.co.il",proxies={'http': '','https':''})
data = response.text
# response.text returns the appropriate html code
# (<!doctype html><html dir="rtl" itemscope=""....)
if response.status_code == requests.codes.ok:
# How do I send those headers to the conn (browser)
print "HEADERS: " + str(response.headers)
conn.send(data)
I'm trying to send a GET request to www.google.co.il, and send the response to the browser (on the example I called it "conn"). The problem is that the browser won't show the received HTML code and instead I'm receiving ERR_EMPTY_RESPONSE.
The headers in the response are :
HEADERS: {'Content-Length': '5451', 'X-XSS-Protection': '1; mode=block', 'Content-Encoding': 'gzip', 'Set-Cookie': 'NID=103=RJzu4RTCNxkh-75dvKBHx-_jen9M8iPes_AdOIQqzBVZ0VPTz1PlQaAVLpwYOmxZlTKmcogiDb1VoY__Es0HqSNwlkmHl3SuBZC8_8XUfqh1PzdWTjrXRnB4S738M1lm; expires=Wed, 08-Nov-2017 10:05:46 GMT; path=/; domain=.google.co.il; HttpOnly', 'Expires': '-1', 'Server': 'gws', 'Cache-Control': 'private, max-age=0', 'Date': 'Tue, 09 May 2017 10:05:46 GMT', 'P3P': 'CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."', 'Content-Type': 'text/html; charset=windows-1255', 'X-Frame-Options': 'SAMEORIGIN'}
Someone told me that the problem is that I'm not sending any header to the browser. Is this really the problem ? Any other suggestions ? and if it is the problem, how do I send the appropriate headers to the browser ?
Edit: I forgot to mention that the connection is
through a Proxy server.
Any help would be great!
Thanks alot, Yahli.

I couldn't find anything about geting the raw http response ( not response.raw ) in requests documentation so i wrote a function :
def http_response(response):
return 'HTTP/1.1 {} {}\r\n{}\r\n\r\n{}'.format(
response.status_code, response.reason ,
'\r\n'.join(k + ': ' + v for k, v in response.headers.items()),
response.content
)
I tested it by setting Firefox HTTP proxy to localhost:port ( with a listening socket on port ) , and it works fine .
Alternatively you can get the host from conn.recv , open a new socket to that host, and send the data . Example :
data = conn.recv(1024)
host = [ l.split(':')[1].strip() for l in data.splitlines() if l.startswith('Host:') ]
if len(host) :
cli = socket.socket()
cli.connect((host[0], 80))
cli.send(data)
response = ''
while True :
data = cli.recv(1024)
if not data.strip() :
break
response += data
conn.send(response)
cli.close()
Where conn is the connection to the web browser . This is just a quick example , assuming you have only HTTP requests ( port 80 ) . There is room for much optimization

Python HttpConnection - write request headers to file

I'm trying to log some connection info in the event that an error occurs. Using httplib's HTTPConnection I can print out the request headers by setting the debug level to 1:
connection = httplib.HTTPConnection('www.example.com')
connection.set_debuglevel(1)
However, this just seems to print directly to the shell without conditions. I need to be able to get this info as a string or something to store in a variable such that I only print it out when an Exception is thrown.
The specific info that I want is the request headers that the library is generating.

I would use requests HTTP library.
To get response headers, you just need this little piece of code:
import requests
try:
r = requests.get("http://www.example.com")
# Raise an exception in case of "bad"
# status code (non-200 response)
r.raise_for_status()
print r.headers
except Exception as e:
print e.message
Output:
{'connection': 'close',
'content-encoding': 'gzip',
'content-length': '1162',
'content-type': 'text/html; charset=UTF-8',
'date': 'Sun, 12 Aug 2012 12:49:44 GMT',
'last-modified': 'Wed, 09 Feb 2011 17:13:15 GMT',
'server': 'Apache/2.2.3 (CentOS)',
'vary': 'Accept-Encoding'}

Turns out the trick is to redirect sys.stdout to a StringIO object, the contents of which can then either be written to file or what ever you like as you can get to the String. Check out StringIO.StringIO.