We Keep Coding

By downloading picture's file-header, how to obtain enough information?

I need to download some pictures from a picture server every day. This server adds thousands of pictures daily, and many pictures have large size. Since the server does not support thumb pics and any pic-description, I have to download the pictures completely to know whether this picture is the one I need. However, my network bandwidth is very low, so it cost considerable time to download every picture. Moreover, the server has strict network flow controlling, thus, I may only download less than 100 pictures every day if the pictures are all large.
I search some related articles and find that picture's file-header contains many useful information, so this is my plan:
Use python code to download all pictures' file-header. If I only download file-header, the network flow will be very small, so I can download all pictures' file-header on the server.
Analyse every pictures' file-header and obtain enough information. From my searching, I know the picture's format(png/jpg/gif), size(XXX,XXX bytes), resolution(XXXX×YYY,such as 1920x1080) can be obtained from picture's file-header which is less than 1000 bytes. Maybe it is possible to get more information from picture's file-header, so if you know more, please help me.
Export the result to an Excel file.
Could you tell me the effective python code to achieve the above three demands?
Added on July 22:
This is some information I got from HTTP header
HTTP/1.1 200 OK
Server: nginx
Date: Sun, 22 Jul 2018 15:13:19 GMT
Content-Type: image/jpeg
Content-Length: 376386
Cache-Control: public,max-age=518400
Expires: Sat, 28 Jul 2018 15:13:19 GMT
Last-Modified: Sun, 22 Jul 2018 15:13:19 GMT
Vary: Origin
ETag: "5be42"
Connection: Keep-alive
Now I can get Content-Type and Content-Length from HTTP header. But it is not enough for me.
I searched and found someone said they could read the image's resolution(XXXX×YYY,such as 1920x1080) from the first 100 bytes of the pic-file data.(100 here is only the maximum number, somebody even said he can get the resolution from the begin 30 bytes of the pic-file data.) I think it is true because many pics I downloaded not-finished can display the resolution and the top of the pic.
Moreover, maybe there is a way to generate thumb without downloading complete pic? I'm not sure it is possible or not, but I think if it can be done, it will be very useful.

You can use PIL library and use getdata.

I don't think that is possible or are there image headers at all? When I do something like
curl -I https://upload.wikimedia.org/wikipedia/de/b/bb/Png-logo.png
to get HTTP headers, I don't see an image size or anything like that:
HTTP/1.1 200 OK
Date: Sat, 21 Jul 2018 17:35:26 GMT
Content-Type: image/png
Content-Length: 811068
Connection: keep-alive
X-Object-Meta-Sha1Base36: tup6ux1u98mkbw32ta64fna0hqw6y09
Last-Modified: Thu, 03 Oct 2013 23:18:32 GMT
Etag: 1f427f6758058528cc0d474a14ee6dc1
X-Timestamp: 1380842311.64879
X-Trans-Id: txdbd33b3337fb497694bd8-005b536ebb
X-Varnish: 185288243, 96001562 108570149, 528370630
Via: 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1), 1.1 varnish (Varnish/5.1)
Accept-Ranges: bytes
Age: 34
X-Cache: cp1062 pass, cp3038 hit/2, cp3039 miss
X-Cache-Status: hit-local
Strict-Transport-Security: max-age=106384710; includeSubDomains; preload
X-Analytics: https=1;nocookies=1
X-Client-IP: 87.152.115.72
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: Age, Date, Content-Length, Content-Range, X-Content-Duration, X-Cache, X-Varnish
Timing-Allow-Origin: *
Content-Security-Policy-Report-Only: default-src 'none'; style-src 'unsafe-inline' data:; font-src data:; img-src data: https://upload.wikimedia.org/favicon.ico; media-src data:; sandbox; report-uri https://commons.wikimedia.org/w/api.php?reportonly=1&source=image&action=cspreport&format=json&
X-Content-Security-Policy-Report-Only: default-src 'none'; style-src 'unsafe-inline' data:; font-src data:; img-src data: https://upload.wikimedia.org/favicon.ico; media-src data:; sandbox; report-uri https://commons.wikimedia.org/w/api.php?reportonly=1&source=image&action=cspreport&format=json&
X-Webkit-CSP-Report-Only: default-src 'none'; style-src 'unsafe-inline' data:; font-src data:; img-src data: https://upload.wikimedia.org/favicon.ico; media-src data:; sandbox; report-uri https://commons.wikimedia.org/w/api.php?reportonly=1&source=image&action=cspreport&format=json&
Even if there is such a thing, the limit of 100 would probably affect the image headers as well.

How to serve a PNG with flask so it can be used as an attachment in Slack?

I am trying to send a PNG as an attachment, but it doesn't show up in Slack. I am setting the right parameters in the POST-method, but Slack refuses to use the image I am providing.
I am using flask to serve the static files:
#app.route('/data/<path:path>')
def send_png(path):
response = make_response(send_file("data/" + path))
return response
When I call the URL in my browser, the files gets displayed without any issues. When I pass the URL to slack as an attachment, the file doesn't show up.
When I pass the URL of an imgur-image, the attachment does get displayed.
For that reason, I assume the issue lies somewhere in the content-type/file-headers of the files flask serves.
My file headers are:
HTTP/1.0 200 OK
Content-Length: 391777
Content-Type: image/png
Last-Modified: Fri, 02 Mar 2018 22:46:41 GMT
Cache-Control: public, max-age=43200
Expires: Sat, 03 Mar 2018 12:48:53 GMT
ETag: "1520030801.2465587-391777-4064615867"
Server: Werkzeug/0.11.11 Python/3.5.2
Date: Sat, 03 Mar 2018 00:48:53 GMT
Connection: keep-alive
I can also verify, that Slack does request my attachment (just doesn't display it, as said before):
[('User-Agent', 'Slackbot 1.0 (+https://api.slack.com/robots)'), ('X-Forwarded-For', '54.89.92.4'), ('Content-Type', ''), ('Accept-Encoding', 'gzip,deflate'), ('Accept', '*/*'), ('Host', 'XXXXXXXXXX'), ('Referer', 'https://slack.com'), ('Content-Length', ''), ('X-Forwarded-Proto', 'https')]

What about this:, the image should go in files= if you want as an attachment
import io
import requests
from PIL import Image
img = Image.open('picture.png')
im = io.BytesIO(img.file.read())
r = requests.post("127.0.0.1", data=im, timeout=5, files=im)

Apache sending Transfer-Encoding: chunked when deflate module is enabled

I have a simple web.py code like below, deployed with mod_wsgi in apache.
import web
urls = (
'/', 'index'
)
class index:
def GET(self):
content = 'hello'
web.header('Content-length', len(content))
return content
app = web.application(urls, globals())
application = app.wsgifunc()
This website runs well, except one minor issue. When mod_deflate is turn on, the response is chunked, even it has a very small response body.
Response Header
HTTP/1.1 200 OK
Date: Wed, 20 May 2015 20:14:12 GMT
Server: Apache/2.4.7 (Ubuntu)
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html
When mod_deflate is turn off, Content-Length header is back.
HTTP/1.1 200 OK
Date: Wed, 20 May 2015 20:30:09 GMT
Server: Apache/2.4.7 (Ubuntu)
Content-Length: 5
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8
I've searched around and someone said reduce the DeflateBufferSize will help, but this response's size is only 5, far from it's default value: 8096, so I don't think it interferes with this issue.
And someone said apache send chunked response because it doesn't know the response's size before begin to send the response to client, but in my code, I do set Content-Length.
I've also tried Flask and Apache/2.2.15 (CentOS), same result.
How do I set content-length when deflate module is enabled? and I don't like to gzip content in python.

The response Content-Length has to reflect the final length of the data sent after the compression has been done, not the original length. Thus mod_deflate has to remove the original Content-Length header and use chunked transfer encoding. The only way it could otherwise know the content length to be able to send the Content-Length before sending the compressed data, would be to buffer up the complete compressed response in memory or into a file and then calculate the length. Buffering all the compressed content isn't practical and in part defeats the point of compressing the data as the response is streamed.
If you don't want mod_deflate enabled for the whole site, then only enable it for certain URL prefixes by scoping it within a Location block.

error in downloading a gif picture from url python

I want to download on disk the gif image:
http://www.portaportese.it/telefono/es_2014043024395.gif
with all the codes I found out around for downloading pictures, I end up with a error in the final saved picture such as:
GIF image was truncated or incomplete.
in a few words the picture is not being saved correctly.
Is there anybody able to provide a correct solution which will download this picture on disk?
Any code returns an empty image.. I tried this:
import urllib2
picture_page = "http://www.portaportese.it/telefono/es_2014043024395.gif"
opener1 = urllib2.build_opener()
page1 = opener1.open(picture_page)
my_picture = page1.read()
filename = "my_image.gif"
fout = open(filename, "wb")
fout.write(my_picture)
fout.close()

The problem does not lie with your Python code, the image that you are trying to download does not exist. If I use curl to place a request at that URL, you can see that no image is stored there.
~ ❯❯❯ curl -I http://www.portaportese.it/telefono/es_2014043024395.gif
HTTP/1.1 200 OK
Date: Mon, 07 Jul 2014 19:35:05 GMT
Server: Apache/2.2.3 (Red Hat)
Connection: close
Content-Type: text/plain; charset=UTF-8
Compare that with this request to a known image source:
~ ❯❯❯ curl -I http://baconmockup.com/300/200
HTTP/1.1 200 OK
Date: Mon, 07 Jul 2014 19:35:42 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.2
Access-Control-Allow-Origin: *
Content-Length: 20564
Content-Disposition: inline; filename=brisket-300-200.jpg
Pragma: public
Cache-Control: public
Expires: Mon, 21 Jul 2014 19:35:43 GMT
Last-Modified: Mon, 20 Aug 2012 19:20:21 GMT
Vary: User-Agent
Content-Type: image/jpeg
If you change the URL in your code to a good image source, then it will work perfectly well.
import urllib2
picture_page = "http://baconmockup.com/300/200"
opener1 = urllib2.build_opener()
page1 = opener1.open(picture_page)
my_picture = page1.read()
filename = "my_image.gif"
fout = open(filename, "wb")
fout.write(my_picture)
fout.close()
I just ran this, and was given a picture of some tasty brisket.

How to send huge files over HTTPPost method in Python, upload large files

I am tring to post huge .ova file through HTTPPost Method in Python
**ResponseHeaders**
Pragma no-cache
Date Thu, 18 Jul 2013 11:17:13 GMT
Content-Encoding gzip
Vary Accept-Encoding
Server Apache-Coyote/1.1
Transfer-Encoding chunked
Content-Language en-US
Content-Type application/json;charset=UTF-8
Cache-Control no-cache, no-store, max-age=0
Expires Thu, 01 Jan 1970 00:00:00 GMT
**RequestHeaders**
Content-Type application/json
Accept application/json
xyzAPIVersion 1.0
X-Requested-With XMLHttpRequest
How to send such a huge file(500 MB) through HTTPPost method through REST API.

You could use requests library:
import requests # $ pip install requests
with open("file.ova", "rb") as file:
requests.post(url, data=file)