We Keep Coding

`document.lastModified` in Python

In python, by using an HTML parser, is it possible to get the document.lastModified property of a web page. I'm trying to retrieve the date at which the webpage/document was last modified by the owner.

A somewhat related question "I am downloading a file using Python urllib2. How do I check how large the file size is?", suggests that the following (untested) code should work:
import urllib2
req = urllib2.urlopen("http://example.com/file.zip")
total_size = int(req.info().getheader('last-modified'))
You might want to add a default value as the second parameter to getheader(), in case it isn't set.

You can also look for a last-modified date in the HTML code, most notably in the meta-tags. The htmldate module does just that.
Here is how it could work:
1. Install the package:
pip/pip3/pipenv (your choice) -U htmldate
2. Retrieve a web page, parse it and output the date:
from htmldate import find_date
find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')
(disclaimer: I'm the author)

python ultrajson: how to use?

I've just installed ultrajson (ujson) to see if I can't get the json decoding to go faster (string to object). However, I'm not seeing any examples of how to use it.
with regular json it's just
import json
my_object = json.loads(my_string)

Change the import statement to import ujson as json
Then you can leave the other parts of your program as it is.

How to save Python in-memory dictionary to a file as a Python source code?

How to get Python source code representation of in-memory Python dictionary?
I decided to ask this question after reading Thomas Kluyver's comment on Rob Galanakis' blog post titled Why bother with python and config files? In his comment Thomas states
But if you want any way to change settings inside the application
(like a preferences dialog), there’s no good way to automatically
write a correct Python file.

Assuming it uses only "basic" Python types, you can write out the repr() of the structure, and then use ast.literal_eval() to read it back in after.

As the article says, you're better off using JSON/YAML or other formats, but if you seriously wanted to use a Python dict and are only using basic Python types...
Writing out (attempt to use pformat to try and make it more human readable):
from pprint import pformat # instead of using repr()
d = dict(enumerate('abcdefghijklmnopqrstuvwxyz'))
open('somefile.py').write(pformat(d))
Reading back:
from ast import literal_eval
d = literal_eval(open('somefile.py').read())

Does converting json to dict with eval a good choice?

I am getting a json object from a remote server, and converting it to a python string like this:
a = eval(response)
Is this stupid in any way, or do I have a better option?

Using eval is not a good way to process JSON:
JSON isn't even valid Python, because of true, false, and null.
eval will execute arbitrary Python code, so you are at the mercy of malicious injection of code.
Use the json module available in the standard library instead:
import json
data = json.loads("[1, 2, 3]")
If you're using a version of Python older than 2.6, you'll need to download the module yourself. It's called simplejson and can be downloaded from PyPi.

Yes, very. Use a json decoder instead:
>>> from simplejson import loads
>>> loads(response)

Alternative to innerhtml that includes header?

I'm trying to extract data from the following page:
http://www.bmreports.com/servlet/com.logica.neta.bwp_PanBMDataServlet?param1=&param2=&param3=&param4=&param5=2009-04-22&param6=37#
Which, conveniently and inefficiently enough, includes all the data embedded as a csv file in the header, set as a variable called gs_csv.
How do I extract this? Document.body.innerhtml skips the header where the data is, what is the alternative that includes the header (or better yet, the value associated with gs_csv)?
(Sorry, new to all this, I've been searching through loads of documentation, and trying a lot of them, but nothing so far has worked).
Thanks to Sinan (this is mostly his solution transcribed into Python).
import win32com.client
import time
import os
import os.path
ie = Dispatch("InternetExplorer.Application")
ie.Visible=False
ie.Navigate("http://www.bmreports.com/servlet/com.logica.neta.bwp_PanBMDataServlet?param1=&param2=&param3=&param4=&param5=2009-04-22&param6=37#")
time.sleep(20)
webpage=ie.document.body.innerHTML
s1=ie.document.scripts(1).text
s1=s1[s1.find("gs_csv")+8:-11]
scriptfilepath="c:\FO Share\bmreports\script.txt"
scriptfile = open(scriptfilepath, 'wb')
scriptfile.write(s1.replace('\n','\n'))
scriptfile.close()
ie.quit

Untested: Did you try looking at what Document.scripts contains?
UPDATE:
For some reason, I am having immense difficulty getting this to work using the Windows Scripting Host (but then, I don't use it very often, apologies). Anyway, here is the Perl source that works:
use strict;
use warnings;
use Win32::OLE;
$Win32::OLE::Warn = 3;
my $ie = get_ie();
$ie->{Visible} = 1;
$ie->Navigate(
'http://www.bmreports.com/servlet/com.logica.neta.bwp_PanBMDataServlet?'
.'param1=&param2=&param3=&param4=&param5=2009-04-22&param6=37#'
);
sleep 1 until is_ready( $ie );
my $scripts = $ie->Document->{scripts};
for my $script (in $scripts ) {
print $script->text;
}
sub is_ready { $_[0]->{ReadyState} == 4 }
sub get_ie {
Win32::OLE->new('InternetExplorer.Application',
sub { $_[0] and $_[0]->Quit },
);
}
__END__
C:\Temp> ie > output
output now contains everything within the script tags.

fetch the source of that page using ajax, and parse the response text like XML using jquery. It should be simple enought to get the text of the first tag you encounter inside the
I'm out of touch with jquery, or I would have posted code examples.
EDIT: I assume you are talking about fetching the csv on the client side.

If this is just a one off script then exctracting this csv data is as simple as this:
import urllib2
response = urllib2.urlopen('http://www.bmreports.com/foo?bar?')
html = response.read()
csv = data.split('gs_csv=')[1].split('</SCRIPT>')[0]
#process csv data here

Thanks to Sinan (this is mostly his solution transcribed into Python).
import win32com.client
import time import os
import os.path
ie = Dispatch("InternetExplorer.Application") ie.Visible=False
ie.Navigate("http://www.bmreports.com/servlet/com.logica.neta.bwp_PanBMDataServlet?param1=&param2=&param3=&param4=&param5=2009-04-22&param6=37#")
time.sleep(20)
webpage=ie.document.body.innerHTML
s1=ie.document.scripts(1).text s1=s1[s1.find("gs_csv")+8:-11]
scriptfilepath="c:\FO Share\bmreports\script.txt"
scriptfile = open(scriptfilepath, 'wb')
scriptfile.write(s1.replace('\n','\n'))
scriptfile.close()
ie.quit