We Keep Coding

Using a variable as a Filename Python

I cannot get using a variable in the filename to work without throwing an error.
def logData(name, info):
currentTime = datetime.datetime.now()
date = currentTime.strftime("%x")
filename = "{}_{}.txt".format(name, date)
f = open(filename, "a")
f.write(info)+ '\n')
f.close()
I have tried formatting the string as shown above as well as concatenating the variables.
Is there anything I can do to solve this?

One issue was the closing parentheses, also the date format contain / (slash ex:07/07/21) which will make the filename with slash i.e not a valid name in windows as it makes it a path.
Solution with least change to your logic:
import datetime
def logData(name, info):
currentTime = datetime.datetime.now()
date = currentTime.strftime("%Y-%m-%d")
filename = "{}_{}.txt".format(name, date)
with open(filename,"a+") as f:
f.write(f'{info}\n')
logData('my_file',"test")

Python3 convert result of DB query to individual strings

I have 2 functions in a python script.
The first one gets the data from a database with a WHERE clause but the second function uses this data and iterates through the results to download a file.
I can get to print the results as a tuple?
[('mmpc',), ('vmware',), ('centos',), ('redhat',), ('postgresql',), ('drupal',)]
But I need to to iterate through each element as a string so the download function can append it onto the url for the response variable
Here is the code for the download script which contains the functions:-
import requests
import eventlet
import os
import sqlite3
# declare the global variable
active_vuln_type = None
# Get the active vulnerability sets
def GetActiveVulnSets() :
# make the variable global
global active_vuln_type
active_vuln_type = con = sqlite3.connect('data/vuln_sets.db')
cur = con.cursor()
cur.execute('''SELECT vulntype FROM vuln_sets WHERE active=1''')
active_vuln_type = cur.fetchall()
print(active_vuln_type)
return(active_vuln_type)
# return str(active_vuln_type)
def ExportList():
vulnlist = list(active_vuln_type)
activevulnlist = ""
for i in vulnlist:
activevulnlist = str(i)
basepath = os.path.dirname(__file__)
filepath = os.path.abspath(os.path.join(basepath, ".."))
response = requests.get('https://vulners.com/api/v3/archive/collection/?type=' + activevulnlist)
with open(filepath + '/vuln_files/' + activevulnlist + '.zip', 'wb') as f:
f.write(response.content)
f.close()
return activevulnlist + " - " + str(os.path.getsize(filepath + '/vuln_files/' + activevulnlist + '.zip'))
Currently it creates a corrupt .zip as ('mmpc',).zip so it is not the actual file which would be mmpc.zip for the first one but it does not seem to be iterating through the list either as it only creates the zip file for the first result from the DB, not any of the others but a print(i) returns [('mmpc',), ('vmware',), ('centos',), ('redhat',), ('postgresql',), ('drupal',)]
There is no traceback as the script thinks it is working.

The following fixes two issues: 1. converting the query output to an iterable of strings and 2. replacing the return statement with a print function so that the for-loop does not end prematurely.
I have also taken the liberty of removing some redundancies such as closing a file inside a with statement and pointlessly converting a list into a list. I am also calling the GetActiveVulnSets inside the ExportList function. This should eliminate the need to call GetActiveVulnSets outside of function definitions.
import requests
import eventlet
import os
import sqlite3
# declare the global variable
active_vuln_type = None
# Get the active vulnerability sets
def GetActiveVulnSets() :
# make the variable global
global active_vuln_type
active_vuln_type = con = sqlite3.connect('data/vuln_sets.db')
cur = con.cursor()
cur.execute('''SELECT vulntype FROM vuln_sets WHERE active=1''')
active_vuln_type = [x[0] for x in cur]
print(active_vuln_type)
return(active_vuln_type)
# return str(active_vuln_type)
def ExportList():
GetActiveVulnSets()
activevulnlist = ""
for i in active_vuln_type:
activevulnlist = str(i)
basepath = os.path.dirname(__file__)
filepath = os.path.abspath(os.path.join(basepath, ".."))
response = requests.get('https://vulners.com/api/v3/archive/collection/?type=' + activevulnlist)
with open(filepath + '/vuln_files/' + activevulnlist + '.zip', 'wb') as f:
f.write(response.content)
print(activevulnlist + " - " + str(os.path.getsize(filepath + '/vuln_files/' + activevulnlist + '.zip')))
While this may solve the problem you are encountering, I would recommend that you write functions with parameters. This way, you know what each function is supposed to take in as an argument and what it spits out as output. In essence, avoid the usage of global variables if you can. They are hard to debug and quite frankly unnecessary in many use cases.
I hope this helps.

Downloading the most recent file from FTP with Python

I have been trying to write a function with Python that would allow to download the most recently added file (by timestamp within filename).
You can see that the format has a big timestamp.
What I have so far with the help of forums is the following code.
In the following code, I tried to sort using the date field (real added date to FTP server). However,
I want to adjust this code so that I can sort the files by the timestamp within filename.
EDIT (Tried to clean the code a bit):
def DownloadFileFromFTPServer2 (server, username, password, directory_to_file, file_to_write):
try:
f = ftplib.FTP(server)
except ((socket.error, socket.gaierror), e):
print ('cannot reach to %s' % server)
return
print ("Connected to FTP server")
try:
f.login(username, password)
except ftplib.error_perm:
print ("cannot login anonymously")
f.quit()
return
print ("Logged on to the FTP server")
try:
f.cwd(directory_to_file)
print ("Directory has been set")
except Exception as inst:
print (inst)
data = []
f.dir(data.append)
datelist = []
filelist =[]
for line in data:
print (line)
col = line.split()
datestr = ' '.join(line.split()[5:8])
date = time.strptime (datestr, '%b %d %H:%M')
datelist.append(date)
filelist.append(col[8])
combo = zip (datelist, filelist)
who = dict ( combo )
# Sort by dates and get the latest file by date....
for key in sorted(iter(who.keys()), reverse = True):
filename = who[key]
print ("File to download is %s" % filename)
try:
f.retrbinary('RETR %s' % filename, open(filename, 'wb').write)
except (ftplib.err_perm):
print ("Error: cannot read file %s" % filename)
os.unlink(filename)
else:
print ("***Downloaded*** %s " % filename)
print ("Retrieving FTP server data ......... DONE")
#VERY IMPORTANT RETURN
return
f.quit()
return 1
Any help is greately appreciated. Thanks.
EDIT [SOLVED]:
The line
date = time.strptime (datestr, '%b %d %H:%M')
should be replaced with:
try:
date = datetime.datetime.strptime (str(col[8]), 'S01375T-%Y-%m-%d-%H-%M-%S.csv')
except Exception as inst:
continue
try-continue is important since the first two path lines such as '.' and '..' will result a ValuError.

Once you have the list of filenames you can simply sort on filename, since the naming convention is S01375T-YYYY-MM-DD-hh-mm.csv this will naturally sort into date/time order. Note that if the S01375T- part varies you could sort on the name split at a fixed position or at the first -.
If this was not the case you could use the datetime.datetime.strptime method to parse the filenames into datetime instances.
Of course if you wished to really simplify things you could use the PyFileSystem FTPFS and it's various methods to allow you to treat the FTP system as if is was a slow local file system.

Try with the -t option in ftp.dir, this orders by date in reverse, then take the first in the list:
data = []
ftp.dir('-t',data.append)
filename = data[0]

You need to extract the timestamp from the filename properly.
You could split the filename at the first '-' and remove the file extensition '.csv' (f.split('-', 1)[1][:-4]).
Then you just need to construct the datetime obj for sorting.
from datetime import datetime
def sortByTimeStampInFile(fList):
fileDict = {datetime.strptime(f.split('-', 1)[1][:-4], '%Y-%m-%d-%H-%M-%S'): f for f in fList if f.endswith('.csv')}
return [fileDict[k] for k in sorted(fileDict.keys())]
files = ['S01375T-2016-03-01-12-00-00.csv', 'S01375T-2016-01-01-13-00-00.csv', 'S01375T-2016-04-01-13-01-00.csv']
print(sortByTimeStampInFile(files))
Returns:
['S01375T-2016-01-01-13-00-00.csv', 'S01375T-2016-03-01-12-00-00.csv', 'S01375T-2016-04-01-13-01-00.csv']
Btw. as long as your time format is 'year-month-day-hour-min-sec', a simple string sort would do it:
sorted([f.split('-', 1)[1][:-4] for f in fList if f.endswith('.csv')])
>>> ['2016-01-01-13-00-00', '2016-03-01-12-00-00', '2016-04-01-13-01-00']

Python: Creating an empty file object

I am attempting to make a logging module for Python that does not work because it fails on creation of the file object.
debug.py:
import os
import datetime
import globals
global fil
fil = None
def init(fname):
fil = open(fname, 'w+')
fil.write("# PyIDE Log for" + str(datetime.datetime.now()))
def log(strn):
currentTime = datetime.datetime.now()
fil.write(str(currentTime) + ' ' + str(os.getpid()) + ' ' + strn)
print str(currentTime) + ' ' + str(os.getpid()) + ' ' + strn
def halt():
fil.close()
fil will not work as None as I get an AttributeError. I also tried creating a dummy object:
fil = open("dummy.tmp","w+")
but the dummy.tmp file is written to instead, even though init() is called before log() is. Obviously you cannot open a new file over an already opened file. I attempted to close fil before init(), but Python said it could not perform write() on a closed file.
This is the code that is accessing debug.py
if os.path.exists(temp):
os.rename(temp, os.path.join("logs","archived","log-" + str(os.path.getctime(temp)) + ".txt"))
debug.init(globals.logPath)
debug.log("Logger initialized!")
I would like to have logging in my program and I cannot find a workaround for this.

Your problem is that you don't assign to the global fil:
def init(fname):
fil = open(fname, 'w+')
This creates a new local variable called fil.
If you want to assign to the global variable fil you need to bring it into the local scope:
def init(fname):
global fil
fil = open(fname, 'w+')

If you want to MAKE your own logging module, then you may want to turn what you already have into a class, so you can import it as a module.
#LoggerThingie.py
import os
import datetime
class LoggerThingie(object):
def __init__(self,fname):
self.fil = open(fname, 'w+')
self.fil.write("# PyIDE Log for" + str(datetime.datetime.now()))
def log(self,strn):
currentTime = datetime.datetime.now()
self.fil.write(str(currentTime) + ' ' + str(os.getpid()) + ' ' + strn)
print str(currentTime) + ' ' + str(os.getpid()) + ' ' + strn
def halt(self):
self.fil.close()
If you did this as a class, you would not have to keep track of globals in the first place (which is generally understood as bad practice in the world of programming: Why are global variables evil? )
Since it is now a module on its own, when you want to use it in another python program you would do this:
from LoggerThingie import LoggerThingie
#because module filename is LoggerThingie.py and ClassName is LoggerThingie
and then use it wherever you want, for example:
x = LoggerThingie('filename.txt') #create LoggerThingie object named x
and every-time you want to insert logs into it:
x.log('log this to the file')
and when you are finally done:
x.halt() # when ur done

If you don't want to start with an empty file you could use StringIO to keep the messages in memory and write them to disk at the end but be careful, if something happened and you didn't wrote the messages they will be lost.

How to use python to turn a .dbf into a shapefile

I have been scouring the internet trying to find a pythonic (sp?!) way to process this data..
Everyday we will recieve a load of data in .dbf format (hopefully) - we then need to save this data as a shapefile.
Does anyone have any links or any suggestions as to my process?

To append the file's creation_date to its name, you need to obtain the creation date with os.stat() and then rename the file with os.rename(). You can format the date string with date.strftime().
import datetime, os
filename = 'original.ext'
fileinfo = os.stat(filename)
creation_date = datetime.date.fromtimestamp(fileinfo.st_ctime)
os.rename(filename, filename + '-' + creation_date.strftime('%Y-%m-%d'))

Off the top of my head:
import os
import datetime
myfile = "test.txt"
creationdate = os.stat(myfile).st_ctime
timestamp = datetime.datetime.fromtimestamp(creationdate)
datestr = datetime.datetime.strftime(timestamp, "%Y%m%d")
os.rename(myfile, os.path.splitext(myfile)[0] + datestr + os.path.splitext(myfile)[1])
renames test.txt to test20110221.txt.

It was in model builder all along!
# (generated by ArcGIS/ModelBuilder)
# Usage: DBF2SHAPEFILE <XY_Table> <Y_Field> <X_Field> <Output_Feature_Class>
# ---------------------------------------------------------------------------
# Import system modules
import sys, string, os, arcgisscripting, datetime
# Adds the creation date to all of the previous shapefiles in that folder
filename = 'D:/test.txt'
fileinfo = os.stat(filename)
creation_date = datetime.date.fromtimestamp(fileinfo.st_ctime)
os.rename(filename, filename + '-' + creation_date.strftime('%Y-%m-%d'))
# Create the Geoprocessor object
gp = arcgisscripting.create()
# Load required toolboxes...
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Data Management Tools.tbx")
# Script arguments...
XY_Table = sys.argv[1]
Y_Field = sys.argv[2]
X_Field = sys.argv[3]
Output_Feature_Class = sys.argv[4]
# Local variables...
Layer_Name_or_Table_View = ""
# Process: Make XY Event Layer...
gp.MakeXYEventLayer_management(XY_Table, X_Field, Y_Field, Layer_Name_or_Table_View, "")
# Process: Copy Features...
gp.CopyFeatures_management(Layer_Name_or_Table_View, Output_Feature_Class, "", "0", "0", "0")

If you wanted to do it without using ArcGIS, you could use OGR's python bindings or the ogr2ogr utility through a subprocess. You could use the utility through a windows batch file, which would be a lot faster than calling the arc process for every file if you have many to do...
As you know it's not a question of changing the extension, there is a specific format required.