100% cpu usage while iterating through dictionary - python

I have python script which synchronizes files from the remote server by comparing with the local files. It uses a file downloading tool written injava. I store filepath, file size and its name in dictionary which looks like:
eddsFilesDict = {'/part07_data07/Science/TGO/ACS/SDU_': ['860904
SDU__DACS_51FC_023D8101_2021-155T02-12-17__00001.EXM', '17660866
SDU__DACS_51FB_023D8101_2021-155T02-10-16__00001.EXM', '17660866
SDU__DACS_51FA_023D8101_2021-155T 02-02-18__00001.EXM', '17660866
SDU__DACS_51F9_023D8101_2021-155T02-00-16__00001.EXM']}
To list up files on the local machine I use next part of the code:
filenames = []
for top, dirs, files in os.walk('/data/local/'):
for fn in files:
filenames.append(os.stat(os.path.join(top, fn)).st_size.__str__() + ' ' + os.path.join(fn))
But in the following part of my script usage of CPU goes up to 100% and it lasts for 1-2 minutes. I cannot understand why. By experimental way I understood that subprocess is not the thing.
for pathname, fileName in eddsFilesDict.items():
for f in fileName:
if f not in filenames:
logger.info('REQUESTING FILE: ' + pathname + '/' + f.split('.')[0].split(' ')[1] + ' FILE SIZE: '
+ f.split(' ')[0])
if 'SDU' in f:
argsFile = shlex.split(
'/home/user1/client/bin/fs_client --arc tar') # here I call java tool to start file downloading
p1 = subprocess.Popen(argsFile, stderr=subprocess.PIPE)
out, err = p1.communicate()
logger.warning('REQUESTING FILE: ' + pathname + '/' + f.split('.')[0].split(' ')[1] + ' FILE SIZE: '
+ f.split(' ')[0] + '\n' + err.__str__())
elif 'SCI' in f:
argsFile = shlex.split(
'/home/user1/client/bin/fs_client --arc tar')
p2 = subprocess.Popen(argsFile, stderr=subprocess.PIPE)
out, err = p2.communicate()
logger.warning('REQUESTING FILE: ' + pathname + '/' + f.split('.')[0].split(' ')[1] + ' FILE SIZE: '
+ f.split(' ')[0] + '\n' + err.__str__())

Related

Python - Path/Folder/File creation

I am running the following block of code to create the path to a new file:
# Opens/create the file that will be created
device_name = target_device["host"].split('.')
path = "/home/user/test_scripts/configs/" + device_name[-1] + "/"
print(path)
# Check if path exists
if not os.path.exists(path):
os.makedirs(path)
# file = open(time_now + "_" + target_device["host"] + "_config.txt", "w")
file = open(path + time_now + "_" + device_name[0] + "_config.txt", "w")
# Time Stamp File
file.write('\n Create on ' + now.strftime("%Y-%m-%d") +
' at ' + now.strftime("%H:%M:%S") + ' GMT\n')
# Writes output to file
file.write(output)
# Close file
file.close()
The code run as intended with the exception that it creates and saves the files on the directory: /home/user/test_scripts/configs/ instead on the indented one that should be: /home/user/test_scripts/configs/device_name[-1]/.
Please advise.
Regards,
./daq
Try using os.path.join(base_path, new_path) [Reference] instead of string concatenation. For example:
path = os.path.join("/home/user/test_scripts/configs/", device_name[-1])
os.makedirs(path, exist_ok=True)
new_name = time_now + "_" + device_name[0] + "_config.txt"
with open(os.path.join(path, new_name), "w+") as file:
file.write("something")
Although I don't get why you're creating a directory with device_name[-1] and as a file name using device_name[0].

Creating multipage tif in Freeimagepy, memory not being freed

I have created a program to merge TIFs into multipage tifs with an rather old version of FreeImagePy from 2009. It actually works pretty well but I have one little hitch. As best I can tell it's not freeing up the memory and eventually crashes. Can anyone tell me what I am missing?
Using Ptyhon 2.7.
import urllib
import FreeImagePy as FIPY
import os.path
import time
# set source files
sourceFile = open('C:\\temp\AGetStatePlats\\PlatsGenesee.csv','r')
# logfile currently gets totally replaced at each run
logFile = open('C:\\temp\\AGetStatePlats\\PlatTifMerge.txt','w')
sourcePath = "c:\\temp\subdownload2\\"
destPath = 'C:\\temp\\subtifMerge\\'
for row in sourceFile:
# prepare filenames
subPath = row.split(',',1)[0]
platID = subPath.split('/',1)[1]
curDocument = sourcePath + platID + '01.tif'
curPage = 0
# check if base file is out there
if not os.path.isfile(destPath + platID + '.tif'):
outImage = FIPY.Image()
outImage.new(multiBitmap = destPath + platID +'.tif')
print (destPath + platID +'.tif')
for n in range (1,100):
#build n
nPad = "{:0>2}".format(n)
if os.path.isfile(sourcePath + platID + nPad + '.tif'):
# delay in case file system is not keeping up may not be needed
time.sleep (1.0/4.0)
outImage.appendPage(sourcePath + platID + nPad + '.tif')
curPage = curPage + 1
else:
outImage.deletePage(0)
outImage.close()
del outImage
logFile.write(sourcePath + platID + '.tif' + '\r')
logFile.write(platID + " PageCount = " + str(curPage) + '\r' )
break
else:
logFile.write(platID + " already exists" + '\r')
# cleanup
sourceFile.close()
logFile.close()
print("Run Complete!")
Here's the error messages:
C:\temp\subtifMerge\05848.tif ('Error returned. ', 'TIFF', 'Warning: parsing error. Image may be incomplete or contain invalid data !') Traceback (most recent call last): File "C:\temp\AGetStatePlats\PlatTIFcombine.py", line 43, in outImage.appendPage(sourcePath + platID + nPad + '.tif') File "C:\Python27\ArcGIS10.4\lib\site-packages\FreeImagePy\FreeIm??agePy.py", line 2048, in appendPage bitmap = self.genericLoader(fileName) File "C:\Python27\ArcGIS10.4\lib\site-packages\FreeImagePy\FreeIm??agePy.py", line 1494, in genericLoader dib = self.Load(fif, fileName, flag); File "C:\Python27\ArcGIS10.4\lib\site-packages\FreeImagePy\FreeIm??agePy.py", line 188, in Load return self.__lib.Load(typ, fileName, flags)
WindowsError: exception: priviledged instruction >>>

Traverse folder to convert all contained files

I've recently started using ffmpeg with the intention of converting my video library to h265 due to its compression benefits. I would like to run one command and have ffmpeg traverse the folder converting each file, one-by-one into h265. I've checked the documentation Here but I can't get my head around it. Does anybody have a template loop script for me to use?
I have ffmpeg installed on a Linux box and have successfully converted single files but I have around 400 files to convert, hence the looping question.
Thanks in advance.
EDIT:
The files I'm waiting to convert are videos with varying containers. I have bee using the python script below, which I have tweaked to suit my needs but isn't working. I will include the error I'm getting and a link to the original below my code.
import os
import sys
import re
import shutil
import subprocess
__author__ = 'Nikhil'
# Edit options here ##################################################
outmode = 'mp4' #Extension of file
remover = True # Delete original file after conversion complete
accept_ext = 'mp4 mkv avi divx m4v mpeg mpg wmv' #Extensions of video files to convert
ffmpeg_exe = "ffmpeg" #Path to ffmpeg executable
ffprobe_exe = "ffprobe" #Path to ffprobe executable
mkvextract_exe = "mkvextract" #Path to mkvextract executable
video_codec = 'libx265' #Video codec to use
video_type = 'h265' #Name of video codec to check for remux
audio_codec = 'aac' #Audio codec to use
audio_type = 'aac' #Name of audio codec to check for remux
crf = "28" #Video quality for libx264
vbr = '' #Audio quality
extract_subtitle = True #Extract subtitles?
subtitle_languages = "en eng english" #Codes for languages to extract
threads = 0 #Number of threads to use in ffmpeg, 0 defaults to all
additional_ffmpeg = '-preset slow -movflags +faststart' #Additional flags for ffmpeg, preset sets speed and compression, movflags to make file web optimized
## END OPTIONS - DO NOT EDIT BELOW THIS LINE UNLESS YOU KNOW WHAT YOU ARE DOING ##
outformat = 'mp4'
if outmode == 'mp4':
outformat = 'mp4'
elif outmode == 'mkv':
outformat = 'matroska'
def ffmpeg(*args, **kwargs):
largs = [ffmpeg_exe, ]
largs.extend(args)
try:
return subprocess.check_output(largs, **kwargs).decode('utf-8')
except:
return None
def getoutput(cmd):
if sys.version < '3':
try:
return subprocess.check_output(cmd.split(' '))
except:
return None
else:
return subprocess.getoutput(cmd)
formats = ""
if getoutput(ffmpeg_exe + ' -formats'):
formats = getoutput(ffmpeg_exe + ' -formats 2')
else:
exit(1)
if ('E mp4' in formats) and ('E matroska' in formats):
print("You have the suitable formats")
else:
print("You do not have both the mkv and mp4 formats...Exiting!")
exit(1)
codecs = getoutput(ffmpeg_exe + ' -codecs 2')
if video_codec in codecs:
print("Check " + video_codec + " Audio Encoder ... OK")
else:
print("Check " + video_codec + " Audio Encoder ... NOK")
exit(1)
if audio_codec in codecs:
print("Check " + audio_codec + " Audio Encoder ... OK")
else:
print("Check " + audio_codec + " Audio Encoder ... NOK")
exit(1)
print("Your FFMpeg is OK\nEntering File Processing\n")
subtitle_languages = subtitle_languages.lower()
def process_file(path, file):
extension = os.path.splitext(file)[1].replace(".", "")
filename = os.path.splitext(file)[0]
if extension in accept_ext:
print(file + " is an acceptable extension. Checking file...")
else:
print(file + " is not an acceptable extension. Skipping...")
return
if ffprobe_exe:
file_info = getoutput('"' + ffprobe_exe + '"' + " " + '"' + os.path.join(path, file) + '"')
else:
file_info = ffmpeg("-i", os.path.join(path, file))
if 'Invalid data found' in file_info:
print("File " + file + " is NOT A VIDEO FILE cannot be converted!")
return
encode_crf = []
if file_info.find("Video: " + video_type) != -1:
vcodec = 'copy'
print("Video is " + video_type + ", remuxing....")
else:
vcodec = video_codec
if crf:
encode_crf = ["-crf", "" + crf]
print("Video is not " + video_type + ", converting...")
encode_vbr = []
if "Audio: " + audio_type in file_info:
acodec = 'copy'
print("Audio is " + audio_type + ", remuxing....")
else:
acodec = audio_codec
if vbr:
encode_vbr = ["-vbr", "" + vbr]
print("Audio is not " + audio_type + ", converting...")
if extension == outmode and vcodec == 'copy' and acodec == 'copy':
print(file + " is already " + outmode + " and no conversion needed. Skipping...")
return
print(
"Using video codec: " + vcodec + " audio codec: " + acodec + " and Container format " + outformat + " for\nFile: " + file + "\nStarting Conversion...\n")
filename = filename.replace("XVID", video_type)
filename = filename.replace("xvid", video_type)
try:
args = ['-i', os.path.join(path, file), '-y', '-f', outformat, '-acodec', acodec]
if encode_vbr:
args.extend(encode_vbr)
args.extend(['-vcodec', vcodec])
if encode_crf:
args.extend(encode_crf)
if additional_ffmpeg:
args.extend(additional_ffmpeg.split(" "))
if threads:
args.extend(['-threads', str(threads)])
args.append(os.path.join(path, filename + '.temp'))
ffmpeg(*args)
print("")
except Exception as e:
print("Error: %s" % e)
print("Removing temp file and skipping file")
if os.path.isfile(os.path.join(path, filename + '.temp')):
os.remove(os.path.join(path, filename + '.temp'))
return
if extract_subtitle and (file_info.find("Subtitle:") != -1):
print("Extracting Subtitles")
matches = re.finditer("Stream #(\d+):(\d+)\((\w+)\): Subtitle: (.*)", file_info)
for m in matches:
if m.group(3).lower() not in subtitle_languages.split(" "):
continue
try:
if 'subrip' in m.group(4):
sub_format = 'copy'
sub_ext = '.srt'
elif mkvextract_exe and 'hdmv_pgs' in m.group(4):
subprocess.check_output([mkvextract_exe, 'tracks', os.path.join(path, file),
m.group(2) + ':' + os.path.join(path, filename + '.' + m.group(
3) + '.' + m.group(2) + '.sup')])
continue
else:
sub_format = 'srt'
sub_ext = '.srt'
ffmpeg("-i", os.path.join(path, file), '-y', '-map', m.group(1) + ':' + m.group(2), '-c:s:0',
sub_format,
os.path.join(path, filename + '.' + m.group(3) + '.' + m.group(2) + sub_ext))
print("")
except Exception as e:
print("Error: %s" % e)
print("Deleting subtitle.")
if os.path.isfile(os.path.join(path, filename + '.' + m.group(3) + '.' + m.group(2) + sub_ext)):
os.remove(os.path.join(path, filename + '.' + m.group(3) + '.' + m.group(2) + sub_ext))
if remover:
print("Deleting original file: " + file)
os.remove(os.path.join(path, file))
if outmode == extension:
shutil.move(os.path.join(path, filename + ".temp"), os.path.join(path, filename + ".enc." + outmode))
filename += ".enc"
else:
shutil.move(os.path.join(path, filename + ".temp"), os.path.join(path, filename + "." + outmode))
def process_directory(path):
if os.path.isfile(os.path.join(path, ".noconvert")):
return
for file in os.listdir(path):
filepath = os.path.join(path, file)
if os.path.isdir(filepath):
process_directory(filepath)
elif os.path.isfile(filepath):
process_file(path, file)
for arg in sys.argv[1:]:
if os.path.isdir(arg):
process_directory(arg)
elif os.path.isfile(arg):
process_file(os.path.dirname(arg), os.path.basename(arg))
The error I am getting is this:
Traceback (most recent call last):
File "/media/569f/ethan1878/bin/convert.py", line 209, in <module>
process_file(os.path.dirname(arg), os.path.basename(arg))
File "/media/569f/ethan1878/bin/convert.py", line 100, in process_file
if 'Invalid data found' in file_info:
TypeError: argument of type 'NoneType' is not iterable
and the original file is hosted Here (as a .txt file)

How to encrypt a .bin file

I need to encrypt 3 .bin files which contain 2 keys for Diffie-Hellman. I have no clue how to do that, all I could think of was what I did in the following Python file. I have an example what the output should look like but my code doesn't seem to produce the right keys. The output file server.ini is used by a client to connect to a server.
import base64
fileList = [['game_key.bin', 'Game'], ['gate_key.bin', 'Gate'], ['auth_key.bin', 'Auth']]
iniList = []
for i in fileList:
file = open(i[0], 'rb')
n = list(file.read(64))
x = list(file.read(64))
file.close()
n.reverse()
x.reverse()
iniList.append(['Server.' + i[1] + '.N "' + base64.b64encode("".join(n)) + '"\n', 'Server.' + i[1] + '.X "' + base64.b64encode("".join(x)) + '"\n'])
iniList[0].append('\n')
#time for user Input
ip = '"' + raw_input('Hostname: ') + '"'
dispName = 'Server.DispName ' + '"' + raw_input('DispName: ') + '"' + '\n'
statusUrl = 'Server.Status ' + '"' + raw_input('Status URL: ') + '"' + '\n'
signupUrl = 'Server.Signup ' + '"' + raw_input('Signup URL: ') + '"' + '\n'
for l in range(1, 3):
iniList[l].append('Server.' + fileList[l][1] + '.Host ' + ip + '\n\n')
for l in [[dispName], [statusUrl], [signupUrl]]:
iniList.append(l)
outFile = open('server.ini', 'w')
for l in iniList:
for i in l:
outFile.write(i)
outFile.close()
The following was in my example file:
# Keys are Base64-encoded 512 bit RC4 keys, as generated by DirtSand's keygen
# command. Note that they MUST be quoted in the commands below, or the client
# won't parse them correctly!
I also tried it without inverting n and x

Trouble with apostrophe in arcpy search cursor where clause

I've put together a tkinter form and python script for downloading files from an ftp site. The filenames are in the attribute table of a shapefile, as well as an overall Name that the filenames correspond too. In other words I look up a Name such as "CABOT" and download the filename 34092_18.tif. However, if a Name has an apostrophe, such as "O'KEAN", it's giving me trouble. I try to replace the apostrophe, like I've done in previous scripts, but it doesn't download anything....
whereExp = quadField + " = " + "'" + quadName.replace("'", '"') + "'"
quadFields = ["FILENAME"]
c = arcpy.da.SearchCursor(collarlessQuad, quadFields, whereExp)
for row in c:
tifFile = row[0]
tifName = quadName.replace("'", '') + '_' + tifFile
#fullUrl = ftpUrl + tifFile
local_filename = os.path.join(downloadDir, tifName)
lf = open(local_filename, "wb")
ftp.retrbinary('RETR ' + tifFile, lf.write)
lf.close()
Here is an example of a portion of a script that works fine by replacing the apostrophe....
where_clause = quadField + " = " + "'" + quad.replace("'", '"') + "'"
#out_quad = quad.replace("'", "") + ".shp"
arcpy.MakeFeatureLayer_management(quadTable, "quadLayer")
select_out_feature_class = arcpy.SelectLayerByAttribute_management("quadLayer", "NEW_SELECTION", where_clause)

Categories