We Keep Coding

How can i optimize my Python loop for speed

I wrote some code that uses OCR to extract text from screenshots of follower lists and then transfer them into a data frame.
The reason I have to do the hustle with "name" / "display name" and removing blank lines is that the initial text extraction looks something like this:
Screenname 1
name 1
Screenname 2
name 2
(and so on)
So I know in which order each extraction will be.
My code works well for 1-30 images, but if I take more than that its gets a bit slow. My goal is to run around 5-10k screenshots through it at once. I'm pretty new to programming so any ideas/tips on how to optimize the speed would be very appreciated! Thank you all in advance :)
from PIL import Image
from pytesseract import pytesseract
import os
import pandas as pd
from itertools import chain
list_final = [""]
list_name = [""]
liste_anzeigename = [""]
list_raw = [""]
anzeigename = [""]
name = [""]
sort = [""]
f = r'/Users/PycharmProjects/pythonProject/images'
myconfig = r"--psm 4 --oem 3"
os.listdir(f)
for file in os.listdir(f):
f_img = f+"/"+file
img = Image.open(f_img)
img = img.crop((240, 400, 800, 2400))
img.save(f_img)
for file in os.listdir(f):
f_img = f + "/" + file
test = pytesseract.image_to_string(PIL.Image.open(f_img), config=myconfig)
lines = test.split("\n")
list_raw = [line for line in lines if line.strip() != ""]
sort.append(list_raw)
name = {list_raw[0], list_raw[2], list_raw[4],
list_raw[6], list_raw[8], list_raw[10],
list_raw[12], list_raw[14], list_raw[16]}
list_name.append(name)
anzeigename = {list_raw[1], list_raw[3], list_raw[5],
list_raw[7], list_raw[9], list_raw[11],
list_raw[13], list_raw[15], list_raw[17]}
liste_anzeigename.append(anzeigename)
reihenfolge_name = list(chain.from_iterable(list_name))
index_anzeigename = list(chain.from_iterable(liste_anzeigename))
sortieren = list(chain.from_iterable(sort))
print(list_raw)
sort_name = sorted(reihenfolge_name, key=sortieren.index)
sort_anzeigename = sorted(index_anzeigename, key=sortieren.index)
final = pd.DataFrame(zip(sort_name, sort_anzeigename), columns=['name', 'anzeigename'])
print(final)

Use a multiprocessing.Pool.
Combine the code under the for-loops, and put it into a function process_file.
This function should accept a single argument; the name of a file to process.
Next using listdir, create a list of files to process.
Then create a Pool and use its map method to process the list;
import multiprocessing as mp
def process_file(name):
# your code goes here.
return anzeigename # Or watever the result should be.
if __name__ is "__main__":
f = r'/Users/PycharmProjects/pythonProject/images'
p = mp.Pool()
liste_anzeigename = p.map(process_file, os.listdir(f))
This will run your code in parallel in as many cores as your CPU has.
For a N-core CPU this will take approximately 1/N times the time as doing it without multiprocessing.
Note that the return value of the worker function should be pickleable; it has to be returned from the worker process to the parent process.

How do you delete a virtual disk with pyvmomi

I am trying to write a Python program using the pyvmomi library to "erase" a virtual hard drive associated with a VM. The way this is done manually is to remove the virtual disk and create a new virtual disk with the same specs. I am expecting that I will need to do the same thing with pyvmomi so I have started down that path. My issue is that I can use ReconfigVM_Task to remove the virtual drive but that leaves the VMDK file itself.
I originally tried using DeleteVStorageObject_Task (since DeleteVirtualDisk_Task is deprecated) to remove the virtual disk file but that requires the ID of the object (the VMDK file) which I am unable to find anywhere. Theoretically that's available from the VirtualDisk property vDiskId but that is null. In further research it seems to only be populated for first class disks.
So I am instead trying to delete the VMDK file directly using DeleteDatastoreFile_Task but when I do that I end up with a XXXX-flat.vmdk file in the datastore so it seems to not actually delete the file.
Any idea on where I'm going wrong here or how to better do this? The VMWare SDK documentation for pyvmomi is...lacking.
Thanks!

You'll have to perform a ReconfigVM_Task operation. The keypoint for this is that the file operation should be destroy. Here's the raw output from performing the operation in the UI:
spec = vim.vm.ConfigSpec()
spec_deviceChange_0 = vim.vm.device.VirtualDeviceSpec()
spec_deviceChange_0.fileOperation = 'destroy'
spec_deviceChange_0.device = vim.vm.device.VirtualDisk()
spec_deviceChange_0.device.shares = vim.SharesInfo()
spec_deviceChange_0.device.shares.shares = 1000
spec_deviceChange_0.device.shares.level = 'normal'
spec_deviceChange_0.device.capacityInBytes = 8589934592
spec_deviceChange_0.device.storageIOAllocation = vim.StorageResourceManager.IOAllocationInfo()
spec_deviceChange_0.device.storageIOAllocation.shares = vim.SharesInfo()
spec_deviceChange_0.device.storageIOAllocation.shares.shares = 1000
spec_deviceChange_0.device.storageIOAllocation.shares.level = 'normal'
spec_deviceChange_0.device.storageIOAllocation.limit = -1
spec_deviceChange_0.device.storageIOAllocation.reservation = 0
spec_deviceChange_0.device.backing = vim.vm.device.VirtualDisk.FlatVer2BackingInfo()
spec_deviceChange_0.device.backing.backingObjectId = ''
spec_deviceChange_0.device.backing.fileName = '[kruddy_2TB_01] web01/web01_2.vmdk'
spec_deviceChange_0.device.backing.split = False
spec_deviceChange_0.device.backing.writeThrough = False
spec_deviceChange_0.device.backing.datastore = search_index.FindByUuid(None, "datastore-14", True, True)
spec_deviceChange_0.device.backing.eagerlyScrub = True
spec_deviceChange_0.device.backing.contentId = 'e26f44020e7897006bec81b1fffffffe'
spec_deviceChange_0.device.backing.thinProvisioned = False
spec_deviceChange_0.device.backing.diskMode = 'persistent'
spec_deviceChange_0.device.backing.digestEnabled = False
spec_deviceChange_0.device.backing.sharing = 'sharingNone'
spec_deviceChange_0.device.backing.uuid = '6000C292-7895-54ee-a55c-49d0036ef1bb'
spec_deviceChange_0.device.controllerKey = 200
spec_deviceChange_0.device.unitNumber = 0
spec_deviceChange_0.device.nativeUnmanagedLinkedClone = False
spec_deviceChange_0.device.capacityInKB = 8388608
spec_deviceChange_0.device.deviceInfo = vim.Description()
spec_deviceChange_0.device.deviceInfo.summary = '8,388,608 KB'
spec_deviceChange_0.device.deviceInfo.label = 'Hard disk 2'
spec_deviceChange_0.device.diskObjectId = '148-3000'
spec_deviceChange_0.device.key = 3000
spec_deviceChange_0.operation = 'remove'
spec.deviceChange = [spec_deviceChange_0]
spec.cpuFeatureMask = []
managedObject.ReconfigVM_Task(spec)

Kyle Ruddy got me pointed in the right direction. Here's a code snippit showing how I made it work for future people searching for information on how to do this:
#Assuming dev is already set to the vim.vm.device.VirtualDisk you want to delete...
virtual_hdd_spec = vim.vm.device.VirtualDeviceSpec()
virtual_hdd_spec.fileOperation = vim.vm.device.VirtualDeviceSpec.FileOperation.destroy
virtual_hdd_spec.operation = vim.vm.device.VirtualDeviceSpec.Operation.remove
virtual_hdd_spec.device = dev
spec = vim.vm.ConfigSpec()
spec.deviceChange = [virtual_hdd_spec]
WaitForTask(vm.ReconfigVM_Task(spec=spec))
The API documentation for this is at https://pubs.vmware.com/vi3/sdk/ReferenceGuide/vim.vm.device.VirtualDeviceSpec.html

Unable to display all the information except for first selection

I am using the following code to process a list of images that is found in my scene, before the gathered information, namely the tifPath and texPath is used in another function.
However, example in my scene, there are 3 textures, and hence I should be seeing 3 sets of tifPath and texPath but I am only seeing 1 of them., whereas if I am running to check surShaderOut or surShaderTex I am able to see all the 3 textures info.
For example, the 3 textures file path is as follows (in the surShaderTex): /user_data/testShader/textureTGA_01.tga, /user_data/testShader/textureTGA_02.tga, /user_data/testShader/textureTGA_03.tga
I guess what I am trying to say is that why in my for statement, it is able to print out all the 3 results and yet anything bypass that, it is only printing out a single result.
Any advices?
surShader = cmds.ls(type = 'surfaceShader')
for con in surShader:
surShaderOut = cmds.listConnections('%s.outColor' % con)
surShaderTex = cmds.getAttr("%s.fileTextureName" % surShaderOut[0])
path = os.path.dirname(surShaderTex)
f = surShaderTex.split("/")[-1]
tifName = os.path.splitext(f)[0] + ".tif"
texName = os.path.splitext(f)[0] + ".tex"
tifPath = os.path.join(path, tifName)
texPath = os.path.join(path, texName)
convertText(surShaderTex, tifPath, texPath)

Only two lines are part of your for loop. The rest only execute once.
So first this runs:
surShader = cmds.ls(type = 'surfaceShader')
for con in surShader:
surShaderOut = cmds.listConnections('%s.outColor' % con)
surShaderTex = cmds.getAttr("%s.fileTextureName" % surShaderOut[0])
Then after that loop, with only one surShader, one surShaderOut, and one surShaderTex, the following is executed once:
path = os.path.dirname(surShaderTex)
f = surShaderTex.split("/")[-1]
tifName = os.path.splitext(f)[0] + ".tif"
texName = os.path.splitext(f)[0] + ".tex"
tifPath = os.path.join(path, tifName)
texPath = os.path.join(path, texName)
Indent that the same as the lines above it, and it'll be run for each element of surShader instead of only once.

How do I get Boto to return EC2 instances - S3 works fine

I'm having some issues with the EC2 bit of Boto (Boto v2.8.0, Python v2.6.7).
The first command returns a list of S3 Buckets - all good! The second command to get a list of EC2 instances blows up with a 403 with "Query-string authentication requires the Signature, Expires and AWSAccessKeyId parameters"
s3_conn = S3Connection(AWSAccessKeyId, AWSSecretKey)
print s3_conn.get_all_buckets()
ec2_conn = EC2Connection(AWSAccessKeyId, AWSSecretKey)
print ec2_conn.get_all_instances()
Also, my credentials are all good (Full admin) - I tested them using the Ruby aws-sdk, both EC2 and S3 work fine.
I also noticed that the host attribute in the ec2_conn object is s3-eu-west-1.amazonaws.com, "s3"...? Surely thats wrong? I've tried retro fixing it to the correct endpoint but no luck.
Any help would be great appreciate
Thanks

Here's some working code I use to list all my instances across potentially multiple regions.
Its doing a lot more than you need, but maybe you can pare it down to what you want.
#!/usr/bin/python
import boto
import boto.ec2
import sys
class ansi_color:
red = '\033[31m'
green = '\033[32m'
reset = '\033[0m'
grey = '\033[1;30m'
def name(i):
if 'Name' in i.tags:
n = i.tags['Name']
else:
n = '???'
n = n.ljust(16)[:16]
if i.state == 'running':
n = ansi_color.green + n + ansi_color.reset
else:
n = ansi_color.red + n + ansi_color.reset
return n
def pub_dns( i ):
return i.public_dns_name.rjust(43)
def pri_dns( i ):
return i.private_dns_name.rjust(43)
def print_instance( i ):
print ' ' + name(i) + '| ' + pub_dns(i) + ' ' + pri_dns(i)
regions = sys.argv[1:]
if len(regions)==0:
regions=['us-east-1']
if len(regions)==1 and regions[0]=="all":
rr = boto.ec2.regions()
else:
rr = [ boto.ec2.get_region(x) for x in regions ]
for reg in rr:
print "========"
print reg.name
print "========"
conn = reg.connect()
reservations = conn.get_all_instances()
for r in reservations:
# print ansi_color.grey + str(r) + ansi_color.reset
for i in r.instances:
print_instance(i)

There is the connect_to_region command:
import boto.ec2
connection = boto.ec2.connect_to_region('eu-west-1', aws_access_key_id=AWSAccessKeyId,
aws_secret_access_key=AWSSecretKey)
The Boto tutorial gives another way. That method would basically work like this:
import boto.ec2
for region in boto.ec2.regions():
if region.name == 'my-favorite-region':
connection = region.connect()
break
This has not been working on older versions of Boto.

Do you have your IAM credentials in order? The given access key should have rights for EC2. If you're not sure, you can add the policy AmazonEC2FullAccess to test, and later tune this down.

How can I count number of Window with specific title on Linux by Python?

Here is my solution, but I think it's not a good one. Can anybody suggest me new solution, I want to enumarize windows, and get the Title property to compare by just using core library of python , not wnck or pygtk ...
def linux_CountWindowsByTitle(title):
import commands
XWinInfoOutput = commands.getoutput("xwininfo -all -root")
resArray = XWinInfoOutput.split("\n")
resRange = len(resArray) - 1
res = 0
#Parse each line of output
for i in range(0, resRange):
index = resArray[i].find("\"") #Get index of Double quote
if (index < 0):
continue #This line does not have title we need
tmp = resArray[i].split("\": (")[0] #Remove tail
windowName = tmp.split("\"",1)[1] #Remove head
if (UTILITY.Compare(title, windowName)):
#LIBRARY.Report(windowName)
res += 1
return res

You can use the module wnck.
import wnck
screen = wnck.screen_get_default()
window_list = wnck.Screen.get_windows(screen)
window_names = [ w.get_name() for w in window_list ]
In order to count the similar windows, you can create a dictionary:
count = window_names.count
wcounts = { item: item.count(item) for item in set(window_names) }
Where the dictionary will have as key the window title and the value will be the number of times the same name is repeated.
Slightly different, but you might find interesting to use is:
wdict = { w.get_name(): w for w in window_list }
wdict.has_key(title)
If you need the window later to other processing, you still have a reference handy in the wdict. For instance, you can check properties, maximize it, minimize, and all typical operations that a window manager would do.
Note: For newer versions of wnck (>=3.0) you have to use PyGObject (GObject Introspection), but you get the idea.