We Keep Coding

Automation of program execution on linux

I have to perform several experiments in order to analyse certain results for a class assignment. For each result, I have to run a line of code by varying the parameters from terminal. I was wondering if there is a way to automate this and thus save me the trouble of running the program and changing the values each time. Attached is the line of code that I have to execute each time:
teaa/examples/scripts/run-python.sh teaa/examples/python/rf_mnist.py \
--numTrees 10 --maxDepth 7 --pcaComponents 40
I would like these lines of code to be executed n times automatically and that in each execution, the parameters 'numTrees, maxDepth and pcaComponents' change in a set range of values.
I have not really tried any solutions yet. I have never programmed in terminal and I have no idea where to start.

For more involved process operations, one should use the subprocess module in stdlib Python... but if you just want a one-off run of this command line, you can just use plain old os.system function - The followin code:
import os
for nt in range(1,21):
for md in range(1,11):
for pc in range(20,100,10):
os.system(f'teaa/examples/scripts/run-python.sh teaa/examples/python/rf_mnist.py --numTrees {nt} --maxDepth {md} --pcaComponents {pc}')
will run your program for numTrees = 1,2...20, maxDepth = 1,..10 and pcaComponents = 20,...90

I created a python bind for CS:GO with pyautogui but it doesnt work

I created a simple bind script. It works on IDLE Python but it doesn't work in CS:GO. Do you know why?
Mayby it must be on background to work?
import keyboard
import pyautogui
import time
def EventListen():
while True:
try:
if keyboard.is_pressed('n'):
pyautogui.press('`')
pyautogui.typewrite('say EZ')
pyautogui.press('enter')
pyautogui.press('`')
EventListen()
except:
EventListen()
EventListen()

I don't see the need to use pyautogui since you are already using keyboard which is sufficient to perform the tasks you need. I have made some changes to your code
import time
import keyboard
def EventListen():
while True:
try:
if keyboard.is_pressed('n'):
keyboard.press('`')
keyboard.write('say EZ')
keyboard.press('enter')
keyboard.press('`')
elif keyboard.is_pressed('/'): #add something to end the process
break
except:
EventListen()
time.sleep(0.001)
EventListen()
There is no need to call the function in the while loop, as it will anyway be executed infinitely unless you kill the process. I don't see why the script wouldn't run in the background, in fact I am typing this
n`say EZ
`
using the script. What might be possible is that your previous program ran continuously, causing high CPU usage which might have competed with the game's demand. I recomend you to add a small delay before every iteration of the while loop, in this case I have added 1 ms delay, which will cause significant reduction in CPU usage. I am not sure if that solved your problem as I am unable to reproduce your exact case, let me know if it helped.
EDIT : I forgot to mention, I have added another binding of keyboard.is_pressed('/') which will make the program break out of the loop and hence terminate it when / key is pressed. You can change this as you like. If you don't want any other binding as such (which I don't recommend) then you can rely on manually killing the task.

you should make an exe with pyinstaller and you run it background

python simple threading won't ends without doing anything (maybe)

When i run the following code (using "sudo python servers.py") the process seem to just finish immediately with just printing "test".
why doesn't the functions "proxy_server" won't run ? or maybe they do but i do not realize that. (because the first line in proxy function doesn't print anything)
this is an impotent code, i didn't want to put unnecessary content, yet it still demonstrate my problem:
import os,sys,thread,socket,select,struct,time
HTTP_PORT = 80
FTP_PORT=21
FTP_DATA_PORT = 20
IP_IN = '10.0.1.3'
IP_OUT = '10.0.3.3'
sys_http = 'http_proxy'
sys_ftp = 'ftp_proxy'
sys_ftp_data = 'ftp_data_proxy'
def main():
try:
thread.start_new_thread(proxy_server, (HTTP_PORT, IP_IN,sys_http,http_handler))
thread.start_new_thread(proxy_server, (FTP_PORT, IP_IN,sys_ftp,http_handler))
thread.start_new_thread(proxy_server, (FTP_DATA_PORT, IP_OUT,sys_ftp_data,http_handler))
print "test"
except e:
print 'Error!'
sys.exit(1)
def proxy_server(host,port,fileName,handler):
print "Proxy Server Running on ",host,":",port
def http_handler(src,sock):
return ''
if __name__ == '__main__':
main()
What am i missing or doing wrong ?

First, you have indentation problems related to using mixed tabs and spaces for indentation. While they didn't cause your code to misbehave in this particular case, they will cause you problems later if you don't stick to consistently using one or the other. They've already broken the displayed indentation in your question; see the print "test" line in main, which looks misaligned.
Second, instead of the low-level thread module, you should be using threading. Your problem is occurring because, as documented in the thread module documentation,
When the main thread exits, it is system defined whether the other threads survive. On SGI IRIX using the native thread implementation, they survive. On most other systems, they are killed without executing try ... finally clauses or executing object destructors.
threading threads let you explicitly define whether other threads should survive the death of the main thread, and default to surviving. In general, threading is much easier to use correctly.

Execute python code without invoking import statement each time

Here is a sample python script. How do I run this script multiple times from command line so that the import line is not called every time? The import statement takes too long to load.
import arcpy
val = arcpy.GetCellValue_management("D:\dem-merged\lidar_wsg84", "-95.090174910630012 29.973962146120652", "")
print str(val)

This problem has no solution if you strictly want this script "to be called from another program. by issuing 'python script.py' on command line".
If you want to do the "heavy import" only once, you have to start python script only once.
Think about starting a daemon, which will start once and then process calls from other program. This way all initialization has to be done only one time and next calls will be fast.
And if you split your python code into two parts (first part for daemon, second for daemon client), you'll be able to call 'python client.py' from another program, but actual computation will be performed by daemon, which is started just one time.
As example:
daemon.py
import socket
#import arcpy
def actual_work():
#val = arcpy.GetCellValue_management("D:\dem-merged\lidar_wsg84", "-95.090174910630012 29.973962146120652", "")
#return str(val)
return 'dummy_reply'
def main():
sock = socket.socket( socket.AF_INET, socket.SOCK_DGRAM )
try:
sock.bind( ('127.0.0.1', 6666) )
while True:
data, addr = sock.recvfrom( 4096 )
reply = actual_work()
sock.sendto(reply, addr)
except KeyboardInterrupt:
pass
finally:
sock.close()
if __name__ == '__main__':
main()
client.py
import socket
import sys
def main():
sock = socket.socket( socket.AF_INET, socket.SOCK_DGRAM )
sock.settimeout(1)
try:
sock.sendto('', ('127.0.0.1', 6666))
reply, _ = sock.recvfrom(4096)
print reply
except socket.timeout:
sys.exit(1)
finally:
sock.close()
if __name__ == '__main__':
main()

It's virtually impossible. Once you leave the interpreter, the modules that were imported are no longer in the memory. It's similar to asking Firefox to save large webpages in memory because the read rate to the cache takes too long. Once Firefox (or Python) is shut off, it's pretty much bye-bye anything in the RAM.
You can make the load time faster, but at your own risk. By running
python -O
you can make it go a bit faster. You can also add another 'O' to make it go just a bit faster. However, this can make some programs buggy and doesn't always work.
You could copy the functions you need into your program by doing
from arcpy import <what you need>
and that might make things go slightly faster.

As far as I know the module gets imported once. So if you do:
import a
import a
it only gets imported once. So instead of running the script many times, maybe you can change it to make all the copies in one go.
If you have to run this specific script many times, I think you can't avoid the import and you'll have to import it every time.

One solution I can think of is to have a server process that runs persistently that does the actual work, while the script that's actually invoked from the command line merely issues requests to that script. This is a fair bit of work, but it may be worth it.

The only solution I can think of is to copy the individual function(s) you need into your code manually, if what you need to execute is small enough.
If you need help on how to do this, just ask in the comments.

Looking at your use case (calling it from a Ruby on Rails webservice), one of the easiest ways would be to use XML-RPC. Use the SimpleXMLRPCServer from the python standard lib, and then use a ruby client (ruby seems to have xmlrpc in the standard lib)?

Easy.
Write your own simple shell using the cmd module and use the runpy module to run your scripts. Import you big module in the shell program and pass it to the programs using init_globals
Look through the docs for http://pypi.python.org/pypi/cmd2/ and it should be fairly clear how you can write your own simple shell, even if it just has two commands, one to edit a file and one to run it.
runpy is part of the Python standard library http://docs.python.org/library/runpy.html and you may not need it, but it is useful to know that the import and module loading mechanism can be controlled and even modified by your command shell.
Have you ever wondered where the name "var1" goes when you execute something like var1 = 25? How does Python find what var1 refers to when you later execute print var1? The answer is that these names are in a dictionary and if you understand what Python dictionaries are and what they can do, it seems like an obvious solution to the problem of connecting names with values. But there's more. Python can have lots of namespaces and you can manipulate those namespaces the same way you manipulate dictionaries. Read this http://www.diveintopython.net/html_processing/locals_and_globals.html to understand the locals and globals namespace. Here is another discussion that will help http://lucumr.pocoo.org/2011/2/1/exec-in-python/
Play around with exec like in this question globals and locals in python exec() until you understand how it works. Then build your command shell to import the module one time at the beginning, and write your scripts to only import the module if it is not already available. When the script is run from inside your shell, the module will already be there.

Get a Global Variable while a Python Program is running

I write a Python 2.5 Command Line program on CentOS 5.5 and it has been running for 1 day and is still running. Now I want to end this program but get the value of a global variable.
I have done some Google. It seems the only way to get the value of a global variable is to attach the Python program to a GDB.
Suppose the global variable is a List, and its name is resultlist. How can I get its value?

This is possible to do, but is extremely tricky, and a single wrong move will cause the program to crash (or be lost in an indeterminate state).
You can use the C function PyEval_GetGlobals() to return the dictionary of globals (as it would if you called globals() in python), and then use PyObject_Print() to print that object to a file (the easiest being whatever stdout is connected to).
You'll want to run GDB and attach it to the instance of python. Then set a breakpoint on a function you know will be called (if your program is printing output, then PyObject_Print() will work; otherwise this page has some functions that probably get called a lot.), then when the program hits the breakpoint, you'll want to disable it, and print the globals.
For example, if my Python program has a PID of 15847:
(gdb) attach 15847
Attaching to process 15847.
Reading symbols for shared libraries . done
Reading symbols for shared libraries ............. done
0x00007fff870b5e52 in select$DARWIN_EXTSN ()
(gdb) break PyObject_Print
Breakpoint 1 at 0x10003d8f4
(gdb) c
Continuing.
The next time your program goes to print something:
Breakpoint 1, 0x000000010003d8f4 in PyObject_Print ()
(gdb) disable
(gdb) call (int)PyObject_Print((void*)PyEval_GetGlobals())
$1 = 0
(gdb) c
Continuing.
Then, in the output of your program, you'll see the global dictionary.

I'm afraid names won't help you much. Since you didn't build in some printing mechanism and your process is already running you're pretty much screwed.
If you have an idea what the values might be your best bet would be using a process memory scanner and start messing around. Though I estimate your chances of success very low, I'm sorry.