I've been looking at examples from other people but I can't seem to get it to work properly.
It'll either use a single core, or basically freeze up maya if given too much to process, but I never seem to get more than one core working at once.
So for example, this is kind of what I'd like it to do, on a very basic level. Mainly just let each loop run simultaneously on a different processor with the different values (in this case, the two values would use two processors)
mylist = [50, 100, 23]
newvalue = [50,51]
for j in range(0, len(newvalue)):
exists = False
for i in range(0, len(mylist)):
#search list
if newvalue[j] == mylist[i]:
exists = True
#add to list
if exists == True:
mylist.append(mylist)
Would it be possible to pull this off? The actual code I'm wanting to use it on can take from a few seconds to like 10 minutes for each loop, but they could theoretically all run at once, so I thought multithreading would speed it up loads
Bear in mind I'm still relatively new to python so an example would be really appreciated
Cheers :)
There are really two different answers to this.
Maya scripts are really supposed to run in the main UI thread, and there are lots of ways they can trip you up if run from a separate thread. Maya includes a module called maya.utils which includes methods for deferred evaluation in the main thread. Here's a simple example:
import maya.cmds as cmds
import maya.utils as utils
import threading
def do_in_main():
utils.executeDeferred (cmds.sphere)
for i in range(10):
t = threading.Thread(target=do_in_main, args=())
t.start()
That will allow you to do things with the maya ui from a separate thread (there's another method in utils that will allow the calling thread to await a response too). Here's a link to the maya documentation on this module
However, this doesn't get you around the second aspect of the question. Maya python isn't going to split up the job among processors for you: threading will let you create separate threads but they all share the same python intepreter and the global interpreter lock will mean that they end up waiting for it rather than running along independently.
You can't use the multiprocessing module, at least not AFAIK, since it spawns new mayas rather than pushing script execution out into other processors in the Maya you are running within. Python aside, Maya is an old program and not very multi-core oriented in any case. Try XSI :)
Any threading stuff in Maya is tricky in any case - if you touch the main application (basically, any function from the API or a maya.whatever module) without the deferred execution above, you'll probably crash maya. Only use it if you have to.
And, BTW, you cant use executeDeferred, etc in batch mode since they are implemented using the main UI loop.
What theodox says is still true today, six years later. However one may go another route by spawning a new process by using the subprocess module. You'll have to communicate and share data via sockets or something similar since the new process is in a seperate interpreter. The new interpreter runs on its own and doesn't know about Maya but you can do any other work in it benefitting from the multi-threaded environment your OS provides before communicating it back to your Maya python script.
Related
I'm building a Python IDE, which needs to highlight all occurrences of the name under cursor (using Jedi library). The process of finding the occurrences can be quite slow.
In order to avoid freezing the GUI, I could run the search in another thread, but when the user moves quickly over several words, the background threads could pile up while working on now obsolete tasks. I would like to cancel the search for previous occurrences when user moves to new name.
Looks like killing a thread is complicated in Python. What are the other options for creating an easily cancellable background tasks in Python 3.4+?
I think concurrent.futures is the answer.
You can create a Thread / Process pool, submit any callable, receive a Future, which you can cancel if needed.
Reference: https://docs.python.org/3/library/concurrent.futures.html
A thread cannot be stopped by another one. This is a OS limitation rather than a Python one. Only thing you can do is periodically inspect a variable and, if set, stop the thread itself (just return).
Moreover, threads in Python suffer from the GIL. This means that CPU intensive operations, when carried out in a separate thread, will still affect your main loop as only one thread per process can run at a time.
I'd recommend you to run the search in a separate process which you can easily cancel whenever you want.
What the guys of YouCompleteMe are doing for example is wrapping Jedi in a HTTP server which they can query in the background. If the user moves the cursor before the completion comes back, the IDE can simply drop the request.
Well, my personal favorites are work queues. If it's a one-time application you should take a look at python rq. Extremely easy and fun to use. If you want to build something more "professional-grade" take a look at something like celery.
You might also want to look at multiprocessing
I have a program, which opens a lot of urls and downloads pictures .
I have a function of the program, which manages link's opening and pictures downloading, which contains a for loop and performs some operations on the priority queue. I want to run this function, but no longer than the set time period. For example if this function is running longer than 1 hour I want to terminate it and run the rest of the program (other functions).
I was trying to find some solutions, and I found two question here on stack.
The first solution use only time module First solution
The second use also the multiprocessing module
Second solution. Can some one suggest which one will be more appropriate to use in my program? I will write a pseudocode of my function:
def fun():
for link in linkList:
if link not in queue:
queue.push(link)
else:
queue.updatePriority(link)
if queue:
top = queue.pop()
fun(top)
This function is called in other function:
def run(startLink):
fun(startLink)
And the run() function is called in other module.
Which method is better to use with a program which contains a lot of modules and performs a lot of
The asyncio module is ideal for this task.
You can create a future, then use asyncio.wait which supports a timeout parameter.
Using multiprocessing here would be a little bit tricky, because fun is consuming a priority queue (I'm assuming a Queue.PriorityQueue) that is coming from some other part of the program. That queue cannot easily be passed between processes - you would need to create a custom multiprocessing.BaseManager subclass, register the Queue.PriorityQueue class with it, and start up the Manager server, instantiate a PriorityQueue on the server, and use a Proxy to that instance everywhere you interact with the queue. That's a lot of overhead, and also hurts performance a bit.
Since it appears you don't actually want any concurrency here - you want the rest of the program to stop while fun is running - I don't think there's a compelling reason to use multiprocessing. Instead, I think using the time-based solution makes more sense.
I've run into situations as of late when writing scripts for both Maya and Houdini where I need to wait for aspects of the GUI to update before I can call the rest of my Python code. I was thinking calling time.sleep in both situations would have fixed my problem, but it seems that time.sleep just holds up the parent application as well. This means my script evaluates the exact same regardless of whether or not the sleep is in there, it just pauses part way through.
I have a thought to run my script in a separate thread in Python to see if that will free up the application to still run during the sleep, but I haven't had time to test this yet.
Thought I would ask in the meantime if anybody knows of some other solution to this scenario.
Maya - or more precisely Maya Python - is not really multithreaded (Python itself has a dodgy kind of multithreading because all threads fight for the dread global interpreter lock, but that's not your problem here). You can run threaded code just fine in Maya using the threading module; try:
import time
import threading
def test():
for n in range (0, 10):
print "hello"
time.sleep(1)
t = threading.Thread(target = test)
t.start()
That will print 'hello' to your listener 10 times at one second intervals without shutting down interactivity.
Unfortunately, many parts of maya - including most notably ALL user created UI and most kinds of scene manipulation - can only be run from the "main" thread - the one that owns the maya UI. So, you could not do a script to change the contents of a text box in a window using the technique above (to make it worse, you'll get misleading error messages - code that works when you run it from the listener but errors when you call it from the thread and politely returns completely wrong error codes). You can do things like network communication, writing to a file, or long calculations in a separate thread no problem - but UI work and many common scene tasks will fail if you try to do them from a thread.
Maya has a partial workaround for this in the maya.utils module. You can use the functions executeDeferred and executeInMainThreadWithResult. These will wait for an idle time to run (which means, for example, that they won't run if you're playing back an animation) and then fire as if you'd done them in the main thread. The example from the maya docs give the idea:
import maya.utils import maya.cmds
def doSphere( radius ):
maya.cmds.sphere( radius=radius )
maya.utils.executeInMainThreadWithResult( doSphere, 5.0 )
This gets you most of what you want but you need to think carefully about how to break up your task into threading-friendly chunks. And, of course, running threaded programs is always harder than the single-threaded alternative, you need to design the code so that things wont break if another thread messes with a variable while you're working. Good parallel programming is a whole big kettle of fish, although boils down to a couple of basic ideas:
1) establish exclusive control over objects (for short operations) using RLocks when needed
2) put shared data into safe containers, like Queue in #dylan's example
3) be really clear about what objects are shareable (they should be few!) and which aren't
Here's decent (long) overview.
As for Houdini, i don't know for sure but this article makes it sound like similar issues arise there.
A better solution, rather than sleep, is a while loop. Set up a while loop to check a shared value (or even a thread-safe structure like a Queue). The parent processes that your waiting on can do their work (or children, it's not important who spawns what) and when they finish their work, they send a true/false/0/1/whatever to the Queue/variable letting the other processes know that they may continue.
I cannot seem to find a simple example of how to schedule an event in Python.
I want to be able to pass a date and time string as an argument into a function.
For example:
String: "m/d/Y HH:MM" would set the time for a future function to run, after the code has been executed. So, like a function that is waiting to go off after I run it.
It seems like the main problem is formatting the string correctly, but a simple example would really help to see how to 'schedule' a function to run.
You don give enough context to understand what are you trying to do in a larger frame - but, generally speaking - "this is not how it works" in Python.
An "ordinary" Python program is a single-threaded, synchronous program - it will run one task, after another, after another, when everything is done, the program exits, and the interpreter exits along with it.
so, something along (with a fictitious "schedule" function):
def main():
print("Hello World")
schedule(60, main)
main()
would not work in Python, if the call to schedule would return immediately - the main function would exit, and the program would try to resume after the main() call, and terminate. There needs to be a piece of code left running, which can count time, and delays, maybe receive network or user generated events, and dispatch them to previously arranged callback functions in order for a program to keep running.
Such a piece of code, which can account for time and dispatch calls, is usually called a "reactor" - and there is none running in a plain Python program. Unlike, say, in a JavaScript program, where the browser, or other JavaScript environment provides such hosting by default.
That is why most Python web or network frameworks, all GUI toolkits, provide such a core - it is usually called at the end of the one main's script and is a method or function named mainloop or serve_forever, start and so on. From that point on, your main script, which had set the appropriate callbacks, scheduled things and so on, stops - the reactor will be the piece of code calling things.
That is where I say your question misses the context of what you want to do: at first you just want to test some scheduling - but afterwards you will want that inside a larger system - that system should be built using an appropriated framework for your "real task" at hand, for example Django, tornado, pyramid, if it is a web-server system, gtk, Qt, Tk if it is a GUI program, PyOgre, kivy, pyglet if it is a multimedia program, twisted for a generic network server of another protocol, or some other thing, like celery or camaelia - these are only general examples.
That said, Python's standard library does offer a "generic" scheduler function - it does implement such a loop, with the bare core of functionality. If you are doing nothing else, and nothing fancy, it will block there until it reaches the time to call your scheduled function, at which point it will exit, and resume the control to your main program. If your called function schedule other things, it will continue running, and so on.
See the documentation and example at:
http://docs.python.org/2/library/sched.html
You can use functions from the datetime module instead of time.time to set r absolute timings as you asking for. Also check the documentation there to threading.Timer - which in a naively way can do more or less what you have in mind, if you want to run a simple function after a given delay, in parallel to whatever other code is running and don't want to rewrite your application to be event based - but simpler as it may seen, it will have many drawbacks in a larger system - you should pick one of the frameworks listed.
I'm writing some code that needs to run on different OS platforms and interact with separate processes. To write tests for it, I need to be able to create processes from python that do nothing but wait to be signaled to stop. I would like to be able to create some processes that recursively create more.
Also (this part might be a little strange), it would be best for my testing if I were able to create processes that weren't children of the creating process, so I could emulate conditions where, e.g., os.waitpid won't have permission to interact with the process, or where one process signals a factory to create a process rather than creating it directly.
If you're using Python 2.6 the multiprocessing package has some stuff you might find useful.
There's a very simple example on my github. If you run spawner it will create 3 processes that run seperately, but use a channel to talk back to the spawner. So if you kill the spawner process the others you have started will die. I'm afraid there's a lot of redundant code in here, I'm in the middle of a refactoring, but I hope it gives a basic idea.