In the script's main thread, I setup a variable called queue and fill it with URLs. Then I create 8 processes using multiprocessing.Process and then those processes spawn 10 threads each using the threading library.
Within the thread worker (which was spawned by another process as per above), I have global queue.
Then will queue.get() act as expected? I've tried it and it seems to be okay during some tests and others not.
Question is, can global variables be accessed from another process and thread?
It is hard to understand what you are exactly asking. But there are two main questions here:
Can global variables be accessed from another process?
No, not without some form of inter-process communication, and even then you would be passing a copy of that variable to the other process. Each process has its own global state.
Can global variables be accessed from another thread?
Yes, a thread who lives in the same process can access a global variable, but you must ensure the safety of any memory that is accessed by multiple threads. Meaning, threads should not access writable memory at the same time as other threads or you run the risk of one thread writing to the memory while another thread attempts to read it.
Answering Question Above
If I understand the setup correctly, each of your child processes has their own global variable queue. Each of those queues should be accessible only to the threads that are spawned within that process.
Related
My program consists of a main core function and a thread that was initiated inside that function to do some tasks (check order status and update the ledger). In my core program, im using objects of multiple classes hundreds of times and i need my thread to be able to make changes and add data to those objects that are shared from the core function. In that thread, i have implemented a thread lock to make sure everything runs smoothly. My question is whether i need to put a lock everytime i use the object in the core function or is it sufficient to only lock the resources in the thread?
My apologies in advance for not sharing the code, i cant.
The effect of holding a lock (i.e. having acquired and not yet released it) is that any other thread that tries to acquire the same lock will be blocked until the first thread has released the lock. In other words, at any given time at most one thread can hold the lock.
If only one thread uses the lock, nothing is achieved.
Using python, how do I share a global variable.
I have main process which calls processA
processA keeps updating the variable and I need that information in the main process.
How is that possible ?
Here is a situation that seems to cause problem to my program. There two threads, one thread has locks shared variables, while the other, wants to access one shared variable. So what will happen in this case, does the second thread just wait till the lock is released? or there will be errors since thread 2 is accessing locked variables?
I have a multithreaded PyQt application that is leaking memory. All the functions that leak memory are worker threads, and I'm wondering if there's something fundamentally wrong with my approach.
When the main application starts, the various worker thread instances are created from the thread classes, but they are not initially started.
When functions run that require a worker thread, the thread is initialized (data and parameters are passed from the main function, and variables are reset from with in the worker instance), and then the thread is started. The worker thread does its business, then completes, but is never formally deleted.
If the function is called again, then again the thread instance is initialized, started, runs, stops, etc...
Because the threads can be called to run again and again, I never saw the need to formally delete them. I originally figured that the same variables were just get re-used, but now I'm wondering if I was mistaken.
Does this sound like the cause of my memory leak? Should I be deleting the threads when they complete even if they're going to be called again?
If this is the root of my problem, can someone point me to a code example of how to handle the thread deleting process properly? (If it matters, I'm using PyQt 4.11.3, Qt 4.8.6, and Python 3.4.3)
I have a python script that is doing some map reduce-ish ETL. I am not originator of the code but working to analyze/diagnose its runtime for some improvements.
In the package, it uses a "Process":
worker = Process(target=grab_worker)
worker.start()
That does a perpectual FTP loop to extract new files from our CDN, can't include the FTP code but shouldn't be relevant for the question
Later in the code, we create an instance of a worker Pool which runs some asynch functions:
workerpool = multiprocessing.Pool(processes=4)
# ...
resultobjs[k] = workerpool.apply_async(func, args=fargs)
Again, the underlying code therein should be irrelevant to question I think so not including code yet.
My question is, in Python, once I create the worker Pool will the workers there be "shared" with the Process?
In other words, if I first create 1 worker with process doing something, later in execution when I create workers with a pool class, when the loop returns and tries to run the function registered with the process, will it then use the previously created workers?
Or, instead, does it keep the "hot side hot and the cold side cold" by allowing each class instance to reference only the workers which it has spawned ( Process reuses its one worker previously created, and Pool continues to use its designated workers vs. Process using the workers generated by Pool ).
The mp.Process knows nothing about the mp.Pool.
So the call to mp.Process will not somehow use a process spawned by mp.Pool.