import threading
import time
start = time.perf_counter()
def do_something():
print("Sleeping in 1 second")
time.sleep(1)
print("Done sleeping")
t1 = threading.Thread(target=do_something)
t2 = threading.Thread(target=do_something)
finish = time.perf_counter()
print(f"Finished in {round(finish-start,1)} seconds(s) ")
Does anyone know why this piece of code returns this error when run and how to fix it?
Traceback (most recent call last):
File "c:/Users/amanm/Desktop/Python/Python Crash Course/threading.py", line 1, in <module>
import threading
File "c:\Users\amanm\Desktop\Python\Python Crash Course\threading.py", line 12, in <module>
t1 = threading.Thread(target=do_something)
AttributeError: partially initialized module 'threading' has no attribute 'Thread' (most likely due to a circular import)
When I run this code in normal IDLE it seems to work but it doesn't work in Visual Studio Code.
It seems like the program file you have created is named threading.py, and you are importing a module also called threading. This causes a circular import because your file is shadowing the built-in module.
Please rename your program (e.g., threading_example.py).
I solved my problem, example code:
Main.py
if __name__ == "__main__":
import package2
pack2class = package2.Package2(main=self)
package2.py
import Main
class Package2(object):
def __init__(self, main:Main.MainClass): # for suggestion code
pass # your codes ...
When importing modules, python checks the files in your current working directory first, before checking other built-in modules. So, you probably have a file named threading.py which doesn't have the necessary attributes. In other words, you made a circular import.
Related
I have a directory tree
working_dir\
main.py
my_agent\
my_worker.py
my_utility\
my_utils.py
Code in each file is as follows
""" main.py """
import os, sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from my_agent.my_worker import MyWorker
import ray
ray.init()
workers = [MyWorker.remote(i) for i in range(10)]
ids = [worker.get_id.remote() for worker in workers]
# print(*ids, sep='\n')
print(*ray.get(ids), sep='\n')
""" worker.py """
from my_utility import my_utils
import ray
#ray.remote
class MyWorker():
def __init__(self, id):
self.id = id
def get_id(self):
return my_utils.f(self.id)
""" my_utils.py """
def f(id):
return '{}: Everything is fine...'.format(id)
Here's a part of the error message I received
Traceback (most recent call last):
File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/function_manager.py", line 616, in fetch_and_register_actor
unpickled_class = pickle.loads(pickled_class)
File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/cloudpickle/cloudpickle.py", line 894, in subimport
import(name)
ImportError: No module named 'my_utility'
Traceback (most recent call last):
File "main.py", line 12, in
print(*ray.get(ids), sep='\n')
File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/worker.py", line 2377, in get
raise value
ray.worker.RayTaskError: ray_worker (pid=30025, host=AiMacbook)
Exception: The actor with name MyWorker failed to be imported, and so cannot execute this method
If I remove all statements related to ray, the above code works fine. Therefore, I boldly guess the reason is that ray runs each actor in a new process and sys.path.append only works in the main process. So I add the following code to worker.py
import os, sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
But it still does not work: the same error message shows up. Now I run out of ideas, what should I do?
You are correct about what the issue is.
In your example, you modify sys.path in main.py in order to be able to import my_agent.my_worker and my_utility.my_utils.
However, this path change is not propagated to the worker processes, so if you were to run a remote function like
#ray.remote
def f():
# Print the PYTHONPATH on the worker process.
import sys
print(sys.path)
f.remote()
You would see that sys.path on the worker does not include the parent directory that you added.
The reason that modifying sys.path on the worker (e.g., in the MyWorker constructor) doesn't work is that the MyWorker class definition is pickled and shipped to the workers. Then the worker unpickles it, and the process of unpickling the class definition requires my_utils to be imported, and this fails because the actor constructor hasn't had a chance to run yet.
There are a couple possible solutions here.
Run the script with something like
PYTHONPATH=$(dirname $(pwd)):$PYTHONPATH python main.py
(from within working_dir/). That should solve the issue because in this case the worker processes are forked from the scheduler process (which is forked from the main Python interpreter when you call ray.init() and so the environment variable will be inherited by the workers (this doesn't happen for sys.path presumably because it is not an environment variable).
It looks like adding the line
parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
os.environ["PYTHONPATH"] = parent_dir + ":" + os.environ.get("PYTHONPATH", "")
in main.py (before the ray.init() call) also works for the same reason as above.
Consider adding a setup.py and installing your project as a Python package so that it's automatically on the relevant path.
The new "Runtime Environments" feature, which didn't exist at the time of this post, should help with this issue: https://docs.ray.io/en/latest/handling-dependencies.html#runtime-environments. (See the working_dir and py_modules entries.)
Hello I did got into circular dependency what is not refactori-zable other than doubling code.
I have something like this (only much more complex):
myParser.py:
import sys
import main #comment this to make it runnable
def parseEvnt():
sys.stdout.write("evnt:")
main.parseCmd(1) #comment this to make it runnable
tbl.py:
import myParser
tblx = {
1:("cmd",),
2:("evnt",myParser.parseEvnt),
}
main.py:
import tbl
def parseCmd(d):
print(tbl.tblx[d][0])
data=[1,2]
for d in data:
if(d<2):
parseCmd(d)
else:
fce = tbl.tblx[d][1]
fce()
Obvious error I'm getting is:
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 1, in <module>
import tbl
File "D:\Data\vbe\workspace\marsPython\testCircular2\tbl.py", line 1, in <module>
import myParser
File "D:\Data\vbe\workspace\marsPython\testCircular2\myParser.py", line 2, in <module>
import main
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 7, in <module>
parseCmd(d)
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 3, in parseCmd
print(tbl.tblx[d][0])
AttributeError: module 'tbl' has no attribute 'tblx'
In C I think I would just tell by declaration in tbl.py hey there is function parseEvnt(). I would not need to include myParser and there would be no circular include.
In python I do not know how to do it.
I read few threads and there is always some wise guy recommending refactorizing. But in this case parseCmd() needs to see tblx which needs to see parseEvnt() (unless function declaration) and parseEvnt() need to call parseCmd() (cos evnt contains triggering cmd and I do not want to double the decoding cmd code).
Is there way how to make it working in python?
You can frequently get away with circular dependencies as long as the modules don't try to use each other's data until all importing is complete - as a practical matter, that means referencing with namespace (from module import something is forbidden) and only using the other modules inside functions and methods (no mystuff = module.mystuff in the global space). That's because when importing starts, python puts the module name in sys.modules and won't try to import that module again.
You ran into trouble because when you run main.py, python adds __main__ to sys.modules. When the code finally came around to import main, there was no "main" in the module list and so main.py was imported again... and its top level code tried to run.
Lets rearrange your test case and throw in a few print statements to tell when import happens.
myParser.py
print(' + importing myParser')
import sys
print('import parsecmd')
import parsecmd
def parseEvnt():
sys.stdout.write("evnt:")
parsecmd.parseCmd(1)
tbl.py
print(' + importing tbl')
print('import myParser')
import myParser
tblx = {
1:("cmd",),
2:("evnt",myParser.parseEvnt),
}
Parsecmd.py (new)
print(' + importing parsecmd')
print('import tbl')
import tbl
def parseCmd(d):
print(tbl.tblx[d][0])
main.py
print('running main.py')
print('import parsecmd')
import parsecmd
if __name__ == "__main__":
data=[1,2]
for d in data:
if(d<2):
parsecmd.parseCmd(d)
else:
fce = parsecmd.tbl.tblx[d][1]
fce()
When I run it I get
running main.py
import parsecmd
+ importing parsecmd
import tbl
+ importing tbl
import myParser
+ importing myParser
import parsecmd <-- didn't reimport parsecmd
cmd
evnt:cmd
If you're insistent on not refactoring (which is the real solution to this - not being a wise guy), you could move your problematic import into your function in myParser.py
import sys
def parseEvnt():
import main ## import moved into function
sys.stdout.write("evnt:")
main.parseCmd(1)
Again, see if you can redesign your code so such interdependencies are avoided.
The above solution is sort of a hack and won't solve future problems you might run into due to this dependency.
Circular imports should be avoided. Refactoring is required, any workaround that still requires a circular import is not a good solution.
That being said, the refactoring doesn't have to be extensive. There are at least a couple of fairly simple solutions.
Solution 1: move shared functions to a shared module
Since you want to use parseCmd from more than one place, move it to a separate file. That way both main.py and myParser.py can import the function.
Solution 2: have main pass parseCmd to parseEvnt.
First, make parseEvnt accept an argument to tell it which function to run:
# myParser.py
import sys
def parseEvnt(parseCmd):
sys.stdout.write("evnt:")
parseCmd(1)
Next, when you call myParser.parseEvnt, pass in a reference to main.parseCmd:
# main.py:
...
else:
fce = tbl.tblx[d][1]
fce(parseCmd)
There are other ways to accomplish the same thing. For example, you could add a "configure" method in myParser, and then have main.py call the configure method and pass in a reference to its parseCmd. The configure method can then store this reference in a global variable.
Another choice is to import main in the function that uses it:
main.py
import sys
def parseEvnt():
import main
sys.stdout.write("evnt:")
main.parseCmd(1)
I have just discovered this strange scoping behaviour of both Python 2 and 3. When I'm adding a late import for a sub-module, the main import of toplevel module stops working. Viable example:
import os
def start():
import sys
print('in modules?', 'os' in sys.modules)
print('in globals?', 'os' in globals())
print('in locals?', 'os' in locals())
print('os =', os)
import os.path
os.path.exists('useless statement')
start()
The output will be:
in modules? True
in globals? True
in locals? False
Traceback (most recent call last):
File "test.py", line 15, in <module>
start()
File "test.py", line 9, in start
print('os =', os)
UnboundLocalError: local variable 'os' referenced before assignment
Any ideas?
This is nothing special about import statements. It's just how the scoping works in Python. If you're assigning a value to a label, it is local to the scope unless explicitly defined global.
Try this code -
a = 2
def start():
print a
a = 3
start()
This also fails with UnboundLocalError as your code because statement a = 3 makes the label a local to function start.
I'm learning by my self Python and I have a mind blowing issue, because I don't understand why is not working. I'm using PyDev and I've download version 2 of Python. I have this code:
class Utils:
#staticmethod
def hello():
print "Hi! I'm the Utils class"
Utils.hello() #Hi! I'm the Utils class
and everything is working fine at this point. But if I import the Utils class and call the static method from another module...
import Utils
Utils.hello()
I get this error:
Traceback (most recent call last):
File "C:\Users\migugonz\Desktop\Docs\Per Folder\WorkSpacePy\Rosalind\src\bioinformatics\stronghold\Pruebas.py", line 40, in <module>
Utils.hello()
AttributeError: 'module' object has no attribute 'hello'
I thing it can't be a big deal, but I've been searching a solution and as long I know this shlould be work.
I believe you need to do Utils.Utils.hello()
or import like from Utils import Utils
I've run into some behavior from Python 2.6.1 that I didn't expect. Here is some trivial code to reproduce the problem:
---- ControlPointValue.py ------
class ControlPointValue:
def __init__(self):
pass
---- ControlPointValueSet.py ----
import ControlPointValue
---- main.py --------------------
from ControlPointValue import *
from ControlPointValueSet import *
val = ControlPointValue()
.... here is the error I get when I run main.py (under OS/X Snow Leopard, if it matters):
jeremy-friesners-mac-pro-3:~ jaf$ python main.py
Traceback (most recent call last):
File "main.py", line 4, in <module>
val = ControlPointValue()
TypeError: 'module' object is not callable
Can someone explain what is going on here? Is Python getting confused because the class name is the same as the file name? If so, what is the best way to fix the problem? (I'd prefer to have my python files named after the classes that are defined in them)
Thanks,
Jeremy
I don't think it's unexpected. What you are basically doing is:
1) the first import in main.py imports the contents of ControlPointValue module into the global namespace. this produces a class bound to that name.
2) the second import in main.py imports the contents of ControlPointValueSet module into the global namespace. This module imports ControlPointValue module. This overwrites the binding in the global namespace, replacing the binding for that name from the class to the module.
To solve, I would suggest you not to import *, ever. Always keep the last module prefix. For example, if you have foo/bar/baz/bruf.py containing a class Frobniz, do
from foo.bar.baz import bruf
and then use bruf.Frobniz()
In addition to the other suggestions about star imports, don't name your module and your class the same. Follow pep8's suggestions and give your modules short all lower case names and name your classes LikeThis. E.g.
---- controlpoints.py ------
class ControlPointValue:
def __init__(self):
pass
---- valuesets.py ----
from controlpoints import ControlPointValue
---- main.py --------------------
from controlpoints import ControlPointValue
from valuesets import *
val = ControlPointValue()