I am stuck in resolving a problem using python. Problem is I have to pass a variable value of module(python_code1.py) to a different module(python_code2.py). Based on the variable value, need to do some calculation in the module python_code2.py and then need to capture output value in the same module(python_code1.py) for further calculations.
Below is the snapshot of my code logic :
python_code2.py
import python_code1
data = python_code1.json_data
'''
Lines of code
'''
output = "some variable attribues"
python_code1.py
import python_code2
json_data = {"val1": "abc3","val1": "abc3","val1": "abc3"}
input_data = python_code2.output
''''
Lines of code using input_data variable
'''''
when I execute python python_code1.py, this is giving error:
AttributeError: module 'python_code2' has no attribute 'output'
I feel like I am not doing it in write way, but considering my code complexity and lines of code, I have to use these 2 module method.
Putting your code at the top-level is fine for quick throw away scripts, but that's about all. The proper way to organize your code for anything non-trivial is to define functions, so you can pass values as arguments and get results as return value.
If there's only on script using those functions, you can keep them in the script itself.
If you have multiple scripts needing to use the same functions, move those functions to a module and import this module from your scripts.
Related
I'm trying to debug Hy's use of bytecode. In particular, each time a module is imported, I want to see the path it was actually imported from, whether source or bytecode. Under the hood, Hy manages modules with importlib. It doesn't explicitly read or write bytecode; that's taken care of by importlib.machinery.SourceFileLoader. So it looks like what I want to do is monkey-patch Python's importing system to print the import path each time an import happens. How can I do that? I should be able to figure out how to do it for Hy once I understand how to do it for Python.
The easiest way that does not involve coding, is to start Python with two(!) verbose flags:
python -vv myscript.py
you'll get a lot of output, including all the import statements and all the files Python tries to access in order to load the module. In this example I have a simple python script that does import json:
lots of output!
[...]
# trying /tmp/json.cpython-310-x86_64-linux-gnu.so
# trying /tmp/json.abi3.so
# trying /tmp/json.so
# trying /tmp/json.py
# trying /tmp/json.pyc
# /usr/lib/python3.10/json/__pycache__/__init__.cpython-310.pyc matches /usr/lib/python3.10/json/__init__.py
# code object from '/usr/lib/python3.10/json/__pycache__/__init__.cpython-310.pyc'
[...]
Alternatively but more complex: you could change the import statement itself. For that, you can overwrite __import__, which is invoked by the import statement itself. This way you could print out all the files import actually opens.
Seems like a good option would be to dynamically patch importlib.machinery.SourceFileLoader(fullname, path) and importlib.machinery.SourcelessFileLoader(fullname, path) to each print or write to a variable (a) the calling method and (b) the argument passed to the function.
If all you need to do is:
I want to see the path it was actually imported from, whether source or bytecode
And you don't need the import to "work properly", perhaps you can do a modified version of something like this. For example, I quickly modified their sample code to get this, I have not tested it so it may not work exactly, but it should get you on the right track:
# custom class to be the mock return value
class MockSourceLoader:
# mock SourceFileLoader method always returns that the module was loaded from source and its path
def SourceFileLoader(fullname, path):
return {"load type": "source", "fullname": fullname, "path": path}
def check_how_imported(monkeypatch):
# Any arguments may be passed and mock_get() will always return our
# mocked object
def mock_get(*args, **kwargs):
return MockSourceLoader
# apply the monkeypatch
monkeypatch.setattr(importlib.machinery, SourceFileLoader, SourceFileLoader)
You would of course provide a similar mock for Sourceless file loading for SourcelessFileLoader
For reference:
https://docs.python.org/3/library/importlib.html#:~:text=importlib.machinery.SourceFileLoader(fullname%2C%20path)%C2%B6
https://docs.python.org/3/library/importlib.html#:~:text=importlib.machinery.SourcelessFileLoader(fullname%2C%20path)
code.py:
"""
Chekcs if isset(ABC) or not.
if not set - sets "ABC = 10".
"""
try: ABC
except: ABC = 10
print(ABC)
Outputs => "10"
cli:
python -c "ABC = 20; import code"
Expected to print "20", but it outputs "10".
Is there any possible way to fix this?
Global variables, despite their name, are not global to every part of your program. Each module has its own global namespace, and the same variable can exist, with different values, in different namespaces.
In your example, you're running a script given on the command line. That gets interpreted as the __main__ module in the interpreter. So in __main__, ABC is going to be equal to 20. But when the main module imports code, it has its own namespace. It doesn't see the __main__.ABC value that already exists, so it creates (and prints) its own ABC value.
As for "fixing" the code, I'm not sure it's worth trying. You could probably have code import __main__ and poke around in its namespace, but that seems like a lot of work for a sort of silly goal. There is almost certainly a better way to achieve whatever your actual end goal is (e.g. printing 10), and messing around with other modules' namespaces is unlikely to be it. I suspect this is an XY problem so while I'm dressing your reasonable question about why the code you have behaves the way it does, I don't really think there's a reasonable fix.
I don't think it's possible to set a value before importing the module: When importing a module it does not neccessarily share the same set of variables. The variables inside a module is essentially scoped within its context.
You might be able to set a temporary environmental variable or use sys.argv to get arguments passed via command line but that's very limited (for example, you can't pass on a Python object).
I personally would use a class to achieve similar functions (however you do need to import it first). For example:
In code.py:
class SampleClassName():
def __init__(self, ABC = 10) -> None:
print(ABC)
# The rest of your logic
Then, you can create an instance of this class using:
python3 -c "from code import SampleClassName; instance = SampleClassName(20)"
Notice that here, ABC = 10 defines the default value for ABC. It it's not set, it would be the default value of 10.
You might want to learn more about classes by reading the Python Docs
Is there a way to perform a python import which is not atomic?
For instance, I have a file as follows:
# Filename: a.py
myvariable = 1
mylist = [1, 2, 3]
raise ImportError
donotimportthis = 5
I then have a separate file which does the following:
import a
a.myvariable == 1 # This is okay as it imported it
a.donotimportthis # <-- raise an exception as this is not imported.
I have a file which contains some python code, this follows the format of:
...variables...
import X
I do not have X installed nor do I want it however I do want the variables.
Note: This file is autogenerated not by me but by a tool whose version is frozen.
Two choices, in descending order of preference:
Change the autogeneration process. Instead of invoking proprietary_autogen_process, invoke custom_autogen_wrapper. This wrapper in turn first invokes the proprietary third-party tool, and then modifies the produced module source code by searching for the code that imports module X, and deletes everything after it.
This is relatively straightforward. You just need to take some care to not introduce false positives or false negatives by performing too loose (or too strict) matching of the import code. Ideally you’d use an AST rewriter but that’s probably overkill; a regular expression search for import X might work, although it will yield wrong results if this text appears inside a comment, string literal or inside a method which isn’t executed.
Generate a stub module X in a location where it will be found by the autogenerated module when importing the latter. I don’t recommend this because it’s tedious: You probably can’t just generate an empty module, since the autogenerated module will want to use X. You need to generate meaningful method stubs.
You can do specific imports with
from a import myvariable
EDIT: The above won't work if anything that is flat in the file raises an error. If you have no way to edit the imported file then I don't know if there is a (resonable) solution to this. Sorry didn't realise.
(an unreasonable solution would be to read in the file as text, slice it, and then run eval on it).
Or, as mentioned in the comments, put the stuff you don't want under
if __name__=="__main__":
<here>
Then it will only be invoked if you run the file directly.
What you can do is removing the donotimportthis variable at the end of the module, as follows: del donotimportthis. I hope it helps
I found the following code snippet that I can't seem to make work for my scenario (or any scenario at all):
def load(code):
# Delete all local variables
globals()['code'] = code
del locals()['code']
# Run the code
exec(globals()['code'])
# Delete any global variables we've added
del globals()['load']
del globals()['code']
# Copy k so we can use it
if 'k' in locals():
globals()['k'] = locals()['k']
del locals()['k']
# Copy the rest of the variables
for k in locals().keys():
globals()[k] = locals()[k]
I created a file called "dynamic_module" and put this code in it, which I then used to try to execute the following code which is a placeholder for some dynamically created string I would like to execute.
import random
import datetime
class MyClass(object):
def main(self, a, b):
r = random.Random(datetime.datetime.now().microsecond)
a = r.randint(a, b)
return a
Then I tried executing the following:
import dynamic_module
dynamic_module.load(code_string)
return_value = dynamic_module.MyClass().main(1,100)
When this runs it should return a random number between 1 and 100. However, I can't seem to get the initial snippet I found to work for even the simplest of code strings. I think part of my confusion in doing this is that I may misunderstand how globals and locals work and therefore how to properly fix the problems I'm encountering. I need the code string to use its own imports and variables and not have access to the ones where it is being run from, which is the reason I am going through this somewhat over-complicated method.
You should not be using the code you found. It is has several big problems, not least that most of it doesn't actually do anything (locals() is a proxy, deleting from it has no effect on the actual locals, it puts any code you execute in the same shared globals, etc.)
Use the accepted answer in that post instead; recast as a function that becomes:
import sys, imp
def load_module_from_string(code, name='dynamic_module')
module = imp.new_module(name)
exec(code, mymodule.__dict__)
return module
then just use that:
dynamic_module = load_module_from_string(code_string)
return_value = dynamic_module.MyClass().main(1, 100)
The function produces a new, clean module object.
In general, this is not how you should dynamically import and use external modules. You should be using __import__ within your function to do this. Here's a simple example that worked for me:
plt = __import__('matplotlib.pyplot', fromlist = ['plt'])
plt.plot(np.arange(5), np.arange(5))
plt.show()
I imagine that for your specific application (loading from code string) it would be much easier to save the dynamically generated code string to a file (in a folder containing an __init__.py file) and then to call it using __import__. Then you could access all variables and functions of the code as parts of the imported module.
Unless I'm missing something?
I'm writing an interpreter for an old in-game scripting language, and so need to compile dictionary that has the name of the command from the language matched up against the symbol for that function.
Now, I've already figured out here: How to call a function based on list entry?
...That you can call functions this way, and I know that you can use dir to get a list of strings of all functions in a module. I've been able to get this list, and using a regex, removed the built-in commands and anything else I don't actually want the script to be able to call. The goal is to sandbox here. :)
Now that I have the list of items that are defined in the module, I need to get the symbol for each definition.
For a more visual representation, this is the test module I want to get the symbol for:
def notify(stack,mufenv):
print stack[-1]
It's pulled in via an init script, and I am able to get the notify function's name in a list using:
import mufprims
import re
moddefs=dir(mufprims)
primsfilter=re.compile('__.+__')
primslist=[ 'mufprims.' + x for x in dir(mufprims) if not primsfilter.match(x) ]
print primslist
This returns:
['mufprims.notify']
...which is the exact name of the function I wish to find the symbol for.
I read over http://docs.python.org/library/symtable.html here, but I'm not sure I understand it. I think this is the key to what I want, but I didn't see an example that I could understand. Any ideas how I would get the symbol for the functions I've pulled from the list?
You want to get the function from the mufprims module by using getattr and the function name. Like so:
primslist=[getattr(mufprims, x) for x in dir(mufprims) if not primsfilter.match(x) ]
I thought I might add another possible suggestion for retrieving the functions of an object:
import inspect
# example using os.path
import os.path
results = inspect.getmembers(os.path, inspect.isroutine)
print results
# truncated result
[...,
('splitdrive', <function splitdrive at 0x1002bcb18>),
('splitext', <function splitext at 0x1002bcb90>),
('walk', <function walk at 0x1002bda28>)]
Using dir on the object would essentially give you every member of that object, including non-callable attributes, etc. You could use the inspect module to get a more controlled return type.