We Keep Coding

Call a function for every script inside a folder

Is there a way (using only python. i.e.: without a bash script nor another language code) to call a specific function in every script inside a folder without needing to import all of them explicitly.
For example, let's say that this is my structure:
main.py
modules/
module1.py
module2.py
module3.py
module4.py
and every moduleX.py has this code:
import os
def generic_function(caller):
print('{} was called by {}'.format(os.path.basename(__file__), caller))
def internal_function():
print('ERROR: Someone called an internal function')
while main.py has this code:
import modules
import os
for module in modules.some_magic_function():
module.generic_function(os.path.basename(__file__))
So if I run main.py, I should get this output:
module1.py was called by main.py
module2.py was called by main.py
module3.py was called by main.py
module4.py was called by main.py
*Please note that internal_function() shouldn't be called (unlike this question). Also, I don't want to declare explicitly every module file even on a __init__.py
By the way, I don't mind to use classes for this. In fact it could be even better.

You can use exec or eval to do that. So it would go roughly this way (for exec):
def magic_execute():
import os
import glob
for pyfl in glob.glob(os.path(MYPATH, '*.py'):
with open(pyfl, 'rt') as fh:
pycode = fh.read()
pycode += '\ngeneric_function({})'.format(__file__)
exec(pycode)
The assumption here is that you are not going to import the modules at all.
Please note, that there are numerous security issues related to using exec in such a non-restricted manner. You can increase security a bit.

While sophros' approach is quickly and enough for implicitly importing the modules, you could have issues related to controlling every module or with complex calls (like having conditions for each calls). So I went with another approeach:
First I created a class with the function(s) (now methods) declared. With this I can avoid checking if the method exists as I can use the default one if I didn't declare it:
# main.py
class BaseModule:
def __init__(self):
# Any code
def generic_function(self, caller):
# This could be a Print (or default return value) or an Exception
raise Exception('generic_function wasn\'t overridden or it was used with super')
Then I created another class that extends the BaseModule. Sadly I wasn't able to get a good way for checking inherence without knowing the name of the child class so I used the same name for every module:
# modules/moduleX.py
from main import BaseModule
class GenericModule(BaseModule):
def __init__(self):
BaseModule.__init__(self)
# Any code
def generic_function(self, caller):
print('{} was called by {}'.format(os.path.basename(__file__), caller))
Finally, in my main.py, I used the importlib for importing the modules dynamically and saving an instance for each one, so I can use them later (for sake of simplicity I didn't save them in the following code, but it's easy as using a list and appending every instance on it):
# main.py
import importlib
import os
if __name__ == '__main__':
relPath = 'modules' # This has to be relative to the working directory
for pyFile in os.listdir('./' + relPath):
# just load python (.py) files except for __init__.py or similars
if pyFile.endswith('.py') and not pyFile.startswith('__'):
# each module has to be loaded with dots instead of slashes in the path and without the extension. Also, modules folder must have a __init___.py file
module = importlib.import_module('{}.{}'.format(relPath, pyFile[:-3]))
# we have to test if there is actually a class defined in the module. This was extracted from [1]
try:
moduleInstance = module.GenericModule(self)
moduleInstance.generic_function(os.path.basename(__file__)) # You can actually do whatever you want here. You can save the moduleInstance in a list and call the function (method) later, or save its return value.
except (AttributeError) as e:
# NOTE: This will be fired if there is ANY AttributeError exception, including those that are related to a typo, so you should print or raise something here for diagnosting
print('WARN:', pyFile, 'doesn\'t has GenericModule class or there was a typo in its content')
References:
[1] Check for class existence
[2] Import module dynamically
[3] Method Overriding in Python

Declare a variable across all files in Python

I have a set of Python files as follows:
py-app/
main.py
module1.py
Inside main.py I declare a variable config as follows
config = {"param1": "value1", "param2":"value2"}
How can I make config variable global in order for it to be accessed in module1.py as follows:
Inside module1.py
def foo()
print(config["param1"])

Please see this from the official website:
https://docs.python.org/3/faq/programming.html#how-do-i-share-global-variables-across-modules
What you can do is - create a special module to store all the global variables there and access them by importing the module.
This singleton design pattern is followed and if you modify the state in one file it will be reflected everywhere. It basically works like a session object.
If required, you can initialize the values as well.

How can I set or mock a variable from a test file?

I have a file that looks like this:
prob = 0.05
def calculate_func(arg1, arg2):
if random.random > prob:
# Do stuff
else:
# Do other things
# Lots more functions
And a test file like this:
from my_project import calculate_func
def test_calculate_func():
arg1, arg2 = 1, 2
assert calculate_func(arg1, arg2) == 'Desired value'
But currently the test file imports error_prob from the main file. I want to be able to set, or mock, error_prob from the test file to test different behaviours. How can I do this?
I don't want to pass error_prob into calculate_func, because error_prob is used by every function in that file.

You can reassign module-scoped variables by importing the module.
import my_project
myproject.prob = 0.1

I don't want to pass error_prob into calculate_func, because error_prob is used by every function in that file.
If you need to inject a dependency, then you need to inject it. One possibility that might simplify things is to move those functions into a class, and set the error_prob implementation as a member when you instantiate it.

Can't we use a single variable in three different module by making it global in python

I am pretty new at programming.
I have three files,
first one is "main.python",
second one is "driver.python",
third one is "global_var.python" which is only used to intialise global variables.
main.python
import global_var
import driver
global_var.init()
driver.fun()
print(shinchan)
driver.python
import global_var
def fun():
shinchan ="hello python"
global_var.python
def init():
global shinchan
upon running this I am getting error
File "main.py", line 13, in <module>
print(shinchan)
NameError: name 'shinchan' is not defined
I understand that in driver.python, python is creating altogether a new variable "shinchan" which has nothing to do with the global variable "shinchan" defined in "global_var.python", which I am trying to implement.
can't we use same variable in these three different files.
please help me out here.

Accessing the variable
If you want to access shinchan in main.py, you need either change the import statement or reference it as an attribute of global_var.
So either change the import to
from global_var import shinchan
or change the call to
print(global_var.shinchan)
You probably don't want to have to call global_var.init() before you can access global_var, so what you probably want to do is this:
global_var.py
shinchan = '' # or whatever you want it to be initially
Setting the variable
global_var.shinchan = 'hello world'
But you probably don't want to do this. Global variables get messy fast, and that's why you should try to avoid them at all costs.
How you probably want do this
If you want to get information from driver.py and store it in a variable in main.py, you should get the data you want from a function. So something like this:
main.py
import driver
shinchan = 'hi'
shinchan = driver.getNewShinchan()
driver.py
def getNewShinchan():
return 'hello world'

Why can't I write a value to a variable that belongs to a different module?

I have 2 .py files.
File a.py:
some_boolean = True
File b.py:
from a import *
def blah():
if some_boolean:
do_something()
some_boolean = not some_boolean
The code works perfectly fine without the some_boolean = not some_boolean line, but as soon as I add it back in I'm getting UnboundLocalError: local variable 'some_boolean' referenced before assignment.
I've been bashing my head for hours now and I can't seem to figure it out. Any ideas?

You need to do qualified import of a and use the full name a.some_boolean:
import a
def blah():
if a.some_boolean:
do_something()
a.some_boolean = not a.some_boolean
I your code some_boolean = not some_boolean tries to create a new local variable. This does not work because the name some_boolean is used for a local variable and shadows the global variable of the same name.
In general, you should never use the same name for a local and global variable.