I have a program consistring of several modules specifying the respective web application handlers and one, specifying the respective router.
The library I use can be found here.
Excerpt from webapp.service (there are more such modules):
from webapp.router import ROUTER
#ROUTER.route('/service/[id:int]')
class ServicePermissions(AuthenticatedService):
"""Handles service permissions."""
NODE = 'services'
NAME = 'services manager'
DESCRIPTION = 'Manages services permissions'
PROMOTE = False
webapp.router:
ROUTER = Router()
When I import the webapp.router module, the webapp.service module does obviously not run. Hence, the #ROUTER.route('/service/[id:int]') decorator is not run and my web aplication will fail with the message, that the respective route is not available.
What is the best practice in that case to run the code in webapp.service to "run" the decorators? I do not really need to import the module itself or any of its members.
As stated in the comments fot the question,
you simply have to import the modules. As for linter complaints, those are the lesser of your problems. Linters are there to help - if they get into the way, just don't listen to them.
So, the simple way just to get your things working is, at the end of your __main__.py or __init__.py, depending on your app structure, to import explicitly all the modules that make use of the view decorator.
If you have a linter, check how to silence it on the import lines - that is usually accomplished with a special comment on the import line.
Python's introspection is fantastic, but it can't find instances of a class, or subclasses, if those are defined in modules that are not imported: such a module is just a text file sitting on the disk, like any data file.
What some frameworks offer as an approach is to have a "discovery" utility that will silently import all "py" files in the project folders. That way your views can "come into existence" without explicit imports.
You could use a function like:
import os
def discover(caller_file):
caller_folder = os.path.dirname(caller_file)
for current, folders, files in os.walk(caller_folder):
if current == "__pycache__":
continue
for file in files:
if file.endswith(".py"):
__import__(os.path.join(current, file))
And call it on your main module with discover(__file__)
Related
I'm not sure what the best way of phrasing this question, but I am noticing that if I have say this file in the project directory:
a.py
import logging
import track
from raven.contrib.django.raven_compat.models import client
from django.conf import settings
from .MetricLogger import log_metrics
_LOG = logging.getLogger('application')
class TrackingClass(track.SubClass):
def record(self, *args, **kwargs):
try:
metrics = {
"metric": 'something',
"trigger": 'request',
"element": 'request'
}
# log_metrics(None, metrics)
except Exception:
client.captureException()
track.tracker.register(EventsappTrackingPixel) # this line gets called at some point without importing this module
The print statement gets called. What is happening and what is triggering it? Is this part of the class indexing that needs to be done or how is this working?
EDIT:
Turns out the track package we depend on is specifically looking for this filename in the root directory and importing it!
Django does not automatically import all files within a directory. It only automatically imports specific predefined files (should they exist), such as models.py and apps.py.
If you have additional custom modules in the directory, like profanityDetector.py, you'll need to manually import them, perhaps within apps.py.
Note that in Django, all modules are imported only once per worker process. This means that if you are not manually importing a.py from another module, it must have been registered as a callback through Django's logging system, possibly due to a pathing or directory mixup.
The call to print is not part of the class definition. It's at the module level, therefore it gets executed when the module is imported.
You can figure out what is triggering the import by inspecting the call stack. Insert these two lines right after (or before) the call to print:
import traceback
traceback.print_stack()
I've run into a circular import problem because I need to import only part of a package. When my program starts, I create a driver class which inherits from a class defined in a package, but another irrelevant part of that package imports the driver; I need to stop the irrelevant part of the package from running until it's needed later.
More info: I have a package of interfaces in my program. They're just parent objects with methods and attributes common to many objects in my program. They have no logical connection other than having similar purposes. They're in a package solely for my convenience; I don't want tons of .py files in the top level, and would rather sort them into subfolders.
The package looks like this:
interfaces
__init__.py
destroyinterface.py
graphicsinterface.py
And the __ init __.py looks like this:
from destroyinterface import DestroyInterface
from graphicsinterface import GraphicsInterface
I want to import DestroyInterface WITHOUT graphicsinterface.py being initialized. graphicsinterface.py imports the driver that's dependent on DestroyInterface, but I can't seem to access DestroyInterface to create the driver without graphicsinterface.py being initialized.
I don't want to remove the GraphicsInterface import from the __ init __.py because I don't want things to have to know it lives in a file called graphicsinterface.py when they import it. Including information about the structure of my packages to every single import both adds boilerplate and makes refactoring harder. I want the classes to be accessible for import directly from the interfaces module but their .py files only be initialized if I explicitly access them.
I don't want to use a lazy import of the driver in graphicsinterface.py either, both because it's messy (I only want the file being initialized when I actually need it) and because an import inside the time-sensitive methods of GraphicsInterface would slow them down.
Am I out of luck? Will I have to sort my files in a different way?
I'd recommend looking at two solutions within the graphics interface. First, if only a couple of functions need the driver, import the driver in those functions. Doing an import of an already imported module is efficient, so it should be fine to import it in the driver.
Another approach is to do something like this in the graphics interface:
driver = None # filled in when driver is imported
Then elsewhere
import driver
import interfaces.graphics
interfaces.graphics.driver = driver
So I came across a hack to fix my problem. Figured I'd share it.
My __ init __.py now looks like this:
class CrazyHack(object):
#property
def DestroyInterface(self):
import crazyhackthing.destroyinterface
return destroyinterface.DestroyInterface
#property
def GraphicInterface(self):
import crazyhackthing.graphicinterface
return graphicinterface.GraphicInterface
import sys
sys.modules["crazyhackthing"] = sys.modules["interfaces"]
sys.modules["interfaces"] = CrazyHack()
This makes any import statements from this package refer to properties of the object defined there, and thus delay the initialization of files until explicit import. No idea if this works on python 3, and it's probably an awful idea in the first place, but it works for me. May God have mercy on my soul.
I have a collection of scripts written in Python. Each of them can be executed independently. However, most of the time they should be executed one after the other, so there is a MainScript.py which calls them in the appropriate order. Each script has some configurable variables (let's call them Root_Dir, Data_Dir and LinWinFlag). If this collection of scripts is moved to a different computer, or different data needs to be processed, these variable values need to be changed. As there are many scripts this duplication is annoying and error-prone. I would like to group all configuration variables into a single file.
I tried making Config.py which would contain them as per this thread, but import Config produces ImportError: No module named Config because they are not part of a package.
Then I tried relying on variable inheritance: define them once in MainScript.py which calls all the others. This works, but I realized that each script would not be able to run on its own. To solve this, I tried adding useGlobal=True in MainScript.py and in other files:
if (useGlobal is None or useGlobal==False):
# define all variables
But this fails when scripts are run standalone: NameError: name 'useGlobal' is not defined. The workaround is to define useGlobal and set it to False when running the scripts independently of MainScript.py. It there a more elegant solution?
The idea is that python wants to access files - including the Config.py - primarily as part of a module.
The nice thing is that Python makes building modules (i.e. python packages) really easy - initializing it can be done by creating a
__init__.py
file in each directory you want as a module, a submodule, a subsubmodule, and so on.
So your import should go through if you have created this file.
If you have further questions, look at the excellent python documentation.
The best way to do this is to use a configuration file placed in your home directory (~/.config/yourscript/config.json).
You can then load the file on start and provide default values if the file does not exist :
Example (config.py) :
import json
default_config = {
"name": "volnt",
"mail": "oh#hi.com"
}
def load_settings():
settings = default_config
try:
with open("~/.config/yourscript/config.json", "r") as config_file:
loaded_config = json.loads(config_file.read())
for key in loaded_config:
settings[key] = loaded_config[key]
except IOError: # file does not exist
pass
return settings
For a configuration file it's a good idea to use json and not python, because it makes it easy to edit for people using your scripts.
As suggested by cleros, ConfigParser module seems to be the closest thing to what I wanted (one-line statement in each file which would set up multiple variables).
i need to call a function from from one python class to another which are at different directories.
I'm using Eclipse and PyDev for developing scripts.
sample.py
class Employee:
def meth1(self,arg):
self.arg=arg
print(arg)
ob=Employee()
ob.meth1("world")
main.py
class main:
def meth(self,arg):
self.arg=arg
print(arg)
obj1=main()
obj1.meth("hello")
I need to access meth1 in main.py
updated code
main.py
from samp.sample import Employee
class main:
def meth(self,arg):
self.arg=arg
print(arg)
obj1=main()
obj1.meth("hello")
After executing main.py it is printing "world" automatically without calling it.
my requirement is i need to call meth1 from main.py explicitly
please find my folder below
import is the concept you need here. main.py will need to import sample and then access symbols defined in that module such as sample.Employee
To ensure that sample.py can be found at import time, the path to its parent directory can be appended to sys.path (to get access to that, of course, you will first need to import sys). To manipulate paths (for example, to turn a relative path like '../samp' into an absolute path, you might want to import os as well and take a look at the standard library functions the sub-module os.path has to offer.
You will have to import the sample.py file as a module to your main.py file. Since the files are in different directories, you will need to use the __init__.py
From the documentation:
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.
Here is a related stack overflow question which explains the solution in more detail.
It's generally not a good idea to alter the sys.path variable. This can make scripts non-portable. Better is to just place your extra modules in the user site directory. You can find out what that is with the following script.
import os
import sys
import site
print(os.path.join(site.USER_BASE, "lib", "python{}.{}".format(*sys.version_info[0:2]), "site-packages"))
Place your modules there. Then you can just import as any other module.
import sample
emp = sample.Employee()
Later you can package it using distutils or setuptools and the imports will still work the same.
I'm wrote a main python module that need load a file parser to work, initially I was a only one text parser module, but I need add more parsers for different cases.
parser_class1.py
parser_class2.py
parser_class3.py
Only one is required for every running instance, then I'm thinking load it by command line:
mmain.py -p parser_class1
With this purpose I wrote this code in order to select the parser to load when the main module will be called:
#!/usr/bin/env python
import argparse
aparser = argparse.ArgumentParser()
aparser.add_argument('-p',
action='store',
dest='module',
help='-p module to import')
results = aparser.parse_args()
if not results.module:
aparser.error('Error! no module')
try:
exec("import %s" %(results.module))
print '%s imported done!'%(results.module)
except ImportError, e:
print e
But, I was reading that this way is dangerous, maybe no stardard..
Then, is this approach ok? or I must find another way to do it?
Why?
Thanks, any comment are welcome.
You could actually just execute the import statement inside a conditional block:
if x:
import module1a as module1
else:
import module1b as module1
You can account for various whitelisted module imports in different ways using this, but effectively the idea is to pre-program the imports, and then essentially use a GOTO to make the proper imports... If you do want to just let the user import any arbitrary argument, then the __import__ function would be the way to go, rather than eval.
Update:
As #thedox mentioned in the comment, the as module1 section is the idiomatic way for loading similar APIs with different underlying code.
In the case where you intend to do completely different things with entirely different APIs, that's not the pattern to follow.
A more reasonable pattern in this case would be to include the code related to a particular import with that import statement:
if ...:
import module1
# do some stuff with module1 ...
else:
import module2
# do some stuff with module2 ...
As for security, if you allow the user to cause an import of some arbitrary code-set (e.g. their own module, perhaps?), it's not much different than using eval on user-input. It's essentially the same vulnerability: the user can get your program to execute their own code.
I don't think there's a truly safe manner to let the user import arbitrary modules, at all. The exception here is if they have no access to the file-system, and therefore cannot create new code to be imported, in which case you're basically back to the whitelist case, and may as well implement an explicit whitelist to prevent future-vulnerabilities if/when at some point in the future the user does gain file-system access.
here is how to use __import__()
allowed_modules = ['os', 're', 'your_module', 'parser_class1.py', 'parser_class2.py']
if not results.module:
aparser.error('Error! no module')
try:
if results.module in allowed_modules:
module = __import__(results.module)
print '%s imported as "module"'%(results.module)
else:
print 'hey what are you trying to do?'
except ImportError, e:
print e
module.your_function(your_data)
EVAL vs __IMPORT__()
using eval allows the user to run any code on your computer. Don't do that. __import__() only allows the user to load modules, apparently not allowing user to run arbitrary code. But it's only apparently safer.
The proposed function, without allowed_modules is still risky since it can allow to load an arbitrary model that may have some malicious code running on when loaded. Potentially the attacker can load a file somewhere (a shared folder, a ftp folder, a upload folder managed by your webserver ...) and call it using your argument.
WHITELISTS
Using allowed_modules mitigates the problem but do not solve it completely: to hardening even more you still have to check if the attacker wrote a "os.py", "re.py", "your_module.py", "parser_class1.py" into your script folder, since python first searches module there (docs).
Eventually you may compare parser_class*.py code against a list of hashes, like sha1sum does.
FINAL REMARKS: At the real end, if user has write access to your script folder you cannot ensure an absolutely safe code.
You should think of all of the possible modules you may import for that parsing function and then use a case statement or dictionary to load the correct one. For example:
import parser_class1, parser_class2, parser_class3
parser_map = {
'class1': parser_class1,
'class2': parser_class2,
'class3': parser_class3,
}
if not args.module:
#report error
parser = None
else:
parser = parser_map[args.module]
#perform work with parser
If loading any of the parser_classN modules in this example is expensive, you can define lambdas or functions that return that module (i.e. def get_class1(): import parser_class1; return parser_class1) and alter the line to be parser = parser_map[args.module]()
The exec option could be very dangerous because you're executing unvalidated user input. Imagine if your user did something like -
mmain.py -p "parser_class1; some_function_or_code_that_is_malicious()"