I am new to Python and recently started writing a script which essentially reads a MySQL database and archives some files by uploading them to Amazon Glacier. I am using the Amazon-provided boto module along with a few other modules.
I noticed that I seem to be replicating the same pattern over and over again when installing and taking advantage of these modules that connect to external services. First, I write a wrapper module which reads my global config values and then defines a connection function, then I start writing functions in that module which perform the various tasks. For example, at the moment, my boto wrapper module is named awsbox and it consists of functions like getConnection and glacierUpload. Here's a brief example:
import config,sys,os
import boto,uuid
_awsConfig = config.get()['aws']
def getGlacierConnection():
return boto.connect_glacier( aws_access_key_id=_awsConfig['access_key_id'],
aws_secret_access_key=_awsConfig['secret_access_key'])
def glacierUpload( filePath ):
if not os.path.isfile( filePath ):
return False
awsConnect = getGlacierConnection()
vault = awsConnect.get_vault( _awsConfig['vault'] )
vault.upload_archive( filePath )
return True
My question is, should I be writing these "wrapper" modules? Is this the Pythonic way to consume these third-party modules? This method makes sense to me but I wonder if creating these interfaces makes my code less portable or modular, or whether or not there is a better way to integrate these various disparate modules into my main script structure.
You're using the modules as intented. You import them and then use them. As I see it, awsbox is the module that holds the implementation of the functions that match your needs.
So answering your cuestion:
should I be writing these "wrapper" modules?, yes (you can stop calling them "wrappers"), an error would be to rewrite those installed modules.
Is this the Pythonic way to consume these third-party modules?, Is the Python way. Authors write modules for you tu use(import).
Related
Ok, so I know this has been asked before in many different threads but I find myself still trying to reduce my doubt.
I have an application that allows the user to pass a dictionary of strings and 'choose' a backend function from a library to process it. The functions are part of a 'workflow' library and loaded by the system admin on the backend. Available functions are stored in the backend in a manifest file.
The mechanics are such that the users send the dictionary as json to the web app and nominate which function from the library should process it. The function is is then loaded and executed via the python exec() or eval() functions.
Before the execution, the requested function is checked against a list of available functions (whitelist) from the manifest file.
My basic question is, can whitelisting make exec() and eval safe? Could it be made 'safer'?
If I understand it, the function is trusted by the admin and that makes it as safe as any python module you install. Just make sure that the exec part is only done on the trusted code. Here is an example where functions with the same name as their file are loaded and executed.
import json
from pathlib import Path
# files named the same as the function, no .py
FUNCTION_DIR = Path("/my/functions/are/here")
def run_func(name, data):
try:
func_namespace = {}
exec(open(FUNCTION_DIR/name).read(), func_namespace)
return func_namespace[name](json.dumps(data))
except Exception as e:
return "Hey, what kind of game are you playing here? " + str(e)
The the function is naturally whitelisted just because its in the known-safe directory.
In order to simplify my code, I have put various functions into external files with I load via:
from (external_file) import (function_name)
...which works fine.
My question though has to do with other modules, such as cv2 or numpy - do I need those listed in my external file (as well as my main file) or is there a way to just list them in my main file?
Each file you put Python code in is its own module. Each module has its own namespace. If some of your code (in any module) uses some library code, it will need some way to access the library from the namespace it is defined in.
Usually this means you need to import the library in each module it's being used from. Don't worry about duplication, modules are cached when they are first loaded, so additional imports from other modules will quickly find the existing module and just add a reference to it in their own namespaces.
Note that it's generally not a good idea to split up your code too much. There's certainly no need for every function or every class to have its own file. Instead, use modules to group related things together. If you have a couple of functions that interoperate a lot, put them in the same module.
I have a Python package with dozens of subpackages/modules. Nearly each of the modules uses open built-in Python function. I have written an own implementation of the file opening function and would like to "redirect" all open calls I have in the package modules to my_open function.
I am aware that it is possible to write open = my_open_file in the top of the module to shadow the open within the module, but this would imply editing each module. Alternatively, putting open = my_open_file in the __init__.py of the package and then doing from package_name import open which also implies adding a single line of code to each module.
Is it possible to define the my_open_file function for the package scope just in a single place? Or adding a single line of code in each module is my only option?
Think about what you're asking to do with this module: you've written your own package, a personal change to the way a built-in function is linked. You want to redefine a standard language feature. That's okay ... part of the power of a language is the capability to do things the way you define.
The problem comes when you want to do this by default, overriding that standard capability for code that hasn't explicitly requested that change. You cannot do this without some high-powered authority. The most straightforward way is to rebuild your Python system from sources, replacing the standard open with your own.
In reality, the "normal" way is to have each of your application modules "opt-in", declaring explicitly that it wants to use the new open instead of the one defined by the language. This is the approach you've outlined in your second paragraph. How is this a problem for you? Inserting a line in each of a list of parameterized files is a single system command.
Some background
I'm using Robot Framework with Python to create a small framework for test automation. I have a few different libraries; a couple are application-specific and one has keywords that I would like to be always available. This always-available library is my common library and I need it to be accessible from the functions in my other libraries as well.
The way I have accomplished this until now has been some boilerplate at the top of my other libraries. Specifically, in my other libraries I have:
try:
self.common_library = BuiltIn().get_library_instance("my_common_lib")
except RuntimeError:
BuiltIn().import_library("my_common_lib", True)
self.common_library = BuiltIn().get_library_instance("my_common_lib")
This code checks the current Robot context for the existence of the common library and gets a reference to it, importing the library first if necessary. This means that I have a reference to the common library within all my other libraries and means that whenever I import any of my libraries in the Robot settings table, I have access to the common library keywords too.
The issue is that when running multiple Robot tests in sequence, the common library seems to disappear. I have a few Robot scripts in a directory and run "robot *.robot". In each test, I run a keyword from the common library. I never import the common library in the settings table, as it should be automatically imported by the other libraries as outlined above. In the first test, the common library exists and the keywords in it work fine. In all the following tests, I am getting a keyword not found error. When I print the results of BuiltIn().get_library_instance(all=True) I can see that, while my application-specific library is still loaded, the common library no longer exists.
The Problem
All my libraries have ROBOT_LIBRARY_SCOPE = 'GLOBAL' in them, including the common library. My common library is dynamically imported via BuiltIn and has a global scope defined, but it seems to fall out of scope when running subsequent tests in one command. Is there a reason that dynamically imported libraries fall out of scope even if they have a global library scope?
Essentially, I want this common library to always be available in my robot scripts and have each of my custom libraries maintain a reference to the common library. If there is a better way to accomplish this or some way to make what I'm currently doing work, please let me know! Thanks.
A solution may be to unconditionally import the common library in all your custom libraries. E.g. in their constructors (__init__()) call this:
BuiltIn().import_library("my_common_lib", True)
Thus you will always have its keywords in scope. Naturally if that common lib does a step that must be ran only once (affects some resource, for example), that had to be accommodated for in it (using a singleton pattern, or something similar).
Edit: come to think of it, this might also not work, the __init__() will be called just once as the libraries are with global scope; and thus the common one's keywords will once again not be imported in the suite's namespace.
Enter RF's listener interface :) : in your custome libraries define the class method suite_start(), and move the try-except block in it. On the start of every suite that uses such library, the method will be executed, and the common keywords - available.
The same precaution as two paragraphs above - make sure reimporting the common library has no side effects.
Another solution may be to change the custom libraries scope to 'TEST SUITE', as you have already deducted yourself (and are reluctant to do, based on the comments :).
Thus the custom libraries will be reinstantiated on every import in the suites, and they will import the common library in the suite's namespace.
I an writing an app to record data given by the user, calculate and print out bills. I have written a script usage.py in MainFolder and want to pull its methods or functions in MainFolder/file/MainFile.py . I have already used the methods or functions in usage.py in MainWin.py
The scripts are in the 3 py extensions are pasted here
How can I use them in methods in MainFolder/usage.py in MainFolder/file/MainFile.py ?
If you make the folder a package that will allow you to use the methods in either location. But, to give further advice it would make more sense to move the methods that are in usage.py into the MainFile.py from a design standpoint. It doesn't make a lot of sense to use functions in a top level file in a nested file.
If you make it a module you can use:
from some.package.usage import policySize
Module documentation:
http://docs.python.org/tutorial/modules.html