Here's what I have:
Project A
----data_I_need.p
Project B
----use_data.py
If data_I_need.p were a source file (and in my path), I could easily import it into Project B. Does such method exist for non-source files?
Right now, I'm having to accomplish this with
with open('C:/......./data_I_need.p', 'rb') as data:
Project A is in my path (defined as 'Content Root' in PyCharm). I can imput source files from Project A.
To recap, the problem is that any Python file in PYTHONPATH can be imported, but resources (non-python files) cannot be loaded without an explicit path.
Project_A
----some_module.py
----data_I_need.p
Project_B
----use_data.py
Here is a solution. In Project B/use_data.py, include the following.
import os
import pathlib
import pickle
# full path to Project_B
here = os.path.dirname(__file__)
# path to folder *above* Project_B ---- joined to Project_A
PROJECT_A = os.path.join(*pathlib.Path(here).parts[:-1], 'Project A')
with open(os.path.join(PROJECT_A, 'data_i_need.p')) as resource:
var = pickle.load resource
This will work even if you're working in a shared drive (e.g., OneDrive) from two different operating systems.
Optionally, you could just set up a "shared_resources" folder and include a python script in that folder to access the resources, again working from os.path.dirname(__file__) then navigating the relative path.
Related
I have a project on Gitlab that should download files to its current directory (regardless of user OS being Mac/Windows) when the user downloads the repo and uses it on locals. Also, I will read the same file again further in the program.
Can anyone suggest to me what directory name should I pass in the python variable to achieve this?
Thank you
If you want to download to the same folder as your python file, you can do
from pathlib import Path
dir_to_download_to = Path(__file__).parent
If you want to download to the same folder as the user is in, then
import os
dir_to_download_to = os.getcwd()
I want to create a multi-serivce app engine app, using the first diagram shown on this page.
https://cloud.google.com/appengine/docs/standard/python/configuration-files#an_example
I want to use third party libraries so I used have a lib folder under the root directory, one yaml under the root directory.
Then I want one microservice called predict. So I created a folder called predict under the root directory as well, then under this folder, I want to write py files using the packages in the lib as well.
What I'm doing is like this:
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
vendor.add(os.path.join(parentdir, 'lib'))
Which didn't work for me as the error says:
ValueError: virtualenv: cannot access /base/data/home/apps/f~project-name/lib: No such virtualenv or site directory
Obviously it didn't work for me, what's the right way to do this?
Update: Dan's solution worked for me! So when I'm deploying my microservice predict, I need to go inside predict directory and deploy it. I think that's why it can't find lib. By symlink the lib library using bash.
ln -s ../lib/ ./lib
I solved this problem.
The problem you're facing is that each service can only access files in its own service directory, which is the directory where its app.yaml file is located, they can't access the app's root directory.
My approach (all my services use standard environment):
created a lib dir in the top app dir and installed in it each package that I want shared across multiple services
each service has its own subdir in the app's dir and a lib directory in it where:
I installed libraries needed only by that service (if any)
I symlinked the needed shared libraries from the top lib dir
This is my app dir structure:
app_dir/
app_dir/dispatch.yaml
app_dir/cron.yaml
app_dir/index.yaml
app_dir/queue.yaml
app_dir/lib/
app_dir/lib/shared_lib_1/
app_dir/lib/shared_lib_2/
app_dir/lib/shared_lib_3/
app_dir/service1/
app_dir/service1/app.yaml
app_dir/service1/lib/shared_lib_1 -> ../../lib/shared_lib_1
app_dir/service1/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service1/lib/service1_lib_1
app_dir/service2/
app_dir/service2/app.yaml
app_dir/service2/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service2/lib/shared_lib_3 -> ../../lib/shared_lib_3
app_dir/service2/lib/service2_lib_1
No fumbling with the lib path is required, all I have in each service is
vendor.add('lib')
See related:
How do I access a vendored library from a module in Python Google App Engine?
Can a default service/module in a Google App Engine app be a sibling of a non-default one in terms of folder structure?
Check all the directories in the path and the final file to make sure they all exist first before code runs. If they don't, prepend the code you showed here with code to create them...
I am attempting to make a Python testing script module self-contained directory-wise for scalability and maintainability purposes.
I have done some research and haven't been able to find the answer i'm looking for. Basically, I would like to determine if there is a function in Python that can do something similar to cd - in shell.
I would generally like to avoid typing in full path-names, and have all the paths relative to a specified environment.
Example:
I have my main directory where my scripts are located python-scripts, I have folders for each testing environment demo, and a screenshot folder for each testing environment demo-screenshots.
If i wanted to save a screenshot to the screenshot with Python, while working in the demo directory, I would just send it to the directory below: demo-screenshots, using the path '/demo-screenshots/example.png'. The problem is that I don't understand how to move back a directory.
Thank you.
You're looking to change the working directory? The OS module in python has a lot of functions to help with this.
import os
os.chdir( path )
path being ".." to go up one directory. If you need to check where you are before/after a change of directory you can issue the getcwd() command:
mycwd = os.getcwd()
os.chdir("..")
#do stuff in parent directory
os.chdir(mycwd) # go back where you came from
path = os.path.dirname(__file__)
print(path)
will print the CWD of the file, say C:\Users\Test\Documents\CodeRevamp\Joke
path2 = os.path.dirname(path)
print(path2)
will print the Parent directory of of the file: C:\Users\Test\Documents\CodeRevamp
I am writing a simple scheduling service. I don't want to hard-code all of the tasks it can schedule and instead would like to support plugins that can be dropped in a folder and loaded dynamically at runtime.
My plan is to have a JSON file (or any configuration file) that maps a task name to the location of a Python file (a module) which will have a class called Plugin. Pretty simple I thought. When someone schedules a task to run, they pass the task name and the time to run it. When the time elapses, the plugin is loaded (or reloaded) and is ran with any additional arguments passed to the scheduler.
I have been looking at the imp module to see how to load modules at runtime. I am not sure whether I want to list plugins using their physical location (file system path) or to use their module names like you'd see in a import statement. It seems imp wants to use physical location.
I got two different versions of this code "working". Here is one that uses importlib:
pluginName = self.__pluginLookup[pluginName]
module = import_module(pluginName)
module = reload(module) # force reload
plugin = module.Plugin()
return plugin
This is one I wrote using imp:
path = self.__pluginLookup[pluginName]
path, moduleName = split(path)
moduleName, extension = splitext(moduleName)
file, path, description = find_module(moduleName, [path])
with file:
module = load_module(moduleName, file, path, description)
plugin = module.Plugin()
return plugin
The problem I am running into is handling dependencies. If I have a plugin.py file that depends on a dependency.py file in the same folder, saying import dependency doesn't seem to work. Instead, it looks up the dependency from the PYTHONPATH.
How can I make the imports relative to the plugins themselves?
You could append path to sys.path:
import sys
sys.path.append(path)
where path is the directory containing the dependency.py.
If you have a plugins directory with an __init__.py, you can add that directory to sys.path. Then modules inside there can do from . import dependency to import another plugin. Or, if the plugin is itself a subpackage (i.e., a directory with its own __init__.py) then it can do from . import dep to import a dependency within the same plugin, or from .. import dep to import a dependency from the global plugins directory. With this setup you don't even need to use imp or the like; you can use the __import__ function, which works by module name.
One possible wrinkle, though, is you say the plugins directory will be "under the current working directory". What does that mean exactly? You mean you want people to be able to add plugins anywhere on the drive and still use them? It would be better to have one central plugins directory for your app, and add that to sys.path, and tell people to put their plugins there.
I have the following directory structure for my GAE project:
project:
library:
lib1.py
lib2/x.py
lib2/y.py
apps:
app1/app1.py
app2/app2.py
app2/async.py
how do I make the library folder visible to any app that will ever be created in the apps dir and its subdirs?
Alternatively it's possible to add the library directory to the sys.path
Create a __init__.py inside the library folder.
import os
import sys
def add_lib_path():
lib_directory = os.path.dirname(os.path.abspath(__file__))
if lib_directory not in sys.path:
sys.path.insert(0, lib_directory)
In every file where you import libraries from the library folder add this code before the import statements:
from lib import add_lib_path
add_lib_path()
In this case all your imported libraries will behave as expected.
PYTHONPATH specifies a series of folders to start searches for imported modules.
GAE adds the folder that contains app.yaml to your PYTHONPATH.
So assuming that app.yaml is in the root of that structure (ie the folder that contains "library" and "apps") then any of your apps can import relative to there...
from library import lib1
from library/lib2 import x