I need to make some changes to the way the OSMnx package gets data from the Overpass API.
To do that, I've forked the repo and cloned that fork into a local directory.
I've made a simple change, adding a print functions to start.
I'm trying to figure out the steps for installing and importing the modified version of the package. I've looked at this question
I'm concerned about a namespace conflict, should I change the name of the folder or the name value in the setup.pyfile. I don't want to write over the working version of the package
What file should I point to when installing? setup.py, core.py, _init_.py?
I haven't found any tutorials on how to modify an existing package safely, just stuff on how to make a package from scratch.
import will search the installed packages list first, then it will check the directory it was called from for files that match the package it's looking for.
So either using pip or conda remove the original version of the package being imported.
Then, be sure that the modified version of the package that you want to import is in a sibdirectory of your project directory, and run import package as xx and it should load the modified package files.
If it doesn't, it's likely that the package wasn't removed from the right environment.
Related
I want to customize a python package. I need to be able to install the customized package in anaconda on other machines. This is the method by which i'm installing the customized package in case it's relevant: How do I install packages to Anaconda from locally built tar.gz
However i would also like to rename the package. For example if i wanted to edit openpyxl, i want my edited package to be called openpyxl_edit. However the name openpyxl is used in multiple places throughout the original package source code.
Is it likely that i have to rename openpyxl to openpyxl_edit in all of the occurrences in all of the source files in the entire package? Is there an easier way of changing the name of a package?
I came across an article on serverlesscode.com about building Python 3 apps for AWS Lambda that recommends using pip (or pip3) to install dependencies in a /vendored subdirectory. I like this idea as it keeps the file structure clean, but I'm having some issues achieving it.
I'm using Serverless Framework and my modules are imported in my code in the normal way, e.g. from pynamodb.models import Model
I've used the command pip install -t vendored/ -r requirements.txt to install my various dependencies (per requirements.txt) in the subdirectory, which seems to work as expected - I can see all modules installed in the subdirectory.
When the function is called, however, I get the error Unable to import module 'handler': No module named 'pynamodb' (where pynamodb is one of the installed modules).
I can resolve this error by changing my pip installation to the project root, i.e. not in the /vendored folder (pip install -t ./ -r requirements.txt). This installs exactly the same files.
There must be a configuration that I'm missing that points to the subfolder, but Googling hasn't revealed whether I need to import my modules in a different way, or if there is some other global config I need to change.
To summarise: how can I use Pip to install my dependencies in a subfolder within my project?
Edit: noting tkwargs' good suggestion on the use of the serverless plugin for packaging, it would still be good to understand how this might be done without venv, for example. The primary purpose is not specifically to make packaging easier (it's pretty easy as-is with pip), but to keep my file structure cleaner by avoiding additional folders in the root.
I've seen some people using the sys module in their lambda function's code to add the subdirectory, vendored in this case, to their python path... I'm not a fan of that as a solution because it would mean needing to do that for every single lambda function and add the need for extra boiler plate code. The solution I ended up using is to modify the PYTHONPATH runtime environment variable to include my subdirectories. For example, in my serverless.yml I have:
provider:
environment:
PYTHONPATH: '/var/task/vendored:/var/runtime'
By setting this as an environment variable at this level it will apply to every lambda function you are deploying in your serverless.yml -- you could also specify it at a per lambda function level if for some reason you didn't want it applied to all of them.
I wasn't sure how to self reference the existing value of PYTHONPATH to ensure I wasn't incorrectly overwriting it while in the process of adding my custom path "/var/task/vendored"... would love to know if anyone else has.
In short, my question is, how do I install the latest version of scikit-image into my usr/lib/python3/dist-packages so I can actually use it? I think there is a problem with my understanding of how third-party modules are installed. As a newb, I don’t know how to rectify that, hence this post.
I need help to understand how to install packages in python3 up until now I have used pip/pip3/apt-get/synaptic etc and it has worked fine for many packages. However, I have hit several barriers (Skimage, opencv, plantcv in python3). I must emphasise, the problem I am having is using these packages in python3, not 2.7.
For example, I want to use the latest version of scikit-image (0.14) with python3. (http://scikit-image.org/) I have tried using the installation instructions and have not yet successfully managed to install it. I have navigated to my usr/lib/python3/dist-packages and copied scikit-image into this directory (I have all the dependencies installed in here already).
Image of my folder for dist-packages as proof
As you can see, the folder containing skimage is in the directory I want to be installed in, how do I actually install it? Do I have to extract skimage out of the folder into the directory and then run the install command? If I navigate to usr/lib/python3/dist-packages/scikit-image and then run pip install -e . I get an error, stating that I need numpy. If I write a python script using python3 I can clearly see I have it installed (and I have been using it for a long time). So, there must be a problem in how I have this package in my file system. I think a janky workaround would be to copy all the modules into my working directory and Import them that way as if they were modules I have made myself, but this obviously negates the whole point of installing packages.
This has also happened with another package called plantcv. Where I went into the directory usr/lib/python3/dist-packages then cloned the source from git hub and installed as per instructions. When I import plantcv in my python3 script. It Imports fine. But, there is nothing in it, as python cannot see the modules which are inside this folder at usr/lib/python3/dist-packages/plantcv/plantcv.
There is clearly some comprehension here that I am missing, as I have a similar problem for two packages now. Please, Internet. Help me understand what I am missing!
You simply need to copy the folder in /usr/lib/python3/dist-packages/package-name
However, there are certain things that are specific to python packages. The folder named package name should be a valid package. A good indicator of that is it will contain a file "__init__.py". It is very likely that every sub-directory inside this package directory will contain a "__init__.py" file. It depends on whether there are modules inside these sub-directories.
In your code simply import the package like the following.
import package-name
where package-name can be skimage
I'm interested in contributing to a GitHub Python module repo, but I'm not entirely sure where to clone it. This is a simple module, just an __init__.py and some .py files. No other files need to be installed or changed outside of the module's folder.
I would like to be able to clone the repository directly in my site-packages folder. When I want to use the library as is, I would switch to the master branch. If I want to develop a new feature, can branch off of devel. If I want to try out a new feature someone else implemented, I can switch to that particular branch. I can even keep it in the development branch, to get the latest, albeit possibly unstable, features. All this without having to change the import statement to point to a different location in any of my scripts. This option, even though is seems to do all the things I want it to do, seems a bit wrong for some reason. Also, I'm not sure what this would do to pip when calling python -m pip list --outdated. I have a feeling it won't know what the current version is.
Another option would be to clone it to some other folder and keep only the pip-installed variant in the site-packages folder. That way I would have a properly installed library in site-packages and I could try out new features by creating a script inside the repo folder. This doesn't seem nearly as flexible as the option above, but it doesn't mess with the site-packages folder.
Which is the best way to go about this? How do you clone repositories when you both want to work on them and use them with the latest features?
I think this is more a question about packaging and open source than Python itself, but I'll try to help you out.
If you want to host your package on Pip, you should go here, and there you'll see how to upload and tag appropriately your package for usage.
If you want to add some functionality to some open source library, what you could do is to try to submit a Pull Request to that library, so everybody can use it. Rules for PR are specific for each project, you you should ask the maintainer.
If your modification doesn't get merged into master, but you still want to use it without changing import statements, you could fork that repo, and publish your own modifications on, for instance, Github.
In that case, you could install you modifications like this:
pip install git+https://github.com/username/amazing-project.git
So in that way, your library will come from your own repo.
If you're going for the third option, I strongly recommend you using virtualenv, where you can create different virtual environments with different packages, dependencies and so on, without messing up with your Python installation. A nice guide is available here.
Why we have to install the python packages before using them?
I am currently working on a small python mysql program. What i tried to download the python connector module from mysql webpage and simply unzip it and place it in the same folder of my code.
And I can import the module properly.
So what is the meaning of installing those packages? Can I use those packages like matplotlib, numpy without installing them ?
Is it possible to have all the required packages installed on a folder so that i can move it to another computer and run my program with only CPython installed (I don't want to install any package on this computer)?
it's not that simple :-)
some packages have dependencies, you also need to download and extract their dependencies (you need pacakge x,and package x uses y) pakcage manager handles that
some package have some c code (they need to be compiled before use (ujson or postgres module) package manager handles that
when your share your code instead of sharing dependencies you simply add a file containing the list of dependencies (requirements.txt) and other user can simply install all dependencies using package manager
Installing a python package enables us to use it anywhere on our system. If we just place the package in the same directory as our script then it may well work, but only for scripts in that directory.
Some packages also rely on others to function properly, and the installation of a package may well install those pre-requisite packages for you. You may be able to do this manually, but you'd have to put them all in the same directory as your script every time you wanted to run it.
So installing the packages is the easiest way to use them.
You don't have to install them, and in some cases you wouldn't install them on your system; if you had split your code across two files and imported one file at the top of the other for example.
In fact, you don't really need install package on your system.
But if you install it, you can use these packages every where on your system.
Also, you can create a requirement.txt file to enable install all packages that you need on other computer. You can check this manual https://pip.pypa.io/en/stable/user_guide/#requirements-files