I have 10 django projects that use over 50 django apps. Each app is separated in its own project and added to pypi and is getting use by few project. Every thing if fine except every time i work on a project and i want to change some code that is in one of my modules (that happens a lot) I have to open the module project, make my changes, test and publish to pypi then come back to my project update requirements.txt file and get the updated module from pip.
I'm looking for a way to be able to edit module right away from all of my projects. For example instead of getting it from pypi i want to get it from git and be able to commit to the git repository in my venv folder!
I know it seems a little bit crazy but i could save a lot of time! publisher and user of all of the modules is me so I don't mind the user to be able to change as well.
Any thought or suggestion will be appreciated. Also any none pip solution will be fine as well like writing a custom shell script.
I don't know about editing in your venv folder, which I think is not a good practice, but you can install from github by pip. You can use 'pip install git+https://github.com/urltoproject/repository.git'. Fill in the necessary details yourself of course. This also works with other systems like gitlab. You could have a separate development requirement file and a production requirement file to separate the two environments, or you install on the commandline directly with pip.
Related
I have a little Python side project which is experiencing some growing pains, wondering how people on larger Python projects manage this issue.
The project is Python/Flask/Docker deployed to AWS. Listed dependencies (that we import directly in the project) are installed from a requirements.txt file with explicit version numbers. We added the version numbers after noticing our new deployments (which rebuild Docker/dependencies etc) would sometimes install newer versions of the packages, causing the project to break.
The issue we're facing now is that an onboarding developer is setting up her environment and facing the same issue - this time with sub-dependencies of the original dependencies. (For example, Flask might install Werkskreug, Jinja2, etc and if some of these are the wrong version, the app breaks.) The obvious solution is to go through each sub-dependency and list out every package, with explicit versions, in requirements.txt. But this is a bit of a pain so I'm asking around to see what people do on Real Projects.
You guys can't be doing this all manually, right? In JS we have NPM and package.lock files and so on - they're automatically built. Is there some equivalent in Python? Have I missed something basic that we should be using here?
Thanks in advance
I wrote a tool that might be helpful for this called realreq.. You can install it from pip pip install realreq. It will generate the requirements you have by reading through your source files and recursively specifying their requirements.
realreq --deep -s /path/to/source will fully specify your dependencies and their sub-dependencies. Note that if you are using a virtual environment you need to have it activated for realreq to be able to find the dependencies, and they must be installed. (i.e realreq needs to be ran in an environment where the dependencies are installed). One of your engineers who has a setup env can run it and then pass the output as a requirements.txt file to your new engineers.
i am new to Python programming language and Django. I am learning about web development with Django, however, each time I create a new project in PyCharm, it doesn´t recognize django module, so i have to install it again. Is this normal? Because i´ve installed django like 5 times. It doesn´t seem correct to me, there must be a way to install Django once and for all and not have the necessity of using 'pip install django' for each new project I create, I am sure there must be a way but I totally ignore it, I think I have to add django to path but I really don´t know how (just guessing). I will be thankful if anyone can help me :)
pycharm runs in a venv. A venv is an isolated duplicate (sort of) of python (interpreter) and other scripts. To use your main interpreter, change your interpreter location. The three folders (where your projects is, along with your other files) are just that. I think there is an option to inherit packages. I like to create a file called requirements.txt and put all my modules there. Comment for further help.
In conclusion, this is normal.
There are 2 ways of starting your project in PyCharm.
To use a virtual environment where you have to install the external libraries into this environment.
To start using the existing interpreter. (If you chose this option you don't have to install it again installing it only once globally would be enough)
So when you start to create a new project choose the existing interpreter.
Doing this will not require you to install it every time(But is not recommended for big projects as there would dependencies on different versions of libraries).
I'm interested in contributing to a GitHub Python module repo, but I'm not entirely sure where to clone it. This is a simple module, just an __init__.py and some .py files. No other files need to be installed or changed outside of the module's folder.
I would like to be able to clone the repository directly in my site-packages folder. When I want to use the library as is, I would switch to the master branch. If I want to develop a new feature, can branch off of devel. If I want to try out a new feature someone else implemented, I can switch to that particular branch. I can even keep it in the development branch, to get the latest, albeit possibly unstable, features. All this without having to change the import statement to point to a different location in any of my scripts. This option, even though is seems to do all the things I want it to do, seems a bit wrong for some reason. Also, I'm not sure what this would do to pip when calling python -m pip list --outdated. I have a feeling it won't know what the current version is.
Another option would be to clone it to some other folder and keep only the pip-installed variant in the site-packages folder. That way I would have a properly installed library in site-packages and I could try out new features by creating a script inside the repo folder. This doesn't seem nearly as flexible as the option above, but it doesn't mess with the site-packages folder.
Which is the best way to go about this? How do you clone repositories when you both want to work on them and use them with the latest features?
I think this is more a question about packaging and open source than Python itself, but I'll try to help you out.
If you want to host your package on Pip, you should go here, and there you'll see how to upload and tag appropriately your package for usage.
If you want to add some functionality to some open source library, what you could do is to try to submit a Pull Request to that library, so everybody can use it. Rules for PR are specific for each project, you you should ask the maintainer.
If your modification doesn't get merged into master, but you still want to use it without changing import statements, you could fork that repo, and publish your own modifications on, for instance, Github.
In that case, you could install you modifications like this:
pip install git+https://github.com/username/amazing-project.git
So in that way, your library will come from your own repo.
If you're going for the third option, I strongly recommend you using virtualenv, where you can create different virtual environments with different packages, dependencies and so on, without messing up with your Python installation. A nice guide is available here.
I'm a Java/Scala dev transitioning to Python for a work project. To dust off the cobwebs on the Python side of my brain, I wrote a webapp that acts as a front-end for Docker when doing local Docker work. I'm now working on packaging it up and, as such, am learning about setup.py and virtualenv. Coming from the JVM world, where dependencies aren't "installed" so much as downloaded to a repository and referenced when needed, the way pip handles things is a bit foreign. It seems like best practice for production Python work is to first create a virtual environment for your project, do your coding work, then package it up with setup.py.
My question is, what happens on the other end when someone needs to install what I've written? They too will have to create a virtual environment for the package but won't know how to set it up without inspecting the setup.py file to figure out what version of Python to use, etc. Is there a way for me to create a setup.py file that also creates the appropriate virtual environment as part of the install process? If not — or if that's considered a "no" as this respondent stated to this SO post — what is considered "best practice" in this situation?
You can think of virtualenv as an isolation for every package you install using pip. It is a simple way to handle different versions of python and packages. For instance you have two projects which use same packages but different versions of them. So, by using virtualenv you can isolate those two projects and install different version of packages separately, not on your working system.
Now, let's say, you want work on a project with your friend. In order to have the same packages installed you have to share somehow what versions and which packages your project depends on. If you are delivering a reusable package (a library) then you need to distribute it and here where setup.py helps. You can learn more in Quick Start
However, if you work on a web site, all you need is to put libraries versions into a separate file. Best practice is to create separate requirements for tests, development and production. In order to see the format of the file - write pip freeze. You will be presented with a list of packages installed on the system (or in the virtualenv) right now. Put it into the file and you can install it later on another pc, with completely clear virtualenv using pip install -r development.txt
And one more thing, please do not put strict versions of packages like pip freeze shows, most of time you want >= at least X.X version. And good news here is that pip handles dependencies by its own. It means you do not have to put dependent packages there, pip will sort it out.
Talking about deploy, you may want to check tox, a tool for managing virtualenvs. It helps a lot with deploy.
Python default package path always point to system environment, that need Administrator access to install. Virtualenv able to localised the installation to an isolated environment.
For deployment/distribution of package, you can choose to
Distribute by source code. User need to run python setup.py --install, or
Pack your python package and upload to Pypi or custom Devpi. So the user can simply use pip install <yourpackage>
However, as you notice the issue on top : without virtualenv, they user need administrator access to install any python package.
In addition, the Pypi package worlds contains a certain amount of badly tested package that doesn't work out of the box.
Note : virtualenv itself is actually a hack to achieve isolation.
In my free time I am developing a small python project based on django. I use several other python packages to improve user and my development experience.
I have understand, that I run pip install pkgname to install a module named pkgname and then I use from pkgname import somewhat and go one and everything is fine and fluffy. I understand why to use virtualenv and already use serious tools like coverage and travis-ci.
Now, I want to the next step. At the moment, my project is not ready for world wide deployment. But I like to prepare for this as soon as possible. It's called Palco and hosted on GitHub.
Usually, I think, pip is the current default to install python modules. But the application I develop is nothing developers will use in their project but users will. So, nobody will do pip install palco and then write from palco import hellyeah. But they will configure their favourite web server to run django's wsgi-handler.
Currently, I would tell people,
please download this archive.zip, extract it and configure your webserver as mentioned
But this is a very php-way of deployment (just throw your files in a directory).
How make I my end-user python project deployable? Do I need a setup.py? Do I miss something?
I know, trac is installed with pip and then I "deploy" my very on trac-instance with trac-admin.py. Should I have a palco-admin.py with the same magic effect as trac-admin?