I've just setup a new environment for my project and uploaded a python repository including bin, lib and project folder. I'm pretty sure I did same previously and it worked without problem. Now when doing the same on an AWS environment I get the error
-bash: /projects/scrapy/bin/python2.7: cannot execute binary file. However when doing source /projects/scrapy/bin/activate it successfully activates the environment.
From what I understand, python should be able to execute without any issue no matter the environment ?
Any help or pointing to the right direction would be much appreciated!
python should be able to execute without any issue no matter the environment ?
No, the Python binary is tied to your specific OS and computer architecture. Python source code can usually be run on different machines (provided you didn't use OS-specific features), but that's only made possible by compiling a Python interpreter for the specific target environment first.
In other words, a Python binary compiled to run on macOS will not work on Linux.
All that source bin/activate achieves is that it configures your terminal setting to use the bin directory as the first directory on the PATH search path. This doesn't make bin/python work in another environment, it just means that both environments have a working shell interpreter that can run that script.
Create a new virtualenv with a Python binary compiled for Linux, and install the same packages there. Use Pipenv or a requirements.txt file to transfer the dependencies from Mac to Linux.
For example, using Pipenv you'd copy over the Pipfile and Pipfile.lock files to the other computer, then run pipenv install in the directory there and re-create the virtualenv and dependencies from those files.
I recommend you read up on Python development best practices in the The Hitchhiker’s Guide to Python; this includes such topics on how to manage an environment for a project.
Related
I have a Ubuntu server with restricted access. There I will be hosting my application.
I trying to run Python scripts which were working with the default packages provided by the server. I want to work with numpy and other modules.
As I cannot install or download or do anything, I created a python server in my local machine (WINDOWS) using WSL to emulate the Linux file system and copied the python environment files to the application directory and deployed in cloud.
The problem is no matter in whatever way I try I cannot import numpy (or any module which I copied). I moved all the site-packages to the location of my Python script (As the current script's path will be there in the system path) and tried to import but no luck.
Please help me with crack this in any possible or impossible way.
I am trying to achieve this for the past 6 days and cannot do it.
Please, I have to achieve this at any cost. I have attached my latest structure.
Thank you in advance.
My Folder structure screenshot:
EDIT:
Ok. Let me get this straight. I have a Linux server (Ubuntu 18.04) where I am hosting an application. From that application, I am calling python scripts for some machine learning purposes. It is restricted server and I cannot access it. The only way that I found out the Linux distro version is through Java code by calling some terminal commands using "ProcessBuilder". As the server is highly restricted I cannot run any of the Linux commands like echo, set, export, sudo, wget/curl,...etc., Since, python3 is already provided by Linux (by default) I am using that python3 command to call my python scripts (from Java code using "ProcessBuilder") and execute them.
If it is a normal script (if I am using python standard libraries) it is working fine. In one of the scripts I am using "numpy". So, I want to import that module. I am doing the development in a windows environment. So, to emulate the Linux file system for importing packages I created a virtual environment in WSL with same Ubuntu version and installed numpy and then replaced all the symlinks inside those packages with the required files. Then I copied the entire environment and pasted in my resources directory (which is in windows environment) and deployed. No luck.
So, I made a zip file for only "site-packages" folder inside that environment. Then I copied the zip file and pasted in my resources folder and deployed. No luck. The error that I always see is "numpy.core._multiarray_umath". All the articles and in GitHub also tell us to re-install the package. But, I cannot install. I don't have any such access.
How can I import numpy without installation? If there is any work around to achieve this please explain, I will do it. Even if it is harder, complex and time-consuming I am okay with it. I want to achieve this.
Let me preface this with:
a warning to please check the AUP (acceptable use policy) of the server you are using, and/or contact the server administrator to make sure you are not violating any rules.
I can think of quite a few reasons why this won't work. If it doesn't, then there may still be workarounds, but they'll be technically complex.
So if I'm understanding you correctly:
You have very limited access to the server; basically only the ability to upload (apparently) and run Java code.
You've also been able to upload Python code and run it through your Java code through ProcessBuild.
You do not have access to log in to a shell, execute arbitrary command other than through ProcessBuild, etc.
Of course, you do not have the ability to install site-packages into the system Python environment.
So ultimately, what you'll probably need to do is something like:
Create a Python3 virtual environment (which doesn't seem to be what you are actually doing) on WSL. By a "Python3 virtual environment", I mean venv, which allows you to create a user-level (not system-level) directory with your packages.
So something like (from inside your project directory):
python3 -v venv venv
source ./venv/bin/activate
Your path will be adjusted so that your python3 and pip3 commands will be found in the venv path. pip3 install numpy will install it into this virtual environment (not the global/system Python).
Upload that entire venv directory to the server. You seem to have some way of doing this already.
You're going to have to have some way of running the Bash shell through ProcessBuilder. Since you have the ability to run python3 through ProcessBuilder, I'm kind of assuming that you will be able to do this as well.
You'll need to (through ProcessBuild) activate the virtual environment on the server, <path_to_project>/venv/bin/activate and, in the same Bash shell run your code.
This will look something like:
bash -c "source ./venv/bin/activate; python3 main.py"
Started to learn python a few weeks ago, i made and ran some basic files. all worked well.
At one point i uninstalled python and reinstalled it.
i continued making and running new files no problem.
now i am trying to go back to and run some of my old files and i am getting an error.
Error:Cannot run program "/Users/paulthomas/PycharmProjects/plotting/venv/bin/python" (in directory "/Users/paulthomas/PycharmProjects/plotting"): error=2, No such file or directory
In pycharm i open the folders and hover over the python file section and it says i have a symlink error. How can i fix it?
i have been using pipenv in my newer projects as they are django projects. But this project im trying to get to run is just a basic python project.
If you have reinstalled python, it is likely that the virtual env is linking to the wrong original python binaries.
You can either relink the files (by finding what they should have linked to, and relinking them) or if you have a requirements.txt file for the environment, delete and recreate the virtual env.
You can either do this through pycharm's environments system, or by using:
python3 -m venv <your environment name>
From your screenshot, your environment name is venv.
If the reinstalled python is a different version, it's likely safer to recreate the virtual env than to try linking things that may end up being inconsistent.
I wanted to replace Python 3.8 32-bit with the 64-bit version to install the face_recognition module, so I deleted the previous version and tried to re-route the project to the new Python version by going to File > Settings > Project Interpreter > Show all > Show Paths for Selected Interpreter, and adding all the Python files from the new folder and getting rid of the old ones.
However, it's still showing me this error when I try to install the module:
(Will2.0) C:\Users\solei\PycharmProjects\Will>pip install face_recognition
No Python at 'C:\Users\solei\AppData\Local\Programs\Python\Python38-32\python.exe'
I've also tried going to the Windows System Properties and changing everything that says "Python38-32" there, but it's still not working. It does work when I make a new environment, though, so at least I know that Python installed properly. It's just this one environment that is tripping me up (I'd prefer not to make a new project for this, btw. I've already installed a lot of modules in it.).
Your selected interpreter is not the system interpreter you've replaced with the 64-bit version, but your project's virtual environment interpreter. The virtual environment's files weren't changed in that process and need to be updated before you can use that environment again.
The system interpreter is your Python interpreter installed using the installation executable. In your case it is located in C:\Users\solei\AppData\Local\Programs\Python\Python38\. You can have multiple system interpreters installed, such as having Python 2.7, Python 3.7 and Python 3.8 side-by-side.
The virtual environment interpreter is a copy of another interpreter created using the venv package from the Python standard library. You can have many virtual environments interpreters in the system (one or more for every project, for example)
The base interpreter is the interpreter that was used as a template for the venv package. Every virtual environment interpreter has its base interpreter (usually a system interpreter) that it requires to run. Changing or upgrading the base interpreter requires updating the virtual environment.
If we take a quick look at the documentation, a virtual environment is described as
a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.
That means you can setup an individual environment for every project, which will contain its own packages. The environment is a very efficient way of managing project packages, that's why PyCharm suggests a creation of such environment over the system interpreter by default. In short, it allows you to have two different versions of the same package used by two different projects, without the packages conflicting with each other.
This also explains why your virtual environment files weren't affected by your upgrade.
Now, I am unfortunately no Python expert. I had to spend some time examining how Python handles virtual environments on Windows and Ubuntu. It seems the environment always requires the base system interpreter present in the system. If you remove or change the location of the base interpreter, the environment will fail to function.
As I mentioned before editing this answer, you can in theory simply edit the pyenv.cfg file located in the root folder of the virtual environment. In practice, that will only work in simple cases and it is not the intended way of updating virtual environments.
You need to upgrade your virtual environment's files to work with your new system interpreter. That can mean the 64-bit version over the 32-bit version, or even a newer version of Python - such us upgrading from 3.7 to 3.8.
Close PyCharm
Check if the system interpreter you want to upgrade to is on the system Path
You can quickly check by running
python -c "import platform; print(platform.architecture())"
For you, the output should look like this
('64bit', 'WindowsPE')
If your output is different, you'll need to prefix the absolute path to the Python executable in step 4).
Navigate to the virtual environment's directory
The directory you're looking for contains the Include, Lib and Scripts directories and the pyenv.cfg file. From your screenshots, it seems this directory is your project's root directory, so in your case:
cd C:\Users\solei\PycharmProjects\Will2.0\
Upgrade the virtual environment
python -m venv --upgrade .
... or if Python is not on your path
C:\Users\solei\AppData\Local\Programs\Python\Python38\python.exe -m venv --upgrade .
The . in the commands refers to the current directory.
Open PyCharm and verify your environment is working correctly
... or simply try to run pip directly from the command line. Note you need to first activate the virtual environment by running the Scripts\activate.bat batch file.
If the above-mentioned method doesn't work, you might have to create a new virtual environment. You can create one easily without making a new PyCharm project. See this PyCharm documentation for reference. However, you'll still need to redownload all the required packages again.
For the simplicity, I recommend creating the new virtual environment in a .venv folder located in the project's root.
Disclaimer
I tested only the Python's behavior alone on a fresh Windows installation inside the Windows Sandbox. I was able to install the 32-bit Python, create a virtual environment, replace Python with the 64-bit version and upgrade the virtual environment to have it launch correctly again.
I was required to install anaconda for a CS course and used spyder and Rstudio.
Then, for a different class I used pycharm.
When I type on the command line "python -V" I get:
Python 3.6.1 :: Anaconda 4.4.0 (x86_64)
and I have no idea why it relates the python version I have installed with Anaconda (and why not pycharm?). I understand that the OS runs python 2.7 (shouldn't I get that instead? and when I type python3 -V get which version of python 3 I have?) and when I use something like Pycharm or Spyder I can choose which version I want from the ones I have installed and use it within the program, not for the terminal.
I just want to have everything in order and under control. I don't think I understand what Anaconda really is (to me is like a program that has more programs in it...). How do I keep anaconda to itself ? 1313
Also, should the packages I installed through Terminal work on both pycharm and spyder/anaconda even though when I used pycharm I used python 3.5 and anaconda 3.6?
I think I need definitions and help to get everything in order in my head and the computer.
Pycharm is just an application to help you write code. Pycharm itself does not run python code. This is why in PyCharm, you need to set the interpreter for a project, which could be any python binary. In PyCharm, go to Preferences > Project > Project Interpreter to see where you would set the python environment being used for a given project. This could point to any python installation on your machine, whether that is the python 2.7 located at /usr/bin/python or a virtual environment in your project dir.
The industry standard way to "keep things in order" is to use what are called virtual environments. See here: https://docs.python.org/3/library/venv.html. A virtual environment is literally just a copy of a python environment (binaries and everything) so whatever directory you specify. This allows you to configure your environment to however you need in your project without interfering with other projects you might have. For example, say project A requires django 1.9.2 but project b requires 1.5.3. By having a virtual environment for each project, dependencies won't conflict.
Since you have python3.6, I would recommend going to you project directory in a terminal window. Running python -m venv .venv to create a hidden directory which contains a local python environment of whatever your 3.6 python installation. You could then set your project interpret to use that environment. to connect to it on the command line, run source .venv/bin/activate from where you created your virtual environment. run which python again and see that python is now referencing your virtual environment :)
If you are using a mac (which I believe you are from what you said about python2.7), what likely happened is that your anaconda installer put the Python bin directory on your PATH environment variable. Type in which python to see what the python alias is referencing. You can undo this if you want by editing your ~/.bash_profile file if you really want.
You are more or less correct about anaconda. It is itself another distribution of python and contains a load of common libraries/dependencies that tend to make life easier. For a lot of data analysis, you likely won't even need to install another dependency with pip after downloading anaconda.
I suspect this won't be all too helpful at first as it is a lot to learn, but hopefully this points you in the right direction.
So I develop a python application and I plan to copy the whole folder for my friend to use it as end-user.
But my friend does not have python installed in the computer and I don't want to make them install it since he is not a developer.
In my project I have set up the virtualenv with python.exe inside it but without the site-packages, and I copy the virtualenv together with the project folder.
Is it possible to do this kind of setup so the application in the other end runs without python installed?
virtualenv is a good option if you are transferring the folder between two same operating systems.
In order to include the correspond site packages that are already installed in your computer, install them inside the virtualenv context by doing pip install in the virtualenv shell.
You could use pip freeze to get a list of installed python packages from your computer.
You could then include a .bat file (if it is a windows system) or .sh file (if its a linux system) so it would run your script with the virtualenv context.