Pipenv and ModuleNotFoundError - python

I have spent hours looking into this issue without any success.
I've looked at various SO discussions and none seem to solve my problem so out of pure frustration here is my question...
I'm trying to launch a script within a windows batch file. The problem is that when I do the script fails because it can not find some of the modules used.
After various attempts I have found that the batch file aspect, at this stage, seems to be irrelevant.
So, ignoring batch files for a minute, If I run the script like this
pipenv run python myscript.py
It works. If I run the following it doesnt
path-to-env\Scripts\activate
python myscript.py
It returns an error ModuleNotFoundError: No module named 'xxx'
It activates the venv OK, but something is not right as it cant find code used in script
Within my IDE (Visual Code) everything works OK
I do have quite a complicated directory structure but given that both the IDE and "pipenv run python myscript.py" work as expected it must be due to something else.
Any ideas or pointers on where I need to be looking? I'm afraid my understanding of pipenv isnt up to solving this ;)
EDIT
In my attempts to solve this I had added the line PYTHONPATH=. to my .env file. This seems to be responsible for allowing this line to work:
pipenv run python myscript.py
If I remove it, then the above ALSO generates the ModuleNotFoundError

OK so after trying lots of various combinations I did finally manage to get this to work.
Although I have no idea why this solution works and others didnt..
It requires two batch files.
One to launch the python script which will contain a line like this
python myscript.py
And another to create the env via pipenv and then call the first batch file
It will have a line like this
pipenv run \path\to\first\batchfile.bat
This combination works and can be successfully called from the Windows Task Scheduler

Related

How to catch and handle a "No python at C:\<some_directory>" error

TL;DR I want to write a batch file that will do some stuff when a "No Python at " error is encountered
I develop and maintain several Python-based automation tools as my job and often, my users will encounter the dreaded "no Python at " error when the tool's virtual environment attempts to activate. I include a batch file that fixes the problem with all downloads of my tools, and it works like a charm, but they currently have to run it manually. I'd like to see if I can set things up so that the fix runs automatically when the error occurs.
Unfortunately, my skill at writing complex behavior in batch files is a bit lacking, and my Googling didn't turn up anything obvious for the tack I'm trying to perform.
Anyway, here's the batch file that runs the Python script:
"venv\Scripts\python.exe" Some_Python_Script.py
And here's what I want to run in the event of a "no Python at <path." error:
for /f "usebackq delims=#" %%a in (`where /r "C:\ProgramData\Microsoft\Windows\Start Menu\Programs" Python?3.??(64*`) do set pypath="%%a"
cd "%~dp0"
%pypath% "%~dp0\fixVenvConfig.py"
^^^The tools are run inside of a secure VM, and the geniuses who install Python on them for us never check the box to add Python to the system path, so I instead must locate a Python executable by following the shortcut in the start menu. Once I have a usable Python interpreter, I pass its file path into a Python script that then fixes the config file of the virtual environment used by the automation tools. I don't remember how half of the stuff in the above batch file script actually works. I just know that it does, so I just don't touch it.
In the end, I'm trying to figure out how to make it so that if what's in the first code block results in a "no Python at " error, the code in the second block gets run, and then the first block gets tried again. As previously mentioned, however, this is a bit beyond my level of batch file expertise. Please help?

How can I access functions inside the anaconda3/bin directory when running a bash script with subprocess.call?

I have the following problem: I wrote a bash script for data analysis that works perfectly fine when I run it from the terminal. To further automate the process I wanted to use a python script that runs the bash script (using subprocess.call), changes the working directory, and reruns the script (and so on). This also worked fine when I did it on my MacBook. However, I need to do the analysis on a Linux machine and here the problem occurred. Again, running the script from the terminal worked fine but once I tried doing this with my python script it fails to find the relevant functions for the analysis. The functions are stored inside the anaconda3/bin folder.
(Python does not even find other functions like "pip")
Of course, I could add the path to all the functions in the bash script but this seems very inefficient to me. So my question is: is there any better way of telling python where to look for the functions? And can you maybe explain to me why running the script from the terminal works but not when I use subprocess.call?
Here is the python script:
import subprocess
import os
path_list = ["Path1",
"Path2"
]
for path in path_list:
os.chdir(path)
subprocess.call("Users/.../bash_script", shell=True)
I'm just posting my series of comments as an answer since I think this at least constitutes a reasonable answer for anyone running into a similar issue (your question could definitely be common enough to index from search engine results).
Issue:
...running the script from the terminal worked fine but once I tried doing this with my python script it fails to find the relevant functions for the analysis
In general, you can troubleshoot this kind of problem with:
import subprocess
subprocess.call('echo $PATH', shell=True)
If the directory that contains the relevant binaries/scripts/etc. is not in the output, then you are facing a PATH issue in the shell created by subprocess.call.
The exact problem as confirmed by the OP in comments is that anaconda3/bin is not part of your PATH. Your script works in a regular terminal session because of the Anaconda initialization function that gets added to your .bashrc when installing.
Part of an answer that is very helpful here: Python - Activate conda env through shell script
The problem with your script, though, lies in the fact that the .bashrc is not sourced by the subshell that runs shell scripts (see this answer for more info). This means that even though your non-login interactive shell sees the conda commands, your non-interactive script subshells won't - no matter how many times you call conda init.
Solution 1: Manually use the Anaconda sourcing function in your script
As the OP mentioned in the comments, their workaround was to use the initialization function added to their .bashrc in the script they are trying to run. Although this perhaps feels like not a great solution, this is a "good enough" workaround. Unfortunately I don't use Anaconda on Linux so I don't have an exact snippet of what this looks like. See the next section for a possibly "cleaner" solution.
Solution 2: Use bash -i to run your script
As mentioned in the same answer linked above, you might be able to use:
bash -i Users/.../bash_script
This will tell bash to run in interactive mode, which then properly sources your .bashrc file when creating the shell. As a result, Anaconda and related functions should work properly.
Solution 3: Manually add anaconda3/bin to PATH
You can check out this answer to decide if this is something you want to do. Keep in mind they are speaking about a Windows OS but most of the same applies to Linux.
When you add the directory to your PATH, you are specifically telling your system to always look in that directory for commands when executing by name, e.g. ping or which. This can have unexpected behavior if you have conflicts (e.g. a command is found with the same name in /usr/bin and .../anaconda3/bin), and as such Anaconda does not add its bin folder to your PATH by default.
This is not necessarily "dangerous" per se, it's just not an ideal solution for most people. However, you are the boss of your own system. If you decide this works for your particular workflow, you can just add the export to your script:
export PATH="path/to/anaconda3/bin:$PATH"
This will set the PATH for use in the current shell and sub-processes.
Solution 4: Manually source the conda script (possibly outdated)
As mentioned in this answer, you can also opt to manually source the conda.sh script (keep in mind your conda.sh might be in another directory):
source /opt/anaconda/etc/profile.d/conda.sh
This will essentially run that shell script and add the included functionality to the current shell (e.g the one spawned by subprocess.call).
Keep in mind this answer is quite a bit older (~2013) and may not apply anymore, depending how much conda has changed over the years.
Notes
As I mentioned in the comments, you may want to post some related questions on https://unix.stackexchange.com/. You have an interesting configuration challenge that may be better suited for answers specifically pertaining to Linux, since your issue is sourcing directly from Linux shell behavior.

Run a python script from bamboo

I'm trying to run a python script from bamboo. I created a script task and wrote inline "python myFile.py". Should I be listing the full path for python?
I changed the working directory to the location of myFile.py so that is not a problem. Is there anything else I need to do within the configuration plan to properly run this script? It isn't running but I know it should be running because the script works fine from terminal on my local machine. Thanks
I run a lot of python tasks from bamboo, so it is possible. Using the Script task is generally painless...
You should be able to use your script task to run the commands directly and have stdout written to the logs. Since this is true, you can run:
'which python' -- Output the path of which python that is being ran.
'pip list' -- Output a list of which modules are installed with pip.
You should verify that the output from the above commands matches the output when ran from the server. I'm guessing they won't match up and once that is addressed, everything will work fine.
If not, comment back and we can look at a few other things.
For the future, there are a handful of different ways you can package things with python which could assist with this problem (e.g. automatically installing missing modules, etc).
You can also use the Script Task directly with an inline Python script to run your myFile.py:
/usr/bin/python <<EOF
print "Hello, World!"
EOF
Check this page for a more complex example:
https://www.langhornweb.com/display/BAT/Run+Python+script+as+a+Bamboo+task?desktop=true&macroName=seo-metadata

Issue with Python Batch file to run Python through Notepad++

EDIT: The code I wrote in my Python file was just this:
print "foo"
I'm using Windows XP Home Premium on this tiny little HP Mini 1000, and I want to run Python files, since we're learning it in school. I am aware of this topic, so I tried to run Python files using a batch file (python.bat), and I'm getting an error that says, "Can't find 'main' module in ''" whenever I run the batch file. I followed the instructions given here. All I did was change "Python26" to "Python33" because of the difference in versions.
Any idea what's wrong here? I really want to run Python files from Notepad++, so I don't want any alternative ways to run them.
This sounds like you don't have PYTHONPATH set up correctly. I suggest you review the documentation here:
http://docs.python.org/2/using/windows.html
Instead of calling Python, call cmd.exe and then use the set command to inspect which variables are set and how they are set. Run the exit command to leave the command shell. When you think you have the variables set up correctly, try again to run Python.
Good luck and have fun!
I use the command line interpreter or IDLE mostly (Win 8.1 now, but I've done so since Win XP SP2), but NPP is my main text editor, so I was curious about this issue.
When I was reproducing this, I was able to generate several errors, but the only one I got that was an exact match was when I failed to configure the Run option correctly.
You need to make sure to follow this step exactly in the instructions you were following. When you navigate to Run -> Run in Notepad++, you have to enter this exactly:
C:\Python33\python.bat "$(FULL_CURRENT_PATH)"
I am pretty sure you left out the "$(FULL_CURRENT_PATH)", or otherwise didn't add it correctly, as failing to do so causes exactly the same error on my end. Failing to include this means that when you run the batch script, you get the wrong input to the Python interpreter, causing the error.

Run a Python script from PackageMaker in OSX

I'm having a difficult time finding any decent documentation for packaging on OSX. My package needs to run a Python script that will do some launchd magic right after the app is installed. I tried using -s scripts with packagemaker and put my script in the directory scripts, but I had no luck. Is there something I'm missing? More importantly, where is all the packagemaker documentation? How can I get my Python script to run? The only thing I've seen is this, which is way outdated and doesn't cover the command line version.
I don't want to use the GUI; it makes my life too difficult.
I've resolved this issue.
The script needs to be named postflight and have a proper shebang line.
The script needs to exit with specific status codes (0 for success, which is the default, I believe).

Categories