So I'm trying to debug an error I got on an HPC setup I have access to. I won't go into details about the error since it's package specific and I'm pretty sure this is an environment variable kind of problem. That said the package is neuron, and if anyone has experience with it and singularity I would appreciate your input.
When I tested everything locally using:
singularity exec --bind ./:/mnt container.sif my_script.py
there were no problems. However the same command ran into an error on the HPC cluster. I set about trying to recreate the error locally to see what the problem was.
For reasons still unknown to me, the error I got on the cluster can be reproduced locally by adding the --containall flag to the exec command. In fact, even the --contain flag can reproduce the error. I can see from the docs that --contain will:
use minimal /dev and empty other directories (e.g. /tmp and $HOME) instead of sharing filesystems from your host
which makes me guess its a path/environment problem, but I'm not 100% sure since I am still new-ish to everything that isn't python.
In order to try and solve the problem I tried using singularity shell to recreate the error. And this is where I hope someone can elucidate matters for me. If I do this:
singularity shell --containall --bind ./:/mnt container.sif
cd /mnt
python3 my_script.py
The script runs fine, I get no errors. However when I run:
singularity exec --containall --bind ./:/mnt container.sif python3 /mnt/my_script.py
I get the same error as I got on the cluster.
What is different about these two approaches? Why might shelling into the container work, and executing it like this not work? I am just looking for help figuring out to debug this.
Additionally, why might the scripts run locally but not on the HPC? My understanding of containers is that they are supposed to allow scripts to be run on different systems because everything is well, contained, in the container. What am I allowing through in these different scenarios that's stopping me from running my code?
My instincts (which aren't exactly experienced) tell me that there is some environment variable that I am carrying through when I shell in (or when I run the scripts locally) that I am losing when I run it in the other ways, but I am not sure where to begin looking for such a thing, or how to keep it in the container.
EDIT:
I also just tried shelling into the container, while in the HPC, and I get the same error. So there's something on my local machine that is being used when I shell in or when I execute the script without the --contain flag
Versions:
Singularity 3.5
Python 3.6.9
NEURON 8.0
Sounds like environment issue: you have something set in your dev env that doesn't exist in your cluster env. By default, all your environment variables are automatically forwarded on to the singularity environment. I recommend using -e/--cleanenv to catch that. When using that, only variables prefixed with SINGULARITYENV_ set in the singularity environment. e.g., to have NEURON_HOME=/mnt/neuron you would use export SINGULARITYENV_NEURON_HOME=/mnt/neuron before running the singularity command.
Once you figure out what the variable to be updated is you can add it normally in %environment or %post, however you prefer. If it's a value that changes depending on the environment, you can export the value in SINGULARITYENV_VARNAME.
Related
I am running a very simple docker container but have to run it in singularity. Every single time I run the image...
singularity --exec --cleanenv dcm2bids.sif dcm2bids $ARG
I get an error telling me that
AttributeError: ‘list’ object has no attribute ‘get’
All of the correct arguments are in the code.
This appears to be some problem python not working within the container file. I don't exactly understand how to modify/fix this. This container works when running only in docker. Could someone explain to me in a very simple way how one might fix this problem?
I also notice that when I go into the container with the singularity run command - if I type "python" there is no version that appears. Not sure if this is supposed to happen or not?
I have the following problem: I wrote a bash script for data analysis that works perfectly fine when I run it from the terminal. To further automate the process I wanted to use a python script that runs the bash script (using subprocess.call), changes the working directory, and reruns the script (and so on). This also worked fine when I did it on my MacBook. However, I need to do the analysis on a Linux machine and here the problem occurred. Again, running the script from the terminal worked fine but once I tried doing this with my python script it fails to find the relevant functions for the analysis. The functions are stored inside the anaconda3/bin folder.
(Python does not even find other functions like "pip")
Of course, I could add the path to all the functions in the bash script but this seems very inefficient to me. So my question is: is there any better way of telling python where to look for the functions? And can you maybe explain to me why running the script from the terminal works but not when I use subprocess.call?
Here is the python script:
import subprocess
import os
path_list = ["Path1",
"Path2"
]
for path in path_list:
os.chdir(path)
subprocess.call("Users/.../bash_script", shell=True)
I'm just posting my series of comments as an answer since I think this at least constitutes a reasonable answer for anyone running into a similar issue (your question could definitely be common enough to index from search engine results).
Issue:
...running the script from the terminal worked fine but once I tried doing this with my python script it fails to find the relevant functions for the analysis
In general, you can troubleshoot this kind of problem with:
import subprocess
subprocess.call('echo $PATH', shell=True)
If the directory that contains the relevant binaries/scripts/etc. is not in the output, then you are facing a PATH issue in the shell created by subprocess.call.
The exact problem as confirmed by the OP in comments is that anaconda3/bin is not part of your PATH. Your script works in a regular terminal session because of the Anaconda initialization function that gets added to your .bashrc when installing.
Part of an answer that is very helpful here: Python - Activate conda env through shell script
The problem with your script, though, lies in the fact that the .bashrc is not sourced by the subshell that runs shell scripts (see this answer for more info). This means that even though your non-login interactive shell sees the conda commands, your non-interactive script subshells won't - no matter how many times you call conda init.
Solution 1: Manually use the Anaconda sourcing function in your script
As the OP mentioned in the comments, their workaround was to use the initialization function added to their .bashrc in the script they are trying to run. Although this perhaps feels like not a great solution, this is a "good enough" workaround. Unfortunately I don't use Anaconda on Linux so I don't have an exact snippet of what this looks like. See the next section for a possibly "cleaner" solution.
Solution 2: Use bash -i to run your script
As mentioned in the same answer linked above, you might be able to use:
bash -i Users/.../bash_script
This will tell bash to run in interactive mode, which then properly sources your .bashrc file when creating the shell. As a result, Anaconda and related functions should work properly.
Solution 3: Manually add anaconda3/bin to PATH
You can check out this answer to decide if this is something you want to do. Keep in mind they are speaking about a Windows OS but most of the same applies to Linux.
When you add the directory to your PATH, you are specifically telling your system to always look in that directory for commands when executing by name, e.g. ping or which. This can have unexpected behavior if you have conflicts (e.g. a command is found with the same name in /usr/bin and .../anaconda3/bin), and as such Anaconda does not add its bin folder to your PATH by default.
This is not necessarily "dangerous" per se, it's just not an ideal solution for most people. However, you are the boss of your own system. If you decide this works for your particular workflow, you can just add the export to your script:
export PATH="path/to/anaconda3/bin:$PATH"
This will set the PATH for use in the current shell and sub-processes.
Solution 4: Manually source the conda script (possibly outdated)
As mentioned in this answer, you can also opt to manually source the conda.sh script (keep in mind your conda.sh might be in another directory):
source /opt/anaconda/etc/profile.d/conda.sh
This will essentially run that shell script and add the included functionality to the current shell (e.g the one spawned by subprocess.call).
Keep in mind this answer is quite a bit older (~2013) and may not apply anymore, depending how much conda has changed over the years.
Notes
As I mentioned in the comments, you may want to post some related questions on https://unix.stackexchange.com/. You have an interesting configuration challenge that may be better suited for answers specifically pertaining to Linux, since your issue is sourcing directly from Linux shell behavior.
I am aware a lot of similar questions exist but I am unable to understand what is happening here. I am trying to follow instructions for this Stanford CoreNLP python wrapper here, one of the steps is to set theCORENLP_HOME environment variable.
I ran the command:
export CORENLP_HOME=/path/to/stanford-corenlp-full-2018-10-05
Restarted the terminal, or actually added to my ~/.bash_profile. Now when I do a echo $CORENLP_HOME in the terminal I am able to see the path correctly. But on the other hand, if the corenlp wrapper code tries to find the same path through python code it returns None.
So I separately checked two python commands, the wrapper code uses os.getenv():
import os
print(os.getenv('CORENLP_HOME')) #prints None
print(os.environ['CORENLP_HOME']) #Throws a KeyError exception
MacOS version: 10.15.4;Python: 3.7.6
I don't have a very deep understanding of environment variables in general, I want to understand what is happening here, or if I am missing something simple. Happy to provide more information!
Environment variables are not global in the UNIX process model. Each process is provided a set of environment variables by the parent process that starts it. That is typically a copy of the parent's environment variables. If you are not starting pycharm from the shell that ran the export command pycharm won't see that shell's environment variables.
The behavior you describe means you are not starting pycharm from the shell that did the export CORENLP_HOME=/path/to/stanford-corenlp-full-2018-10-05.
P.S., The UNIX process model also means that a process cannot modify the environment variables of a different process.
I have spent hours looking into this issue without any success.
I've looked at various SO discussions and none seem to solve my problem so out of pure frustration here is my question...
I'm trying to launch a script within a windows batch file. The problem is that when I do the script fails because it can not find some of the modules used.
After various attempts I have found that the batch file aspect, at this stage, seems to be irrelevant.
So, ignoring batch files for a minute, If I run the script like this
pipenv run python myscript.py
It works. If I run the following it doesnt
path-to-env\Scripts\activate
python myscript.py
It returns an error ModuleNotFoundError: No module named 'xxx'
It activates the venv OK, but something is not right as it cant find code used in script
Within my IDE (Visual Code) everything works OK
I do have quite a complicated directory structure but given that both the IDE and "pipenv run python myscript.py" work as expected it must be due to something else.
Any ideas or pointers on where I need to be looking? I'm afraid my understanding of pipenv isnt up to solving this ;)
EDIT
In my attempts to solve this I had added the line PYTHONPATH=. to my .env file. This seems to be responsible for allowing this line to work:
pipenv run python myscript.py
If I remove it, then the above ALSO generates the ModuleNotFoundError
OK so after trying lots of various combinations I did finally manage to get this to work.
Although I have no idea why this solution works and others didnt..
It requires two batch files.
One to launch the python script which will contain a line like this
python myscript.py
And another to create the env via pipenv and then call the first batch file
It will have a line like this
pipenv run \path\to\first\batchfile.bat
This combination works and can be successfully called from the Windows Task Scheduler
I am stuck since 2 days trying to set up a small automatic deployment script.
The thing is: I have been using Git for some months now, but I always used it locally just by myself, just with the purpose of easily saving version of my code. All good until here.
Now I have to find a way to "publish" the code as soon as new functionalities are implemented and I think the code is stable enough.
Searching around I've discovered these 'hooks', which are scripts that are executed by Git in certain situations. Basically the idea is to have my master branch sync'd with my published code, so that everytime I merge a branch to the master and 'push', the files are automatically copied into '/my/published/folder'.
That said, I've found this tutorial that explains to do exactly what I want using a 'hooks' post-receive script, which is written in Ruby. Since at my studio I don't have and don't want to use Ruby at this time, I've found a Python version of the same script.
I tested and tested, but I couldn't make it work. I keep getting the same error:
remote: GIT_WORK_TREE is not recognized as as internal or external command,
Consider this is based on the tutorial I've shared above. Same prj name, same structure, etc.
I even installed Ruby on my personal laptop and tried the original script, but it still doesn't work...
I'm using Windows, and the Git env variable is set and accessible. But nevertheless it seems like it's not recognizing the GIT_WORK_TREE command. If I run it from the Git Bash it works just fine, but if I use the Windows Shell I get the same error message.
I suppose that when in my py script use the call() function, it runs the cmd using the Windows Shell. That's my guess, but I don't really know how to solve it. Google didn't help, as if no one ever had this problem before.
Maybe I'm just not seeing something obvious here, but I spent the whole day on this and I cannot get out of this bog!
Does anyone know how to solve it, or at least have an idea for a workaround?
Hope someone can help...
Thanks a lot!
The Ruby script you are talking about generates "bash" command:
GIT_WORK_TREE=/deploy/path git checkout -f ...
It means: define environment variable "GIT_WORK_TREE" with value "/deploy/path" and execute "git checkout -f ...".
As I understand it doesn't work for Windows command line.
Try to use something like:
set GIT_WORK_TREE=c:\temp\deploy && git checkout -f ...
I've had this problem as well - the best solution I've found is to pass the working tree across as one of the parameters
git --work-tree="/deploy/path" checkout -f ...