We Keep Coding

How can I access functions inside the anaconda3/bin directory when running a bash script with subprocess.call?

I have the following problem: I wrote a bash script for data analysis that works perfectly fine when I run it from the terminal. To further automate the process I wanted to use a python script that runs the bash script (using subprocess.call), changes the working directory, and reruns the script (and so on). This also worked fine when I did it on my MacBook. However, I need to do the analysis on a Linux machine and here the problem occurred. Again, running the script from the terminal worked fine but once I tried doing this with my python script it fails to find the relevant functions for the analysis. The functions are stored inside the anaconda3/bin folder.
(Python does not even find other functions like "pip")
Of course, I could add the path to all the functions in the bash script but this seems very inefficient to me. So my question is: is there any better way of telling python where to look for the functions? And can you maybe explain to me why running the script from the terminal works but not when I use subprocess.call?
Here is the python script:
import subprocess
import os
path_list = ["Path1",
"Path2"
]
for path in path_list:
os.chdir(path)
subprocess.call("Users/.../bash_script", shell=True)

I'm just posting my series of comments as an answer since I think this at least constitutes a reasonable answer for anyone running into a similar issue (your question could definitely be common enough to index from search engine results).
Issue:
...running the script from the terminal worked fine but once I tried doing this with my python script it fails to find the relevant functions for the analysis
In general, you can troubleshoot this kind of problem with:
import subprocess
subprocess.call('echo $PATH', shell=True)
If the directory that contains the relevant binaries/scripts/etc. is not in the output, then you are facing a PATH issue in the shell created by subprocess.call.
The exact problem as confirmed by the OP in comments is that anaconda3/bin is not part of your PATH. Your script works in a regular terminal session because of the Anaconda initialization function that gets added to your .bashrc when installing.
Part of an answer that is very helpful here: Python - Activate conda env through shell script
The problem with your script, though, lies in the fact that the .bashrc is not sourced by the subshell that runs shell scripts (see this answer for more info). This means that even though your non-login interactive shell sees the conda commands, your non-interactive script subshells won't - no matter how many times you call conda init.
Solution 1: Manually use the Anaconda sourcing function in your script
As the OP mentioned in the comments, their workaround was to use the initialization function added to their .bashrc in the script they are trying to run. Although this perhaps feels like not a great solution, this is a "good enough" workaround. Unfortunately I don't use Anaconda on Linux so I don't have an exact snippet of what this looks like. See the next section for a possibly "cleaner" solution.
Solution 2: Use bash -i to run your script
As mentioned in the same answer linked above, you might be able to use:
bash -i Users/.../bash_script
This will tell bash to run in interactive mode, which then properly sources your .bashrc file when creating the shell. As a result, Anaconda and related functions should work properly.
Solution 3: Manually add anaconda3/bin to PATH
You can check out this answer to decide if this is something you want to do. Keep in mind they are speaking about a Windows OS but most of the same applies to Linux.
When you add the directory to your PATH, you are specifically telling your system to always look in that directory for commands when executing by name, e.g. ping or which. This can have unexpected behavior if you have conflicts (e.g. a command is found with the same name in /usr/bin and .../anaconda3/bin), and as such Anaconda does not add its bin folder to your PATH by default.
This is not necessarily "dangerous" per se, it's just not an ideal solution for most people. However, you are the boss of your own system. If you decide this works for your particular workflow, you can just add the export to your script:
export PATH="path/to/anaconda3/bin:$PATH"
This will set the PATH for use in the current shell and sub-processes.
Solution 4: Manually source the conda script (possibly outdated)
As mentioned in this answer, you can also opt to manually source the conda.sh script (keep in mind your conda.sh might be in another directory):
source /opt/anaconda/etc/profile.d/conda.sh
This will essentially run that shell script and add the included functionality to the current shell (e.g the one spawned by subprocess.call).
Keep in mind this answer is quite a bit older (~2013) and may not apply anymore, depending how much conda has changed over the years.
Notes
As I mentioned in the comments, you may want to post some related questions on https://unix.stackexchange.com/. You have an interesting configuration challenge that may be better suited for answers specifically pertaining to Linux, since your issue is sourcing directly from Linux shell behavior.

How to install SageMath kernel in Anaconda?

I'm trying to use Sage in Anaconda 3 but it looks that the libraries are not imported.
I firstly created a new environment 'ipykernel_py2' and then installed Python 2 as explained in here. With this I can have both Python 3 and Python 3 up and running in Anaconda 3.
Then I went to the kernel's folder created (C:\Users\YOUR_USERNAME\AppData\Local\Continuum\anaconda3\envs\ipykernel_py2\share\jupyter\kernels) and pasted Sage's kernel (taken from C:\Program Files\SageMath 8.2\runtime\opt\sagemath-8.2\local\share\jupyter\kernels). This allows to create new SageMath files in Jupyter but the kernel is dead.
To activate the kernel I used Anaconda Prompt and typed:
activate ipykernel_py2
python -m ipykernel install --user --name sagemath --display-name "SageMath 8.2"
So the kernel is now activated and I can create and run Sage files. However the libraries are still not working. It seems that the file is running like a normal Python 2 file.
Does anyone know how to fix this? Do I need to create a seperate environment?

Sage for Windows runs under a UNIX emulation environment called Cygwin. Looking at the sagemath/kernel.json it contains:
{"display_name": "SageMath 8.2", "argv": ["/opt/sagemath-8.2/local/bin/sage", "--python", "-m", "sage.repl.ipython_kernel", "-f", "{connection_file}"]}
You can see here that it has a UNIX-style path to the sage executable. This path only makes sense to other programs running under Sage's Cygwin environment, and is meaningless to native Windows programs. Simply converting it to the equivalent Windows path won't work either, because bin/sage is actually a shell script. At the very least you need to provide a Windows path to the bash that comes with Cygwin and pass it the UNIX path to the sage executable (the same as the one above). Without a login shell, most environment variables needed won't be set either, so you probably need bash -l.
So, something like:
{"display_name": "SageMath 8.2", "argv": ["C:\\Program Files\\SageMath 8.2\\runtime\\bin\\bash.exe", "-l", "/opt/sagemath-8.2/local/bin/sage", "--python", "-m", "sage.repl.ipython_kernel", "-f", "{connection_file}"]}
might work. The one thing I'm not sure about is whether the {connection_file} argument will be handled properly either. I haven't tested it.
Update: Indeed, the above partially works, but there are a few problems: The {connection_file} argument as passed as the absolute Windows path to the file. While Cygwin can normally translate transparently from Windows paths to a corresponding UNIX path, there is a known issue that Python's os.path module on Cygwin does not handle Windows-style paths well, and this leads to issues.
The other major problem I encountered was that IPKernelApp, the class that drives generic Jupyter kernels, has a thread which polls to see if the kernel's parent process (in this case the notebook server) has exited, so it can appropriately shut down if the parent shuts down. This is how kernels know to automatically shut down when you kill the notebook server.
How this is done is very different depending on the platform--Windows versus UNIX-like. Because Sage's kernel runs in Cygwin, it chooses the UNIX-like poller. However, this is wrong if the notebook server happens to be a native Windows process, as is the case when running the Sage kernel in a Windows-native Jupyter. Remarkably, the parent poller for Windows can work just as well on Cygwin since it accesses the Windows API through ctypes. Therefore, this can be worked around by providing a wrapper to IPKernelApp that forces uses of ParentPollerWindows.
A possible solution then looks something like this: From within the SageMath Shell do:
$ cd "$SAGE_LOCAL"
$ mkdir -p ./share/jupyter/kernels/sagemath
$ cd ./share/jupyter/kernels/sagemath
$ cat <<_EOF_ > kernel-wrapper.sh
#!/bin/sh
here="$(dirname "$0")"
connection_file="$(cygpath -u -a "$1")"
exec /opt/sagemath-8.2/local/bin/sage --python "${here}/kernel-wrapper.py" -f "${connection_file}"
_EOF_
$ cat <<_EOF_ > kernel-wrapper.py
from ipykernel.kernelapp import IPKernelApp as OrigIPKernelApp
from ipykernel.parentpoller import ParentPollerWindows
from sage.repl.ipython_kernel.kernel import SageKernel
class IPKernelApp(OrigIPKernelApp):
"""
Although this kernel runs under Cygwin, its parent is a native Windows
process, so we force use of the ParentPollerWindows.
"""
def init_poller(self):
if self.interrupt or self.parent_handle:
self.poller = ParentPollerWindows(self.interrupt,
self.parent_handle)
IPKernelApp.launch_instance(kernel_class=SageKernel)
_EOF_
Now edit the kernel.json (in its existing location under share\jupyter\kernels\sagemath) to read:
{"display_name": "SageMath 8.2", "argv": ["C:\\Program Files\\SageMath 8.2\\runtime\\bin\\bash.exe", "-l", "/opt/sagemath-8.2/local/share/jupyter/kernels/sagemath/kernel-wrapper.sh", "{connection_file}"]}
This runs kernel-wrapper.sh which in turn runs kernel-wrapper.py. (There are a few simplifications I could make to get rid of the need for kernel-wrapper.sh completely, but that would be easier in SageMath 8.3 which includes PyCygwin.)
Make sure to change every "8.2" to the appropriate "X.Y" version for your Sage installation.
Update: Made some updates thanks to feedback from a user, but I haven't tested these changes yet, so please make sure instead of blindly copy/pasting that every file/directory path in my instructions exists and looks correct.
As you can see, this was not trivial, and was never by design meant to be possible. But it can be done. Once the kernel itself is up and running it's just a matter of talking to it over TCP/IP sockets so there's not too much magic involved after that. I believe there are some small improvements that could be made on both the Jupyter side and on the Sage side that would facilitate this sort of thing in the future...

Linux desktop file Exec doesn't used the user session

I have a python script file that works perfectly when I use it from the terminal.
Now I have created the following .desktop file in order to launch it easily:
[Desktop Entry]
Name=Test
GenericName=Test
Comment=My test script
Type=Application
Exec=/opt/test.py
Icon=/opt/test.png
Categories=Utils;
When I launch it the GTK window appear but clicking a button that call an init.d script make it working not properly.
Therefore adding Terminal=true make it working perfectly but I don't want to have that terminal open.
So I have then put the following code in order to log the environment variables:
import os
with open("/tmp/py_env.log", "w") as env_log:
env_log.write(str(os.environ))
and found differences.
So my question is how to write the .desktop file so that my application is running like if I start it from my terminal (without having an opened terminal :))

The problem is valid, but I think "replicating the terminal environment" is the wrong approach to solve it.
Indeed, what makes the application work is not the fact that it's launched from the terminal, it's that the terminal happens to have some environment variables which matter to your application.
Therefore, what you should aim for is to have those environment variables set properly at all times, rather than assuming the terminal environment will always happen to contain them all the time for all your users.
Thus, you should:
Check which environment variables are different between the two environments
Make a list of those which matter (i.e. those which would make the .desktop file work properly), and of what their value needs to be for the script to work
Either:
Create a wrapper script for your Python script, which initializes those environment variables properly, OR
Set those environment variables from inside the Python script itself.

this question is similar to .bashrc not read when shell script is invoked from desktop shortcut
either initialize your environment in ~/.bash_profile instead of
~/.bashrc
OR
make your *.desktop file call a wrapper that initializes your
environment - e.g. by sourcing ~/.bashrc (or whatever script is
responsible now).
the second solution is more specific (does not effect all other unrelated launches of your shell) in should thus be preferred.

Thanks anyone to have participate to this question.
I have solved this issue by implemented use of pkexec instead of gksudo.
pkexec seems to reuse the current user environment then I don't have this issue anymore.
Thanks.

How to install python 3.2.3 on Windows 7 enterprise

although I have been using python a long time very easily in a Linux environment, I have tremendous trouble to even install it correctly in a windows environment. I hope this is a question to be asked here, as it is not directly a programming question.
Especially, I have the following problems:
When on the command line, python is not a recognized command. Do I have to set the Windows path manually myself? If so, how to do that?
When starting a python script, should this be done with python.exe or pythonw.exe? What is the difference?
I also tried to install ipython several times, it never got installed (even after following the starting ipythonenter link description here thread.
When starting a script with python.exe, a window pops up and closes immediately. I saw some hints in putting in a readline command, which is of no help if there is a syntax error in the script. So how to be able to keep the window open, or how to run the command on the cmd.exe?
Thank you for any help on these items.
Alex

1) Look here: www.computerhope.com/issues/ch000549.htm
2) It has already been answered, always try to use search before asking question:
pythonw.exe or python.exe?
4) When using cmd.exe just navigate to your script folder using dir for changing directories and C:,D:,etc. for changing drives. Then run script by typing just the script name. When installed correctly, Python automatically launches .py scripts with python, so you don't have to write 'python' before script name. When run in cmd, window will stay open. If you want it to stay open even when launching script with double-click, use function waiting for user input, see here How to keep a Python script output window open?

You might want to use Python3.3, there is a new launcher for Python scripts in it. By that, you can start Python scripts with py <scriptname> which has the benefit of being installed in your path (C:\Windows\system32) and you can use a shebang to tell whether the script is for Python2 or Python3.
Also
In addition to the launcher, the Windows installer now includes an
option to add the newly installed Python to the system PATH
(contributed by Brian Curtin in issue 3561).

Is it possible to launch a Paster shell with some modules pre-imported?

Is it possible to run "paster shell blah.ini" (or a variant thereof) and have it automatically load certain libraries?
I hate having to always type "from foo.bar import mystuff" as the first command in every paster shell, and would like the computer to do it for me.

An option to try would be to create a sitecustomize.py script. If you have this in the same folder as your paster shell, the python interpreter should load it up on startup.
Let me clarify, sitecustomize.py, if found, is always loaded on startup of the interpreter. So if you put it where it can be found, ideally somewhere that is only found when the paster shell starts, then you should be able to add your imports to it and have them be ready.
This is probably your best bet. If the paster shell is a packaged app (a la py2exe) it should still work.
See also:
http://www.rexx.com/~dkuhlman/pylons_quick_site.html#using-an-ipython-embedded-shell
http://pylonshq.com/project/pylonshq/ticket/428

If you set the environment variable PYTHONSTARTUP to the name of a file, it will execute that file on opening the interactive prompt.
I don't know anything about paster shell, but I assume it works similarly.
Alternatively you could look into iPython, which has much more powerful features (particularly when installed with the readline library). For example %run allows you to run a script in the current namespace, or you can use history completion.
Edit:
Okay. Having looked into it a bit more, I'm fairly certain that paster shell just does a set of useful imports, and could be easily replicated with a short script and ipython and then %run myscript.py
Edit:
Having looked at the source, it would be very hard to do (I was wrong about the default imports. It parses your config file as well), however if you have Pylons and iPython both installed, then paster shell should use iPython automagically. Double check that both are installed properly, and double check that paster shell isn't using iPython already (it might be looking like normal python prompt).