We Keep Coding

A shell script initiated by Python's os.system fails to run, but the script do run when called from terminal

I have a Python3 script that needs to call a shell script with some parameters. When I call this shell script directly form the terminal - it works. The shell script call from terminal:
source $HW/scripts/gen.sh -top $TOP -proj opy_fem -clean
But when I try to call the shell script exactly the same way from Python 3 using os.system (or os.popen - same result), the shell script fails to run. Python call to the shell script:
os.system("source $HW/scripts/gen.sh -top $TOP -proj opy_fem -clean")
Get the next errors:
/project/users/alona/top_fabric_verif_env/logic/hw/scripts/gen.sh: line 18: syntax error near unexpected token `('
/project/users/alona/top_fabric_verif_env/logic/hw/scripts/gen.sh: line 18: `foreach i ( $* )'
Could you please shed light on why the same shell script fails to run from Python?
Thank you for any help

foreach is a C-shell command. csh (and derivates like tcsh) are not standard system shells in Unix/Linux.
If you need to use a specific shell, for instance the C-shell:
os.system('/bin/csh -c "put the command here"')
This will execute the /bin/csh in the standard shell, but starting two shells instead of one creates an additional overhead. A better solution is:
subprocess.run(['/bin/csh', '-c', 'put the command here'])
Note that using the shell's source ... command does not make much sense when the shell exits after the command.

Printing .py file output in command line

I am trying to access a python function from the command line, and I would like to write such a command that will print the output in the terminal. The below doesn't work. What could I change?
python -c 'from laser import Laser; laser = Laser();l = laser.embed_sentences("hello", lang = "en").shape == (1, 1024); print(l)'

(base) ~ % python -c 'print("hello, world")'
hello, world
Printing works fine for me when running python through python -c. Are you sure your terminal isn't truncating your output by omitting the last (and in this case, only) line? You could try creating a single line file (no newline at the end) and then running cat [filename] (which is how I sometimes discover that my terminal is doing this)

-c cmd : program passed in as string (terminates option list)
That is the correct flag to be used. This must be a CLI config issue. Or the script is taking longer than you are expecting to run and it appears no output is generated.
Does python -c 'print("hello")' work?

determine file type of a file without extension

I want to use pygmentize to highlight some script files (python/bash/...) without extension. But pygmentize requires me to specify the lexer using -l. It does not automatically identify the file type from the content.
I have the following options at hand, but none of them work now:
use file -b --mime-type. But this command output x-python and x-shellscript instead of python and bash and I don't know the rules
use vim -e -c 'echo &ft|q' the_file. For any file with or without file extension, vim has a mechanism to guess the file type. But it doesn't work. Since the output goes to the vim window and disappears after q.
What can I do?
#Samborski's method works fine in normal case but it does not work in python subprocess.check_output since the pts is not allocated. If you use nvim, you can use this more straightforward way:
HOME=_ nvim --headless -es file <<EOF
call writefile([&ft], "/dev/stdout")
EOF

You can use vim this way:
vim -c ':silent execute ":!echo " . &ft . " > /dev/stdout"' -c ':q!' the_file
It simply constructs command to run in the shell as a string concatenation.

run python script from a .bat file and using Powershell

I run several python scripts on my W10 server and I'm looking for a way to open them all together.
I run manually my scripts opening a PowerShell on the script folder and executing pythonw script.py
I have tried several ways, with no result...
PowerShell -NoProfile -ExecutionPolicy Bypass -Command "& 'C:\Users\pc2\AppData\Local\Programs\Python\Python36-32\pythonw.exe C:\Users\pc2\script.py'"
also tested:
powershell.exe -noexit -command "'pythonw C:\Users\pc2\script.py'"
And:
powershell -command "& {&'C:\Users\pc2\AppData\Local\Programs\Python\Python36-32\pythonw.exe ' C:\Users\pc2\script.py}"
None of the above does anything... they should be opening the script.py file in using pythonw.
Can anyone point me in the correct direction to get this done?

To execute a single script:
powershell -c "pythonw 'C:\Users\pc2\script1.py'"
Note: The enclosing '...' around the file path are only needed if the latter contains shell metacharacters such as whitespace.
Alternatively,
use ""..."" when calling from cmd.exe (batch file), and `"...`"
when calling from PowerShell.
To execute multiple scripts, in sequence:
powershell -c "& { $Args | % { pythonw $_ } } 'C:\Users\pc2\script1.py' 'C:\Users\pc2\script2.py'"
Note: This only works when calling from cmd.exe (batch file).
(From within PowerShell, simply execute what's inside "..." directly)
As for what you tried, focusing just on what command string PowerShell ended up seeing:
& 'C:\Users\pc2\AppData\Local\Programs\Python\Python36-32\pythonw.exe C:\Users\pc2\script.py'
The problem is that you're passing the entire command line to &, whereas it only expects the executable name;
C:\Users\pc2\AppData\Local\Programs\Python\Python36-32\pythonw.exe C:\Users\pc2\script.py is obviously not a valid executable name / path.
'pythonw C:\Users\pc2\script.py'
By virtue of being enclosed in '...', this is a string literal, which PowerShell simply outputs as such - no command is ever executed.
& {&'C:\Users\pc2\AppData\Local\Programs\Python\Python36-32\pythonw.exe ' C:\Users\pc2\script.py}
Since you're using a script block ({ ... }), you're passing C:\Users\pc2\script.py as an argument, so you'd have to refer to it as such inside the block, either via the automatic $Args variable, or via a parameter defined in a param() block.
(As an aside: your file path has an extraneous trailing space, but PowerShell seems to ignore that.)

From the powershell console itself, you can directly run like this:
PS C:\Python27> python C:\Users\pc2\script.py
If necessary please edit the PATHTEXT environmental variable in the powershell profile:
$env:PATHEXT += ";.py"
In the batch also you can put the same command to execute the python. Save the file as something.bat :
python C:\Users\pc2\script.py
Hope it helps.

Execute bash script from URL using python

Assume I have a file at http://mysite.com/myscript.sh that contains:
#!/bin/bash
echo "Hello $1"
From the command line, I can execute my script (without downloading it) using the following command:
bash <(curl -s http://mysite.com/myscript.sh) World
Now, instead of executing the above command from the command line, I want to execute it from a python script. I tried doing the following:
import os
os.system('bash <(curl -s http://mysite.com/myscript.sh) World')
...but I get the following error:
sh: -c: line 0: syntax error near unexpected token `('
How do I make this execute correctly in python?

Evidently, os.system runs its command through /bin/sh, which usually causes whichever shell it's linked to to drop to a compatibility mode that doesn't include the <(...) construction. You can get around it by either storing the result in a temporary file or using another level of shell. Ugly, but it works.
os.system('bash -c "bash <(curl -s http://mysite.com/myscript.sh) World"')

There is a libcurl for python so you don't have to go the way around to command line behaviour. Here's the function list that should really do it - have never run remote scripts myself though. If you need installing the python binding, the instructions are here.
import curl