We Keep Coding

How does this Perl shebang work? (For someone not coming from Perl)

Could someone explain how this shebang works?
#!/usr/bin/perl -e$_=$ARGV[0];exec(s/\w+$/python3/r,$_)
I've seen it posted twice here but not coming from Perl it looks like magic to me.
I ask as I would like to adjust the directory to a python environment relative to the script.
i.e #!../env/bin/python3 to (I'm just guessing here) #!/usr/bin/perl -e$_=$ARGV[0];exec(s/\w+$/env/bin/python3/r,$_)
Edit: I am trying to execute a simple ''Hello world" program.
#!/usr/bin/perl -e'$_=$ARGV[0];exec(s{\w+$}{exploit-env/bin/python3}r,$_)'
###############################
def main():
print('Hello world')
###############################
if __name__ == '__main__':
main()

Shebang handling isn't consistent across all systems[1], but your system apparently apparently executes what the following shell command would execute (assuming the file containing the shebang is /path/to/script):
/usr/bin/perl -e'$_=$ARGV[0];exec(s/\w+$/python3/r,$_)' /path/to/script
(The path to the script might not be an absolute path —it might be relative to the current directory— but doesn't matter here.)
The script produces /path/to/python3 from /path/to/script (by replacing the trailing "word" characters, which include letters, digits and underscore, but not /), then evaluates
exec('/path/to/python3', '/path/to/script')
The replaces the program running in the current process with a Python interpreter, passing the path to the script as an argument.
If I read between the lines correctly, you want to to run /path/to/../env/bin/python3 instead of /path/to/python3. In order to achieve that, use either of the following:
#!/usr/bin/perl -e$_=$ARGV[0];exec(s/\w+$/..\/env\/bin\/python3/r,$_)
or
#!/usr/bin/perl -e$_=$ARGV[0];exec(s{\w+$}{../env/bin/python3}r,$_)
/ needs to be escaped by \ when / is used as the delimiter (first solution), but we can change the delimiter to produce a more readable result (second solution).
That shebang you presented causes the arguments to be absorbed. Replace the $_ with #ARGV to pass them on.
#!/usr/bin/perl -e$_=$ARGV[0];exec(s{\w+$}{../env/bin/python3}r,#ARGV)
At least historically, some systems treat the entire sequence that follows #! as the path (i.e. no arguments allows), and some have very strict limits as to the length of the path.

How could I get the command line that I entered in the terminal to run a python file?

Suppose I have a python file main.py, and it has some optional parameters, --learning-rate, --batch-size, and etc.
If I want to run this file, I can input the following into the terminal(Ubuntu Linux for example).
python3 main.py --learning-rate 0.1 --batch-size 100
And now, I want to write some code in main.py, in order that after I enter the command above, I can get this command in a string by executing those code. The following is the string I want to get:
"python3 main.py --learning-rate 0.1 --batch-size 100"
The reason I want to do this is that I want to write this string into my recording file so that I can know better what command I have run.
Could anyone tell me what package should I import and what code should I write to get that command information during running the python file?
Thanks!

You cannot always get precisely what you typed, because the shell will have first done substitutions and expanded filenames before starting your script. For example, if you type python "foo.py" *.txt, your script won't see *.txt, it will see the list of files, and it won't see the quotes around foo.py.
With that caveat out of the way, the sys module has a variable named argv that contains all of the arguments. argv[0] is the name of the script.
To get the name of the python executable you can use sys.executable.
To tie it all together, you can do something like this:
print(sys.executable + " " + " ".join(sys.argv))

Why not just remake the command using the arguments you parsed? It won't be exactly what you typed, but that might be nice as it will be in a common format.
Ex (assuming the learning rate and batch size are stored in similarly named variables):
command = "python3 main.py --learning-rate {} --batch-size {}".format(learning_rate, batch_size)
Of course it will be a little more complicated if some commands are optional, but I assume in that case there would be a default value for these parameters, since your network would need those parameters every time.

Applying a perl script to every file in a directory and obtain output using Python

I am trying to make a python script that will open a directory, apply a perl script to every file in that directory and get its out put in either multiple text files or just one.
I currently have:
import shlex, subprocess
arg_str = "perl tilt.pl *.pdb > final.txt"
arg = shlex.split(arg_str)
import os
framespdb = os.listdir("prac_frames")
for frames in framespdb:
subprocess.Popen(arg, stdout=True)
I keep getting *.pdb not found. I am very new to all of this so any help trying to complete this script would help.

*.pdb not found means exactly that - there won't be a *.pdb in whatever directory you're running the script... and as I read the code - I don't see anything to imply it's within 'frames' when it runs the perl script.
you probably need os.chdir(path) before the Popen.
How do I "cd" in Python?
...using a python script to run somewhat dubious syscalls to perl may offend some people but everyone's done it.. aside from that I'd point out:
always specify full paths (this becomes a problem if you will later say, want to run your job automatically from cron or an environment that doesn't have your PATH).
i.e. 'which perl' - put that full path in.
./.pdb would be better but not as good as the fullpath/.pdb (which you could use instead of the os.chdir option).

subprocess.Popen(arg, stdout=True)
does not expand filename wildcards. To handle your *.pdb, use shell=True.

Passing individual lines from files into a python script using a bash script

This might be a simple question, but I am new to bash scripting and have spent quite a bit of time on this with no luck; I hope I can get an answer here.
I am trying to write a bash script that reads individual lines from a text file and passes them along as argument for a python script. I have a list of files (which I have saved into a single text file, all on individual lines) that I need to be used as arguments in my python script, and I would like to use a bash script to send them all through. Of course I can take the tedious way and copy/paste the rest of the python command to individual lines in the script, but I would think there is a way to do this with the "read line" command. I have tried all sorts of combinations of commands, but here is the most recent one I have:
#!/bin/bash
# Command Output Test
cat infile.txt << EOF
while read line
do
VALUE = $line
python fits_edit_head.py $line $line NEW_PARA 5
echo VALUE+"huh"
done
EOF
When I do this, all I get returned is the individual lines from the input file. I have the extra VALUE there to see if it will print that, but it does not. Clearly there is something simple about the "read line" command that I do not understand but after messing with it for quite a long time, I do not know what it is. I admit I am still a rookie to this bash scripting game, and not a very good one at that. Any help would certainly be appreciated.

You probably meant:
while read line; do
VALUE=$line ## No spaces allowed
python fits_edit_head.py "$line" "$line" NEW_PARA 5 ## Quote properly to isolate arguments well
echo "$VALUE+huh" ## You don't expand without $
done < infile.txt
Python may also read STDIN so that it could accidentally read input from infile.txt so you can use another file descriptor:
while read -u 4 line; do
...
done 4< infile.txt
Better yet if you're using Bash 4.0, it's safer and cleaner to use readarray:
readarray -t lines < infile.txt
for line in "${lines[#]}; do
...
done

How to work around no output redirection in Windows powershell [duplicate]

This question already has answers here:
How to redirect the output of a PowerShell to a file during its execution
(10 answers)
Redirecting standard input\output in Windows PowerShell
(5 answers)
Closed 9 years ago.
I have written a lot of code that relies on output redirection in cygwin.
I now have to integrate some libraries that aren't compatible with cygwin and that need in my case Windows or Linux.
Is there a smooth way to go from cygwin's python script.py 42 <in.txt >out.txt to powershell's equivalent? I know powershell can't use input/output redirection, at least not with the <, > (the documentation says it can weirdly) but is there another way?

PowerShell definitely can do input/output redirection, it just doesn't do it the exact same way as bash.
I'm assuming you don't want to actually learn PowerShell, just get a quick & dirty answer. So, here goes.
For simple cases, bash-like syntax works fine:
foo > out.txt
When that doesn't work, often just wrapping up the tool in Powershell scriptlet is all you need. In other words, create a foo.ps1 file with this in it:
foo > out.txt
Then just run foo.ps1 instead of foo.
You may also want to look at the Tee-Object, Set-Content, Add-Content, and Start-Transcript, etc. cmdlets. Start with A Task-Based Guide to Windows PowerShell Cmdlets.
For example, Start-Transcript captures standard output as a "dumb string" and writes it to a file. So, one equivalent of foo > out.txt is:
Start-Transcript -path out.txt
foo
Stop-Transcript
Or, for foo >> out.txt:
Start-Transcript -path out.txt --append
foo
Stop-Transcript
Similarly, in the other direction, Get-Content reads a text file and prints it out as a dumb string, which you can pipe to your program:
Get-Content in.txt | foo
Of course another obvious possibility is to change your Python script to take filenames.
In many cases, just using fileinput.input() instead of sys.stdin gives you this for free—you can pass it a file's content via stdin, or a filename on the command line (or all kinds of fancier combinations) and the script will see it the same way.