We Keep Coding

Passing ipython variable to bash command one line for loop

I'm sorry for asking a duplicate as this and this are very similar but with those answers I don't seem to be able to make my code work.
If have a jupyter notebook cell with:
some_folder = "/some/path/to/files/*.gif"
!for name in {some_folder}; do echo $name; done
The output I get is just {folder}
If I do
some_folder = "/some/path/to/files/*.gif"
!for name in "/some/path/to/files/*.gif"; do echo $name; done # <-- gives no newlines between file names
# or
!for name in /some/path/to/files/*.gif; do echo $name; done # <-- places every filename on its own line
My gif files are printed to screen.
So my question why does it not use my python variable in the for loop?
Because the below, without a for loop, does work:
some_folder = "/some/path/to/files/"
!echo {some_folder}
Follow up question: I actually want my variable to just be the folder and add the wildcard only in the for loop. So something like this:
some_folder = "/some/path/to/files/"
!for name in {some_folder}*.gif; do echo $name; done
For context, later I actually want to rename the files in the for loop and not just print them. The files have an extra dot (not the one from the .gif extension) which I would like to remove.

There's an alternative way to use shell bash in a Jupyter cell with cell magic, see here. It seems to allow what you are trying to do.
If you already ran in a normal cell some_folder = r"/some/path/to/files/*.gif" or some_folder = "/some/path/to/files/*.gif", then you can try in a separate cell:
%%bash -s {some_folder}
for i in {1..5}; do echo $1; done
That said, what you seems to be trying to do with some_folder = "/some/path/to/files/*.gif" isn't going to work as such. If you try to pass "/some/path/to/files/*.gif" from Python to bash, it isn't going to work like passing /some/path/to/files/*.gif directly to bash. Bash isn't passing "/some/path/to/files/*.gif" directly to a command, it expands it and then passes it. There's not going to be an expansion passing from Python. And there's other peculiarities you'll come across. Tar you can pass a Python list of files directly using the bracket notation and it will handle that.
The solutions are to either do more on the Python side or more in the shell side. Python has it's own glob module, see here. You can combine that with working with os.system(). Python has fnamtch that is nice because you can use Unix-like file name matching. Plus there's shutil that allows moving/renaming, see shutil.move(). In Python os.remove(temp_file_name) can delete files. If you aren't working on a Windows machine there's the sh module that makes things nice. See here and here.

ValueError: need more than 0 values to unpack (Python 2)

I am trying to replicate another researcher's findings by using the Python file that he added as a supplement to his paper. It is the first time I am diving into Python, so the error might be extremely simple to fix, yet after two days I haven't still. For context, in the Readme file there's the following instruction:
"To run the script, make sure Python2 is installed. Put all files into one folder designated as “cf_dir”.
In the script I get an error at the following lines:
if __name__ == '__main__':
cf_dir, cf_file, cf_phys_file = sys.argv[1:4]
os.chdir(cf_dir)
cf = pd.read_csv(cf_file)
cf_phys = pd.read_csv(cf_phys_file)
ValueError: need more than 0 values to unpack
The "cf_file" and "cf_phys_file" are two major components of all files that are in the one folder named "cf_dir". The "cf_phys_file" relates only to two survey question's (Q22 and Q23), and the "cf_file" includes all other questions 1-21. Now it seems that the code is meant to retrieve those two files from the directory? Only for the "cf_phys_file" the columns 1:4 are needed. The current working directory is already set at the right location.
The path where I located "cf_dir" is as follows:
C:\Users\Marc-Marijn Ossel\Documents\RSM\Thesis\Data\Suitable for ML\Data en Artikelen\Per task Suitability for Machine Learning score readme\cf_dir
Alternative option in readme file,
In the readme file there's this option, but also here I cannot understand how to direct the path to the right location:
"Run the following command in an open terminal (substituting for file names
below): python cfProcessor_AEAPnP.py cf_dir cf_file cf_phys_file task_file jobTaskRatingFile
jobDataFile OESfile
This should generate the data and plots as necessary."
When I run that in "Command Prompt", I get the following error, and I am not sure how to set the working directory correctly.
- python: can't open file 'cfProcessor_AEAPnP.py': [Errno 2] No such file or directory
Thanks for the reading, and I hope there's someone who could help me!
Best regards & stay safe out there during Corona!!
Marc

cf_dir, cf_file, cf_phys_file = sys.argv[1:4]
means, the python file expects few arguments when called.
In order to run
python cfProcessor_AEAPnP.py cf_dir cf_file cf_phys_file task_file jobTaskRatingFile jobDataFile OESfile
the command prompt should be in that folder.
So, open command prompt and type
cd path_to_the_folder_where_ur_python_file_is_located
Now, you would have reached the path of the python file.
Also, make sure you give full path in double quotes for the arguments.

How could I get the command line that I entered in the terminal to run a python file?

Suppose I have a python file main.py, and it has some optional parameters, --learning-rate, --batch-size, and etc.
If I want to run this file, I can input the following into the terminal(Ubuntu Linux for example).
python3 main.py --learning-rate 0.1 --batch-size 100
And now, I want to write some code in main.py, in order that after I enter the command above, I can get this command in a string by executing those code. The following is the string I want to get:
"python3 main.py --learning-rate 0.1 --batch-size 100"
The reason I want to do this is that I want to write this string into my recording file so that I can know better what command I have run.
Could anyone tell me what package should I import and what code should I write to get that command information during running the python file?
Thanks!

You cannot always get precisely what you typed, because the shell will have first done substitutions and expanded filenames before starting your script. For example, if you type python "foo.py" *.txt, your script won't see *.txt, it will see the list of files, and it won't see the quotes around foo.py.
With that caveat out of the way, the sys module has a variable named argv that contains all of the arguments. argv[0] is the name of the script.
To get the name of the python executable you can use sys.executable.
To tie it all together, you can do something like this:
print(sys.executable + " " + " ".join(sys.argv))

Why not just remake the command using the arguments you parsed? It won't be exactly what you typed, but that might be nice as it will be in a common format.
Ex (assuming the learning rate and batch size are stored in similarly named variables):
command = "python3 main.py --learning-rate {} --batch-size {}".format(learning_rate, batch_size)
Of course it will be a little more complicated if some commands are optional, but I assume in that case there would be a default value for these parameters, since your network would need those parameters every time.

Python for Data Analysis, Chapter 2, first example

I'm following along with the examples in a translated version of Wes McKinney's "Python for Data Analysis" and I was blocked in first example of Chapter 2
I think my problem arose because I saved a data file in a wrong path. is that right?
I stored a file, usagov_bitly_data2012-03-16-1331923249.txt, in C:\Users\HRR
and also stored folder, pydata-book-mater, that can be downloaded from http://github.com/pydata-book in C:\Users\HRR\Anaconda2\Library\bin.

Depends.
You might change the location you save your File or eddit the path you give to your code in Line 10. Since you're yousing relativ Paths i guess your script runs in C:\Users\HRR\Anaconda2\Library\bin, which means you have to go back to C:\Users\HRR or use an absolute Path ... or move the File, but hell you don't want to move a file every time you want to open it, like moving word files into msoffice file to open it, so try to change the Path.
And allways try harder ;)

In python open() will open from the current directory down unless given a full path (in linux that starts with / and windows <C>://). In your case the command is open the folder ch02 in the directory the script is running from and then open usagov_bitly_data2012-03-16-1331923249.txt in that folder.
Since you are storing the text file in C:\Users\HRR\usagov_bitly_data2012-03-16-1331923249.txt and you did not specify the directory of the script. I recommend the following command instead open(C:\\Users\\HRR\\usagov_bitly_data2012-03-16-1331923249.txt)
Note: the double \ is to escape the characters and avoid tabs and newlines showing up in the path.

Python 3 from CMD

I am attempting to run this .PY file from Command Prompt:
# Merge two .BSG files
# Starting block and world position are taken from the first file
# Example: "bsgmerge input.bsg output.bsg merged.bsg"
import io, sys
one = open(sys.argv[1]).readlines()
two = open(sys.argv[2]).readlines()
for n in [1,3,5,7,9,11,17,19,21,23]:
one[n] = one[n][:-1]+"|"+two[n].partition("|")[2]
open(sys.argv[3],"w").write("".join(one))
It is a program that takes a creation from the game Beseige and merges it with another saved creation so that opening the merged file results in both creations being present. If you want more details, you can read up on that here.
I am having trouble figuring out how to call this program from the command line. At first I thought the problem was me having Python 2 (it requires Python 3), so I uninstalled 2 and installed 3. This did not help.
What I am doing is entering the "python" command to pull up the Python environment within CMD, then entering the command to call the program based on the third comment in the file ("bsgmerge input.bsg output.bsg merged.bsg").
I tried using full file paths or simply changing to the correct directory before typing the "python" command and using only the file names, but so far I've had no luck.
When I am in the correct directory, then enter the Python environment, typing the command "bsgmerge 1.bsg 2.bsg M.bsg" (my existing files to be merged are 1.bsg and 2.bsg), this error occurs:
File "<stdin>", line 1
bsgmerge 1.bsg 2.bsg M.bsg
^
SyntaxError: invalid syntax
I took a Python course (which is why I used to have Python 2 on my machine) last fall, so I noticed that there is no "def" defining a function in the above code, which is something I've never encountered, so I'm thinking that is the root of my problems.
Thanks in advance for the help.

I was probably same problem with python launcher.
If you use Linux, first line shoud be:
#! /path/to/your/python/3
In Windows it some more complicated:
In registry by regedit change
HKEY_CLASSES_ROOT\Python.File\shell\open\command from "C:\Python27\python.exe" "%1" %* to "C:\Windows\py.exe" "%1" %*.
And first line of script shoud be:
#! python3
Now it shoud work properly.