We Keep Coding

How can I handle index when reading from stdin [duplicate] - python

This question already has answers here:
embedding short python scripts inside a bash script
(9 answers)
Closed 4 years ago.
I'm working with bash scripts and would like to embed a Python snippet inside a bash function.
So I got this working Python snippet, which simply reads from stdin and parse it to get the title of the entry[0]:
import feedparser, sys
root = feedparser.parse(sys.stdin.read())
print root['entries'][0].title
And everything seems fine with it:
$ curl -sf https://feedforall.com/sample.xml | python xmlparser.py
RSS Solutions for Restaurants
An IndexError happens when I execute this way:
$ curl -sf https://feedforall.com/sample.xml | python - <<EOF
import feedparser, sys
root = feedparser.parse(sys.stdin.read())
print root['entries'][0].title
EOF
Got this IndexError:
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
IndexError: list index out of range
Seems like root['entries'] in this case is returning an empty list..and I don't know why.
Thanks for help

Worked fine executing like that:
$ curl -sf https://feedforall.com/sample.xml | python <( cat <<EOF
import feedparser, sys
root = feedparser.parse(sys.stdin.read())
print root['entries'][0].title
EOF
)

Related

This question already has an answer here:
How to get bash arguments with leading pound-sign (octothorpe)
(1 answer)
Closed 1 year ago.
I am trying to pass a command line argument like this because the actual string contains a # symbol before and after it
python3 test.py #fdfdf#
This is the error message I am getting
Traceback (most recent call last):
File "test.py", line 3, in <module>
print(sys.argv[1])
IndexError: list index out of range
How can I pass a string which contains a # symbol in the beginning and the end as a command line argument?
Update:
OS: Ubuntu 20.04

You can try one of these and see which one works:
python3 test.py "#fdfdf#" or python3 test.py \#fdfdf\#

This question already has an answer here:
Passing a url as argument
(1 answer)
Closed 2 years ago.
I have a Python script which uses sys to accept arguments -
import sys
url = sys.argv[1]
I need to provide it with a bunch of urls to parse. The script works perfectly on a jupytor notebook (after hard-coding the arguments in the code: url = 'http://www.hello.com') notebook but when I try to execute it as a script I get errors like these, for various URLs -
for 'http://www.blog.example.com:123/path/to/file.html?key1=value1'
[1] 85926
zsh: no matches found: http://www.blog.example.com:123/path/to/file.html?key1=value1
[1] + exit 1 python -m urlparser
for 'https://www.hello.com/photo.php?id=2064343443411&set=a.2634433167446&type=3&hall'
zsh: parse error near `&'
Meanwhile, the script works fine for simpler URLs like https://blog.hello.com/one/two
What could be the issue? Encoding problems?

I figured -
Have to put the arguments in quotes like - 'http://www.blog.example.com:123/path/to/file.html?key1=value1'

This question already has answers here:
Actual meaning of 'shell=True' in subprocess
(7 answers)
Closed 2 years ago.
I'm trying to use some particle physics software and run it in a loop with slightly changed variables each time. It has specific functionality to take a txt file and scan it for relevant commands for this, so naturally I am trying to make use of this. I have written a python script (this was originally written in python3 to run on my laptop, but I need to move over to the university's cluster. I've converted to python2 as the particle physics software itself is python 2 and trying to run both versions of python on the cluster simultaneously is not something I know how to do. Specifically the cluster has python2.6.9 installed, I do not have any ability to change this, I also cannot update it or install new modules. The problematic part of my code is:
def run_madgraph():
# starts madgraph and tells it to run mg5_aMC using mg_run_iridis.txt as a file for it.
print('starting....')
subprocess.check_call(['python', './madgraph/bin/mg5_aMC mg_run_iridis.txt'])
mg5_aMC starts up the physics software, mg_run_iridis.txt is the text file with the information about variables. I don't understand why this doesn't work as the following command works just fine in the bash terminal (on the cluster):
python ./madgraph/bin/mg5_aMC mg_run_iridis.txt
As far as I can tell the issue is on the python-bash side rather than it being anything in the physics software. The error I get is:
(MGenv) [cb27g11#green0155 Development]$ python MainIridis_script.py
about to run madgraph 1.0 0.9
starting....
python: can't open file '/home/cb27g11/Development/./madgraph/bin/mg5_aMC mg_run_iridis.txt': [Errno 2] No such file or directory
Traceback (most recent call last):
File "/home/cb27g11/Development/MainIridis_script.py", line 41, in <module>
the_main_event()
File "/home/cb27g11/Development/MainIridis_script.py", line 21, in the_main_event
run_madgraph()
File "/home/cb27g11/Development/MainIridis_script.py", line 38, in run_madgraph
subprocess.check_call(['python', './madgraph/bin/mg5_aMC mg_run_iridis.txt'])
File "/home/cb27g11/.conda/envs/MGenv/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', './madgraph/bin/mg5_aMC mg_run_iridis.txt']' returned non-zero exit status 2.
Just to be clear, /home/cb27g11/Development, definitely exists:
(MGenv) [cb27g11#green0155 Development]$ pwd
/home/cb27g11/Development
As do the various files involved:
(MGenv) [cb27g11#green0155 Development]$ ls
AD_noCharge_021020.tar.gz NoChar_Allh_1.0_0.9 mg_run_basic.txt
MainIridis_script.py NoChar_Allh_1_0.9 mg_run_iridis.txt
NoCh_h1Only.tar.gz iridis_script.pbs py.py
NoCh_h2Only.tar.gz madgraph
Including mg5_aMC:
(MGenv) [cb27g11#green0155 Development]$ cd madgraph/
(MGenv) [cb27g11#green0155 madgraph]$ cd bin
(MGenv) [cb27g11#green0155 bin]$ ls
mg5 mg5_aMC py.py
I don't know how else to go about this, and ultimately however I do it, I still need to call this script file inside another script (bash this time) in order to submit it to the cluster.
I'm unsure if it's possible to just avoid the python script all together and switch to bash, I suspect at least some of it needs to remain in python but I'm a physicist not a programmer so I could easily be wrong! Here's the full script in anycase:
from __future__ import absolute_import, division, print_function
import numpy as np
import subprocess
import fileinput
tan_beta = np.arange(1, 21.2, 0.2) #Array of tan(beta) values
sin_bma = np.arange(0.9, 1.001, 0.001)#Array of sin(beta-alpha) values
hcm = 1.500000e+05 # Mass of charged Higgses in GeV
textfilepath = 'mg_run_iridis.txt' #path to txt file madgraph will use
Process = 'NoChar_Allh'
def the_main_event():
with open('mg_run_basic.txt','r') as old_card:
text =old_card.read() #stores string of old_card ready for editing
for i in range(0, len(tan_beta)):
tb = tan_beta[i] # loops over tan(beta) values
for j in range(0, len(sin_bma)):
sbma = sin_bma[j] # loops over sin(beta-alpha) values
make_input(tb, sbma, hcm, text)
print('about to run madgraph ' + str(tb) + ' ' + str(sbma))
run_madgraph()
def make_input(Tb, Sbma, Hcm, Text):
# inputs are the value of tan_beta, the value of sin(beta-alpha) values, the desired mass for the charged higgses and a string of text
with open(textfilepath, 'w') as new_card:
#simulation card, the .txt file that gets fed to madgraph
sim_card = Text.replace('TBV', str(Tb))
sim_card = sim_card.replace('SBMAV', str(Sbma))
sim_card = sim_card.replace('HCMV', str(Hcm))
sim_card = sim_card.replace('NAME', str(Process)+'_' + str(Tb) +'_' + str(Sbma))
new_card.write(sim_card) # saves new txt file for madgraph
def run_madgraph():
# starts madgraph and tells it to run mg5_aMC using mg_run_iridis.txt as a file for it.
print('starting....')
subprocess.check_call(['python', './madgraph/bin/mg5_aMC mg_run_iridis.txt'])
the_main_event()
It seems like from the errors it's treating the madgraph executable (I'm not sure if thats correct terminology, I mean the file that starts up the propgram) as a module?
Any help would be really appreciated!

Try:
subprocess.run('python ./madgraph/bin/mg5_aMC mg_run_iridis.txt', shell=True)
Note this run part. See documentaion for details. Currently, you are trying to call the subprocess module itself, rather then a particular function from the module.

This question already has answers here:
Command not found error in Bash variable assignment
(5 answers)
Closed 3 years ago.
I want to run a python script which is parsed from a bash file. When I try to run it, I get errors everywhere and I do not know what the issue could be. Any help?
The bash script (run_dmd_atomic.sh) looks like this:
#!/bin/bash
BATCH_SIZE=${1}
CODE_LENGTH=${2}
EPOCHS=${3}
SP=${4}
A1=${5}
A2=${6}
A3=${7}
srun python3 dmd_solver.py \
--batch_size ${BATCH_SIZE} \
--code_length ${CODE_LENGTH} \
--epochs ${EPOCHS} \
--sp ${SP} \
--a1 ${A1} \
--a2 ${A2} \
--a3 ${A3} \
The error I get is:
[tmarta#eu-login-09-ng training]$ ./run_dmd_atomic.sh
./run_dmd_atomic.sh: line 20: srun: command not found

In bash script, no white space is allowd in variable assignment expression. Otherwise, bash interpret it as a command or excutable.
Your error log is pointing out that.
For example, in your script
BATCH_SIZE = ${1}
should be
BATCH_SIZE=${1}

This question already has answers here:
Can't find any info on Python's read() method (python 2.7)
(2 answers)
Closed 7 years ago.
I'm learning python from LPTH.
In exercise 15 in study drills, I'm supposed to know what read() does using pydoc; However when I try to do so with, python -m pydoc read(), I get an error like this.
an expression is expected after << ( >>
at line : 1 character 23
python -m pydoc read ( <<<< )
*category info : parser error : (:), parentcontainsErrorRecordException
* FullyQualifiedErrorID: ExpectedExpression
I don't understand what I did wrong.
I used the same way for: raw_input,os,open, but apparently I am doing something wrong with read().

You probably want this:
python -m pydoc read
But that will give you: no Python documentation found for 'read'
In what context are you trying to use read()... Example: to open a file and read and see associated docs, use
python -m pydoc file