Retrieving args from running a script via sagemaker processing: unrecognized arguments - python

I am testing a job. For that, I run the script via notebook using the following code.
I want to be able to retrieve the value of the arg
"dataset" via "argparse". I don't understand what I am doing wrong here:
The notebook that runs the job:
from sagemaker.processing import Processor, ProcessingInput, ProcessingOutput
from sagemaker.network import NetworkConfig
netconfig = NetworkConfig(XXX)
dataset= 'A'
processor = Processor(image_uri='XXX',
role='XXX',
instance_count=1,
instance_type="XXX",
network_config=netconfig)
processor.run(inputs=[ProcessingInput(
source='s3://input-bucket/settings_dataset_A.json',
destination='/opt/ml/processing/input')],
outputs=[ProcessingOutput(
source='/opt/ml/processing/output',
destination='s3://output-bucket')],
arguments=["--verbose", "--dataset", dataset],
wait=True,
logs=True)
and the code in the script:
parser = argparse.ArgumentParser()
parser.add_argument('--verbose', help="verbose",action="store_true")
parser.add_argument("--dataset", type=str, required =False)
args = parser.parse_args()
print('Arguments:', sys.argv)
print(args.dataset)
It gives an error: "unrecognized argument" but it is able to print the correct value ! And then it just stops running the rest of the code.
Do you see anything wrong with the code above ? Thanks a lot

Related

TypeError: 'NoneType' object is not iterable when i use argparse

def getOptions():
parser = argparse.ArgumentParser(description='Parses Command.')
parser.add_argument('-t','--train',nargs='+',help='Training data directories.')
parser.add_argument('-i','--iteration',type=int,nargs='+',help='Number of iteration.')
options = parser.parse_args()
return options
I know that parser.parse_args() returns an not iterable object
i want to use "options.train" in a for loop but i cant go over that error. Also, vars dont work for me
The reason you are encountering this error is that, you need to provide train
arguments to parse_args() method either like following or providing it in command line:
import argparse
def get_options():
parser = argparse.ArgumentParser(description='Parses Command.')
parser.add_argument('-t', '--train', nargs='+', help='Training data directories.')
parser.add_argument('-i', '--iteration', type=int, nargs='+', help='Number of iteration.')
options = parser.parse_args(['-t', 'train1', 'train2'])
return options
options = get_options()
print('train arguments:', options.train)
for option in options.train:
print('train:', option)
Result of above code is the following:
train arguments: ['train1', 'train2']
train: train1
train: train2
Assuming the above Python code is saved into a file called train.py, it can be run at the command line without specifying arguments to parse_args() method:
$ python train.py -t train/etc train/etc2
train arguments: ['train/etc', 'train/etc2']
train: train/etc
train: train/etc2
In case there is no -t args to parse there is a possible way to handle occurred exception in try-except code block or refer to Python argparse: Make at least one argument required for more solutions of ensuring availability of inputs.

Running Argpars but got this error SystemExit 2

I am trying to set up training arguments and parse but I got this error could anyone help please!
parser = argparse.ArgumentParser(description='Explore pre-trained AlexNet')
parser.add_argument(
'--image_path', type=str,
help='Full path to the input image to load.')
parser.add_argument(
'--use_pre_trained', type=bool, default=True,
help='Load pre-trained weights?')
args = parser.parse_args()
got this error
usage: ipykernel_launcher.py [-h] [--image_path IMAGE_PATH]
[--use_pre_trained USE_PRE_TRAINED]
ipykernel_launcher.py: error: unrecognized arguments: -f /root/.local/share/jupyter/runtime/kernel-ff8e2476-e39b-4e40-b8f9-6b8113fe8f1f.json
An exception has occurred, use %tb to see the full traceback.
SystemExit: 2
In a Jupyter notebook cell:
import sys
sys.argv
I get get
['/usr/local/lib/python3.8/dist-packages/ipykernel_launcher.py',
'-f',
'/home/paul/.local/share/jupyter/runtime/kernel-7923bfd2-9f96-45cf-8b44-1859a2185715.json']
The Jupyter server is using the sys.argv to set up the communication channel with your kernel. argparse parses this list too.
So commandline and argparse cannot be used to provide arguments to your notebook when run this way.
How did you start this script? Did you even try to provide the commandline values that the script expected?
'--image_path'
'--use_pre_trained'
If you did, you probably would have gotten a different parser's error, about 'unexpected arguments'. That's coming from the server.
If you use a Colab may this solution help you, this solution suggest to you to write argparse in the other python file introduction-argparse-colab
%%writefile parsing.py
import argparse
parser = argparse.ArgumentParser()
parser.parse_args()
In a Jupyter notebook code works like this:
parser = argparse.ArgumentParser()
parser.add_argument('--image_folder', type=str, default='/content/image', help='path to image folder')
parser.add_argument("-f", "--file", required=False)
You need to add in the end of " parser.add_argument " line ( This is the reason you are getting 'SystemExit: 2' error ):
parser.add_argument("-f", required=False)
in your case this should work:
parser = argparse.ArgumentParser(description='Explore pre-trained AlexNet')
parser.add_argument('--image_path', type=str, default='your_path', help='Full path to the input image.')
parser.add_argument('--use_pre_trained', type=bool, default=True, help='Load pre-trained weights?')
parser.add_argument("-f", "--file", required=False)
args = parser.parse_args()
Now you can call:
image = args.image_path
Or
from PIL import Image
image = Image.open(args.image_path)
Tested in Google colab

argparse error with parsing required arguments

I have a script saved as workspace.py
import argparse
import os
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('title', type=str, help="Will be displayed as the title")
parser.add_argument('-f', '--folder', help='Point to the folder you want to read from (defaults to current folder in command prompt)', type=str, default=os.getcwd())
args = parser.parse_args()
print(args)
someFunction(args.folder, args.title)
Which I call from terminal with:
workspace.py myTitle
Resulting in the error
workspace.py: error: the following arguments are required: title
I have no idea why this is happening because I supply "myTitle" in the terminal. If I specify a default= for the title argument it works perfectly with that value. The part that is throwing me is it doesn't even get to the print(args) so I cannot see what the program thinks is what, but instead fails at args = parser.parse_args()
I tried to even redo the exact example at: https://docs.python.org/2/howto/argparse.html#introducing-positional-arguments (copied below)
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("echo", help="echo the string you use here")
args = parser.parse_args()
print args.echo
Running
workspace.py hello
Results in (after adding parenthesis to the print for 3.X)
workspace.py: error: the following arguments are required: echo
Is there something I'm missing? Why does it not just print "hello"? Is there some Python 3 specific syntax I'm missing or something?
I've gotten it to work if I run python workspace.py someString instead of workspace.py someString. I do not understand why this version works, since command prompt obviously recognizes it as Python and runs it correctly until args = parser.parse_args(). There were no errors like 'workspace.py' is not recognized as an internal or external command, operable program or batch file. There was no problem importing modules either. Consider the below command prompt session if you are running into a similar error. Maybe you will simply have to include python in your commands like I have to...
C:\Users\rparkhurst\PycharmProjects\Workspace>workspace.py MyTitle
usage: workspace.py [-h] [-f FOLDER] title
workspace.py: error: the following arguments are required: title
C:\Users\rparkhurst\PycharmProjects\Workspace>python workspace.py MyTitle
Namespace(folder='C:\\Users\\rparkhurst\\PycharmProjects\\Workspace', title='MyTitle')

Print any error from the script to a file - stderr

I have a python program that accepts three command line arguments -f (file1) -n (file2) -o (file3).
The program is structured as follows:
ffile=''
nfile=''
ofile=''
try:
myopts, args = getopt.getopt(sys.argv[1:],"f:n:o:h:")
for x,y in myopts:
if x == '-f':
ffile=y
elif x == '-n':
nfile=y
elif x == '-o':
ofile=y
<do something here with the files>
except:
sys.stderr("err.txt",w)
sys.exit(2)
What I am trying to do is:
If any of the arguments (-f,-n) are missing, then print a corresponding error.
Print the error message - whatever it might be - to a file "err.txt".
However, the err.txt file is empty. I am not sure what is going on. I have seen many questions where stderr is written to a file, but it does not seem to answer my question.
If you are open to changing your approach, I would recommend not trying to re-invent the wheel, and checking out some of the existing python libraries to accomplish this functionality.
There is a library called argparse that does well at parsing out command line arguments and returning appropriate error messages, maybe you would find it helpful.
import argparse
p = argparse.ArgumentParser(description='My Program', formatter_class=argparse.ArgumentDefaultsHelpFormatter)
p.add_argument("-f", type=str, help="-f file", required=True)
p.add_argument("-n", type=str, help="-n file", required=True)
p.add_argument("-o", type=str, help="-o file", default=None)
args = p.parse_args() # will throw error if required -f and -n args not passed
In terms of error logging, you could try using the python logging library. This library can be configured to log information to stdout, text files, etc. and is likely flexible enough for your requirements.

error: too few arguments argparse

I'm trying to import a json file into firebase using a python script. Then I came across a similar script in Github. I tried running it after the replacing the required parameters. I'm getting a too few arguments error. This is the Github project if you would like to refer
https://github.com/firebase/firebase-streaming-import
Snippet of the code I'm running is given below. I'm using IDLE to run the program. Thanks in advance
argParser = argparse.ArgumentParser(description="Import a large json file into a Firebase via json Streaming.\
Uses HTTP PATCH requests. Two-pass script, run once normally,\
then again in --priority_mode.")
argParser.add_argument('firebase_url', help="Specify the Firebase URL (e.g. https://test.firebaseio.com/dest/path/).")
argParser.add_argument('json_file', help="The JSON file to import.")
argParser.add_argument('-a', '--auth', help="Optional Auth token if necessary to write to Firebase.")
argParser.add_argument('-t', '--threads', type=int, default=8, help='Number of parallel threads to use, default 8.')
argParser.add_argument('-s', '--silent', action='store_true',
help="Silences the server response, speeding up the connection.")
argParser.add_argument('-p', '--priority_mode', action='store_true',
help='Run this script in priority mode after running it in normal mode to write all priority values.')
main(argParser.parse_args())
Since you cannot parse arguments with parameters like in terminal/cmd in IDLE. Try parsing the args as such : args = argParser.parse_args(['URL','FILE_PATH'])
import argparse
argParser = argparse.ArgumentParser(description="Import a large json file into a Firebase via json Streaming.\
Uses HTTP PATCH requests. Two-pass script, run once normally,\
then again in --priority_mode.")
argParser.add_argument('firebase_url', help="Specify the Firebase URL (e.g. https://test.firebaseio.com/dest/path/).")
argParser.add_argument('json_file', help="The JSON file to import.")
argParser.add_argument('-a', '--auth', help="Optional Auth token if necessary to write to Firebase.")
argParser.add_argument('-t', '--threads', type=int, default=8, help='Number of parallel threads to use, default 8.')
argParser.add_argument('-s', '--silent', action='store_true',
help="Silences the server response, speeding up the connection.")
argParser.add_argument('-p', '--priority_mode', action='store_true',
help='Run this script in priority mode after running it in normal mode to write all priority values.')
args = argParser.parse_args(['URL','FILE_PATH'])
print(args)
main(args)
Output:
Namespace(auth=None, firebase_url='URL', json_file='FILE_PATH', priority_mode=False, silent=False, threads=8)

Categories