Problems passing argument to python function with decorator - python

I have the following code, which results in this error:
TypeError('smallTask() takes exactly 1 argument (2 given)',)
#task
def master():
count = 0
obj = { 'var1':'val1', 'var2':'val2' }
while count < 10:
subtask('smallTask',obj).apply_async()
count += 1
#task(name='smallTask')
def smallTask(obj):
print obj
Passing a dictionary to a function, I imagine I need to use **kwargs but if I do that, I get the error that the function takes no arguments yet 2 have been supplied.
I assume the issue here is with either the decorator (have a basic understanding of this but not enough to solve the problem) or the subtask function in Celery.
I don't have enough python knowledge to really proceed..could anyone give me an idea of what's happening and how I can pass the smallTask function a dictionary?

You need to pass arguments for a subtask in the args keyword argument, which must be a tuple according to the celery.subtask() documentation:
subtask('smallTask', args=(obj,)).apply_async()
or use the Task.subtask() method on your smallTask task, but again pass the arguments as a tuple:
smallTask.subtask((obj,)).apply_async()
Alternatively, use star arguments with the Task.s() method:
smallTask.s(obj).apply_async()
The subtasks documentation you yourself linked to use a tuple in the examples; arguments and keyword arguments are two pieces of data that Celery has to store for you until it can run that task, then it'll apply those arguments and keyword arguments for you.
But the celery.subtask() function takes more than just the arguments and keyword arguments for your task; it also takes additional options. In order to work with arbitrary arguments (positional or keyword) to your task, and support other arguments that are not passed to your task, the function signature has no choice but to accept positional arguments as an explicit tuple, and keyword arguments as an explicit dictionary.
The Task.s() method does not accept any arguments other than what the task itself would accept, so it does support passing arguments as if you called the task directly. Internally, this uses catch-all arguments: Task.s(*args, **kwarg), and just passes the captured arguments as a tuple and dictionary on to Task.subtask().

Related

What are practical use case of Positional only arguments and Keyword only arguments in python? [duplicate]

This question already has answers here:
Why use positional only arguments in python 3.8?
(1 answer)
Mandatory keywords arguments
(2 answers)
Closed 11 months ago.
Python has support for postional only and keyword only arguments
Positional only arguments
def postional_only_func(a,b,/):
...
Keyword only arguments
def keyword_only_func(*,a,b):
...
But, what are its practical use case when writing a function? Is it just to make code more readable?
Both of these features were added to make writing certain functions easier. IF you want to read the full rational and motivation, you should read the corresponding PEPs:
PEP 570 – Python Positional-Only Parameters
PEP 3102 – Keyword-Only Arguments
For positional only, here are concrete examples for use-cases, using examples from the built-in functions:
There are functions with other interesting semantics:
range(), an overloaded function, accepts an optional parameter to the left of its required parameter. [4]
dict(), whose mapping/iterator parameter is optional and semantically must be positional-only. Any externally visible name for
this parameter would occlude that name going into the **kwarg keyword
variadic parameter dict. [3]
One can emulate these semantics in Python code by accepting (*args,
**kwargs) and parsing the arguments manually. However, this results in a disconnect between the function definition and what the function
contractually accepts. The function definition does not match the
logic of the argument handling.
And for keyword-only arguments it says the following regarding motivation:
There are often cases where it is desirable for a function to take a
variable number of arguments. The Python language supports this using
the ‘varargs’ syntax (*name), which specifies that any ‘left over’
arguments be passed into the varargs parameter as a tuple.
One limitation on this is that currently, all of the regular argument
slots must be filled before the vararg slot can be.
This is not always desirable. One can easily envision a function which
takes a variable number of arguments, but also takes one or more
‘options’ in the form of keyword arguments. Currently, the only way to
do this is to define both a varargs argument, and a ‘keywords’
argument (**kwargs), and then manually extract the desired keywords
from the dictionary.

Python: array as an argument for a function

I wanted to know how to work with an array as a functional argument in Python. I will show a short example:
def polynom(x, coeff_arr):
return coeff_arr[0]+ coeff_arr[1]+x +coeff_arr[2]*x**2
I obviously get the error that 2 positional arguments are needed but 4 were given when I try to run it, can anybody tell me how to do this accept just using (coeff_arr[i]) in the argument of the function?
Cheers
Your question is missing the code you use to call the function, but from the error I infer that you are calling it as polynom(x, coefficient1, coefficient2, coefficient3). Instead you need to either pass the coefficients as a list:
polynom(x, [coefficient1, coefficient2, coefficient3])
Or use the unpacking operator * to define the function as follows, which will take all positional arguments after x and put them into coeff_arr as a list:
def polynom(x, *coeff_arr):
(The unpacking operator can also be used in a function call, which will do the opposite of taking a list and passing its elements as positional arguments:
polynom(x, *[coefficient1, coefficient2, coefficient3])
is equivalent to
polynom(x, coefficient1, coefficient2, coefficient3)
)

Understanding positional arguments in Python

For the following Python script:
from sys import argv
script, input_encoding, error = argv
def main(language_file, encoding, errors):
line = language_file.readline()
if line:
print_line(line, encoding, errors)
return main(language_file, encoding, errors)
def print_line(line, encoding, errors):
next_lang = line.strip()
raw_bytes = next_lang.encode(encoding, errors=errors)
cooked_string = raw_bytes.decode(encoding, errors=errors)
print(raw_bytes, "<===>", cooked_string)
languages = open("languages.txt", encoding="utf-8")
main(languages, input_encoding, error)
Looking at the main function I do not understand the following line:
print_line(line, encoding, errors)
Why are we calling the print_line function and passing arguments to it that are named exactly the same as its parameters?
print_line()
When I attempt to call the print_line() argument without passing arguments Python is outputting:
print_line() missing 3 required positional arguments: 'line',
'encoding', and 'errors'
OP: print_line() missing 3 required positional arguments: 'line', 'encoding', and 'errors'
The error is obvious since it is the way the function print_line() was defined.
Furthermore:
def print_line(line, encoding, errors):
print(line, encoding, errors)
line = 1
encoding = 2
errors = 3
print_line(errors, encoding, line)
OUTPUT:
3 2 1
Note: It is positional, not naming arguments
EDIT: 1
def abc(a,b,c=2):
return a+b+c
abc(1,2) #both positional argument and c is default
5
abc(2, b=3) # positional, named and again c is default
7
abc(a=2,b=4) # both named argument and c is default
8
EDIT 2:
OP: What is the purpose of a positional argument please?
Well ..
Short answer: A positional argument is any argument that's not supplied as a key=value pair.
To understand what that means, unfortunately, is somewhat involved.
The term "argument" is used somewhat imprecisely throughout the programming community and especially in Python documentation.
Technically arguments are what you pass into functions and parameters are what you define as the names/placeholders for those arguments.
So, when I define a function thus:
def foo(a,b):
return a+b
... and call it like so:
foo(1,3)
... then a and b are my parameters while 1 and 3 are arguments for a given call to that function.
Now this is a quibble. People will often refer to a and b as "arguments" to their function when they are actually the names (parameters) which will contain the arguments while the function is executing.
Now, with this point made, understand that Python supports four classes of parameters: "required positional" (the kind you've seen for just about any other programming language), "optional" ... or "defaulted" ... positional parameters with some default value specified, "star args" (similar to the VARARGS support in some other languages such as C/C++ and Java), and "kwargs."
The latter is almost unique to Python (though Ruby has very similar support).
So you can define a function with a parameter list like:
def bar(a, b, c=None, d=[], *e, **opts):
'''bar takes all sorts of arguments
'''
results = dict()
results['positional 1'] = a
results['positional 2'] = b
results['sum']=a+b
results['optional'] = list()
for each in e:
results['optional'].append(each)
if c is not None:
results['default over-ridden']=c
else:
results['default']='no arg supplied for parameter c'
d.append(results)
if 'verbose' in opts and opts['verbose']:
for k,v in results:
print '%s:%s' % (k, v)
return (results, d)
... this, rather contrived, example has a couple of normal, traditional positional parameters (a and b), and optional third and fourth parameters (one of which will default to the special Python singleton value None if bar() is called with only two arguments and the other which will default to a list) as well as an optional variable number of additional arguments (which are passed into our function through the parameter named "e") and, finally, bar() can accept any number of "keyword" arguments (passed to it as key-value pairs and referenced through the parameter "opts" (in my example).
There's a lot to digest there. First there are a and b. Those are just like you'd see in most programming languages. If you call this with only one argument you'll get an error because the function requires two arguments. Then there are c and d ... these parameters can be supplied with arguments ... but if the function is called with only the two required arguments than parameter c will refer to "None" and d will refer to the list ... which was instantiated at the time we defined the function!
Whoa! Re-read that last bit because it's a common source of confusion to people who make the mistake of defining functions with parameters that default to mutable types (lists, dictionaries, sets, or most instances of your custom defined classes).
When you're defining a function in Python the interpreter is executing code. It's executing the def statement and evaluating the arguments (which become your function's parameters). So a Python virtual machine instantiates a list (the [] --- empty list literal) as the function is defined. The parameter (e in my example) is now bound to that list, just as any Python "variable" (name) is bound to any other object. And the object to which it refers will be accessible any time bar() is called with three or fewer arguments.
Here's the tricky bit: any time you call bar with more than three arguments then the parameter e will be bound (for the duration of that particular call) to the fourth argument. The underlying list will be hidden for the duration of that call. Such the default list object is contained in a closure and would normally be completely inaccessible outside of the function. (In this example I've including a reference to it in my return statement, showing how we can export a reference to that enclosed object if we choose).
So, if I pass bar() four arguments then it will attempt to modify the object referred to through its "e" parameter using that object's 'append' method. (Obviously that will fail and raise an exception for any object that doesn't support an append method). Every time I call bar() with only three arguments then I'm modifying that enclosed list object within its closure.
This can be useful but is rarely what new programmers expect when they are learning Python. So, as a rule, don't define functions with mutable objects as defaults to your parameters. When you understand the semantics well enough then you know when to break that rule.
I used "None" as my default for the other parameter for a reason. It's often useful to use "None" as the default for any parameter for which you want to have a default value and, then, to explicitly test "is None" and supply your own default from within the body of your function. In this way you can distinguish between your default parameter value and any argument that was explicitly passed to you (and happened to match your default). (This will also prevent the case where you might inadvertently create a mutable closure as previously described. The assignments/bindings occurring in the body of your function will result in a new instantiation for every call that reaches that condition).
So we've covered the required positional parameters, and those which supply defaults (and are, thus, optionally positional parameters).
With parameter 'e' we see Python's support for variable numbers of arguments. Any arguments specified after the fourth will all be gathered up into a tuple and passed to us through our parameter 'e' ... except for any arguments of the form: this=that.
Which brings us, at last. to "opts." Conventionally Python programmers will refer to this parameter as "kwargs" (key/word arguments). I named it "opts" to emphasize that point that "kwargs" is only a conventional bit of terminology and, technically, a bit confusing since, as I've belabored throughout this text, it's a parameter which is referring to any keyword arguments which might have been passed as parameters through some function call.
It's possible to write all your functions such that they take NO positional arguments and are only defined with a "key/words arguments" parameter. You can then ensure that callers to your function must always spell out which argument is bound to what name every time that call on you. This can be handy if your function might have catastrophic consequences in any case where it was called with arguments in the wrong order.
I realize this is confusing ... and I definitely recommend that you play with different function definitions and a wide variety of calls to see how these interact.
I'll also note one additional "gotchya" that can bite you. Your callers CANNOT pass kwargs (opts) values with keys that have names conflicting with your parameter names. Try calling bar() with an argument list like: bar(1,2,3,4,a=99) and you'll get an exception like this: "TypeError: bar() got multiple values for keyword argument 'a'"
Python is parsing the arguments into parameters by managing a namespace (like a dictionary). Attempting to supply a key/word argument which any keys that match your parameter names creates an ambiguity ... and thus raises an exception.
I'll also add two additional notes to this already cumbersome answer.
When you're calling functions in Python you can pass arguments like so:
myargs=(1,2,3)
bar(*myargs)
... and this will be treated as though you'd called it with bar(1,2,3) --- or as if you make the following function call: apply(bar, myargs).
In fact it used to be that you had to use the apply() function in order to accomplish this layer of indirection (back when you could define functions with *foo parameters but not call them with *foo arguments).
(The addition of the *args syntax on function calls largely displaced the use of Python's apply() builtin function).
... and lastly it's possible to pass a dictionary of kwargs using this syntax:
mykwargs={'z':99, 'whatever':'yikes'}
bar(1,2,3, **mykwargs)
... or combined with my previous example:
bar(*myargs, **mykwargs)
So it's vital to understand this distinction between what * and ** mean when defining functions and what they mean when calling them. The meanings are complementary to one another and intuitive if you understand the distinction between arguments and parameters (despite how the term "arguments" is more commonly used).
Why are we calling the print_line function and passing arguments to it that are named exactly the same as it's parameters?
That is really just a coincidence. The following does exactly the same as your example:
from sys import argv
script, input_encoding, error = argv
def main(language_file, what_encoding_do_you_want, list_of_errors):
next_line = language_file.readline()
if next_line:
print_line(next_line, what_encoding_do_you_want, list_of_errors)
return main(language_file, what_encoding_do_you_want, list_of_errors)
def print_line(line, encoding, errors):
next_lang = line.strip()
raw_bytes = next_lang.encode(encoding, errors=errors)
cooked_string = raw_bytes.decode(encoding, errors=errors)
print(raw_bytes, "<===>", cooked_string)
languages = open("languages.txt", encoding="utf-8")
main(languages, input_encoding, error)
It all comes down to 'scope'. The function-definition of print_line declares that it has three (positional) arguments, which can be referred to inside the function as 'line', 'encoding' and 'errors'.
The function-call to print_line from main adds three arguments to the call that refer to existing variables or arguments at that point in code: line refers to the variable created there; encoding and encoding refer to the argument to main function itself.
The function requires three arguments, and normally, you provide them in the order specified.
Python allows you to pass them in any order you like, though:
print_line(encoding='ascii', line='hello', errors=None)
or even
my_dict = {
'line': 'privet, mir!',
'errors': None,
'encoding': 'utf-8'
}
print_line(**my_dict)
However, the function requires all these arguments; you have to pass them in somehow, or you get an error message, like you well noticed.
The variables' scope is local: Anything in the def is local to the function it defines, and separate from any other variables with the same name outside of that block.
A parameter can be made optional by defining a default value. Optional parameters must always come after mandatory parameters in Python.
def another_fun(confetti, fireworks, bassoons=None, candy='sugar fluff')
If you call another_fun with only two parameters, bassoons will default to None and candy will be set to the string 'sugar fluff'.
Why are we calling the print_line function
Well, to execute it obviously ;-)
and passing arguments to it
Because the function is defined to take arguments - it cannot do its job if you don't give what it needs to work on.
that are named exactly the same as its parameters?
This is actually totally irrelevant - it just happens that the variables in main() are named the same as the function's arguments (which makes sense since we're taling about the same things - a "line of text", an encoding name and a value that describes how to handle encoding errors), but you pass litteral (unnamed) values or variables with just any arbitrary name, it would work just the same.
A function is a (mostly) self-contained unit of work. It usually uses some internal ("local") variables which only it can sees and that only exists during the function's execution. It also usually takes arguments: values that must be passed by the caller, and are bound to the matching arguments names (in our case, the first value will be bound to the line name, the second to the encoding name etc). The local names (local variables and arguments) are totally unrelated to the name under which those values are known in the caller's scope (if even they are bound to names - as I said you can as well pass literal values or other anonymous objects)
When I attempt to call the print_line() argument without passing arguments Python is outputting "print_line() missing 3 required positional arguments: 'line', 'encoding', and 'errors'"
Yes, of course. The function needs three arguments, so you have to pass three arguments, plain and simple. The fact that you have three local variables by the same name in the caller scope (the main function) will not automagically "fill in" those arguments for you, and the print_line function knows absolutely nothing about it's caller's scope anyway.
Note that the terms "positional" and "named" arguments mostly to refer to how you pass the arguments themselves - by position (the default), which I already explained above, or by name (ie print_line(line="hello", errors="ignore", encoding="utf-8"), which allow you to pass the arguments in a different order, or from a dictionnary you've built dynamically etc etc, but you first need to understand the concepts of function, arguments and scope before going further...
I strongly suggest that you do the official tutorial, which does have a chapter on functions - it's mostly intended at peoples with already some programming experience (it's not a CS101 course) but it's still well explained.

Difference between Positional , keyword, optional and required argument?

I am learning about functions in python and found many good tutorials and answers about functions and their types, but I am confused in some places. I have read the following:
If function has "=" then it's a keyword argument i.e (a,b=2)
If function does not have "=" then it's positional argument i.e (a,b)
My doubts :
What is the meaning of a required argument and an optional argument? Is the default argument also a keyword argument? (since because both contain "=")
Difference between the positional, keyword, optional, and required
arguments?
python official documentation says that there are two types of arguments. If so, then what are *args and **kargs (I know how they work but don't know what they are)
how *args and **kargs store values? I know how *args and
**kargs works but how do they store values? Does *args store values in a tuple and **kargs in the dictionary?
please explain in deep. I want to know about functions because I am a newbie :)
Thanks in advance
Default Values
Let's imagine a function,
def function(a, b, c):
print a
print b
print c
A positional argument is passed to the function in this way.
function("position1", "position2", "position3")
will print
position1
position2
position3
However, you could pass in a keyword argument as below,
function(c="1",a="2",b="3")
and the output will become:
2
3
1
The input is no longer based on the position of the argument, but now it is based on the keyword.
The reason that b is optional in (a,b=2) is because you are giving it a default value.
This means that if you only supply the function with one argument, it will be applied to a. A default value must be set in the function definition. This way when you omit the argument from the function call, the default will be applied to that variable. In this way it becomes 'optional' to pass in that variable.
For example:
def function(a, b=10, c=5):
print a
print b
print c
function(1)
and the output will become:
1
10
5
This is because you didn't give an argument for b or c so they used the default values. In this sense, b and c are optional because the function will not fail if you do not explicitly give them.
Variable length argument lists
The difference between *args and **kwargs is that a function like this:
def function(*args)
for argument in args:
print argument
can be called like this:
function(1,2,3,4,5,6,7,8)
and all of these arguments will be stored in a tuple called args. Keep in mind the variable name args can be replaced by any variable name, the required piece is the asterisk.
Whereas,
def function(**args):
keys = args.keys()
for key in keys:
if(key == 'somethingelse'):
print args[key]
expects to be called like this:
function(key1=1,key2=2,key3=3,somethingelse=4,doesnt=5,matter=6)
and all of these arguments will be stored in a dict called args. Keep in mind the variable name args can be replaced by any variable name, the required piece is the double asterisk.
In this way you will need to get the keys in some way:
keys = args.keys()
Optional arguments are those that have a default or those which are passed via *args and **kwargs.
Any argument can be a keyword argument, it just depends on how it's called.
these are used for passing variable numbers of args or keyword args
yes and yes
For more information see the tutorial:
https://docs.python.org/2/tutorial/controlflow.html#more-on-defining-functions

What is the best way to pass multiple parameters to class init in python?

I am creating a module in python that can take multiple arguments. What would be the best way to pass the arguments to the definition of a method?
def abc(arg):
...
abc({"host" : "10.1.0.100", "protocol" : "http"})
def abc(host, protocol):
...
abc("10.1.0.100", "http")
def abc(**kwargs):
...
abc(host = "10.1.0.100", protocol = "http")
Or something else?
Edit
I will actually have those arguments (username, password, protocol, host, password2) where none of them are required.
def abc(host, protocol):
...
abc("10.1.0.100", "http")
abc(host="10.1.0.100", protocol="http")
If you call it using positional args or keyword args depends on the number of arguments and if you skip any. For your example I don't see much use in calling the function with keyword args.
Now some reasons why the other solutions are bad:
abc({"host" : "10.1.0.100", "protocol" : "http"})
This is a workaround for keyword arguments commonly used in languages which lack real keyword args (usually JavaScript). In those languages it is nice, but in Python it is just wrong since you gain absolutely nothing from it. If you ever want to call the functions with args from a dict, you can always do abc(**{...}) anyway.
def abc(**kwargs):
People expect the function to accept lots of - maybe even arbitrary - arguments. Besides that, they don't see what arguments are possible/required and you'd have to write code to require certain arguments on your own.
If all the arguments are known ahead, use an explicit argument list, optionally with default values, like def abc(arg1="hello", arg2="world",...). This will make the code most readable.
When you call the function, you can use either abd("hello", "world") or abc(arg1="hello", arg2="world"). I use the longer form if there are more than 4 or 5 arguments, it's a matter of taste.
This really depends on the context, the design and the purpose of your method.
If the parameters are defined and compulsory, then the best option is:
def abc(host, protocol):
...
abc('10.1.0.100', 'http')
in case the parameters are defined but optional, then:
def abc(host=None, protocol=None): # I used None, but put a default value
...
you can call it thorugh positional arguments as abc('10.1.0.100', 'http') or by their name abc(protocol='http')
And if you don't know at first what are the arguments it will receive are (for example in string formatting) then the best option is using the **kwargs argument:
def abc(**kwargs):
...
And in this way you must call it using named arguments.

Categories