Am a newbie to Python. I have come across this code:
Xtrain is a pandas dataframe object with a shape of X train shape= (42000, 784)
But am able to call the python any() builtin function on the object returned by isnull function of a pandas dataframe
train = pd.read_csv(datapath+"/" +"train.csv")
X_train = train.drop(labels=["label"],axis=1)
X_train.isnull().any().describe()
I understood that builtin functions can be called directly from anywhere. Does the above mean that they are accessible to all objects also ? i.e. in some way every class inherits builtin functions ?
The comments already address the question, but for the sake of posting a solution...
builtin functions can be called from anywhere, on anything - though whether or not they will work depends on whether or not you are passing in a valid input for that function.
any is a builtin function.
HOWEVER.. in this case, you are actually working with an any method that belongs to the pd.DataFrame class.
try dir(pd.DataFrame) .. you will see any listed in there.
this means it's a function that belongs to the class (making it a method).
this is NOT the same as the builtin any.
Further proof...
import inspect
print(inspect.getfile(any))
output
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\deden\AppData\Local\Continuum\anaconda3\lib\inspect.py", line 666, in getfile
type(object).__name__))
TypeError: module, class, method, function, traceback, frame, or code object was expected, got builtin_function_or_method
print(inspect.getfile(pd.DataFrame.any))
output
C:\Users\deden\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\generic.py
Related
I start with the assumption that I have already looked at many other posts related to the "builtin_function_or_method" error, but I have not found a solution to my problem. I really hope that someone has a moment to read because I write this post out of exhaustion.
To summarize: the following piece of code invokes the "get_valid_locations_" function
valid_locations = eval.get_valid_locations_(board)
The method I call is a function of the Evaluate class that I have previously imported in this way (I attach both how I imported the file and how I instantiated the object)
from Utilities import evaluate
eval = evaluate.Evaluate()
While the function is as follows
def get_valid_locations_(self, board):
valid_locations = []
for col in range(NUMS_COL):
if eval.is_valid_location(board, col):
valid_locations.append(col)
return valid_locations
I cannot understand what generates the error. It seems to me that the assignment and the way in which I import the class is correct, also because in the Evaluate class there are other functions that do not generate errors. So the problem I think is restricted to either the function or how it is invoked. Unless the Python compiler sees the method. I also tried to add the _ to the end of the function name but nothing.
I hope someone can help me or have some advice on how to better structure the files for projects. Thanks a lot in advance and have a nice day.
In this line that's failing:
valid_locations = eval.get_valid_locations_(board)
It sees eval as the standard Python builtin function eval. Wherever you defined it like this:
eval = evaluate.Evaluate()
The first line is not seeing that definition.
First of all, choose a different name that isn't a Python builtin to reduce confusion.
Then make sure that your use of the object eval has the correct value. If you defined it globally in a module, then from some_module import eval. Or you could pass it as an argument to a function where eval.get_valid_locations_(board) is called.
Your definition of get_valid_locations_ has a similar problem where it calls eval.is_valid_location(board, col). If get_valid_locations_ and is_valid_location are both methods of the class Evaluate, then you should call self.is_valid_location instead. Not all instances of Evaluate should necessarily be called eval, nor should they have to be defined in the same file as the class.
you created an eval object, and then trying to call get_valid_locations_ function on it.
valid_locations = eval.get_valid_locations_(board)
this is what generates the error. eval object has no get_valid_locations_ function.
The attr's package somehow ruins pytorch's parameter() method for a module. I am wondering if anyone has any work-arounds or solutions, so that the two packages can seamlessly integrate?
If not, any advice on which github to post the issue to? My instinct would be to post this onto attr's github, but the stack trace is almost entirely relevant to pytorch's codebase.
Python 3.7.3
attrs== 19.1.0
torch==1.1.0.post2
torchvision==0.3.0
import attr
import torch
class RegularModule(torch.nn.Module):
pass
#attr.s
class AttrsModule(torch.nn.Module):
pass
module = RegularModule()
print(list(module.parameters()))
module = AttrsModule()
print(list(module.parameters()))
The actual output is:
$python attrs_pytorch.py
[]
Traceback (most recent call last):
File "attrs_pytorch.py", line 18, in <module>
print(list(module.parameters()))
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 814, in parameters
for name, param in self.named_parameters(recurse=recurse):
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 840, in named_parameters
for elem in gen:
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 784, in _named_members
for module_prefix, module in modules:
File "/usr/local/anaconda3/envs/bgg/lib/python3.7/site-packages/torch/nn/modules/module.py", line 975, in named_modules
if self not in memo:
TypeError: unhashable type: 'AttrsModule'
The expected output is:
$python attrs_pytorch.py
[]
[]
You may get it to work with one workaround and using dataclasses (which you should, as it's in standard Python library since 3.7 which you are apparently using). Though I think simple __init__ is more readable. One could do something similar using attrs library (disabling hashing), I just prefer the solution using standard libraries if possible.
The reason (if you manage to handle hashing related errors) is that you are calling torch.nn.Module.__init__() which generates _parameters attribute and other framework-specific data.
First solving hashing with dataclasses:
#dataclasses.dataclass(eq=False)
class AttrsModule(torch.nn.Module):
pass
This solves hashing issues as, as stated by the documentation, section about hash and eq:
By default, dataclass() will not implicitly add a hash() method
unless it is safe to do so.
which is needed by PyTorch so the model can be used in C++ backed (correct me if I'm wrong), furthermore:
If eq is false, hash() will be left untouched meaning the
hash() method of the superclass will be used (if the superclass is object, this means it will fall back to id-based hashing).
So you are fine using torch.nn.Module __hash__ function (refer to documentation of dataclasses if any further errors arise).
This leaves you with the error:
AttributeError: 'AttrsModule' object has no attribute '_parameters'
Because torch.nn.Module constructor is not called. Quick and dirty fix:
#dataclasses.dataclass(eq=False)
class AttrsModule(torch.nn.Module):
def __post_init__(self):
super().__init__()
__post_init__ is a function called after __init__ (who would of guessed), where you can initialize torch-specific parameters.
Still, I would advise against using those two modules together. For example, you are destroying PyTorch's __repr__ using your code, so repr=False should be passed to the dataclasses.dataclass constructor, which gives this final code (obvious collisions between libraries eliminated I hope):
import dataclasses
import torch
class RegularModule(torch.nn.Module):
pass
#dataclasses.dataclass(eq=False, repr=False)
class AttrsModule(torch.nn.Module):
def __post_init__(self):
super().__init__()
module = RegularModule()
print(list(module.parameters()))
module = AttrsModule()
print(list(module.parameters()))
For more on attrs please see hynek answer and his blog post.
attrs has a chapter on hashability that also explains the pitfalls of hashing in Python: https://www.attrs.org/en/stable/hashing.html
You’ll have to decide what behavior is adequate for your concrete problem. For more general information check out https://hynek.me/articles/hashes-and-equality/ — turns out hashing is surprisingly tricky in Python.
I'm working on a web-server type of application and as part of multi-language communication I need to serialize objects in a JSON file. The issue is that I'd like to create a function which can take any custom object and save it at run time rather than limiting the function to what type of objects it can store based on structure.
Apologies if this question is a duplicate, however from what I have searched the other questions and answers do not seem to tackle the dynamic structure aspect of the problem, thus leading me to open this question.
The function is going to be used to communicate between PHP server code and Python scripts, hence the need for such a solution
I have attempted to use the json.dump(data,outfile) function, however the issue is that I need to convert such objects to a legal data structure first
JSON is a rigidly structured format, and Python's json module, by design, won't try to coerce types it doesn't understand.
Check out this SO answer. While __dict__ might work in some cases, it's often not exactly what you want. One option is to write one or more classes that inherit JSONEncoder and provides a method that turns your type or types into basic types that json.dump can understand.
Another option would be to write a parent class, e.g. JSONSerializable and have these data types inherit it the way you'd use an interface in some other languages. Making it an abstract base class would make sense, but I doubt that's important to your situation. Define a method on your base class, e.g. def dictify(self), and either implement it if it makes sense to have a default behavior or just have it it raise NotImplementedError.
Note that I'm not calling the method serialize, because actual serialization will be handled by json.dump.
class JSONSerializable(ABC):
def dictify(self):
raise NotImplementedError("Missing serialization implementation!")
class YourDataType(JSONSerializable):
def __init__(self):
self.something = None
# etc etc
def dictify(self):
return {"something": self.something}
class YourIncompleteDataType(JSONSerializable):
# No dictify(self) implementation
pass
Example usage:
>>> valid = YourDataType()
>>> valid.something = "really something"
>>> valid.dictify()
{'something': 'really something'}
>>>
>>> invalid = YourIncompleteDataType()
>>> invalid.dictify()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in dictify
NotImplementedError: Missing dictify implementation!
Basically, though: You do need to handle this yourself, possibly on a per-type basis, depending on how different your types are. It's just a matter of what method of formatting your types for serialization is the best for your use case.
I am starting to play around with python a little, and as a novice i tried this:
>>> s="";str=""
>>> for x in [ 1,2,3,4,5,6 ] :
... s += str(x)
...
Traceback (most recent call last):
File "<console>", line 3, in <module>
TypeError: 'str' object is not callable
I accidentally declared a variable called str (str is supposed to be a function).
Assuming it would break the semantics too much, even in a dynamically
typed language, is there a namespace i can use to qualify methods
like str and ensure this does not happen or make it difficult?
This is what import <module> instead of from <module> import * is used for. As long as you use str in the only meaning of local variable value in <module>, you can use
module.str elswhere, without mangling namespace.
The only tokens that can't be clashed are keywords. This is intended functionality and there is no way to prevent this: everything is an object in Python
You might want to use some IDE tools, p.ex. Eclipse+PyDev, that checks your code and warn for possible errors.
As per your question you have already defined str=""
so when you will call str method which converts values into string it will not call actual method in place of that it will call str="".
that's why you are getting error because you can not call a str object to convert int to string.
I have a problem similar to the first problem in this question, which as far as I can see went unanswered.
I have a file "config.py" which contains a lot of parameters to be used by a class (this config.py file will change), however I can't get these to propagate into the class via execfile.
In an example piece of code:
class Class():
def __init__(self):
execfile("config.py")
print x
# config.py
x = "foo"
>>> t = Class()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __init__
NameError: global name 'x' is not defined
Any help welcome, or any better methods of retrieving parameters from a file to be used in a class.
Many Thanks.
I don't get what you're trying to do (but i don't like it, and this is just me) but to fix your problem do (test in python2.6):
class Class():
def __init__(self):
execfile('config.py', locals()) # Not recommanded, maybe you want globals().
print x
But from the doc:
Note
The default locals act as described
for function locals() below:
modifications to the default locals
dictionary should not be attempted.
Pass an explicit locals dictionary if
you need to see effects of the code on
locals after function execfile()
returns. execfile() cannot be used
reliably to modify a function’s
locals.
and about :
Any help welcome, or any better
methods of retrieving parameters from
a file to be used in a class.
You can use import.
Even though it might be convenient to keep configuration settings in a Python file I would recommend against it. I think it opens up a whole set of problems that you don't really want to have do deal with. Anything could be placed in your configuration file, including malicious code.
I would use either the json module or the ConfigParser module to hold my configuration.
If you have trouble choosing between those two I would recommend the json module. Json is a simple yet flexible format for structured data.