I am currently re-writing some code that uses python's select.select() method, but these will only return socket objects in which I have to go and manually match that socket to the socket in a class that was put in there under __init__. the pseudo-code for that would basically be [classobject for classobject in classList if SocketFromSelection == class.socketobject][0] (which I'm pretty much using).
I found in python documentation that in select.select(), "You may also define a wrapper class yourself, as long as it has an appropriate fileno() method (that really returns a file descriptor, not just a random integer)."
My question is, how would I attach a fileno() method in a class so that I can pass a sequence of these classes into select.select() so that it returns the classes and not just the sockets? Also, would this run on windows? If not, is there a better way to match the socket to the socket in a class in a list of classes?
From the code you included in your question, it sounds like you have a class that contains a socket inside of it (as the socketobject attribute). In this case, you can make your wrapper object selectable by proxying the socket's fileno method on your wrapper:
def SocketWrapper(object):
def __init__(self, socket):
self.socketobj = socket # use whatever you are already doing
def fileno(self):
return self.socketobj.fileno()
Now you can pass instances of SocketWrapper directly to select, rather than passing the sockets and then later having to sort out which socket corresponds to which instance.
Related
Is it possible to nest an arbitrary number Singleton classes within a Singleton class in Python?
There's no problem in changing my approach to solving this issue if a simpler alternative exists. I am just using the "tools in my toolset", if you will. I'm simulating some larger processes so bear with me if it seems a bit far-fetched.
An arbitrary number of gRPC servers can be started up and each server will be listening on a different port. So for a client to communicate with these servers, separate channels and thus separate stubs will need to be created for a client to communicate to a given server.
You could just create a new channel and a new stub every time a client needs to make a request to a server, but I am trying to incorporate some best practices and reuse the channels and stubs. My idea is to create a Singleton class that is comprised of Singleton subclasses that house a channel and stub pair as instance attributes. I would have to build the enclosing class in a way that allows me to add additional subclasses whenever needed, but I have never seen this type of thing done before.
The advantage to this approach is that any module that instantiates the main Singleton class will have access to the existing state of the channel and stub pairs, without having to recreate anything.
I should note that I already initialize channels and stubs from within Singleton classes and can reuse them with no problem. But the main goal here is to create a data structure that allows me to reuse/share a variable amount of gRPC channel and stub pairs.
The following is code for reusing the gRPC Channel object; the stubs are built in a very similar way, only difference is they accept the channel as an arguement.
class gRPCChannel(object):
_instance, _channel = None, None
port: int = 50051
def __new__(cls):
"""Subsequent calls to instance() return the singleton without repeating the initialization step"""
if cls._instance is None:
cls._instance = super(gRPCChannel, cls).__new__(cls)
# The following is any initialization that needs to happen for the channel object
cls._channel = grpc.insecure_channel(f'localhost:{cls.port}', options=(('grpc.enable_http_proxy', 0),))
return cls._channel
def __exit__(self, cls):
cls._channel.close()
I think this is simpler than the "singleton pattern":
# grpc.py
import functools as ft
class GrpcChannel:
pass # normal class, no funny __new__ overload business
#ft.lru_cache
def channel():
return GrpcChannel()
Usage:
import grpc
channel = grpc.channel()
assert grpc.channel() is channel
If you really want it all namespaced under the class (IMO no reason to, takes more typing & syntax for no extra benefit), then:
class GrpcChannel:
#classmethod
#ft.lru_cache
def instance(cls):
return cls()
# usage
assert grpc.GrpcChannel.instance() is grpc.GrpcChannel.instance()
I am struggling to understand when it makes sense to use an instance method versus a static method. Also, I don't know if my functions are static since there is not a #staticmethod decorator. Would I be able to access the class functions when I make a call to one of the methods?
I am working on a webscraper that sends information to a database. It’s setup to run once a week. The structure of my code looks like this
import libraries...
class Get:
def build_url(url_paramater1, url_parameter2, request_date):
return url_with_parameters
def web_data(request_date, url_parameter1, url_parameter2): #no use of self
# using parameters pull the variables to look up in the database
for a in db_info:
url = build_url(a, url_parameter2, request_date)
x = requests.Session().get(url, proxies).json()
#save data to the database
return None
#same type of function for pulling the web data from the database and parsing it
if __name__ == ‘__main__’:
Get.web_data(request_date, url_parameter1, url_parameter2)
Parse.web_data(get_date, parameter) #to illustrate the second part of the scrapper
That is the basic structure. The code is functional but I don’t know if I am using the methods (functions?) correctly and potentially missing out on ways to use my code in the future. I may even be writing bad code that will cause errors down the line that are impossibly hard to debug only because I didn’t follow best practices.
After reading about when class and instance methods are used. I cannot see why I would use them. If I want the url built or the data pulled from the website I call the build_url or get_web_data function. I don’t need an instance of the function to keep track of anything separate. I cannot imagine when I would need to keep something separate either which I think is part of the problem.
The reason I think my question is different than the previous questions is: the conceptual examples to explain the differences don't seem to help me when I am sitting down and writing code. I have not run into real world problems that are solved with the different methods that show when I should even use an instance method, yet instance methods seem to be mandatory when looking at conceptual examples of code.
Thank you!
Classes can be used to represent objects, and also to group functions under a common namespace.
When a class represents an object, like a cat, anything that this object 'can do', logically, should be an instance method, such as meowing.
But when you have a group of static functions that are all related to each other or are usually used together to achieve a common goal, like build_url and web_data, you can make your code clearer and more organized by putting them under a static class, which provides a common namespace, like you did.
Therefore in my opinion the structure you chose is legitimate. It is worth considering though, that you'd find static classes more in more definitively OOP languages, like Java, while in python it is more common to use modules for namespace separation.
This code doesn't need to be a class at all. It should just be a pair of functions. You can't see why you would need an instance method because you don't have a reason to instantiate the object in the first place.
The functions you have wrote in your code are instance methods but they were written incorrectly.
An instance method must have self as first parameter
i.e def build_url(self, url_paramater1, url_parameter2, request_date):
Then you call it like that
get_inst = Get()
get_inst.build_url(url_paramater1, url_parameter2, request_date)
This self parameter is provided by python and it allow you to access all properties and functions - static or not - of your Get class.
If you don't need to access other functions or properties in your class then you add #staticmethod decorator and remove self parameter
#staticmethod
def build_url(url_paramater1, url_parameter2, request_date):
And then you can call it directly
Get.build_url(url_paramater1, url_parameter2, request_date)
or call from from class instance
get_inst = Get()
get_inst.build_url(url_paramater1, url_parameter2, request_date)
But what is the problem with your current code you might ask?
Try calling it from an instance like this and u will see the problem
get_inst = Get()
get_inst.build_url(url_paramater1, url_parameter2, request_date)
Example where creating an instance is useful:
Let's say you want to make a chat client.
You could write code like this
class Chat:
def send(server_url, message):
connection = connect(server_url)
connection.write(message)
connection.close()
def read(server_url):
connection = connect(server_url)
message = connection.read()
connection.close()
return message
But a much cleaner and better way to do it:
class Chat:
def __init__(server_url):
# Initialize connection only once when instance is created
self.connection = connect(server_url)
def __del__()
# Close connection only once when instance is deleted
self.connection.close()
def send(self, message):
self.connection.write(message)
def read(self):
return self.connection.read()
To use that last class you do
# Create new instance and pass server_url as argument
chat = Chat("http://example.com/chat")
chat.send("Hello")
chat.read()
# deleting chat causes __del__ function to be called and connection be closed
delete chat
From given example, there is no need to have Get class after all, since you are using it just like a additional namespace. You do not have any 'state' that you want to preserve, in either class or class instance.
What seems like a good thing is to have separate module and define these functions in it. This way, when importing this module, you get to have this namespace that you want.
I need some help in terms of 'pythonic' way of handling a specific scenario.
I'm writing an Ssh class (wraps paramiko) that provides the capability to connect to and executes commands on a device under test (DUT) over ssh.
class Ssh:
def connect(some_params):
# establishes connection
def execute_command(command):
# executes command and returns response
def disconnect(some_params):
# closes connection
Next, I'd like to create a Dut class that represents my device under test. It has other things, besides capability to execute commands on the device over ssh. It exposes a wrapper for command execution that internally invokes the Ssh's execute_command. The Ssh may change to something else in future - hence the wrapper.
def Dut:
def __init__(some params):
self.ssh = Ssh(blah blah)
def execute_command(command)
return self.ssh.execute_command(command)
Next, the device supports a custom command line interface for device under test. So, a class that accepts a DUT object as an input and exposes a method to execute the customised command.
def CustomCli:
def __init__(dut_object):
self.dut = dut_object
def _customize(command):
# return customised command
def execute_custom_command(command):
return self.dut.execute_command(_customize(command))
Each of the classes can be used independently (CustomCli would need a Dut object though).
Now, to simplify things for user, I'd like to expose a wrapper for CustomCli in the Dut class. This'll allow the creator of the Dut class to exeute a simple or custom command.
So, I modify the Dut class as below:
def Dut:
def __init__(some params):
self.ssh = Ssh(blah blah)
self.custom_cli = Custom_cli(self) ;# how to avoid this circular reference in a pythonic way?
def execute_command(command)
return self.ssh.execute_command(command)
def execute_custom_command(command)
return self.custom_cli.execute_custom_command(command)
This will work, I suppose. But, in the process I've created a circular reference - Dut is pointing to CustomCli and CustomCli has a reference to it's creator Dut instance. This doesn't seem to be the correct design.
What's the best/pythonic way to deal with this?
Any help would be appreciated!
Regards
Sharad
In general, circular references aren't a bad thing. Many programs will have them, and people just don't notice because there's another instance in-between like A->B->C->A. Python's garbage collector will properly take care of such constructs.
You can make circular references a bit easier on your conscience by using weak references. See the weakref module. This won't work in your case, however.
If you want to get rid of the circular reference, there are two way:
Have CustomCLI inherit from Dut, so you end up with just one instance. You might want to read up on Mixins.
class CLIMerger(Dut):
def execute_custom_command(command):
return self.execute_command(_customize(command))
# use self^ instead of self.dut
class CLIMixin(object):
# inherit from object, won't work on its own
def execute_custom_command(command):
return self.execute_command(_customize(command))
# use self^ instead of self.dut
class CLIDut(Dut, CLIMixin):
# now the mixin "works", but still could enhance other Duts the same way
pass
The Mixin is advantageous if you need several cases of merging a CLI and Dut.
Have an explicit interface class that combines CustomCli and Dut.
class DutCLI(object):
def __init__(self, *bla, **blah):
self.dut = Dut(*bla, **blah)
self.cli = CustomCLI(self.dut)
This requires you to write boilerplate or magic to forward every call from DutCLI to either dut or cli.
As it is common knowledge, the python __del__ method should not be used to clean up important things, as it is not guaranteed this method gets called. The alternative is the use of a context manager, as described in several threads.
But I do not quite understand how to rewrite a class to use a context manager. To elaborate, I have a simple (non-working) example in which a wrapper class opens and closes a device, and which shall close the device in any case the instance of the class gets out of its scope (exception etc).
The first file mydevice.py is a standard wrapper class to open and close a device:
class MyWrapper(object):
def __init__(self, device):
self.device = device
def open(self):
self.device.open()
def close(self):
self.device.close()
def __del__(self):
self.close()
this class is used by another class myclass.py:
import mydevice
class MyClass(object):
def __init__(self, device):
# calls open in mydevice
self.mydevice = mydevice.MyWrapper(device)
self.mydevice.open()
def processing(self, value):
if not value:
self.mydevice.close()
else:
something_else()
My question: When I implement the context manager in mydevice.py with __enter__ and __exit__ methods, how can this class be handled in myclass.py? I need to do something like
def __init__(self, device):
with mydevice.MyWrapper(device):
???
but how to handle it then? Maybe I overlooked something important? Or can I use a context manager only within a function and not as a variable inside a class scope?
I suggest using the contextlib.contextmanager class instead of writing a class that implements __enter__ and __exit__. Here's how it would work:
class MyWrapper(object):
def __init__(self, device):
self.device = device
def open(self):
self.device.open()
def close(self):
self.device.close()
# I assume your device has a blink command
def blink(self):
# do something useful with self.device
self.device.send_command(CMD_BLINK, 100)
# there is no __del__ method, as long as you conscientiously use the wrapper
import contextlib
#contextlib.contextmanager
def open_device(device):
wrapper_object = MyWrapper(device)
wrapper_object.open()
try:
yield wrapper_object
finally:
wrapper_object.close()
return
with open_device(device) as wrapper_object:
# do something useful with wrapper_object
wrapper_object.blink()
The line that starts with an at sign is called a decorator. It modifies the function declaration on the next line.
When the with statement is encountered, the open_device() function will execute up to the yield statement. The value in the yield statement is returned in the variable that's the target of the optional as clause, in this case, wrapper_object. You can use that value like a normal Python object thereafter. When control exits from the block by any path – including throwing exceptions – the remaining body of the open_device function will execute.
I'm not sure if (a) your wrapper class is adding functionality to a lower-level API, or (b) if it's only something you're including so you can have a context manager. If (b), then you can probably dispense with it entirely, since contextlib takes care of that for you. Here's what your code might look like then:
import contextlib
#contextlib.contextmanager
def open_device(device):
device.open()
try:
yield device
finally:
device.close()
return
with open_device(device) as device:
# do something useful with device
device.send_command(CMD_BLINK, 100)
99% of context manager uses can be done with contextlib.contextmanager. It is an extremely useful API class (and the way it's implemented is also a creative use of lower-level Python plumbing, if you care about such things).
The issue is not that you're using it in a class, it's that you want to leave the device in an "open-ended" way: you open it and then just leave it open. A context manager provides a way to open some resource and use it in a relatively short, contained way, making sure it is closed at the end. Your existing code is already unsafe, because if some crash occurs, you can't guarantee that your __del__ will be called, so the device may be left open.
Without knowing exactly what the device is and how it works, it's hard to say more, but the basic idea is that, if possible, it's better to only open the device right when you need to use it, and then close it immediately afterwards. So your processing is what might need to change, to something more like:
def processing(self, value):
with self.device:
if value:
something_else()
If self.device is an appropriately-written context manager, it should open the device in __enter__ and close it in __exit__. This ensures that the device will be closed at the end of the with block.
Of course, for some sorts of resources, it's not possible to do this (e.g., because opening and closing the device loses important state, or is a slow operation). If that is your case, you are stuck with using __del__ and living with its pitfalls. The basic problem is that there is no foolproof way to leave the device "open-ended" but still guarantee it will be closed even in the event of some unusual program failure.
I'm not quite sure what you're asking. A context manager instance can be a class member - you can re-use it in as many with clauses as you like and the __enter__() and __exit__() methods will be called each time.
So, once you'd added those methods to MyWrapper, you can construct it in MyClass just as you are above. And then you'd do something like:
def my_method(self):
with self.mydevice:
# Do stuff here
That will call the __enter__() and __exit__() methods on the instance you created in the constructor.
However, the with clause can only span a function - if you use the with clause in the constructor then it will call __exit__() before exiting the constructor. If you want to do that, the only way is to use __del__(), which has its own problems as you've already mentioned. You could open and close the device just when you need it using with but I don't know if this fulfills your requirements.
According to twisted documentation
a new Protocol instance is created every time a connection is made however i am observing some sort of data sharing between the two.
Briefly I have defined a class that each protocol will use for state
class JSONCollector():
char_buffer = StringIO.StringIO()
...
def process_data(self, data):
...
self.char_buffer.write(char)
The protocol instantiates it
class JSONProtocol(protocol.Protocol):
def __init__(self):
self.json_collector = JSONCollector()
def dataReceived(self, data):
self.json_collector.process_data(data)
self.transport.write(str(self))
However each connection seems to get the same instance of JSONCollector, when I added the following statements to dataReceived
self.transport.write(str(self.json_collector.char_buffer))
self.transport.write(str(self))
I get the following:
connection 1: StringIO.StringIO instance at 0x968ae2c><main.JSONProtocol instance at 0x969036c>
connection 2: StringIO.StringIO instance at 0x968ae2c><main.JSONProtocol instance at 0x969068c>
Also everytime I type in text, text that was typed in from other connection(s) gets displayed.
So it seems that for some strange reason StringIO() instances are shared, am I missing something. I suppose I can use a factory to separate buffers by addr and make sure each Protocol only uses its own buffer, but without having the need for shared storage I would rather not jump through hoops
Thank you.
This:
class JSONCollector():
char_buffer = StringIO.StringIO()
Seems like an error, and the source of your troubles. Instead, try this:
class JSONCollector():
def __init__(self):
self.char_buffer = StringIO.StringIO()
Otherwise, you are making one char_buffer for the entire class type, as opposed to one per instance.