I'm having a problem using the Perspective Broker feature of Twisted Python. The structure of my code is like this:
class DBService(service.Service):
def databaseOperation(self, input):
# insert input into DB
class PerspectiveRoot(pb.Root):
def __init__(self, service):
self.service = service
def remote_databaseOperation(self, input):
return self.service.databaseOperation(input)
db_service = DBService()
pb_factory = pb.PBServerFactory(PerspectiveRoot(db_service))
I hook up the factory to a TCP server and then multiple clients connect, who are able to insert records into the database via the remote_databaseOperation function.
This works fine until the number of requests gets big, then I end up with duplicate inputs and missing inputs. I assume this is because DBService's 'input' variable persists and gets overwritten during simultaneous requests. Is this correct? And, if so, what's the best way to rewrite my code so it can deal with simultaneous requests?
My first thought is to have DBService maintain a list of DB additions and loop through it, while clients are able to append to the list. Is this the most 'Twisted' way to do it?
Alternatively, is there a separate pb.Root instance for every client? In which case, I could move the database operation into there since its variables won't get overwritten.
Related
I am currently developing a system with multiple communicating entites. It models the communication in a production plant. I am using the freeopcua Python implementation for the communication. To break it down:
There is a script that is running a simulation and hosts the OPC UA server. The other script has the client and connects to the server of the first script. It reads some of the variables, while it writes others. In essence, it controls the simulation.
The structure of the script is:
# - connect to the server
# - map the information model variables onto Python to make them accessible
try:
while True:
# - read some of the variables using node.get_value()
# - perform some logic
# - write some variables node.set_value(new_value)
except:
# - shut down the server
I want to implement PubSub, but I seem to be missing something. The following is a slightly changed version of the example of the freeopcua Github page:
class SubHandler(object):
"""
Client to subscription. It will receive events from server
"""
def datachange_notification(self, node, val, data):
print("Python: New data change event", node, val)
# pass
def event_notification(self, event):
print("Python: New event", event)
if __name__ == "__main__":
# configure client
client = Client("opc.tcp://127.0.0.1:10000")
try:
# connect client
client.connect()
# map the nodes of the informaiton model to Pyhton
# e.g. node1, node2
handler = SubHandler()
sub = client.create_subscription(500, handler)
handle1 = sub.subscribe_data_change(node1)
handle2 = sub.subscribe_data_change(node2)
while True:
# use the data of the read-nodes ?
except:
sub.unsubscribe(handle1)
sub.unsubscribe(handle2)
sub.delete()
client.disconnect()
This subscription accurately prints changes of the nodes. However, I want to use the data in Python to perform some logic and need the values in Python variables. Do I still need to call
node1_py = node1.get_value()
? If I do call get_value(), am I circumventing the subscription and making it obsolete?
Or am I missing some other way to access the values in the loop?
I really appreciate some help.
Thanks in advance
isn't the sense behind the handler, that he handles the subscribed datas?
Just add some logic into your handler. It provides you the node and the value, for me there is no reason, to request the data again.
When you need to compare datas as example, then you will need to request the other datas inside the handler. To prevent further requests to the server you could also store the values in variables inside the handler. Not only the last one, you can also store as example the last 10 values inside the variable.
Depending on your structure, it's maybe better to provide different handlers for different subscription. So the handler won't be a big fat class and just cares about his own nodes.
I feel like I am missing something, but I hope after 3 months this could help you.
I am writing a Python app to work with Mongodb on the backend with pymongo.
I have created a model classes to mirror documents inserted on the database
this is a quick example of models in action:
if __name__ == '__main__':
class MyApp(App):
db_client = MongoClient(replicaset='erpRS').test
def build(self):
model = models.UserModel(name='NewUser', cpf='01234567890')
print(model)
model.save()
model_from_db = models.UserModel.objects.find_one()
print(model_from_db)
def on_stop(self):
self.db_client.close()
My model classes (ex: UserModel) create an instance of Collection as a class property to be used for CRUD operations. So on the snippet above, I create a user and save it to the db. then I fetch a previously saved user from the db. When app stops, on_stop() is called and pymongo client is and its connections are closed.
This code is working as it is intended.
But monitoring the database I can see that just this snipped above, with a single model instance, and thusly, a single Collection instance, and two calls to CRUD operations, open as much as 6 connections to the database.
On further inspection mongodb iself seems to keep a number of connections open. After a single run of this program, my 3 node ReplicaSet, all running on the same machine, has 19 connections open.
Is this a reasonable behavior? Are there supposed to be this many open connections?
MongoDB has connection pool. You shouldn't worry too much about having 19 connections open. The cost of one connection is 1mb file on the node.
What you should think about is refactoring your code and use DI to pass around MongoClient object.
Creating a new MongoClient and a connection every time you instantiate a class is way too expensive.
I want to open only one single connection to my cassandra database in Django.
I did not find anything on this topic unfortunately.
At the moment I got a class which i iniate whenever I need to query:
class CassConnection():
def __init__(self):
self.auth_prov = PlainTextAuthProvider(settings.runtime_settings[k.CASSANDRA_USER],
settings.runtime_settings[k.CASSANDRA_PASSWORD])
self.cluster = Cluster(settings.runtime_settings[k.CASSANDRA_CLUSTER], auth_provider=self.auth_prov)
self.session = self.cluster.connect(keyspace=settings.runtime_settings[k.KEYSPACE_TIREREADINGS])
self.session.row_factory = dict_factory
def get_session(self):
return self.session
I open a new Session in other classes for every query I make by:
self.con = CassConnection()
self.session = self.con.get_session()
Anyone a hint, how to keep the session open and make it accesible via multiple packages?
For a "one connection per Django process" scheme, basically, what you want is
a connection proxy class (which you already have) with a "lazy" connection behaviour (doesn't connect until someone tries to use the connection),
a module-global instance of this class that other packages can import, and
a way to ensure the connection gets properly closed (which cassandra requires for proper operation)
This last point well be the main difficulty as none of the available options (mainly atexit.register() and the __del__(self) method) are 101% reliable. Implementing __del__(self) on your connection proxy might be the most reliable still, just beware of circular depencies (http://eli.thegreenplace.net/2009/06/12/safely-using-destructors-in-python might be a good read here).
Also note that a "one single connection per django process" mean your connections must be totally thread-safe, as you usually will have many threads per Django process (depending on your wsgi container configuration).
Another solution - if thread-safety is an issue - might be to have a single connection per request...
I'm constructing my app such that each user has their own database (for easy isolation, and to minimize the need for sharding). This means that each web request, and all of the background scripts, need to connect to a different database based on which user is making the request, and use that connection for all function calls.
I figure I can make some sort of middleware that would pass the right connection to my web requests by attaching it to the request variable, but I don't know how I should ensure that all functions and model methods called by the request use this connection.
Well how to "ensure that all functions and model methods called by the request use this connection" is easy. You pass the connection into your api as with any well-designed code that isn't relying on global variables for such things. So you have a database session object loaded per-request, and you pass it down. It's very easy for model objects to turtle that session object further without explicitly passing it because each managed object knows what session owns it, and you can query it from there.
db = request.db
user = db.query(User).get(1)
user.add_group('foo')
class User(Base):
def add_group(self, name):
db = sqlalchemy.orm.object_session(self)
group = Group(name=name)
db.add(group)
I'm not recommending you use that exact pattern but it serves as an example of how to grab the session from a managed object, avoiding having to pass the session everywhere explicitly.
On to your original question, how to handle multi-tenancy... In your data model! Designing a system where you are splitting things up at that low of a level is a big maintenance burden and it does not scale well. For example it becomes very difficult to use any type of connection pooling when you have an arbitrary number of independent connections. To get around that people commonly use the SQL SCHEMA feature supported by some databases. That allows you to use the same connection but have access to a different table structure per session. That's better, but again managing all of those schemas independently should raise some red flags, violating DRY with all of that duplication in your data model. Any duplication at that level quickly becomes a burden that you need to be ready for.
I have a Pylons-based web application which connects via Sqlalchemy (v0.5) to a Postgres database. For security, rather than follow the typical pattern of simple web apps (as seen in just about all tutorials), I'm not using a generic Postgres user (e.g. "webapp") but am requiring that users enter their own Postgres userid and password, and am using that to establish the connection. That means we get the full benefit of Postgres security.
Complicating things still further, there are two separate databases to connect to. Although they're currently in the same Postgres cluster, they need to be able to move to separate hosts at a later date.
We're using sqlalchemy's declarative package, though I can't see that this has any bearing on the matter.
Most examples of sqlalchemy show trivial approaches such as setting up the Metadata once, at application startup, with a generic database userid and password, which is used through the web application. This is usually done with Metadata.bind = create_engine(), sometimes even at module-level in the database model files.
My question is, how can we defer establishing the connections until the user has logged in, and then (of course) re-use those connections, or re-establish them using the same credentials, for each subsequent request.
We have this working -- we think -- but I'm not only not certain of the safety of it, I also think it looks incredibly heavy-weight for the situation.
Inside the __call__ method of the BaseController we retrieve the userid and password from the web session, call sqlalchemy create_engine() once for each database, then call a routine which calls Session.bind_mapper() repeatedly, once for each table that may be referenced on each of those connections, even though any given request usually references only one or two tables. It looks something like this:
# in lib/base.py on the BaseController class
def __call__(self, environ, start_response):
# note: web session contains {'username': XXX, 'password': YYY}
url1 = 'postgres://%(username)s:%(password)s#server1/finance' % session
url2 = 'postgres://%(username)s:%(password)s#server2/staff' % session
finance = create_engine(url1)
staff = create_engine(url2)
db_configure(staff, finance) # see below
... etc
# in another file
Session = scoped_session(sessionmaker())
def db_configure(staff, finance):
s = Session()
from db.finance import Employee, Customer, Invoice
for c in [
Employee,
Customer,
Invoice,
]:
s.bind_mapper(c, finance)
from db.staff import Project, Hour
for c in [
Project,
Hour,
]:
s.bind_mapper(c, staff)
s.close() # prevents leaking connections between sessions?
So the create_engine() calls occur on every request... I can see that being needed, and the Connection Pool probably caches them and does things sensibly.
But calling Session.bind_mapper() once for each table, on every request? Seems like there has to be a better way.
Obviously, since a desire for strong security underlies all this, we don't want any chance that a connection established for a high-security user will inadvertently be used in a later request by a low-security user.
Binding global objects (mappers, metadata) to user-specific connection is not good way. As well as using scoped session. I suggest to create new session for each request and configure it to use user-specific connections. The following sample assumes that you use separate metadata objects for each database:
binds = {}
finance_engine = create_engine(url1)
binds.update(dict.fromkeys(finance_metadata.sorted_tables, finance_engine))
# The following line is required when mappings to joint tables are used (e.g.
# in joint table inheritance) due to bug (or misfeature) in SQLAlchemy 0.5.4.
# This issue might be fixed in newer versions.
binds.update(dict.fromkeys([Employee, Customer, Invoice], finance_engine))
staff_engine = create_engine(url2)
binds.update(dict.fromkeys(staff_metadata.sorted_tables, staff_engine))
# See comment above.
binds.update(dict.fromkeys([Project, Hour], staff_engine))
session = sessionmaker(binds=binds)()
I would look at the connection pooling and see if you can't find a way to have one pool per user.
You can dispose() the pool when the user's session has expired