We Keep Coding

How can I do parallel tests in Django with Elasticsearch-dsl?

Has anyone gotten parallel tests to work in Django with Elasticsearch? If so, can you share what configuration changes were required to make it happen?
I've tried just about everything I can think of to make it work including the solution outlined here. Taking inspiration from how Django itself does the parallel DB's, I currently have created a custom new ParallelTestSuite that overrides the init_worker to iterate through each index/doctype and change the index names roughly as follows:
_worker_id = 0
def _elastic_search_init_worker(counter):
global _worker_id
with counter.get_lock():
counter.value += 1
_worker_id = counter.value
for alias in connections:
connection = connections[alias]
settings_dict = connection.creation.get_test_db_clone_settings(_worker_id)
# connection.settings_dict must be updated in place for changes to be
# reflected in django.db.connections. If the following line assigned
# connection.settings_dict = settings_dict, new threads would connect
# to the default database instead of the appropriate clone.
connection.settings_dict.update(settings_dict)
connection.close()
### Everything above this is from the Django version of this function ###
# Update index names in doctypes
for doc in registry.get_documents():
doc._doc_type.index += f"_{_worker_id}"
# Update index names for indexes and create new indexes
for index in registry.get_indices():
index._name += f"_{_worker_id}"
index.delete(ignore=[404])
index.create()
print(f"Started thread # {_worker_id}")
This seems to generally work, however, there's some weirdness that happens seemingly randomly (i.e. running the test suite again doesn't reliably reproduce the issue and/or the error messages change). The following are the various errors I've gotten and it seems to randomly fail on one of them each test run:
Raise a 404 when trying to create the index in the function above (I've confirmed that it's the 404 coming back from the PUT request, however in the Elasticsearch server logs it says that it's created the index without issue)
a 500 when trying to create the index, although this one hasn't happened in a while so I think this was fixed by something else
query responses will sometimes not have an items dictionary value inside the _process_bulk_chunk function from the elasticsearch library
I'm thinking that there's something weird going on at the connection layer (like somehow the connections between Django test runner processes are getting the responses mixed up?) but I'm at a loss as to how that would be even possible since Django uses multiprocessing to parallelize the tests and thus they are each running in their own process. Is it somehow possible that the spun-off processes are still trying to use the connection pool of the original process or something? I'm really at a loss of other things to try from here and would greatly appreciate some hints or even just confirmation that this is in fact possible to do.

I'm thinking that there's something weird going on at the connection layer (like somehow the connections between Django test runner processes are getting the responses mixed up?) but I'm at a loss as to how that would be even possible since Django uses multiprocessing to parallelize the tests and thus they are each running in their own process. Is it somehow possible that the spun-off processes are still trying to use the connection pool of the original process or something?
This is exactly what is happening. From the Elasticsearch DSL docs:
Since we use persistent connections throughout the client it means that the client doesn’t tolerate fork very well. If your application calls for multiple processes make sure you create a fresh client after call to fork. Note that Python’s multiprocessing module uses fork to create new processes on POSIX systems.
What I observed happening is that the responses get very weirdly interleaved with a seemingly random client that may have started the request. So a request to index a document might end up with a response to create an index which have very different attributes on them.
The fix is to ensure that each test worker has its own Elasticsearch client. This can be done by creating worker-specific connection aliases and then overwriting the current connection aliases (with the private attribute _using) with the worker-specific one. Below is a modified version of the code you posted with the change
_worker_id = 0
def _elastic_search_init_worker(counter):
global _worker_id
with counter.get_lock():
counter.value += 1
_worker_id = counter.value
for alias in connections:
connection = connections[alias]
settings_dict = connection.creation.get_test_db_clone_settings(_worker_id)
# connection.settings_dict must be updated in place for changes to be
# reflected in django.db.connections. If the following line assigned
# connection.settings_dict = settings_dict, new threads would connect
# to the default database instead of the appropriate clone.
connection.settings_dict.update(settings_dict)
connection.close()
### Everything above this is from the Django version of this function ###
from elasticsearch_dsl.connections import connections
# each worker needs its own connection to elasticsearch, the ElasticsearchClient uses
# global connection objects that do not play nice otherwise
worker_connection_postfix = f"_worker_{_worker_id}"
for alias in connections:
connections.configure(**{alias + worker_connection_postfix: settings.ELASTICSEARCH_DSL["default"]})
# Update index names in doctypes
for doc in registry.get_documents():
doc._doc_type.index += f"_{_worker_id}"
# Use the worker-specific connection
doc._doc_type._using = doc.doc_type._using + worker_connection_postfix
# Update index names for indexes and create new indexes
for index in registry.get_indices():
index._name += f"_{_worker_id}"
index._using = doc.doc_type._using + worker_connection_postfix
index.delete(ignore=[404])
index.create()
print(f"Started thread # {_worker_id}")

How to make sure data over UNX Sockets gets sent in order using Twisted Python

With my current setup, I'm running a server with Django and I'm trying to automate backing up to the cloud whenever a POST/PUT action is made. To circumvent the delay (Ping to server hovers around 100ms and an action can reach upwards of 10 items posted at once), I decided to create a separate entity with a requests client and simply have this handle all backing up functions.
To do this, I have that entity listen via UNX using twisted and I send it a string through it whenever I hit an endpoint. The problem however is that if too many end points get called at once or get called in rapid succession, the data sent over the socket no longer comes in order. Is there any way to prevent this? Code below:
UNX Server:
class BaseUNXServerProtocol(LineOnlyReceiver):
rest_client = RestClient()
def connectionMade(self):
print("UNIX Client connected!")
def lineReceived(self, line):
print("Line Received!")
def dataReceived(self, data):
string = data.decode("utf-8")
jstring = json.loads(data)
if jstring['command'] == "upload_object":
self.rest_client.upload(jstring['model_name'], jstring['model_id'])
Unix Client:
class BaseUnixClient(object):
path = BRANCH_UNX_PATH
connected = False
def __init__(self):
self.init_vars()
self.connect()
def connect(self):
if os.path.exists(self.path):
self.client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
self.client.connect(self.path)
self.connected = True
else:
print("Could not connect to path: {}".format(self.path))
def call_to_upload(self, model_class, model_id, upload_type):
self.send_string(_messages.branch_upload_message(model_class, model_id, upload_type))
Endpoint perform_create: (Essentially a hook that gets called whenever a new object is POSTed)
def perform_create(self, serializer):
instance = serializer.save()
# Call for upload/notify
UnixClient().call_to_upload(model_class=type(instance).__name__, model_id=instance.id, upload_type="create")

SOCK_STREAM connections are always ordered. Data on one connection comes out in the same order it went in (or the connection breaks).
THe only obvious problem with the code you shared is that you shouldn't override dataReceived on a LineOnlyReceiver subclass. All your logic belongs in lineReceived.
That wouldn't cause out-of-order data problems but it could lead to framing issues (like partial JSON messages being processed, or multiple messages being combined) which would probably cause json.loads to raise an exception.
So, to answer your question: data is delivered in order. If you are seeing out-of-order operation, it's because the data is being sent in a different order than you expect or because there is a divergence between the order of data delivery and the order of observable side-effects. I don't see any way to provide a further diagnosis without seeing more of your code.
Seeing your sending code, the problem is that you're using a new connection for every perform_create operation. There is no guarantee about delivery order across different connections. Even if your program does:
establish connection a
send data on connection a
establish connection b
send data on connection b
close connection a
close connection b
The receiver may decide to process data on connection b before data on connection a. This is because the underlying event notification system (select, epoll_wait, etc) doesn't (ever, as far as I know) preserve information about the ordering of the events it is reporting on. Instead, results come out in a pseudo-random order or a boring deterministic order (such as ascending by file descriptor number).
To fix your ordering problem, make one UnixClient and use it for all of your perform_create calls.

Module to control SSL handshake in python?

Is there a module to control the SSL handshake in python, Both client side and server? The python default SSL module is great but does the handshake automatic. I was wondering if there is a module that will allow me to do it manual similar to this:
import SSLManuel
import socket
s = socket.socket()
s.connect(("server.com",9999))
ClientHello = SSLManuel.generateClientHelloMessage(ssl=TLSv1_2, cipher="ECDHE-RSA-AES128-GCM-SHA256", server="www.server.com")
s.send(ClientHello)
ServerHello = s.recv()#this would receive the server hello
#This would verify the certificate of the server
if SSLManuel.check_cert(ServerHello) == true:
Pre-Master-Key = SSLManuel.generatePreMasterKey()
ClientKeyExchange = SSLManuel.generateClientKeyExchange(Pre-Master-Key)
ChangeCiherSpec = SSLManuel.generateChangeCipherSpec()
ClientFinished = SSLManuel.generateClientFinished()
Sessionkey = SSLManuel.generateMasterKey(Pre-Master-Key)
s.send(ClientKeyExchange)
s.send(ChangeCiherSpec)
s.send(ClientFinished)
ServerFinished = s.recv()
#This will check if the server is ready to communicate securely.
if SSLManuel.checkServerFinshed(ServerFinished) == true:
#I can now use the SessionKey to encrypt data to and from the server
s.send(SSLManuel.encrypt(SessionKey, "GET / HTTP/1.0\n\n"))
response = s.recv()
print(SSLManuel.decrypt(SessionKey, response))
I hope the naming conventions used in this example can help you understand what I'm trying to accomplish. Most of my knowledge of SSL comes from This Article. I have tried to write my own but have failed and I can't seem to find any module that will allow me to do this.

There are several pure-python implementations of SSL/TLS. Any of them will allow you to do this:
https://github.com/pyca/tls
https://github.com/DinoTools/python-flextls
https://github.com/tomato42/tlslite-ng (maintained fork of https://github.com/trevp/tlslite)
As far as I understand your question, your aim is to improve your understanding of the protocol. I would personally use the latter for this purpose, because it has an extensive inline documentation. tlslite.tlsconnection.handshakeClientAnonymous is a good starting point for your investigation, the function eventually calls _handshakeClientAsyncHelper to perform the actual handshake.

Repeated transient-socket connections - memory leak risk?

I'm writing a script that opens a text file and loops through each line (pausing a few seconds between each one). For each line, it opens a transient client socket connection and sends the text to a host server. A host response may or may not come; it doesn't matter either way.
I've already bumped into the Python socket limitation where you can't reconnect with an existing socket object (because doing so triggers the exception EBADF, 'Bad file descriptor'). So I'm creating a new socket instance for each transient connection. The trick then of course becomes how to avoid a memory leak.
The way I've approached this is to push the whole portion of creating, using, and closing the socket to a function -- relying on Python's garbage collection to remove each instance after I'm done with it:
import socket,select,time
def transientConnect(host,port,sendData):
response = ''
sendSocket = socket.socket()
sendSocket.connect((serverHost,serverPort))
sendSocket.send(line)
gotData = select.select([sendSocket],[],[],2)
if (gotData[0]):response = sendSocket.recv(65535)
sendSocket.close()
return response
scriptLines = open('testScript.txt','r').readlines()
serverHost = '127.0.0.1'
serverPort = 15004
for line in scriptLines:
response = transientConnect(serverHost,serverPort,line)
print(response)
time.sleep(3.0)
My questions: (1) Will this approach work to avert any memory leaks? (2) Is there a more direct way to make sure each instance is eliminated after I'm finished with it?

First off, it is normal to only use a socket for a single exchange. See the socket HOWTO.
One of the nice things about python is that in general you don't have to worry about garbage collection. And you shouldn't unless you have real memory use problems.
From this webpage, keep in mind that:
"Python won’t clean up an object when it goes out of scope. It will clean it up when the last reference to it has gone out of scope."
So if the socket created inside the function isn't referenced elsewhere, it should go out of scope and be deallocated (but not gc-ed). What follows is probably specific to cpython. Read the documentation of gc.set_threshold() for an idea how garbage collection works in cpython. Especially:
"When the number of allocations minus the number of deallocations exceeds threshold0, collection starts."
The standard values for the thresholds (in cpython) are:
In [2]: gc.get_threshold()
Out[2]: (700, 10, 10)
So there would heva to be a fair number of allocations before you get a gc run. You can force garbage collection by running gc.collect().

How to connect to Cassandra inside a Pylons app?

I created a new Pylons project, and would like to use Cassandra as my database server. I plan on using Pycassa to be able to use cassandra 0.7beta.
Unfortunately, I don't know where to instantiate the connection to make it available in my application.
The goal would be to :
Create a pool when the application is launched
Get a connection from the pool for each request, and make it available to my controllers and libraries (in the context of the request). The best would be to get a connexion from the pool "lazily", i.e. only if needed
If a connexion has been used, release it when the request has been processed
Additionally, is there something important I should know about it ? When I see some comments like "Be careful when using a QueuePool with use_threadlocal=True, especially with retries enabled. Synchronization may be required to prevent the connection from changing while another thread is using it.", what does it mean exactly ?
Thanks.
--
Pierre

Well. I worked a little more. In fact, using a connection manager was probably not a good idea as this should be the template context. Additionally, opening a connection for each thread is not really a big deal. Opening a connection per request would be.
I ended up with just pycassa.connect_thread_local() in app_globals, and there I go.

Okay.
I worked a little, I learned a lot, and I found a possible answer.
Creating the pool
The best place to create the pool seems to be in the app_globals.py file, which is basically a container for objects which will be accessible "throughout the life of the application". Exactly what I want for a pool, in fact.
I just added at the end of the file my init code, which takes settings from the pylons configuration file :
"""Creating an instance of the Pycassa Pool"""
kwargs = {}
# Parsing servers
if 'cassandra.servers' in config['app_conf']:
servers = config['app_conf']['cassandra.servers'].split(',')
if len(servers):
kwargs['server_list'] = servers
# Parsing timeout
if 'cassandra.timeout' in config['app_conf']:
try:
kwargs['timeout'] = float(config['app_conf']['cassandra.timeout'])
except:
pass
# Finally creating the pool
self.cass_pool = pycassa.QueuePool(keyspace='Keyspace1', **kwargs)
I could have done better, like moving that in a function, or supporting more parameters (pool size, ...). Which I'll do.
Getting a connection at each request
Well. There seems to be the simple way : in the file base.py, adding something like c.conn = g.cass_pool.get() before calling WSGIController, something like c.conn.return_to_pool() after. This is simple, and works. But this gets a connection from the pool even when it's not required by the controller. I have to dig a little deeper.
Creating a connection manager
I had the simple idea to create a class which would be instantiated at each request in the base.py file, and which would automatically grab a connection from the pool when requested (and release it after). This is a really simple class :
class LocalManager:
'''Requests a connection from a Pycassa Pool when needed, and releases it at the end of the object's life'''
def __init__(self, pool):
'''Class constructor'''
assert isinstance(pool, Pool)
self._pool = pool
self._conn = None
def get(self):
'''Grabs a connection from the pool if not already done, and returns it'''
if self._conn is None:
self._conn = self._pool.get()
return self._conn
def __getattr__(self, key):
'''It's cooler to write "c.conn" than "c.get()" in the code, isn't it?'''
if key == 'conn':
return self.get()
else:
return self.__dict__[key]
def __del__(self):
'''Releases the connection, if needed'''
if not self._conn is None:
self._conn.return_to_pool()
Just added c.cass = CassandraLocalManager(g.cass_pool) before calling WSGIController in base.py, del(c.cass) after, and I'm all done.
And it works :
conn = c.cass.conn
cf = pycassa.ColumnFamily(conn, 'TestCF')
print cf.get('foo')
\o/
I don't know if this is the best way to do this. If not, please let me know =)
Plus, I still did not understand the "synchronization" part in Pycassa source code. If it is needed in my case, and what should I do to avoid problems.
Thanks.