We’re using grpc and protobuf objects as our communication method for a reinforcement learning project, Using python as a client and an external program as a server.
We’re currently in the process of trying to get multiple environments running in parallel to speed up training times, hence the use of ray.
However I am running into issues getting our existing communication methods to serialize properly for ray.remote etc.
This is the main error message that comes up:
“TypeError: cannot pickle ‘google.protobuf.pyext._message.MessageDescriptor’ object”
Any potential solutions or ideas? My python knowledge is lacking a little some i’m not exactly sure what the best way around this is.
Related
I'm using Pyomo library to solve an optimization problem with Gurobi optimizer. I have an academic license for Gurobi, but afaik this should not pose any kind of limitation. I'd like to optimize the model with different parameters in parallel. In other words, I have many instances of the same model but different parameters that could be solved independently.
I've followed the instruction that I found on the documentation page of Pyomo regarding pyro and dispatching (here the reference). Basically, I started the Pyomo name server with pyomo_ns, then the dispatch server with dispatch_srvr, and finally I have launched four instances of pyro_mip_server.
When I launch my python script it works. No errors occur. The problem is that it takes the same amount of time as the serial execution. I've also monitored the activity of all eight cores that my CPU has and only one is constantly at 100% of the load. It's like no concurrent execution is happening at all.
I'm using SolverManager provided by Pyomo library to submit from the Python script different instances of my model. You can find here the guide of how to use it.
I'm on a Linux machine, the latest LTS Ubuntu distro.
Have somebody had such an experience or does someone know what's the problem here? If you need any additional info just let me know :). Thank you!
I've got a following setup:
VB.NET Web-Service is running and it needs to regularly call Python script with machine learning model to predict some stuff. To do this my Web-Service generates a file with input for Python and runs Python script as a subprocess. The script makes predictions and returns them, as standard output, back to Web-Service.
The problem is, that the script requires a few seconds to import all the machine learning libraries and load saved model from drive. It's much more than doing actual prediction. During this time Web-Service is blocked by running subprocess. I have to reduce this time drastically.
What I need is a solution to either:
1. Improve libraries and model loading time.
2. Communicate Python script with VB.NET Web-Service and run Python all the time with imports and ML model already loaded.
Not sure I understood the question but here are some things I can think of.
If this is a network thing you should have python compress the code before sending it over the web.
Or if you're able to, use multithreading while reading the file from the web.
Maybe you should upload some more code so we can help you better.
I've found what I needed.
I've used web.py to convert the Python script into a Web-Service and now both VB.NET and Python Web-Services can communicate. Python is running all the time, so there is no delay for loading libraries and data each time a calculation have to be done.
I am planning on writing some software for a web server that uses machine learning to process large amounts of data. This will be real estate data from a MySQL server. I will be using the CUDA framework from Nvidia with python/caffe or the c++ library. I will be using a Tesla P100. Although python is more widely used for machine learning I presume it is hard to write a server app in python without sacrificing performance. Is this true? Is c++ well supported for machine learning? Will anything be sacrificed by writing a professional server app in python (ex: connecting to MySQL database)?
Python is a language that performs worse than c++ in terms of runtime for several reasons:
First and foremost, Python is a scripting language that runs with an interpreter as opposed to c++ which compiled into machine code before running.
Secondly: python runs in the background a garbage collector system while in c++ the memory management is done manually by the programmer.
In your case, I recommend that you work with Python for several reasons:
Writing in Python in CUDA allows you to compile the code even though it is Python (CUDA provides JIT - Just In Time compiler, as well as a compiler and other effective tools), Which greatly improves performance
Python provides many, rich varied libraries that will help you a lot in the project, especially in the field of machine learning.
The development time and code length will be significantly shorter in Python.
From my experience with working at CUDA in the Python language I recommend you use numba and numbapro libraries, they are comfortable to work with and provide support for many libraries like numpy.
Best of luck.
Both C++ and Python are perfectly reasonable languages for implementing web servers. Python is actually quite common, and there are many frameworks for writing web servers in Python such as flask, bottle, django, etc. Architecturally, I wonder whether you really need the machine learning (which I imagine would be a data processing pipeline) and the web server to be the same process / binary; however, even if you do them in the same server, I suspect that either language would be perfectly reasonable; moreover, if you ever came to the point where you needed to run a piece of computation in C++ for performance, using SWIG to call C++ from Python or using some form of message passing from Python to a C++ helper process (such as via gRPC) are options.
I've written an abstraction layer in Python for a piece of commercial software that has an API used for accessing the database back end. The API is exposed via a Windows DLL, and my library is written in Python.
My Python package loads the necessary libraries provided by the application, initializes them, and creates a couple of Python APIs on top. There are low level functions that simply wrap the API, and make the functions callable from Python, as well as a higher level interface that makes interaction with the native API more efficient.
The problem I'm encountering is that when running a daemon that uses the library, it seems there is a memory leak. (Several hundred KB/s) I've used several Python memory profiling tools, as well as tested each function individually, and only one function seems to leak, yet no tool reports that memory has been lost during execution of that function.
On Linux, I would use Valgrind to figure out if the vendor's library was the culprit, but the application only runs on Windows. How can I diagnose whether the vendor is at fault, or if it's the way I'm accessing their library?
I ended up writing a program in C without dynamic memory allocation to test the library. The leak is indeed in one of the functions I'm calling, not the Python program.
Let me explain what I'm trying to achieve. In the past while working on Java platform, I used to write Java codes(say, to push or pull data from MySQL database etc.) then create a war file which essentially bundles all the class files, supporting files etc and put it under a servlet container like Tomcat and this becomes a web service and can be invoked from any platform.
In my current scenario, I've majority of work being done in Java, however the Natural Language Processing(NLP)/Machine Learning(ML) part is being done in Python using the NLTK, Scipy, Numpy etc libraries. I'm trying to use the services of this Python engine in existing Java code. Integrating the Python code to Java through something like Jython is not that straight-forward(as Jython does not support calling any python module which has C based extensions, as far as I know), So I thought the next option would be to make it a web service, similar to what I had done with Java web services in the past. Now comes the actual crux of the question, how do I run the ML engine as a web service and call the same from any platform, in my current scenario this happens to be Java. I tried looking in the web, for various options to achieve this and found things like CherryPy, Werkzeug etc but not able to find the right approach or any sample code or anything that shows how to invoke a NLTK-Python script and serve the result through web, and eventually replicating the functionality Java web service provides. In the Python-NLTK code, the ML engine does a data-training on a large corpus(this takes 3-4 minutes) and we don't want the Python code to go through this step every time a method is invoked. If I make it a web service, the data-training will happen only once, when the service starts and then the service is ready to be invoked and use the already trained engine.
Now coming back to the problem, I'm pretty new to this web service things in Python and would appreciate any pointers on how to achieve this .Also, any pointers on achieving the goal of calling NLTK based python scripts from Java, without using web services approach and which can deployed on production servers to give good performance would also be helpful and appreciable. Thanks in advance.
Just for a note, I'm currently running all my code on a Linux machine with Python 2.6, JDK 1.6 installed on it.
One method is to build an XML-RPC server, but you may wish to fork a new process for each connection to prevent the server from seizing up. I have written a detailed tutorial on how to go about this: https://speakerdeck.com/timclicks/case-studies-of-python-in-parallel?slide=68.
NLTK based system tends to be slow at response per request, but good throughput can be achieved given enough RAM.