The final outcome of my work should be a Python function that takes a JSON object as the only input and return another JSON object as output. To keep it more specific, I am a data scientist, and the function that I am speaking about, is derived from data and it delivers predictions (in other words, it is a machine learning model).
So, my question is how to deliver this function to the "tech team" that is going to incorporate it into a web-service.
At the moment I face few problems. First, the tech team does not necessarily work in Python environment. So, they cannot just "copy and paste" my function into their code. Second, I want to make sure that my function runs in the same environment as mine. For example, I can imagine that I use some library that the tech team does not have or they have a version that differ from the version that I use.
ADDED
As a possible solution I consider the following. I start a Python process that listen to a socket, accept incoming strings, transforms them into JSON, gives the JSON to the "published" function and returns the output JSON as a string. Does this solution have disadvantages? In other words, is it a good idea to "publish" a Python function as a background process listening to a socket?
You have the right idea with using a socket but there are tons of frameworks doing exactly what you want. Like hleggs, I suggest you checkout Flask to build a microservice. This will let the other team post JSON objects in an HTTP request to your flask application and receive JSON objects back. No knowledge of the underlying system or additional requirements required!
Here's a template for a flask app that replies and responds with JSON
from flask import Flask, request, jsonify
app = Flask(__name__)
#app.route('/', methods=['POST'])
def index():
json = request.json
return jsonify(your_function(json))
if __name__=='__main__':
app.run(host='0.0.0.0', port=5000)
Edit: embeded my code directly as per Peter Britain's advice
My understanding of your question boils down to:
How can I share a Python library with the rest of my team, that may not be using Python otherwise?
And how can I make sure my code and its dependencies are what the receiving team will run?
And that the receiving team can install things easily mostly anywhere?
This is a simple question with no straightforward answer... as you just mentioned that this may be integrated in some webservice, but you do not know the actual platform for this service.
You also ask:
As a possible solution I consider the following. I start a Python process that listen to a socket, accept incoming strings, transforms them into JSON, gives the JSON to the "published" function and returns the output JSON as a string. Does this solution have disadvantages? In other words, is it a good idea to "publish" a Python function as a background process listening to a socket?
In the most simple case and for starting I would say no in general. Starting network servers such as an HTTP server (which is built-in Python) is super easy. But a service (even if qualified as "micro") means infrastructure, means security, etc.
What if the port you expect is not available on the deployment machine? - What happens when you restart that machine?
How will your server start or restart when there is a failure?
Would you need also to eventually provide an upstart or systemd service (on Linux)?
Will your simple socket or web server support multiple concurrent requests?
is there a security risk to expose a socket?
Etc, etc. When deployed, my experience with "simple" socket servers is that they end up being not so simple after all.
In most cases, it will be simpler to avoid redistributing a socket service at first. And the proposed approach here could be used to package a whole service at a later stage in a simpler way if you want.
What I suggest instead is a simple command line interface nicely packaged for installation.
The minimal set of things to consider would be:
provide a portable mechanism to call your function on many OSes
ensure that you package your function such that it can be installed with all the correct dependencies
make it easy to install and of course provide some doc!
Step 1. The simplest common denominator would be to provide a command line interface that accepts the path to a JSON file and spits JSON on the stdout.
This would run on Linux, Mac and Windows.
The instructions here should work on Linux or Mac and would need a slight adjustment for Windows (only for the configure.sh script further down)
A minimal Python script could be:
#!/usr/bin/env python
"""
Simple wrapper for calling a function accepting JSON and returning JSON.
Save to predictor.py and use this way::
python predictor.py sample.json
[
"a",
"b",
4
]
"""
from __future__ import absolute_import, print_function
import json
import sys
def predict(json_input):
"""
Return predictions as a JSON string based on the provided `json_input` JSON
string data.
"""
# this will error out immediately if the JSON is not valid
validated = json.loads(json_input)
# <....> your code there
with_predictions = validated
# return a pretty-printed JSON string
return json.dumps(with_predictions, indent=2)
def main():
"""
Print the JSON string results of a prediction, loading an input JSON file from a
file path provided as a command line argument.
"""
args = sys.argv[1:]
json_input = args[0]
with open(json_input) as inp:
print(predict(inp.read()))
if __name__ == '__main__':
main()
You can process eventually large inputs by passing the path to a JSON file.
Step 2. Package your function. In Python this is achieved by creating a setup.py script. This takes care of installing any dependent code from Pypi too. This will ensure that the version of libraries you depend on are the ones you expect. Here I added nltk as an example for a dependency. Add yours: this could be scikit-learn, pandas, numpy, etc. This setup.py also creates automatically a bin/predict script which will be your main command line interface:
#!/usr/bin/env python
# -*- encoding: utf-8 -*-
from __future__ import absolute_import, print_function
from setuptools import setup
from setuptools import find_packages
setup(
name='predictor',
version='1.0.0',
license='public domain',
description='Predict your life with JSON.',
packages=find_packages(),
# add all your direct requirements here
install_requires=['nltk >= 3.2, < 4.0'],
# add all your command line entry points here
entry_points={'console_scripts': ['predict = prediction.predictor:main']}
)
In addition as is common for Python and to make the setup code simpler I created a "Python package" directory moving the predictor inside this directory.
Step 3. You now want to package things such that they are easy to install. A simple configure.sh script does the job. It installs virtualenv, pip and setuptools, then creates a virtualenv in the same directory as your project and then installs your prediction tool in there (pip install . is essentially the same as python setup.py install). With this script you ensure that the code that will be run is the code you want to be run with the correct dependencies. Furthermore, you ensure that this is an isolated installation with minimal dependencies and impact on the target system. This is tested with Python 2 but should work quite likely on Python 3 too.
#!/bin/bash
#
# configure and installs predictor
#
ARCHIVE=15.0.3.tar.gz
mkdir -p tmp/
wget -O tmp/venv.tgz https://github.com/pypa/virtualenv/archive/$ARCHIVE
tar --strip-components=1 -xf tmp/venv.tgz -C tmp
/usr/bin/python tmp/virtualenv.py .
. bin/activate
pip install .
echo ""
echo "Predictor is now configured: run it with:"
echo " bin/predict <path to JSON file>"
At the end you have a fully configured, isolated and easy to install piece of code with a simple highly portable command line interface.
You can see it all in this small repo: https://github.com/pombredanne/predictor
You just clone or fetch a zip or tarball of the repo, then go through the README and you are in business.
Note that for a more engaged way for more complex applications including vendoring the dependencies for easy install and not depend on the network you can check this https://github.com/nexB/scancode-toolkit I maintain too.
And if you really want to expose a web service, you could reuse this approach and package that with a simple web server (like the one built-in in the Python standard lib or bottle or flask or gunicorn) and provide configure.sh to install it all and generate the command line to launch it.
Your task is (in generality) about productionizing a machine learning model, where the consumer of the model may not be working in the same environment as the one which was used to develop the model. I've been trying to tackle this problem since past few years. The problem is faced by many companies and it is aggravated due to skill set, objectives as well as environment (languages, run time) mismatch between data scientists and developers. From my experience, following solutions/options are available, each with its unique advantages and downsides.
Option 1 : Build the prediction part of your model as a standalone web service using any lightweight tool in Python (for example, Flask). You should try to decouple the model development/training and prediction part as much as possible. The model that you have developed, must be serialized to some form so that the web server can use it.
How frequently is your machine learning model updated? If it is not done very frequently, the serialized model file (example: Python pickle file) can be saved to a common location accessible to the web server (say s3), loaded in memory. The standalone web server should offer APIs for prediction.
Please note that exposing a single model prediction using Flask would be simple. But scaling this web server if needed, configuring it with right set of libraries, authentication of incoming requests are all non-trivial tasks. You should choose this route only if you have dev teams ready to help with these.
If the model gets updated frequently, versioning your model file would be a good option. So in fact, you can piggyback on top of any version control system by checking in the whole model file if it is not too large. The web server can de-serialize (pickle.load) this file at startup/update and convert to a Python object on which you can call prediction methods.
Option 2 : use predictive modeling markup language. PMML was developed specifically for this purpose: predictive modeling data interchange format independent of environment. So data scientist can develop model, export it to a PMML file. The web server used for prediction can then consume the PMML file for doing predictions. You should definitely check the open scoring project which allows you to expose machine learning models via REST APIs for deploying models and making predictions.
Pros: PMML is standardized format, open scoring is a mature project with good development history.
Cons: PMML may not support all models. Open scoring is primarily useful if your tech team's choice of development platform is JVM. Exporting machine learning models from Python is not straightforward. But R has good support for exporting models as PMML files.
Option 3 : There are some vendors offering dedicated solutions for this problem. You will have to evaluate cost of licensing, cost of hardware as well as stability of the offerings for taking this route.
Whichever option you choose, please consider the long term costs of supporting that option. If your work is in a proof of concept stage, Python flask based web server + pickled model files will be the best route. Hope this answer helps you!
As already suggested in other answers the best option would be creating a simple web service. Besides Flask you may want to try bottle which is very thin one-file web framework. Your service may looks as simple as:
from bottle import route, run, request
#route('/')
def index():
return my_function(request.json)
run(host='0.0.0.0', port=8080)
In order to keep environments the same check virtualenv to make isolated environment for avoiding conflicts with already installed packages and pip to install exact version of packages into virtual environment.
I guess you have 3 possibilities :
convert python function to javascript function:
Assuming the "tech-team" use Javascript for web-service, you may try to convert your python function directly to a Javascript function (which will be really easy to integrate on web page) using empythoned (based on emscripten)
The bad point of this method is that each time you need update/upgrade your python function, you need also to convert to Javascript again, then check & validate that the function continue to work.
simple API server + JQuery
If the conversion method is impossible, I am agree with #justin-bell, you may use FLASK
getting JSON as input > JSON to your function parameter > run python function > convert function result to JSON > serve the JSON result
Assuming you choose the FLASK solution, "tech-team" will only need to send an async. GET/POST request containing all the arguments as JSON obj, when they need to get some result from your python function.
websocket server + socket.io
You can also use take a look on Websocket to dispatch to the webservice (look at flask + websocket for your side & socket.io for webservice side.)
=> websocket is really usefull when you need to push/receive data with low cost and latency to (or from) a lot of users (Not sure that websocket will be the best fit to your need)
Regards
What widely-used Python library is there to make test doubles of the behaviour of low-level HTTP APIs from the Python standard library?
I have a code base (Python 2 code) which performs HTTP sessions using the various HTTP-level classes in the standard library: httplib.HTTPConnection and urllib2.HTTPBasicAuthHandler and so on.
Testing this code will be made much easier if I can create numerous test doubles with pre-determined behaviour and instrumentation for later inspection, to allow the functions I'm testing to interact with those doubles instead of the real thing.
Mocking the API is no problem, of course; the standard mock library makes that easy. What is needed here, though, is to pre-set specific responses at specific points during the HTTP session, without duplicating the rest of the low-level protocol implementation.
By “widely-known”, I'm hoping that the library will already be available in OS package repositories. Requiring a package from a third party would be a significant cost for this project, so it would help a lot if it's already packaged and in Debian.
What standard code libraries provide classes which make it easy to double the behaviour of httplib classes and urllib2 classes, etc. for test cases?
I'm trying to get the rendered markup for http://www.epicurious.com/recipes/food/reviews/Breaded-Chicken-Cutlets-aka-Grandma-Jodys-Chicken-51114400; in theory the very same markup given by the 'View Page Source' menu option in Firefox.
I'm using a Python 2.7 script and the httplib library (http://docs.python.org/2/library/httplib.html). I've created an HTTPConnection object and when I try to get the markup via the HTTPResponse object's functions, I inevitably get a getaddrinfo - 11004 error. This script has been executed in Windows 7 and Ubuntu environments.
None of the other solutions for this error that I've read fit the bill: I am not behind any firewall, and I have no problem pinging www.google.com. I wonder if that website just doesn't conform to some standard I'm unaware of, as I haven't been able to successfully ping my target website.
I'm open to alternate approaches, let me know if there is a better way.
You might want to check out the reqests library. It makes simple things like this much easier:
import requests
r = requests.get('http://www.epicurious.com/recipes/food/reviews/Breaded-Chicken-Cutlets-aka-Grandma-Jodys-Chicken-51114400')
print r.text
Here are the docs: http://docs.python-requests.org/en/latest/
Ran the above and verified it works.
Particularly, I'm working with python and vmware vsphere to make virtual machine creation/power on and stuff like that automated. I know how to print attributes of virtual machines, however, I still can't call any methods because I don't know how.
this is code snippet I'm having trouble with:
for vm in virtual_machines:
print "VM: "+vm['name']+" ("+vm['runtime.powerState']+") ",
if hasattr(vm['guest'],'ipAddress'):
print vm['guest'].ipAddress,
if vm['runtime.powerState'] == 'poweredOn':
print 'RAM: '+str(vm['summary.quickStats'].distributedMemoryEntitlement)+'MB, CPU: '+str(vm['summary.quickStats'].distributedCpuEntitlement)+'MHz',
if vm['name'] == 'VIRT_VZ_114':
# This is the trouble maker, it says types mismatch "suds.TypeNotFound: Type not found: 'guest'"
client.service.PowerOnVM_Task(mo_VirtualMachine,vm)
print
How do I call methods correctly?
Have you looked at psphere at all?
It's a python project to provide native bindings for the vSphere Web Services SDK.
You may be able to get your problems solved quicker with a dedicated library, rather than struggle with suds, SOAP and WDSL issues. It actually uses suds under the hood, but provides you with a pythonic API instead.
My plan is to develop a multi-tier, multi-platform database application.
I would like to consume the data from cocoa/objective c apps, .net apps, and web browsers.
I don’t really know where to start and have been looking a Python, but can’t find if cocoa/objective c apps can consume python data objects.
Can anyone point me in the right direction as to how to achieve my goal?
My requirements are:
Data layer should be platform independent.
Whole system is scalable. Therefore multi tier.
Data access can be from cocoa, .net and web based clients.
You can make python and objective-c work together. Since you can use 100% normal C you can use the Python C interface. It's very tedious though.
There's also PyObjC. This acts as a bridge between Objective-C and Python. The documentation is pretty good and it will be much simpler than using the Python C interface directly.
You could also try using Thrift. Thrift is like Protocol Buffers by Google, but has support for generating Objective-C classes. You will have to write some boiler plate code to convert the data object into a thrift object; but after that is done you can pass information amongst any of the languages thrift supports. Documentation is on the thin side; I wrote a tutorial on using with Objective-C available on the thrift wiki here some time ago, not sure if it us up-to-date though as there have been several releases of thrift since then.