Could you give me an idea/concepts (not in code) on how could I link NodeJS and Python?
Let's say,
I have NodeJS up and running in PM2 (assuming I already know REST API) and I have a ton of data sets that I need to be ready to display to client side using socket.io (assumming I already know socket.io) as soon as possible.
I'm thinking to use Python. This is for me to implement the basics of machine learning.
In what concept should I start? I'd really love to hear your ideas.
Well you seem to be assuming way too many things, okay from your description I would suggest you to have a look at concept called microservice architecture.
This is how it will work let us assume you want to build an online shopping application where you have 2 main scenarios first is sell all items on your website and second you want to recommend products to your user(Your ML comes into play over here)
So as you said you already know REST API so what you would do is create a microservice (Consider it as a small nodejs application(Using either express or sails or any other framework) which has APIs exposed for all shopping related business logic) also you end up using fromtend technology viz. angularjs for your client side code. You'll show all this shopping stuff by calling your nodejs REST APIs from your angularjs client code. Node provides socket support via socket.io.
Similarly you write a small microservice in python(using Flask and Python-SocketIO) which takes your huge amount of data from datastore does all ML magic and returns recommended products for the particular user(which you received from your angularjs client application), and return it using Python-SocketIO to angularjs(or node application if you're maintaining your frontend logic there instead of angular).
You have provided very less detail so this is abstract view of what you can look into.
Since you're Python oriented for your ML code I'd suggest you to reduce the list of skills you need to learn and/or improve using Python for everything.
You could use Python-SocketIO and Flask, for example.
Related
I got 6 real-time videos which are inference output from heavy deep learning calculation from python, and I've tried to display them to PyQt that lots of threading issues with ugly GUI!!. So, I want to use a framework only to display well and fancy GUI! There seem lots of frameworks based on my google search, and I have no idea
which one is the best for my current project among Node js, Django, and Flask!
I need to display 6 real-time videos with 10~15 FPS.
Communicating well with python.
Easy to build GUI, I have some features like logging, displaying real-time graph(optional)
For video streaming, you will require to setup a socket URL for listening to incoming packet. Definitely, you can dedicate a server just for this to handle high amount of traffic.
Now, as such it dependents on your application, if you are just going to use it for streaming then Flask will do, it is lightweight. On the other hand, Django and NodeJS provides ready made function for socket programming and they are very useful.
I would say it doesn't really matter much as Django would be good because of its pre built features... Plus data transfer is very reliable in django along with your expertise in python, django is preferred. NodeJs would be easier to implement tho. Really prefers upon you, but both django and nodeJs works pretty well.
Sorry, no idea on flask
My opinion - flask faster development, node - better efficiency, some thoughts why are explained here: https://hinty.io/ivictbor/flask-vs-node/
I wanted to ask you about microservices in Python. As of writing this, i got pretty good with writing well structured flask-restful APIs and I wanted to go and learn about microservices in python.
Right now I have read up quite a lot of info regarding this and even searched online to find examples for this (1 example here) but I am not really sure exactly where to start as I don't want to invest too much time in a inefficient pattern.
So I wanted to know if anyone know any courses with examples for Python on building high load services. My only hints so far as asyncio and aiohttp for request handling and i'm not sure if using a message broker (such as zeromq or rabbitmq) would be a good idea as from what I read, it adds request lag.
Any advice would be great.
PS: The current pattern I'm stuck on is the API Gateway pattern and I would also want to know if it is a good direction as a start.
There are plenty of microservice frameworks in Python that can handle high load and get you a long way towards following best practices.
Try for example pymacaron (http://pymacaron.com/). Pymacaron is basically a flask app whose endpoints are auto-spawn from a swagger specification. To write a pymacaron microservice, you mostly have to:
(1) write a swagger specification for your api (which is always a good starting point, whatever language you are using). Your swagger file describes the get/post/etc calls of your api and which objects (json dicts) they get and return, but also which python method in your code that implement the endpoint.
(2) and implement your endpoints' methods.
Once you have done that, you get loads of things for free: you can package your code as a docker container, deploy it to amazon beanstalk, start asynchronous tasks from within your api calls, or get the api documentation with no extra work.
Here is an example of an helloworld api implemented with pymacaron: https://github.com/pymacaron/pymacaron-helloworld
I'm trying to learn web programming in python, and have the following project in mind: mine the yahoo finance api for instrument data, and display it in real time, as well as plot charts based on instrument data.
I already did something similar using wxpython, and I'm interested in how I would accomplish this in a web application.
My first thought was to use django and matplotlib on the server, and have the client request updated chart images through jquery at a certain time interval, but after a bit of research I came upon libraries like twisted and tornado...and now I'm confused. Would they work better for this web app than django ?
After the above rambling, my question is: what library should I use for writing the web app i have in mind ? I'm also thinking that I should abandon matplotlib, and generate the chart on client side, but I'm not sure what javascript library would allow me to do that, if any.
Few tips:
1/ Do not plot your data at backend . Instead use the browsers to generate charts.I would recommend
using jqplot, or highcharts.
2/ Yes, you can use tornado or twisted instead of django, as they are asynchronous servers, and would provide faster handling of requests.
3/ You should create a REST interface of your application, with server side only sending JSON data, and do all the UI templating and charting on client side.
4/ Backbone.js (recommended, but you can use some other MVC framework), would also prove to be helpful if your app grows too complex.
I'm working on a tracking proxy (for want of a better term) written in Python. It's a simple http (wsgi) application that will run on one (maybe more) server and accepts event data from a desktop client. This service would then forward the tracking data on to some actual tracking platform (DeskMetrics, MixPanel, Google Analytics) so that we don't have to deal with the slicing and dicing of data.
The reason for this implementation is that it's much easier and faster to make changes to a server process that we control rather than having to ensure every client in the wild gets updated if the tracking backend changes in some way.
I've been looking up info on the various options and I was hoping somebody here would have some good advice from their own experiences. Ideally we'd be able to use Google Analytics as it's free for any amount of usage but paid options are fine.
My only real requirement is either a good Python library or a well documented api that I can write a wrapper for (this seems somewhat lacking in GA when it comes to triggering events through any method other than their js or other provided libs).
N.B. We're not really tracking server code so something like NewRelic isn't appropriate, we're just decoupling a desktop application from the specifics of the tracking backend.
We ran into this same problem a bunch of times, we ended up building a suite of server-side analytics libraries to make this easier.
Segment.io has libraries for Python, Ruby, Java, Node, .NET and PHP that abstract the APIs for Mixpanel, KISSmetrics, Google Analytics and a bunch of other analytics services.
You could integrate the Python library once, and then send your data wherever you want. The data is proxied through Segment.io's hosted service. Hopefully this cleans up the mess of integrating a bunch of libraries, each with slightly different APIs. (The service is free for the first million events.)
Have you tried anything below?
The Google Data APIs Python Client Library has source specific to analytics
http://code.google.com/p/gdata-python-client/
http://code.google.com/p/gdata-python-client/source/browse/#hg%2Fsamples%2Fanalytics
https://developers.google.com/gdata/articles/python_client_lib
You might be able to borrow from these sources as well;
Google has something they are working on for mobile and source is available in PHP, JSP, ASP.net and Perl: https://developers.google.com/analytics/devguides/collection/other/mobileWebsites
I also came accross this in PHP http://code.google.com/p/php-ga/
As for others:
KissMetrics: http://support.kissmetrics.com/apis/python
MixPanel: https://mixpanel.com/docs/integration-libraries/python
DeskMetrics: don't seem to have python, http://docs.deskmetrics.com/index.html
Sorry I cannot provide information based off extensive experience with anything python related other then providing a few of these resources. I would be interested to see what you come up with.
We use a lot of of python to do much of our deployment and would be handy to connect to our TFS server to get information on iteration paths, tickets etc. I can see the webservice but unable to find any documentation. Just wondering if anyone knew of anything?
The web services are not documented by Microsoft as it is not an officially supported route to talk to TFS. The officially supported route is to use their .NET API.
In the case of your sort of application, the course of action I usually recommend is to create your own web service shim that lives on the TFS server (or another server) and uses their API to talk to the server but allows you to present the data in a nice way to your application.
Their object model simplifies the interactions a great deal (depending on what you want to do) and so it actually means less code over-all - but better tested and testable code and also you can work around things such as the NTLM auth used by the TFS web services.
Hope that helps,
Martin.
So, this question is friggin' old, but let me take a whack at it (since it keeps coming up in my google searches).
There's no officiall supported API for the on premise TFS (the MSFT hosted one has http://www.visualstudio.com/en-us/integrate/api/overview).
That said, you can always use Fiddler (http://www.telerik.com/fiddler) or something like it to inspect the calls that the web client for TFS is making to the server and do your magic to turn those into the scripts in python you want.
You'll need to run your python scripts under a service account that has TFS privs appropriate to what it is trying to do (read, update, confugure... whatever).
Since it sounds like you are just trying to read from TFS, this might be a really easy way for you to get what you want since an HTTP get to
http://yourserver/tfs/yourcollection/yourproject/_workitems#id=yourworkitemid
will hand you back (halfway) sane html payloads.
If you want lists of iterations or teams or whatever, then your service account needs to have the appropriate admin privileges and hit things like
http://yourserver/tfs/yourcollection/yourproject/_admin/_iterations
and use that response.