I'm working on a tracking proxy (for want of a better term) written in Python. It's a simple http (wsgi) application that will run on one (maybe more) server and accepts event data from a desktop client. This service would then forward the tracking data on to some actual tracking platform (DeskMetrics, MixPanel, Google Analytics) so that we don't have to deal with the slicing and dicing of data.
The reason for this implementation is that it's much easier and faster to make changes to a server process that we control rather than having to ensure every client in the wild gets updated if the tracking backend changes in some way.
I've been looking up info on the various options and I was hoping somebody here would have some good advice from their own experiences. Ideally we'd be able to use Google Analytics as it's free for any amount of usage but paid options are fine.
My only real requirement is either a good Python library or a well documented api that I can write a wrapper for (this seems somewhat lacking in GA when it comes to triggering events through any method other than their js or other provided libs).
N.B. We're not really tracking server code so something like NewRelic isn't appropriate, we're just decoupling a desktop application from the specifics of the tracking backend.
We ran into this same problem a bunch of times, we ended up building a suite of server-side analytics libraries to make this easier.
Segment.io has libraries for Python, Ruby, Java, Node, .NET and PHP that abstract the APIs for Mixpanel, KISSmetrics, Google Analytics and a bunch of other analytics services.
You could integrate the Python library once, and then send your data wherever you want. The data is proxied through Segment.io's hosted service. Hopefully this cleans up the mess of integrating a bunch of libraries, each with slightly different APIs. (The service is free for the first million events.)
Have you tried anything below?
The Google Data APIs Python Client Library has source specific to analytics
http://code.google.com/p/gdata-python-client/
http://code.google.com/p/gdata-python-client/source/browse/#hg%2Fsamples%2Fanalytics
https://developers.google.com/gdata/articles/python_client_lib
You might be able to borrow from these sources as well;
Google has something they are working on for mobile and source is available in PHP, JSP, ASP.net and Perl: https://developers.google.com/analytics/devguides/collection/other/mobileWebsites
I also came accross this in PHP http://code.google.com/p/php-ga/
As for others:
KissMetrics: http://support.kissmetrics.com/apis/python
MixPanel: https://mixpanel.com/docs/integration-libraries/python
DeskMetrics: don't seem to have python, http://docs.deskmetrics.com/index.html
Sorry I cannot provide information based off extensive experience with anything python related other then providing a few of these resources. I would be interested to see what you come up with.
Related
Hello Python friends I am developing a commercial application in python that backs up databases, directory synchronization, finally ...
I need to implement a bandwidth limit feature.
For example the user defines that the software cannot exceed 40kbps for upload.
I know there is a way to do it because the Dropbox client is written in python and has this functionality.
I searched the net how to implement this in other languages, I even found an example using lib Twisted, but it doesn't fit my case,
because data can be configured to send to multiple clouds, AWS, Google Cloud, FTP, Google Drive, and more.
The only solution left for me was to hook the Windows API, specifically the WinSock API, to intercept system calls to
send, recv and similar functions and implement my own bandwidth limiter, but this is going to be so complicated to implement.
Does anyone have any ideas how to help me?
After a little o research I found this question Adjust speed of Socket, maybe Dropbox use that... I tried and works like a charm, but as I said, this not fit in my case, because I send to multiple clouds, is Hooking Windows API the only solution or anyone has another idea?
;)
I wanted to ask you about microservices in Python. As of writing this, i got pretty good with writing well structured flask-restful APIs and I wanted to go and learn about microservices in python.
Right now I have read up quite a lot of info regarding this and even searched online to find examples for this (1 example here) but I am not really sure exactly where to start as I don't want to invest too much time in a inefficient pattern.
So I wanted to know if anyone know any courses with examples for Python on building high load services. My only hints so far as asyncio and aiohttp for request handling and i'm not sure if using a message broker (such as zeromq or rabbitmq) would be a good idea as from what I read, it adds request lag.
Any advice would be great.
PS: The current pattern I'm stuck on is the API Gateway pattern and I would also want to know if it is a good direction as a start.
There are plenty of microservice frameworks in Python that can handle high load and get you a long way towards following best practices.
Try for example pymacaron (http://pymacaron.com/). Pymacaron is basically a flask app whose endpoints are auto-spawn from a swagger specification. To write a pymacaron microservice, you mostly have to:
(1) write a swagger specification for your api (which is always a good starting point, whatever language you are using). Your swagger file describes the get/post/etc calls of your api and which objects (json dicts) they get and return, but also which python method in your code that implement the endpoint.
(2) and implement your endpoints' methods.
Once you have done that, you get loads of things for free: you can package your code as a docker container, deploy it to amazon beanstalk, start asynchronous tasks from within your api calls, or get the api documentation with no extra work.
Here is an example of an helloworld api implemented with pymacaron: https://github.com/pymacaron/pymacaron-helloworld
I want to send automated payments using the PayPal APIs. As far as I know, currently there are only 2 ways to do it:
Using the Payouts Endpoints of the modern REST APIs
Using the Adaptative Payments from the "classic" APIs
The preferred option would be option 1, because there is already an official SDK available for Python, sadly this feature seems to be only available on the United States, the documentation states:
Merchants in the United States can register to use Payouts in the Live
environment as outlined in the following steps. Other merchants should
contact their Account Manager or customer service to enable Payouts.
So it seems there are some extra steps to enable Payouts, but they are very opaque about it and don't give so much information, it's unclear to me whether it's really available or not. (I have already contacted their tech support to get assistance, I'm waiting for their reply, I'm just asking this here as a backup plan).
If I want to use the Adaptative Payments on Python, more specifically on Django I have 3 options:
Implement a small Java wrapper around the official SDK and call it through Python using JPype
Implement a small Java wrapper and call it through the command line, or run it as a service and call it internally from the server.
Implement the required subset of the PayPal SDK in Python myself.
Option 3 is the most complicated, but probably the most efficient, however since it will not be officially supported, maintaining it can become problematic. Option 2 seems so easy to implement, but since the memory overhead of the JVM will be there anyway maybe it's better to interface it through JPype, which is the main point of this question.
Question details here:
Sorry for the large introduction, it ended up much larger than expected, now going to the point.
If I start the JVM in Django with JPype. Will this mean that each request (there will be a thread for each request, that's how web servers work) will start it's own JVM? Is it a good approach?
I have developed a python program that parses a webpage and creates a new text document with the parsed data. I want to deliver this new information to the web. I have no idea where to start with something like this. Are there any free options where I can have a site automatically call this python code upon request and update the new data to its page? Or is the only feasible solution here to have my own website/server that uses my code? I'm honestly pretty overwhelmed with many of the options when I try to begin doing a web-search for a solution like this. I have done a decent amount of application programming before so i'm confident in my ability to learn new things, but web protocols are all new to me so its hard to find a starting point.
Ultimately I want this python code to run automatically, or per request of a user, and deliver to the data to them. It could even be through an email, although that is probably less practical.
I personally have good experience using Google Appengine (and its free for a limited amount of requests). The downside is that it does not allow C-extensions or Python3.
If you want to host your own server, tornado is a good option I think. Tornado supports both Python2 and Python3.
There are a great deal of options available.. from 'traditional' virtual server or website hosts like a2hosting or godaddy to 'Cloud Application Hosts' such as Amazon EC2, Heroku or OpenShift.
For your case, and without knowing more, I would suggest that an application hosting is more appropriate, and that you should take a look at Heroku and Openshift in particular.
Define carefully what you want to achieve (how the users access your application, what they see, how they interact with it... etc..) and then evaluate these options based on those requirements.
Most offer a free trial, or even free services, depending on what you need! Good luck
If you've never worked with web technologies before this will be a overwhelming task, since there's a lot of different technologies involved, and many possible ways to combine them.
You'll probably want to start by familiarizing yourself with the very basics of the HTTP protocol.
Then you should read a bit on CGI server-side programming (the article also has a quick overview on HTTP).
Python can run both on CGI and WSGI (if the server provider allows such access), so you may also want to read about WSGI.
Once you grasp all these concepts, you should check this question for actual python techniques.
Also, since you seem to be under the impression you must pay to have a website/app deployed, you should know there are companies that host python apps for free
We use a lot of of python to do much of our deployment and would be handy to connect to our TFS server to get information on iteration paths, tickets etc. I can see the webservice but unable to find any documentation. Just wondering if anyone knew of anything?
The web services are not documented by Microsoft as it is not an officially supported route to talk to TFS. The officially supported route is to use their .NET API.
In the case of your sort of application, the course of action I usually recommend is to create your own web service shim that lives on the TFS server (or another server) and uses their API to talk to the server but allows you to present the data in a nice way to your application.
Their object model simplifies the interactions a great deal (depending on what you want to do) and so it actually means less code over-all - but better tested and testable code and also you can work around things such as the NTLM auth used by the TFS web services.
Hope that helps,
Martin.
So, this question is friggin' old, but let me take a whack at it (since it keeps coming up in my google searches).
There's no officiall supported API for the on premise TFS (the MSFT hosted one has http://www.visualstudio.com/en-us/integrate/api/overview).
That said, you can always use Fiddler (http://www.telerik.com/fiddler) or something like it to inspect the calls that the web client for TFS is making to the server and do your magic to turn those into the scripts in python you want.
You'll need to run your python scripts under a service account that has TFS privs appropriate to what it is trying to do (read, update, confugure... whatever).
Since it sounds like you are just trying to read from TFS, this might be a really easy way for you to get what you want since an HTTP get to
http://yourserver/tfs/yourcollection/yourproject/_workitems#id=yourworkitemid
will hand you back (halfway) sane html payloads.
If you want lists of iterations or teams or whatever, then your service account needs to have the appropriate admin privileges and hit things like
http://yourserver/tfs/yourcollection/yourproject/_admin/_iterations
and use that response.