Please let me know how can i perform data reconciliation process for Rest API end point call i.e the data that is returned as part of response object from rest api call is matching with the data that is written to a output file. These Rest API are setup in Apigee API management system.
Please let me know what are the criteria i can utilize to perform data reconciliation process. Any python code example will be helpful
Related
maybe it is simple but I didn't find a satisfactory answer yet.
I have a python application that collects data over CAN bus (temperature, weight, ...) and I want to visualize them over Angular.
On the one side, I wrote the Python application that cyclic read the CAN-bus data and writes them to the console and on the other hand I wrote a small Angular application that contains the first step a simple table.
Now I want to fill in the table every 10 seconds with data from the Python application instead of printing them to the console.
How can I connect these both?
My first thought was a simple file where I save the values from Python and read them with Angular.
The second solution is a database, but I think this is too much for only a few values
So is there a direct way to access the Python data from Angular?
Basic idea is to create an api in python and let angular consume that
then there is the question of weather you want to have backup data in python,
if so then save it a db or file and use that as response for angular
if you want to do some fancy real time stuff may be look into long polling or http event stream
There are several ways you can access Python data from an Angular application:
One way is to use a REST API. You can create a REST API in Python
using a web framework like Flask or Django, and then use Angular's
HTTP client to make requests to the API and retrieve the data.
Another option is to use WebSockets. You can use a Python library
like asyncio or websockets to set up a WebSocket server, and then
use Angular's WebSocket client to connect to the server and receive
updates in real-time.
You can also use a message queue like RabbitMQ or ZeroMQ to allow
your Python and Angular applications to communicate with each other
asynchronously.
Overall, the best approach will depend on your specific requirements and how you want to structure your application. A REST API is a good choice if you need to retrieve data from the Python application on demand, while WebSockets or a message queue can be used for real-time communication and updates.
Apologies for my ignorance.
I am a data analyst in a small organisation (no data engineer/architect/web developer) and I am trying to set up a connection to an API with a company whose software/app we have purchased.
They have stated that for them to set up access to their API i need to create a callback url.
I don't really know where to start with this and neither does the IT manager.
I will likely be using Zapier or Power Automate to build the workflow needed but not sure how to go about setting up the callbackurl as requested. there last response was
'we would need a callback URL or local host to be able to generate API key'
any guidance would be great
This is what I want to achieve:
Ask the user to authorize the collection of their data on a Google Analytics 4 property (or Universal Analytics but I would rather not)
Programmatically retrieve and store the data every n-hours
I was able to do (1) client-side by asking for authorization with google's OAUTH2 and making a call to Reporting API v4 https://developers.google.com/analytics/devguides/reporting/core/v4 using gapi on the front-end.
However, I'm not sure how to do it on a schedule without user interaction. I've searched Google's API docs and I believe there's a way to do it in python https://developers.google.com/analytics/devguides/reporting/core/v4/quickstart/service-py but I am currently limited to Node and the browser. I guess I could make a server in python that does the data fetching and connects with the Node application, but that's yet another layer of complications that I'm trying to avoid. Is there a way to do everything in Node?
GCP APIs are all documented in a way which allows everyone to generate client libraries in a variety of languages, including node.js. The documentation for the node.js client for Analytics Reporting is here.
For the question of how to schedule this on GCP, I would recommend you to use Cloud Scheduler. This will hit an endpoint running on Cloud Run, which will do the actual work. Alternatively, if you already have a service running somewhere else, you can simply add the required endpoints there and point Cloud Scheduler to it.
The overall design which I would suggest you goes something like this:
Build a site which takes the user through the OAUTH2 login process,
requesting the relevant Google Analytics Reporting API scopes
required to make the request.
Store the obtained credentials in their user database.(preferably
Firestore in Datastore mode)
Set up a Cloud Run service (or anything else), with two endpoints
Iteration endpoint: Iterate through the list of users and add tasks
to Cloud Tasks to hit the download endpoint for each one.
Download endpoint: Takes a user ID (e.g. as a query parameter) and
performs the download for this user. You will need to load the
credentials for the user from the database and use this to access the
reporting API.
Store the downloaded data in the desired location, e.g. Cloud
Storage, Firestore, Cloud SQL, etc.
Set up Cloud Scheduler to hit the iteration endpoint at the desired
frequency.
For the GCP services mentioned above, basically everything other than Analytics, you may use the "cloud" clients for node.js, which are available here
Note : The question you have asked is a very broad question and this answer is just a suggestion. You may think about other designs whichever works best for you.
I am trying to build an app where users will be able to connect to my app, enter a keyword for searching on twitter and then the results will be stored on a database. From the moment the user enters a keyword I want to keep track of what is being said on twitter.Those results will be further analyzed and some statistics will be presented to the user.
So far I have used tweppy and twitter streaming api for getting the tweets. But I realized that I can not have more than one open streaming connections (for searching in parallel for multiple keywords).
I searched the stackoverflow and found solutions like disconnect, connect and then search with a new keyword, but in that case I am going to lose data.
Also I checked the Twitter API, which gives you 450 results max/15 min:
https://dev.twitter.com/docs/rate-limiting/1.1/limits
Stream API:
- public stream doesn't give the oppurtunity to have more than connections
- Site stream doesn't give you the oppurtunity for search
Firehose API is not option since is too expensive.
How can I solve this problem? I am seeing many apps searching live for more instances than one. Have anyone met this before?
You could use tweepy to collect all tweets from the sample or filter streaming endpoint and save this to a database. Then use the database to only return tweets for your search term.
If you don't want tweets to persist for too long, you might have better results using noSQL databases like redis and using an expiration timestamp, so it doesn't fill up infinitely.
I'm running an RESTful API with Python (Flask).
I want to be able to track:
which requests have been made
when did those requests happen
how long did it take to send the response
I want to use Google Analytics for this, because of it's nice dashboard and extended functionalities.
My question
How can I implement Google Analytics into a REST API?
OR does anyone know another tool/library that can be implemented?
This is what I found at the moment:
a tracking app that uses MongoDB
the Google data API - but this is for reading GA data, not tracking the API?
There are actually two ways to send server-side data to Google Analytics. The standard method is the GIF Image Request API, which is the same API that ga.js uses on the client-side. Google has started developing a new REST API known as the Measurement Protocol, but this is only in developer preview.
Server-Side GA
There are a few issues to work through when trying to send server-side data to GA.
Like #mehaase pointed out above, the gif API takes the ip address from the request, so all of your server-side requests will appear as users coming from the location of your servers. The measurement protocol doesn't let you change the request's ip either. I'll assume the publicly available gif API in this answer.
Another issue is that the gif endpoint requires a client-side cookie. You can fake this cookie on every request but this will cause each event to look like a new visitor. That's fine as long as you keep the server-side API and website in separate Google Analytics profiles.
Also beware that Google can take up to an hour to show your events once you've sent them. This can make debugging a bit painful, so be patient.
Here's the breakdown of what each variable in the GA cookie means, and a good node.js example of sending server-side data to GA.
Other Event Tracking Options
Even though GA is excellent for tracking website metrics, it's not built for tracking server-side events. A category of analytics known as event tracking is the perfect application for restful API usage tracking.
The API generally looks like this:
analytics.track('API Response', {
method : 'POST',
endpoint: '/comments'
duration: 124
status : 500
});
And lets you see reports on the frequencies and distributions of each event and event property You can answer questions like: how many /comments API calls happened today? How many were 200s? How many had a response higher than 200ms? etc.
Here are some event tracking tools that can help you do this:
Mixpanel
KissMetrics
Keen.IO
I'm the co-founder of Segment.io, a company that provides a simple API for client-side, server-side and mobile analytics. We let you send data from python, php, ruby, node, java, .net, javascript, and iOS, and we'll forward it to Google Analytics, Mixpanel, KissMetrics, Keen.IO, or any of our other supported services without you having to learn their API.
And finally, here's an article from our analytics academy that explains why event tracking is useful.
I know this is very old post. I came across google analytics support in Python
https://developers.google.com/api-client-library/python/apis/analytics/v3
Thought this is right place to document as well (y)