I have a function in Python, defined in an API from my broker, "getCalendar" which recieves a list of news announcements and expected impact on market. how can I transform this list which arrives as a JSON object to a pandas dataframe, so I can analyze it?
P.S.: the API is a connection to a server, which is first established and only then can data be streamed from there, so using url address and converting that to pandas dataframe is not possible.
Thanks in advance for your help.
Seems your getCalendar is writing stuff to stdout and you first need to capture that into a string variable. Use this post solution to capture: Can I redirect the stdout in python into some sort of string buffer?
(e.g. write a wrapper like getCalendarStdout() used below)
Once you got the json output into a variable (say calendar), try this:
calendar = getCalendarStdout()
import json
import pandas as pd
data=json.loads(calendar)
dafr = pd.DataFrame(data,columns=['col1','col2'])
Here we are trying to get only certain fields from the json output into the dafr DataFrame. If you can paste the calendar json data (part of it) people can help you get desired dataframe.
Related
I am trying to export a tableau view to csv using python tableau server client.
This is the code piece I'm using:
view_object = <my view object>
csv_req_option = TSC.CSVRequestOptions(maxage=-1)
# csv_req_option.vf('Category', 'Furniture')
server.views.populate_csv(view_object, csv_req_option)
with open('view_data.csv', 'wb') as f:
# Perform byte join on the CSV data
f.write(b''.join(view_object.csv))
However, the above code gave me an error indicating 504 Gateway Time-out, I am guessing this is because I have too large data to export, so I am trying to add a filter using csv_req_option.vf(...) to make the data size smaller.
I want to filter based on the date from 2022-06-28 to 2022-07-07, I have Googled to see how to apply a filter for csv, but looks like the answer I found did not filter based on date. (The line which I commented out was the answer I found previously, but not relevant to my request.) So could anyone help me filter based on datetime, or even some guidance also helps?
Thank you so much!
I am trying to query Google BigQuery using the Pandas/Python client interface. I am following the tutorial here: https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas. I was able to get it to work but I want to query the data as the JSON format that can be downloaded directly from the WebUI (see screenshot). Is there a way to download data as the JSON structure pictured instead of converting it to the data frame object?
I imagine the command would be somewhere around this part of the code from the tutorial:
dataframe = (
bqclient.query(query_string)
.result()
.to_dataframe(bqstorage_client=bqstorageclient)
)
Just add .to_json(orient='records') call after converting to dataframe:
json_data = bqclient.query(query_string).result().to_dataframe(bqstorage_client=bqstorageclient).to_json(orient='records')
pandas docs
I'm a Ruby dev doing a lot of data work that's decided to switch to Python. I'm enjoying making the transition so far and have been blown away by Pandas, Jupyter Notebooks etc.
My current task is to write a lightweight RESTful API that under the hood is running queries against Google BigQuery.
I have a really simple test running in Flask, this works fine, but I did have trouble rendering the BigQuery response as JSON. To get around this, I used Pandas and then converted the dataframe to JSON. While it works, this feels like an unnecessary step and I'm not even sure this is a legitimate use case for Pandas. I have also read that creating a dataframe can be slow as data volume increases.
Below is my little mock up test in Flask. It would be really helpful to hear from experienced Python Devs how you'd approach this and if there are any other libraries I should be looking at here.
from flask import Flask
from google.cloud import bigquery
import pandas
app = Flask(__name__)
#app.route("/bq_test")
def bq_test():
client = bigquery.Client.from_service_account_json('/my_creds.json')
sql = """select * from `my_dataset.my_table` limit 1000"""
query_job = client.query(sql).to_dataframe()
return query_job.to_json(orient = "records")
if __name__ == "__main__":
app.run()
From the BigQuery documentation-
BigQuery supports functions that help you retrieve data stored in
JSON-formatted strings and functions that help you transform data into
JSON-formatted strings:
JSON_EXTRACT or JSON_EXTRACT_SCALAR
JSON_EXTRACT(json_string_expr, json_path_string_literal), which returns JSON values as STRINGs.
JSON_EXTRACT_SCALAR(json_string_expr, json_path_string_literal), which returns scalar JSON values as STRINGs.
Description
The json_string_expr parameter must be a JSON-formatted string. ...
The json_path_string_literal parameter identifies the value or values you want to obtain from the JSON-formatted string. You construct this parameter using the JSONPath format.
https://cloud.google.com/bigquery/docs/reference/standard-sql/json_functions
Are there any tools for parsing a .json file format
You cannot selectively parse JSON. This makes no sense as how can yo tell where is the section you selectively demanding, without parsing whole file? That said, JSON have to be read usually as whole, however you can selectively process loaded JSON by own criteria. I'd sufficient to iterate over activities node, check value of action_type and act accordingly. You need to write that code yourself. There's no makePortal() magic in the language to help you with that.
Load the full json data and filter it after loading
import json
data = json.load(open('data.json'))
activities = [act for act in data['activities'] if '_STAGE_CHANGE' in act['composite_id']]
I exhaustively searched for my question on the web , but the closest answer I found was this:
Answer for converting list to string
To be precise, I fetched some data in a variable in java using mysql-python connector module.
After I printed the variable,The fetched data looked something like this:
[(16,'abc#ail.com','lenbox','+91','16','ACTIVE'),
(17,'xyz#ail.com','lenro','+41','17','ACTIVE'),
(18,'sdf#ail.com','meru','+51','18','NOTACTIVE')]
I want to use this data to hit an API whose URL and other details are known to me. For that the above data should be in a JSON string format.Please suggest a way?
I am newbie in python language and don't know so well about JSON.