got some functions with sqlstatements. My first func is fine because i get only 1 result.
My second function returns a large list of errorcodes and i dont know how to get them back for response.
TypeError: <sqlalchemy.engine.result.ResultProxy object at 0x7f98b85ef910> is not JSON serializable
Tried everything need help.
My Code:
def topalarms():
customer_name = request.args.get('customer_name')
machine_serial = request.args.get('machine_serial')
#ts = request.args.get('ts')
#ts_start = request.args.get('ts')
if (customer_name is None) or (machine_serial is None):
return missing_param()
# def form_response(response, session):
# response['customer'] = customer_name
# response['serial'] = machine_serial
# return do_response(customer_name, form_response)
def form_response(response, session):
result_machine_id = machine_id(session, machine_serial)
if not result_machine_id:
response['Error'] = 'Seriennummer nicht vorhanden/gefunden'
return
#response[''] = result_machine_id[0]["id"]
machineid = result_machine_id[0]["id"]
result_errorcodes = error_codes(session, machineid)
response['ErrorCodes'] = result_errorcodes
return do_response(customer_name, form_response)
def machine_id(session, machine_serial):
stmt_raw = '''
SELECT
id
FROM
machine
WHERE
machine.serial = :machine_serial_arg
'''
utc_now = datetime.datetime.utcnow()
utc_now_iso = pytz.utc.localize(utc_now).isoformat()
utc_start = datetime.datetime.utcnow() - datetime.timedelta(days = 30)
utc_start_iso = pytz.utc.localize(utc_start).isoformat()
stmt_args = {
'machine_serial_arg': machine_serial,
}
stmt = text(stmt_raw).columns(
#ts_insert = ISODateTime
)
result = session.execute(stmt, stmt_args)
ts = utc_now_iso
ts_start = utc_start_iso
ID = []
for row in result:
ID.append({
'id': row[0],
'ts': ts,
'ts_start': ts_start,
})
return ID
def error_codes(session, machineid):
stmt_raw = '''
SELECT
name
FROM
identifier
WHERE
identifier.machine_id = :machineid_arg
'''
stmt_args = {
'machineid_arg': machineid,
}
stmt = text(stmt_raw).columns(
#ts_insert = ISODateTime
)
result = session.execute(stmt, stmt_args)
errors = []
for row in result:
errors.append(result)
#({'result': [dict(row) for row in result]})
#errors = {i: result[i] for i in range(0, len(result))}
#errors = dict(result)
return errors
My problem is func error_codes somethiing is wrong with my result.
my Output should be like this:
ABCNormal
ABCSafety
Alarm_G01N01
Alarm_G01N02
Alarm_G01N03
Alarm_G01N04
Alarm_G01N05
I think you need to take a closer look at what you are doing correctly with your working function and compare that to your non-working function.
Firstly, what do you think this code does?
for row in result:
errors.append(result)
This adds to errors one copy of the result object for each row in result. So if you have six rows in result, errors contains six copies of result. I suspect this isn't what you are looking for. You want to be doing something with the row variable.
Taking a closer look at your working function, you are taking the first value out of the row, using row[0]. So, you probably want to do the same in your non-working function:
for row in result:
errors.append(row[0])
I don't have SQLAlchemy set up so I haven't tested this: I have provided this answer based solely on the differences between your working function and your non-working function.
You need a json serializer. I suggest using Marshmallow: https://marshmallow.readthedocs.io/en/stable/
There are some great tutorials online on how to do this.
Related
I'm very confused. I'm trying to run a simple query on a database:
def get_not_tested_calculations(self, location_id: int):
with self.session.begin() as sess:
sel = select(MHICalculation).filter(
MHICalculation.applicable_location == location_id,
MHICalculation.is_tested == False
)
res = [i[0] for i in sess.execute(sel).all()]
return res
What I keep getting is a whole load of <db.schema.MHICalculation object at 0x000002795B13B4F0> objects.
When I inspect the select object the query itself looks correct:
SELECT mhi_calculation.id, mhi_calculation.mhi_type, mhi_calculation.added_date, mhi_calculation.added_by, mhi_calculation.mhi_cutoff, mhi_calculation.truck_window_size, mhi_calculation.ochreous_goethite_min, mhi_calculation.ochreous_goethite_max, mhi_calculation.kaolinite_min, mhi_calculation.kaolinite_max, mhi_calculation.hematite_ultrafines_min, mhi_calculation.hematite_ultrafines_max, mhi_calculation.hematite_lump_min, mhi_calculation.hematite_lump_max, mhi_calculation.hematite_moisture_min, mhi_calculation.hematite_moisture_max, mhi_calculation.goethite_ultrafines_min, mhi_calculation.goethite_ultrafines_max, mhi_calculation.goethite_lump_min, mhi_calculation.goethite_lump_max, mhi_calculation.goethite_moisture_min, mhi_calculation.goethite_moisture_max, mhi_calculation.truck_count, mhi_calculation.is_tested, mhi_calculation.applicable_location
FROM mhi_calculation
WHERE mhi_calculation.applicable_location = :applicable_location_1 AND mhi_calculation.is_tested = false
It works fine when I specify a column, for example:
def get_not_tested_calculations(self, location_id: int):
with self.session.begin() as sess:
sel = select(MHICalculation.id).filter(
MHICalculation.applicable_location == location_id,
MHICalculation.is_tested == False
)
res = [i[0] for i in sess.execute(sel).all()]
return res
And the query is pretty much the same, albeit shorter, of course:
SELECT mhi_calculation.id
FROM mhi_calculation
WHERE mhi_calculation.applicable_location = :applicable_location_1 AND mhi_calculation.is_tested = false
I've been trying all the different ways but for some strange reason can't get it to work at all. I'm using SQLAlchemy 1.4.27 with Python 3.9.7.
Any ideas what is happening here?
The ultimate goal of this is to output select data columns to a .csv. I had it working once to where it only got the first table on the page but I needed both. Now it says this. Im quite new to python and IDK how I got to this point in the first place. I needed the call and put table but on the web page the calls came first and when I did .find I only got the calls. I am working on this with a friend and he put in the last two functions. He could get the columns I wanted but now we only get the calls. I tried to fix it and now it say the error in the title.
import bs4
import requests
import pandas as pd
import csv
from bs4 import BeautifulSoup
#sets desired ticker. in the future you could make this long
def ticker():
ticker = ['GME','NYMT']
return ticker
#creates list of urls for scrapet to grab
def ticker_site():
ticker_site = ['https://finance.yahoo.com/quote/'+x+'/options?p='+x for x in ticker()]
return ticker_site
optionRows = []
for i in range(len(ticker_site())):
optionRows.append([])
def ticker_gets():
option_page = ticker_site()
requested_page = requests.get(option_page[i])
ticker_soup = BeautifulSoup(requested_page.text,'html.parser')
return ticker_soup
def soup_search():
table = ticker_gets()
both_tables = table.find_all('table')
call_table = both_tables[0]
put_table= both_tables[1]
call_rows = call_table.find('tr')
put_rows = put_table.find('tr')
#makes the call table
for call in call_rows:
whole_call_table = call.find_all('td')
call_row = [y.text for y in whole_call_table]
optionRows[call].append(call_row)
#makes the put table
for put in put_rows:
whole_put_table = put.find_all('td')
put_row = [z.text for z in whole_put_table]
optionRows[put].append(put_row)
for i in range(len(optionRows)):
optionRows[i] = optionRows[i][1:len(optionRows[i])]
return optionRows
def getColumns(columnIndexes=[2, 4, 5]):
newList = []
for tickerIndex in range(len(soup_search())):
newList.append([])
indexCount = 0
for j in soup_search()[tickerIndex]:
newList[tickerIndex].append([])
for i in columnIndexes:
newList[tickerIndex][indexCount].append(j[i])
indexCount += 1
return newList
def csvOutputer():
rows = getColumns()
fields = ["Ticker", "Strike", "Bid", "Ask"]
with open('newcsv', 'w') as f:
write = csv.writer(f)
write.writerow(fields)
for i in range(len(ticker())):
for j in rows[i]:
j.insert(0, ticker()[i])
write.writerow(j)
csvOutputer()
I've trying to find a good way to accumulate errors in case when function returns two values (result, errors)
Example:
def do_something(inp):
"""just for example"""
if type(inp) is not str:
return None, 'inp is not string'
return inp.upper(), None
results_dict = {}
errors = []
r, e = do_something(handler1(inp1))
results_dict[inp1] = r
if e:
errors.append(r)
r, e = do_something(handler2(inp2))
results_dict[inp2] = r
if e:
errors.append(r)
# and so on
Is there the right way to accumulate errors in one string
like this pseudocode
errors = ErrorsAccumulator()
results_dict = {}
results_dict[inp1], errors = do_something(handler1(inp1))
results_dict[inp2], errors = do_something(handler2(inp2))
# and so on
I can't use loop for it because there is a lot different handlers
As I understand it, what you want to do is collect outputs and results in 2 different lists.
As it is written right now, the code will simply replace whatever the variable errors contains:
errors = ErrorsAccumulator()
results_dict = {}
results_dict[inp1], errors = do_something(handler1(inp1))
results_dict[inp2], errors = do_something(handler2(inp2))
Assuming that errors is a list, you could instead do this:
def unpack(t):
# this function will unpack the result and error of
# your calls to do_something
return t[0], [t[1]]
errors = []
results_dict = {}
results_dict[inp1], errors[len(errors):] = unpack(do_something(handler1(inp1)))
results_dict[inp2], errors[len(errors):] = unpack(do_something(handler2(inp2)))
However, that is not very clean (in my opinion). What I propose you do instead is:
inputs_results_errors = []
inputs_results_errors.append(inp1, *do_something(handler1(inp1)))
inputs_results_errors.append(inp1, *do_something(handler2(inp1)))
# append many more ...
# and then separate into lists or dicts
inputs, results, errors = zip(*inputs_results_errors)
What is the best way to look into how the loop works thourh iterations?
I have defined 2 functions which have to go in one after another (the 2nd gets the result of the 1st and works it through).
Ultimately I need 2-line pandas dataframe as the output.
Sample code below.
def candle_data (
figi,
int1 = candle_resolution,
from1 = previous_minutemark,
to1 = last_minutemark
):
response = market_api.market_candles_get(figi = ticker_figi_test, from_ = from1, to = to1, interval = int1)
if response.status_code == 200:
return response.parse_json().dict()
else:
return print(response.parse_error())
def response_to_pandas_df (response):
df_candles = pd.DataFrame(response['payload'])
df_candles = pd.json_normalize(df_candles['candles'])
df_candles = df_candles[df_candles['time'] >= previous_minutemark]
df_candles = df_candles[['c', 'figi','time']]
df_candles_main = df_candles_template.append(df_candles)
return df_candles_main
then I call the functions in a loop:
ticker_figi_list = ["BBG000CL9VN6", "BBG000R7Z112"]
df_candles_main = df_candles_template
for figi in ticker_figi_list:
response = candle_data(figi)
df_candles_main = response_to_pandas_df(response)
But in return I get only 1 row of data for the 1st FIGI in the list.
I suppose, that I define the candle_data() function with figi_ticker_test which contain only 1 value may be the case. But I'm not sure how to work this around.
Thank you in advance.
It looks like the problem is you are calling the api with figi = ticker_figi_test. I assume ticket_figi_test is equal to the first figi in your list and so you aren't actually calling the api with different figi on each iteration. Try changing to the following:
response = market_api.market_candles_get(figi = figi, from_ = from1, to = to1, interval = int1)
Im working on a small project of retrieving information about books from the Google Books API using Python 3. For this i make a call to the API, read out the variables and store those in a list. For a search like "linkedin" this works perfectly. However when i enter "Google", it reads the second title from the JSON input. How can this happen?
Please find my code below (Google_Results is the class I use to initialize the variables):
import requests
def Book_Search(search_term):
parms = {"q": search_term, "maxResults": 3}
r = requests.get(url="https://www.googleapis.com/books/v1/volumes", params=parms)
print(r.url)
results = r.json()
i = 0
for result in results["items"]:
try:
isbn13 = str(result["volumeInfo"]["industryIdentifiers"][0]["identifier"])
isbn10 = str(result["volumeInfo"]["industryIdentifiers"][1]["identifier"])
title = str(result["volumeInfo"]["title"])
author = str(result["volumeInfo"]["authors"])[2:-2]
publisher = str(result["volumeInfo"]["publisher"])
published_date = str(result["volumeInfo"]["publishedDate"])
description = str(result["volumeInfo"]["description"])
pages = str(result["volumeInfo"]["pageCount"])
genre = str(result["volumeInfo"]["categories"])[2:-2]
language = str(result["volumeInfo"]["language"])
image_link = str(result["volumeInfo"]["imageLinks"]["thumbnail"])
dict = Google_Results(isbn13, isbn10, title, author, publisher, published_date, description, pages, genre,
language, image_link)
gr.append(dict)
print(gr[i].title)
i += 1
except:
pass
return
gr = []
Book_Search("Linkedin")
I am a beginner to Python, so any help would be appreciated!
It does so because there is no publisher entry in volumeInfo of the first entry, thus it raises a KeyError and your except captures it. If you're going to work with fuzzy data you have to account for the fact that it will not always have the expected structure. For simple cases you can rely on dict.get() and its default argument to return a 'valid' default entry if an entry is missing.
Also, there are a few conceptual problems with your function - it relies on a global gr which is bad design, it shadows the built-in dict type and it captures all exceptions guaranteeing that you cannot exit your code even with a SIGINT... I'd suggest you to convert it to something a bit more sane:
def book_search(search_term, max_results=3):
results = [] # a list to store the results
parms = {"q": search_term, "maxResults": max_results}
r = requests.get(url="https://www.googleapis.com/books/v1/volumes", params=parms)
try: # just in case the server doesn't return valid JSON
for result in r.json().get("items", []):
if "volumeInfo" not in result: # invalid entry - missing volumeInfo
continue
result_dict = {} # a dictionary to store our discovered fields
result = result["volumeInfo"] # all the data we're interested is in volumeInfo
isbns = result.get("industryIdentifiers", None) # capture ISBNs
if isinstance(isbns, list) and isbns:
for i, t in enumerate(("isbn10", "isbn13")):
if len(isbns) > i and isinstance(isbns[i], dict):
result_dict[t] = isbns[i].get("identifier", None)
result_dict["title"] = result.get("title", None)
authors = result.get("authors", None) # capture authors
if isinstance(authors, list) and len(authors) > 2: # you're slicing from 2
result_dict["author"] = str(authors[2:-2])
result_dict["publisher"] = result.get("publisher", None)
result_dict["published_date"] = result.get("publishedDate", None)
result_dict["description"] = result.get("description", None)
result_dict["pages"] = result.get("pageCount", None)
genres = result.get("authors", None) # capture genres
if isinstance(genres, list) and len(genres) > 2: # since you're slicing from 2
result_dict["genre"] = str(genres[2:-2])
result_dict["language"] = result.get("language", None)
result_dict["image_link"] = result.get("imageLinks", {}).get("thumbnail", None)
# make sure Google_Results accepts keyword arguments like title, author...
# and make them optional as they might not be in the returned result
gr = Google_Results(**result_dict)
results.append(gr) # add it to the results list
except ValueError:
return None # invalid response returned, you may raise an error instead
return results # return the results
Then you can easily retrieve as much info as possible for a term:
gr = book_search("Google")
And it will be far more tolerant of data omissions, provided that your Google_Results type makes most of the entries optional.
Following #Coldspeed's recommendation it became clear that missing information in the JSON file caused the exception to run. Since I only had a "pass" statement there it skipped the entire result. Therefore I will have to adapt the "Try and Except" statements so errors do get handled properly.
Thanks for the help guys!