Related
I wanted to reach out for any advice on normalizing JSON within brackets. I have made several attempts with json_normalize and also a user defined function to denest any JSON which has worked effortlessly until now (credit goes to Serge de Gosson de Varennes for sharing, thank you!) but I keep running into errors or the user defined functon does not flatten the json data below.
Here is the CSV examples
total_count,start_index,items,items_per_page,filing_history_status
14750,0,"[{'category': 'capital', 'date': '2023-01-18', 'description': 'capital-return-purchase-own-shares', 'links': {'self': '/company/00102498/filing-history/MzM2NjI2NjAwOGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/KFIPMRklI9VLmGnwmLTteT757IXrdIXEweOGPopDAtc'}, 'paper_filed': True, 'type': 'SH03', 'annotations': [{'annotation': 'Clarification hmrc confirmation duty paid', 'category': 'annotation', 'date': '2023-01-18', 'description': 'annotation', 'description_values': {'description': 'Clarification hmrc confirmation duty paid'}, 'type': 'ANNOTATION'}], 'pages': 3, 'barcode': 'ABVEZEGG', 'transaction_id': 'MzM2NjI2NjAwOGFkaXF6a2N4'}, {'category': 'capital', 'date': '2023-01-18', 'description': 'capital-return-purchase-own-shares', 'links': {'self': '/company/00102498/filing-history/MzM2NjI2NjAwNGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/Vnn3SQ93Bjt1i9j0bbte52sVb9GA5gbT9P7J8RRy35I'}, 'paper_filed': True, 'type': 'SH03', 'pages': 5, 'barcode': 'ABVEZEEG', 'transaction_id': 'MzM2NjI2NjAwNGFkaXF6a2N4'}, {'action_date': '2023-01-10', 'category': 'capital', 'date': '2023-01-17', 'description': 'capital-sale-or-transfer-treasury-shares-with-date-currency-capital-figure', 'description_values': {'capital': [{'currency': 'USD', 'figure': '251,229,378', 'date': '2023-01-10'}], 'date': '2023-01-10'}, 'links': {'self': '/company/00102498/filing-history/MzM2NTYyMDc0NmFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/kn0cgguEz1QENk-MUsFXRcLUSaUkcc2ds4dLJxRVLDg'}, 'paper_filed': True, 'type': 'SH04', 'pages': 2, 'barcode': 'YBV5MT5S', 'transaction_id': 'MzM2NTYyMDc0NmFkaXF6a2N4'}, {'action_date': '2022-12-23', 'category': 'capital', 'date': '2023-01-17', 'description': 'capital-sale-or-transfer-treasury-shares-with-date-currency-capital-figure', 'description_values': {'capital': [{'currency': 'USD', 'figure': '251,022,732.5', 'date': '2022-12-23'}], 'date': '2022-12-23'}, 'links': {'self': '/company/00102498/filing-history/MzM2NTYyMTIxMmFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/9P23zv_wuYQkXdRMEbyDv5wbLM9b06hqGtfOXGyeHPE'}, 'paper_filed': True, 'type': 'SH04', 'pages': 2, 'barcode': 'YBV5MVAI', 'transaction_id': 'MzM2NTYyMTIxMmFkaXF6a2N4'}, {'category': 'capital', 'date': '2023-01-12', 'description': 'capital-return-purchase-own-shares', 'links': {'self': '/company/00102498/filing-history/MzM2NTQzMzkxMmFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/EgbaDfsLiwXqwi-w5Hus-QKPdcI1kYmjrTOoLJmiI_4'}, 'paper_filed': True, 'type': 'SH03', 'annotations': [{'annotation': 'Clarification HMRC confirmation received that appropriate duty has been paid on this repurchase.', 'category': 'annotation', 'date': '2023-01-12', 'description': 'annotation', 'description_values': {'description': 'Clarification hmrc confirmation received that appropriate duty has been paid on this repurchase.'}, 'type': 'ANNOTATION'}], 'pages': 3, 'barcode': 'ABUTWG01', 'transaction_id': 'MzM2NTQzMzkxMmFkaXF6a2N4'}, {'category': 'capital', 'date': '2023-01-12', 'description': 'capital-return-purchase-own-shares', 'links': {'self': '/company/00102498/filing-history/MzM2NTQzMzkxMWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/t0ey0FXW0kaaTi9P8Ps-5zM95IG3iTkM3fTHJK2b6dI'}, 'paper_filed': True, 'type': 'SH03', 'annotations': [{'annotation': 'Clarification HMRC confirmation received that appropriate duty has been paid on this repurchase.', 'category': 'annotation', 'date': '2023-01-12', 'description': 'annotation', 'description_values': {'description': 'Clarification hmrc confirmation received that appropriate duty has been paid on this repurchase.'}, 'type': 'ANNOTATION'}], 'pages': 3, 'barcode': 'ABUTWFZU', 'transaction_id': 'MzM2NTQzMzkxMWFkaXF6a2N4'}, {'action_date': '2022-12-22', 'category': 'capital', 'date': '2023-01-09', 'description': 'capital-allotment-shares', 'description_values': {'date': '2022-12-22', 'capital': [{'figure': '12,706,252', 'currency': 'GBP'}, {'currency': 'USD', 'figure': '4,760,695,780.25'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2NTAyOTczNGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/XJE1I-VE224cxMQKzzJFSs6hTPS0X0edMLFe2y8ViEU'}, 'type': 'SH01', 'pages': 4, 'barcode': 'XBUUT623', 'transaction_id': 'MzM2NTAyOTczNGFkaXF6a2N4'}, {'action_date': '2022-10-20', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-10-20', 'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'figure': '4,866,938,331.50', 'currency': 'USD'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMTgwMWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/WOIUjTKXXPVduvRy3kkNfq6jiwU6fTZPmFAyq9Tc6k8'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXNZF', 'transaction_id': 'MzM2MzUwMTgwMWFkaXF6a2N4'}, {'action_date': '2022-11-07', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-11-07', 'capital': [{'figure': '12,706,252', 'currency': 'GBP'}, {'figure': '4,841,613,617', 'currency': 'USD'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjMzN2FkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/vGcWIhrp2T51Xjwvh-DxGZG2s-qX0pee3-uyvVdhq-s'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXQ7U', 'transaction_id': 'MzM2MzUwMjMzN2FkaXF6a2N4'}, {'action_date': '2022-11-08', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'figure': '4,839,188,723.50', 'currency': 'USD'}], 'date': '2022-11-08'}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjMzOGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/hX4_Z1O_rV2y6hrqQrxs9qIWNOhTwBlDajSien0XGjM'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXQ83', 'transaction_id': 'MzM2MzUwMjMzOGFkaXF6a2N4'}, {'action_date': '2022-10-26', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'figure': '4,855,749,979.50', 'currency': 'USD'}], 'date': '2022-10-26'}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjE0MWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/esS-R31w94VMTfbEvz1ak5wBWK-dZta5rzlTu6BMTQc'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXPGJ', 'transaction_id': 'MzM2MzUwMjE0MWFkaXF6a2N4'}, {'action_date': '2022-10-28', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-10-28', 'capital': [{'figure': '12,706,252', 'currency': 'GBP'}, {'currency': 'USD', 'figure': '4,850,257,261'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjE0NWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/tvtkhuXRyEhqol5f-NDu1b5D93tykfMTfIJQDWr_iyE'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXPBU', 'transaction_id': 'MzM2MzUwMjE0NWFkaXF6a2N4'}, {'action_date': '2022-11-04', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-11-04', 'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'currency': 'USD', 'figure': '4,843,953,910.25'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjE0OWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/rJTbehtmiovQhzX625TilsiLBPLN0BtTQcQQPQyTz2g'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXPCI', 'transaction_id': 'MzM2MzUwMjE0OWFkaXF6a2N4'}, {'action_date': '2022-10-31', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'capital': [{'figure': '12,706,252', 'currency': 'GBP'}, {'figure': '4,847,654,519', 'currency': 'USD'}], 'date': '2022-10-31'}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjE0OGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/vnZ4v6xLe61-aj509lCrSZyrciGWfd3JYArL0pJk-h0'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXPC2', 'transaction_id': 'MzM2MzUwMjE0OGFkaXF6a2N4'}, {'action_date': '2022-10-25', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-10-25', 'capital': [{'figure': '12,706,252', 'currency': 'GBP'}, {'currency': 'USD', 'figure': '4,858,505,435.25'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjAwNmFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/NBMwIr_YjIXCPunxhAZDmZyKXFnMgvC-UCbEPIQYkQA'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXOMA', 'transaction_id': 'MzM2MzUwMjAwNmFkaXF6a2N4'}, {'action_date': '2022-11-03', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'currency': 'USD', 'figure': '4,846,406,664.25'}], 'date': '2022-11-03'}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjE0N2FkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/JlVWlMQwiT0uw0GsFDi5N3l6YuY6jpLMIIve7Sdxs7c'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXPCA', 'transaction_id': 'MzM2MzUwMjE0N2FkaXF6a2N4'}, {'action_date': '2022-10-27', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'figure': '4,852,987,329', 'currency': 'USD'}], 'date': '2022-10-27'}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjE0M2FkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/pA1ufNYMqxg1IqjVXcFx1v2aVCbu8Tt3HYP2jRezWRc'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXPGR', 'transaction_id': 'MzM2MzUwMjE0M2FkaXF6a2N4'}, {'action_date': '2022-10-21', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'capital': [{'figure': '12,706,252', 'currency': 'GBP'}, {'figure': '4,864,093,329.75', 'currency': 'USD'}], 'date': '2022-10-21'}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjAwM2FkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/RXHig4q8GhyTAm8n6IQ2adQsYVK6tHOFwvx9j8KJ6dM'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXOKI', 'transaction_id': 'MzM2MzUwMjAwM2FkaXF6a2N4'}, {'action_date': '2022-10-24', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'currency': 'USD', 'figure': '4,861,296,627.75'}], 'date': '2022-10-24'}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMjAwNWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/Xab3kKnqCQez6IkyCNR98_krrmTSNuR5usQ7ECT9oLY'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXOM2', 'transaction_id': 'MzM2MzUwMjAwNWFkaXF6a2N4'}, {'action_date': '2022-10-14', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'figure': '4,878,250,683.50', 'currency': 'USD'}], 'date': '2022-10-14'}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMTY0MWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/J3Zv7zI7JWvJJX5SjVruBwthLEgx6KfM7PqRC_ahLnU'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXN37', 'transaction_id': 'MzM2MzUwMTY0MWFkaXF6a2N4'}, {'action_date': '2022-10-18', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-10-18', 'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'figure': '4,872,592,202.25', 'currency': 'USD'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMTc5OWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/Bpg0KCk-scjaq6aGkI_XXEi7Hq-ia34Yevn0AZN-x9s'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXNVV', 'transaction_id': 'MzM2MzUwMTc5OWFkaXF6a2N4'}, {'action_date': '2022-10-12', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-10-12', 'capital': [{'figure': '12,706,252', 'currency': 'GBP'}, {'figure': '4,884,232,443.25', 'currency': 'USD'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMTYzN2FkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/rtHP4aZ76T2H9ufCF5saoxCTydKp9x-6b62xuyLTLeM'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXN5F', 'transaction_id': 'MzM2MzUwMTYzN2FkaXF6a2N4'}, {'action_date': '2022-10-19', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-10-19', 'capital': [{'figure': '12,706,252', 'currency': 'GBP'}, {'figure': '4,869,745,587.75', 'currency': 'USD'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMTgwMGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/8wMFoih73gPzsW3cKAEHxpPHcNUUfWRc7_jdJCvJTo4'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXO34', 'transaction_id': 'MzM2MzUwMTgwMGFkaXF6a2N4'}, {'action_date': '2022-10-13', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-10-13', 'capital': [{'figure': '12,706,252', 'currency': 'GBP'}, {'figure': '4,881,240,830.50', 'currency': 'USD'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMTYzOGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/ugvcHhgmk9x3IthyitanO5Vfk3wRf7q5V366uuMhfcU'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXN6Z', 'transaction_id': 'MzM2MzUwMTYzOGFkaXF6a2N4'}, {'action_date': '2022-10-17', 'category': 'capital', 'date': '2023-01-03', 'description': 'capital-cancellation-shares', 'description_values': {'date': '2022-10-17', 'capital': [{'currency': 'GBP', 'figure': '12,706,252'}, {'figure': '4,875,370,334', 'currency': 'USD'}]}, 'links': {'self': '/company/00102498/filing-history/MzM2MzUwMTY0MmFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/7qNdyy_cXHh2bD_kNvz6Tx4CCdLrjqZVuIdCo-ytSx0'}, 'paper_filed': True, 'type': 'SH06', 'pages': 8, 'barcode': 'YBJEXNC8', 'transaction_id': 'MzM2MzUwMTY0MmFkaXF6a2N4'}]",25,filing-history-available
12,0,"[{'action_date': '2022-09-23', 'category': 'confirmation-statement', 'date': '2022-09-30', 'description': 'confirmation-statement-with-no-updates', 'description_values': {'made_up_date': '2022-09-23'}, 'links': {'self': '/company/12098424/filing-history/MzM1MzcxMDIyOWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/wm97CdEs1DD7j6mZuW5lQg1Z9c0OINd9yTIotgAhqtI'}, 'type': 'CS01', 'pages': 3, 'barcode': 'XBDNG7WZ', 'transaction_id': 'MzM1MzcxMDIyOWFkaXF6a2N4'}, {'action_date': '2021-07-31', 'category': 'accounts', 'date': '2022-04-29', 'description': 'accounts-with-accounts-type-total-exemption-full', 'description_values': {'made_up_date': '2021-07-31'}, 'links': {'self': '/company/12098424/filing-history/MzMzNzg3MDQyMGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/4tOH4j8gi_Y20nCkuVkboyhIS6Lzj9hp-5AjXvRCmfE'}, 'type': 'AA', 'pages': 9, 'barcode': 'XB2XP2YY', 'transaction_id': 'MzMzNzg3MDQyMGFkaXF6a2N4'}, {'action_date': '2021-09-23', 'category': 'confirmation-statement', 'date': '2021-10-06', 'description': 'confirmation-statement-with-no-updates', 'description_values': {'made_up_date': '2021-09-23'}, 'links': {'self': '/company/12098424/filing-history/MzMxNjI0Njc4OGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/gejKaHDsYPaEReCs17xJu0zS4PNfMHP43FkjygYeslA'}, 'type': 'CS01', 'pages': 3, 'barcode': 'XAEKJKRD', 'transaction_id': 'MzMxNjI0Njc4OGFkaXF6a2N4'}, {'action_date': '2020-07-31', 'category': 'accounts', 'date': '2021-04-30', 'description': 'accounts-with-accounts-type-total-exemption-full', 'description_values': {'made_up_date': '2020-07-31'}, 'links': {'self': '/company/12098424/filing-history/MzI5OTY1MDE0NGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/G42SZuUKIYaMwqBTv4WFQ_46ITwTCso8JoMQmQBZ-qw'}, 'type': 'AA', 'pages': 9, 'barcode': 'XA3HJGOR', 'transaction_id': 'MzI5OTY1MDE0NGFkaXF6a2N4'}, {'action_date': '2020-09-23', 'category': 'confirmation-statement', 'date': '2020-10-09', 'description': 'confirmation-statement-with-no-updates', 'description_values': {'made_up_date': '2020-09-23'}, 'links': {'self': '/company/12098424/filing-history/MzI4MDA5MDQxMmFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/RpF_HoJZui0F68sDrMM3H6hWW66MIevkwtRsGsRoVYg'}, 'type': 'CS01', 'pages': 3, 'barcode': 'X9FAGITS', 'transaction_id': 'MzI4MDA5MDQxMmFkaXF6a2N4'}, {'type': 'CS01', 'action_date': '2019-09-23', 'description_values': {'made_up_date': '2019-09-23'}, 'links': {'self': '/company/12098424/filing-history/MzI0NDg5NDUxM2FkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/Ekb8A2MZsDtRU7tiVXYx5sXLXuEbhf52kYZ8PdNz9lI'}, 'category': 'confirmation-statement', 'date': '2019-09-23', 'description': 'confirmation-statement-with-updates', 'pages': 4, 'barcode': 'X8EMUH23', 'transaction_id': 'MzI0NDg5NDUxM2FkaXF6a2N4'}, {'type': 'PSC01', 'subcategory': 'notifications', 'action_date': '2019-09-10', 'description_values': {'psc_name': 'Bartosz Pazik', 'notification_date': '2019-09-10'}, 'category': 'persons-with-significant-control', 'links': {'self': '/company/12098424/filing-history/MzI0NDg5MzYzNmFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/rAlwZsYfyqfYAV4n_TlFX3c5hqyXmBnKakzKhDL3QL0'}, 'date': '2019-09-23', 'description': 'notification-of-a-person-with-significant-control', 'pages': 2, 'barcode': 'X8EMUDUJ', 'transaction_id': 'MzI0NDg5MzYzNmFkaXF6a2N4'}, {'description_values': {'officer_name': 'Mr Bartosz Pazik', 'appointment_date': '2019-09-10'}, 'category': 'officers', 'links': {'self': '/company/12098424/filing-history/MzI0NDg5MjYxOWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/tK86XekvMQmckMA9Xs8tE9yaBKnCyr0GxrrAcmGUYcU'}, 'date': '2019-09-23', 'description': 'appoint-person-director-company-with-name-date', 'subcategory': 'appointments', 'type': 'AP01', 'action_date': '2019-09-10', 'pages': 2, 'barcode': 'X8EMU9L6', 'transaction_id': 'MzI0NDg5MjYxOWFkaXF6a2N4'}, {'type': 'TM01', 'subcategory': 'termination', 'action_date': '2019-09-10', 'description_values': {'termination_date': '2019-09-10', 'officer_name': 'Lukasz Wojewodzic'}, 'links': {'self': '/company/12098424/filing-history/MzI0NDg5MDc0NGFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/RANnOGoDeSbMP2kqvYY-fh2OyyUfGLqmIZfbgl215oE'}, 'category': 'officers', 'date': '2019-09-23', 'description': 'termination-director-company-with-name-termination-date', 'pages': 1, 'barcode': 'X8EMU1SY', 'transaction_id': 'MzI0NDg5MDc0NGFkaXF6a2N4'}, {'description_values': {'cessation_date': '2019-09-09', 'psc_name': 'Ducers Ltd'}, 'category': 'persons-with-significant-control', 'links': {'self': '/company/12098424/filing-history/MzI0NDg5MDU2OWFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/HdTB7gx7881GBsfsDkfFTEbWnL5kSpnXrtk7k8Fqgz8'}, 'date': '2019-09-23', 'description': 'cessation-of-a-person-with-significant-control', 'subcategory': 'termination', 'type': 'PSC07', 'action_date': '2019-09-09', 'pages': 1, 'barcode': 'X8EMU1CW', 'transaction_id': 'MzI0NDg5MDU2OWFkaXF6a2N4'}, {'description': 'resolution', 'subcategory': 'certificate', 'type': 'RESOLUTIONS', 'description_values': {'description': 'Resolutions'}, 'date': '2019-09-21', 'links': {'self': '/company/12098424/filing-history/MzI0NDgzOTExNmFkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/1RFExUTiTyve6UBTLFym0OQqNVSQ2hecc_5EMI0Z7TU'}, 'category': 'resolution', 'associated_filings': [{'category': 'change-of-name', 'date': '2019-09-21', 'description': 'change-of-name-by-resolution', 'type': 'NM01'}], 'resolutions': [{'category': 'change-of-name', 'description': 'resolution-change-of-name', 'description_values': {'resolution_date': '2019-09-20'}, 'subcategory': 'resolution', 'type': 'RES15'}], 'pages': 3, 'barcode': 'X8EF5GMI', 'transaction_id': 'MzI0NDgzOTExNmFkaXF6a2N4'}, {'links': {'self': '/company/12098424/filing-history/MzIzOTA5NTI0M2FkaXF6a2N4', 'document_metadata': 'https://frontend-doc-api.company-information.service.gov.uk/document/nICiy7Oydb83beqjy1DHvNzoFSFLShsXcDaDnWLt3Cs'}, 'associated_filings': [{'action_date': 1562803200000, 'category': 'capital', 'date': '2019-07-11', 'description': 'statement-of-capital', 'description_values': {'capital': [{'currency': 'GBP', 'figure': '1'}], 'date': '2019-07-11'}, 'type': 'SH01'}, {'category': 'incorporation', 'date': '2019-07-11', 'description': 'model-articles-adopted', 'type': 'MODEL ARTICLES'}], 'date': '2019-07-11', 'category': 'incorporation', 'description': 'incorporation-company', 'type': 'NEWINC', 'pages': 10, 'barcode': 'X89I45EQ', 'transaction_id': 'MzIzOTA5NTI0M2FkaXF6a2N4'}]",25,filing-history-available
The csv is read via Pandas and then I try to flatten it.
import pandas as pd
#credit for this amazing function below and work goes to Serge de Gosson de Varennes
def flatten_nested_json_df(df):
df = df.reset_index()
s = (df.applymap(type) == list).all()
list_columns = s[s].index.tolist()
s = (df.applymap(type) == dict).all()
dict_columns = s[s].index.tolist()
while len(list_columns) > 0 or len(dict_columns) > 0:
new_columns = []
for col in dict_columns:
exploded = pd.json_normalize(df[col]).add_prefix(f'{col}.')
exploded.index = df.index
df = pd.concat([df, exploded], axis=1).drop(columns=[col])
new_columns.extend(exploded.columns) # inplace
for col in list_columns:
# print(f"exploding: {col}")
df = df.drop(columns=[col]).join(df[col].explode().to_frame())
new_columns.append(col)
s = (df[new_columns].applymap(type) == list).all()
list_columns = s[s].index.tolist()
s = (df[new_columns].applymap(type) == dict).all()
dict_columns = s[s].index.tolist()
return df
df1 = pd.read_csv('stackexample.csv')
df1.head() # screenshot highlights data before flattening attempt
Data before attempting to flatten JSON
#attempt at flattening
df2 = flatten_nested_json_df(df1)
df2.head()
Data after attempt to flatten JSON
Thank you for any advice on this matter! I have been stuck on this issue for many days looking at alternatives
Thank you for any suggestions
The function you are using is working, however it needs the data to be in the correct format. The data in column items is in string format, you need to convert it back to data structure (lists and dictionaries).
# https://www.geeksforgeeks.org/python-convert-a-string-representation-of-list-into-list/
from ast import literal_eval
df["items"] = df["items"].apply(literal_eval)
So just use the function in the DataFrame normally.
flatten_nested_json_df(df)
Output:
I have the following info stored in a dict called "info1":
[{'symbol': 'ETHUSDT', 'orderId': 480977, 'orderListId': -1, 'clientOrderId': 'S3', 'price': '0.00000000', 'origQty': '0.02000000', 'executedQty': '0.02000000', 'cummulativeQuoteQty': '25.07260000', 'status': 'FILLED', 'timeInForce': 'GTC', 'type': 'MARKET', 'side': 'BUY', 'stopPrice': '0.00000000', 'icebergQty': '0.00000000', 'time': 1672948804160, 'updateTime': 1672948804160, 'isWorking': True, 'workingTime': 1672948804160, 'origQuoteOrderQty': '0.00000000', 'selfTradePreventionMode': 'NONE'}, {'symbol': 'ETHUSDT', 'orderId': 566634, 'orderListId': -1, 'clientOrderId': 'S3', 'price': '0.00000000', 'origQty': '0.02000000', 'executedQty': '0.02000000', 'cummulativeQuoteQty': '24.98800000', 'status': 'FILLED', 'timeInForce': 'GTC', 'type': 'MARKET', 'side': 'SELL', 'stopPrice': '0.00000000', 'icebergQty': '0.00000000', 'time': 1672971274007, 'updateTime': 1672971274007, 'isWorking': True, 'workingTime': 1672971274007, 'origQuoteOrderQty': '0.00000000', 'selfTradePreventionMode': 'NONE'}]
How would I go about updating the values for both 'time' keys in this dict?
Have tried to identify the index of each, without luck.
First off, you are not dealing with a dictionary, but a list of dictionaries.
isinstance(info1, list)
#True
That is why you can simply try to fetch a key and make an update here. First, you have to iterate through the list of dictionaries and then use dict.update to update specific key values. Here is how you can do this with a simple loop -
new_times = [1672948804190, 1672971274120] #one for each dict in list
for d,t in zip(info1, new_times):
d.update({'time':t})
print(info1)
[{'symbol': 'ETHUSDT',
'orderId': 480977,
'orderListId': -1,
'clientOrderId': 'S3',
'price': '0.00000000',
'origQty': '0.02000000',
'executedQty': '0.02000000',
'cummulativeQuoteQty': '25.07260000',
'status': 'FILLED',
'timeInForce': 'GTC',
'type': 'MARKET',
'side': 'BUY',
'stopPrice': '0.00000000',
'icebergQty': '0.00000000',
'time': 1672948804190, #<---------
'updateTime': 1672948804160,
'isWorking': True,
'workingTime': 1672948804160,
'origQuoteOrderQty': '0.00000000',
'selfTradePreventionMode': 'NONE'},
{'symbol': 'ETHUSDT',
'orderId': 566634,
'orderListId': -1,
'clientOrderId': 'S3',
'price': '0.00000000',
'origQty': '0.02000000',
'executedQty': '0.02000000',
'cummulativeQuoteQty': '24.98800000',
'status': 'FILLED',
'timeInForce': 'GTC',
'type': 'MARKET',
'side': 'SELL',
'stopPrice': '0.00000000',
'icebergQty': '0.00000000',
'time': 1672971274120, #<---------
'updateTime': 1672971274007,
'isWorking': True,
'workingTime': 1672971274007,
'origQuoteOrderQty': '0.00000000',
'selfTradePreventionMode': 'NONE'}]
I'm receiving data from FTX websocket api.
wsEndpoint ='wss://ftx.com/ws/'
msg = json.dumps({'op': 'subscribe', 'channel': 'trades', 'market': 'ETH-PERP'})
def on_open(ws):
ws.send(msg)
def on_message(ws, message):
trades = json.loads(message)['data']
print(trades)
ws = websocket.WebSocketApp(wsEndpoint, on_message=on_message, on_open=on_open)
ws.run_forever()
Seems like it generates a new list whenever it sends back data.
[{'id': 5124414968, 'price': 1277.6, 'size': 0.03, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:24.362764+00:00'}]
[{'id': 5124415008, 'price': 1277.7, 'size': 0.168, 'side': 'buy', 'liquidation': False, 'time': '2022-10-11T03:44:25.501504+00:00'}]
[{'id': 5124415161, 'price': 1277.6, 'size': 1.321, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415162, 'price': 1277.6, 'size': 0.05, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415163, 'price': 1277.6, 'size': 0.006, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415164, 'price': 1277.6, 'size': 0.2, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415165, 'price': 1277.6, 'size': 0.001, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415166, 'price': 1277.6, 'size': 0.152, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415167, 'price': 1277.6, 'size': 4.29, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415168, 'price': 1277.6, 'size': 7.087, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415169, 'price': 1277.6, 'size': 2.6, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415170, 'price': 1277.6, 'size': 3.02, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415171, 'price': 1277.6, 'size': 2.6, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415172, 'price': 1277.6, 'size': 0.133, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415173, 'price': 1277.6, 'size': 0.04, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415174, 'price': 1277.6, 'size': 0.652, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}, {'id': 5124415175, 'price': 1277.6, 'size': 9.42, 'side': 'sell', 'liquidation': False, 'time': '2022-10-11T03:44:28.612715+00:00'}]
How can I put them into one list or dataframe so I can plot them? Is it to assign each new list a variable with loop and store them in an empty list? And maybe it is a dumb question to ask, but is it possible to put a for loop in a list?
I have this tuple with nested dictionaries:
Orders = ({'ret_code': 0, 'ret_msg': 'OK', 'ext_code': '', 'ext_info': '', 'result': [{'user_id': 121212, 'stop_order_id': 'ssdf3215641d215415d', 'symbol': 'XRPUSDT', 'side': 'Sell', 'order_type': 'Market', 'price': 0, 'qty': 2, 'time_in_force': 'ImmediateOrCancel', 'order_status': 'Untriggered', 'trigger_price': 0.8839, 'base_price': '0.8860', 'order_link_id': '', 'created_time': '2021-06-14T14:21:50.000Z', 'updated_time': '2021-06-14T14:21:50.000Z', 'take_profit': 0, 'stop_loss': 0, 'tp_trigger_by': 'UNKNOWN', 'sl_trigger_by': 'UNKNOWN', 'trigger_by': 'LastPrice', 'reduce_only': False, 'close_on_trigger': False}, {'user_id': 121212, 'stop_order_id': 's65d4654sd234165sd4', 'symbol': 'XRPUSDT', 'side': 'Buy', 'order_type': 'Market', 'price': 0, 'qty': 2, 'time_in_force': 'ImmediateOrCancel', 'order_status': 'Untriggered', 'trigger_price': 0.8929, 'base_price': '0.8820', 'order_link_id': '', 'created_time': '2021-06-14T14:03:20.000Z', 'updated_time': '2021-06-14T14:18:59.000Z', 'take_profit': 0, 'stop_loss': 0, 'tp_trigger_by': 'UNKNOWN', 'sl_trigger_by': 'UNKNOWN', 'trigger_by': 'LastPrice', 'reduce_only': True, 'close_on_trigger': True}, {'user_id': 121212, 'stop_order_id': 'sd654s6d5465416ds546s5d4', 'symbol': 'XRPUSDT', 'side': 'Sell', 'order_type': 'Market', 'price': 0, 'qty': 3, 'time_in_force': 'ImmediateOrCancel', 'order_status': 'Untriggered', 'trigger_price': 0.88, 'base_price': '0.8817', 'order_link_id': '', 'created_time': '2021-06-14T14:03:04.000Z', 'updated_time': '2021-06-14T14:18:59.000Z', 'take_profit': 0, 'stop_loss': 0, 'tp_trigger_by': 'UNKNOWN', 'sl_trigger_by': 'UNKNOWN', 'trigger_by': 'LastPrice', 'reduce_only': True, 'close_on_trigger': True}, {'user_id': 121212, 'stop_order_id': 's65d46s5d46s5d46s5d46sd5', 'symbol': 'XRPUSDT', 'side': 'Buy', 'order_type': 'Market', 'price': 0, 'qty': 2, 'time_in_force': 'ImmediateOrCancel', 'order_status': 'Untriggered', 'trigger_price': 0.88, 'base_price': '0.8839', 'order_link_id': '', 'created_time': '2021-06-14T13:58:30.000Z', 'updated_time': '2021-06-14T14:18:59.000Z', 'take_profit': 0, 'stop_loss': 0, 'tp_trigger_by': 'UNKNOWN', 'sl_trigger_by': 'UNKNOWN', 'trigger_by': 'LastPrice', 'reduce_only': True, 'close_on_trigger': True}, {'user_id': 121212, 'stop_order_id': 'sd654s6d54s6d51s6d54s65d4', 'symbol': 'XRPUSDT', 'side': 'Sell', 'order_type': 'Market', 'price': 0, 'qty': 3, 'time_in_force': 'ImmediateOrCancel', 'order_status': 'Untriggered', 'trigger_price': 0.8929, 'base_price': '0.8855', 'order_link_id': '', 'created_time': '2021-06-14T13:49:20.000Z', 'updated_time': '2021-06-14T14:18:59.000Z', 'take_profit': 0, 'stop_loss': 0, 'tp_trigger_by': 'UNKNOWN', 'sl_trigger_by': 'UNKNOWN', 'trigger_by': 'LastPrice', 'reduce_only': True, 'close_on_trigger': True}], 'time_now': '1623682473.336069', 'rate_limit_status': 598, 'rate_limit_reset_ms': 1623682473333, 'rate_limit': 600}, <bravado.requests_client.RequestsResponseAdapter object at 0x00789C86BBFA0>)
I want to search every dictionary for:
'reduce_only': False, 'close_on_trigger': False
If available print (True) and get the 'side' for exactly this dictionary:
True
Sell
The problem is that the dictionary changes depending on the orders
It means that there can be two or three dictionaries in the list
I tried to use For Loop but it only prints variables and keys
for x in Orders:
for y in x:
print(x)
Your data doesn't look like a proper Python data.
As for the dictionaries, I see no problem:
orders = ({
'ret_code': 0,
'ret_msg': 'OK',
'ext_code': '',
'ext_info': '',
'result': [{
'user_id': 121212,
'stop_order_id': 'ssdf3215641d215415d',
'symbol': 'XRPUSDT',
'side': 'Sell',
'order_type': 'Market',
'price': 0,
'qty': 2,
'time_in_force': 'ImmediateOrCancel',
'order_status': 'Untriggered',
'trigger_price': 0.8839,
'base_price': '0.8860',
'order_link_id': '',
'created_time': '2021-06-14T14: 21: 50.000Z',
'updated_time': '2021-06-14T14: 21: 50.000Z',
'take_profit': 0,
'stop_loss': 0,
'tp_trigger_by': 'UNKNOWN',
'sl_trigger_by': 'UNKNOWN',
'trigger_by': 'LastPrice',
'reduce_only': False,
'close_on_trigger': False
},
{
'user_id': 121212,
'stop_order_id': 's65d4654sd234165sd4',
'symbol': 'XRPUSDT',
'side': 'Buy',
'order_type': 'Market',
'price': 0,
'qty': 2,
'time_in_force': 'ImmediateOrCancel',
'order_status': 'Untriggered',
'trigger_price': 0.8929,
'base_price': '0.8820',
'order_link_id': '',
'created_time': '2021-06-14T14: 03: 20.000Z',
'updated_time': '2021-06-14T14: 18: 59.000Z',
'take_profit': 0,
'stop_loss': 0,
'tp_trigger_by': 'UNKNOWN',
'sl_trigger_by': 'UNKNOWN',
'trigger_by': 'LastPrice',
'reduce_only': True,
'close_on_trigger': True
},
{
'user_id': 121212,
'stop_order_id': 'sd654s6d5465416ds546s5d4',
'symbol': 'XRPUSDT',
'side': 'Sell',
'order_type': 'Market',
'price': 0,
'qty': 3,
'time_in_force': 'ImmediateOrCancel',
'order_status': 'Untriggered',
'trigger_price': 0.88,
'base_price': '0.8817',
'order_link_id': '',
'created_time': '2021-06-14T14: 03: 04.000Z',
'updated_time': '2021-06-14T14: 18: 59.000Z',
'take_profit': 0,
'stop_loss': 0,
'tp_trigger_by': 'UNKNOWN',
'sl_trigger_by': 'UNKNOWN',
'trigger_by': 'LastPrice',
'reduce_only': True,
'close_on_trigger': True
},
{
'user_id': 121212,
'stop_order_id': 's65d46s5d46s5d46s5d46sd5',
'symbol': 'XRPUSDT',
'side': 'Buy',
'order_type': 'Market',
'price': 0,
'qty': 2,
'time_in_force': 'ImmediateOrCancel',
'order_status': 'Untriggered',
'trigger_price': 0.88,
'base_price': '0.8839',
'order_link_id': '',
'created_time': '2021-06-14T13: 58: 30.000Z',
'updated_time': '2021-06-14T14: 18: 59.000Z',
'take_profit': 0,
'stop_loss': 0,
'tp_trigger_by': 'UNKNOWN',
'sl_trigger_by': 'UNKNOWN',
'trigger_by': 'LastPrice',
'reduce_only': True,
'close_on_trigger': True
},
{
'user_id': 121212,
'stop_order_id': 'sd654s6d54s6d51s6d54s65d4',
'symbol': 'XRPUSDT',
'side': 'Sell',
'order_type': 'Market',
'price': 0,
'qty': 3,
'time_in_force': 'ImmediateOrCancel',
'order_status': 'Untriggered',
'trigger_price': 0.8929,
'base_price': '0.8855',
'order_link_id': '',
'created_time': '2021-06-14T13: 49: 20.000Z',
'updated_time': '2021-06-14T14: 18: 59.000Z',
'take_profit': 0,
'stop_loss': 0,
'tp_trigger_by': 'UNKNOWN',
'sl_trigger_by': 'UNKNOWN',
'trigger_by': 'LastPrice',
'reduce_only': True,
'close_on_trigger': True
}
],
'time_now': '1623682473.336069',
'rate_limit_status': 598,
'rate_limit_reset_ms': 1623682473333,
'rate_limit': 600
}, '<bravado.requests_client.RequestsResponseAdapter object at 0x00789C86BBFA0 >')
dicts = orders[0]['result']
for i, d in enumerate(dicts):
reduce = d['reduce_only']
close = d['close_on_trigger']
sell = d['side']
try:
prnt = d['print']
except:
prnt = ""
print(f'dict {i} : {reduce=} {close=} {sell=} {prnt=}')
Output:
dict 0 : reduce=False close=False sell='Sell' prnt=''
dict 1 : reduce=True close=True sell='Buy' prnt=''
dict 2 : reduce=True close=True sell='Sell' prnt=''
dict 3 : reduce=True close=True sell='Buy' prnt=''
dict 4 : reduce=True close=True sell='Sell' prnt=''
I'm confused. None of the stuff I tried from other related questions worked and I can't seem to find it by googling.
I want to filter a json api response with Python, without just sorting the datafeed I get back - that's not much good to me. I also want to explicitly select one field, rather than deleting all the other fields, because sometimes there are more fields than other times. It has to be accomplished with the structure of the request, not by modifying anything after the fact.
I use the following code to get a list of sports from the matchbook api, but how would I just show the names of those sports, filtering out the id and types?
r7 = s.get('https://api.matchbook.com/edge/rest/lookups/sports')
r8 = r7.json()
pprint.pprint (r8)
This is what I get back:
{'offset': 0,
'per-page': 20,
'sports': [{'id': 112, 'name': 'Australian Rules', 'type': 'SPORT'},
{'id': 13, 'name': 'Auto Racing', 'type': 'SPORT'},
{'id': 3, 'name': 'Baseball', 'type': 'SPORT'},
{'id': 4, 'name': 'Basketball', 'type': 'SPORT'},
{'id': 14, 'name': 'Boxing', 'type': 'SPORT'},
{'id': 110, 'name': 'Cricket', 'type': 'SPORT'},
{'id': 11, 'name': 'Current Events', 'type': 'SPORT'},
{'id': 115, 'name': 'Cycling', 'type': 'SPORT'},
{'id': 116, 'name': 'Darts', 'type': 'SPORT'},
{'id': 117, 'name': 'Gaelic Football', 'type': 'SPORT'},
{'id': 8, 'name': 'Golf', 'type': 'SPORT'},
{'id': 241798357140019,
'name': 'Greyhound Racing',
'type': 'SPORT'},
{'id': 24735152712200, 'name': 'Horse Racing', 'type': 'SPORT'},
{'id': 222109340250019,
'name': 'Horse Racing (Ante Post)',
'type': 'SPORT'},
{'id': 231138347942400,
'name': 'Horse Racing Beta',
'type': 'SPORT'},
{'id': 118, 'name': 'Hurling', 'type': 'SPORT'},
{'id': 6, 'name': 'Ice Hockey', 'type': 'SPORT'},
{'id': 126, 'name': 'MMA', 'type': 'SPORT'},
{'id': 5, 'name': 'NCAA Basketball', 'type': 'SPORT'},
{'id': 2, 'name': 'NCAA Football', 'type': 'SPORT'}],
'total': 28}
401
[Finished in 5.3s]
How would I get rid of the 'id' and 'type' fields?
This is an extract of a large set of json data:
{'events': [{'allow-live-betting': True,
'category-id': [9,
297063445660036,
410444698700010,
410468520880009,
477274740480010,
486941505790009],
'id': 488243069570009,
'in-running-flag': True,
'markets': [{'allow-live-betting': True,
'back-overround': 102.04249,
'event-id': 488243069570009,
'id': 488243069830010,
'in-running-flag': True,
'lay-overround': 97.95751,
'live': True,
'market-type': 'money_line',
'name': 'Moneyline',
'runners': [{'event-id': 488243069570009,
'event-participant-id': 488243069720009,
'id': 488243069860009,
'market-id': 488243069830010,
'name': 'G Mager',
'prices': [{'available-amount': 48.42639,
'currency': 'GBP',
'decimal-odds': 1.53191,
'exchange-type': 'back-lay',
'odds': 1.53191,
'odds-type': 'DECIMAL',
'side': 'back'},
{'available-amount': 473.49579,
'currency': 'GBP',
'decimal-odds': 1.51546,
'exchange-type': 'back-lay',
'odds': 1.51546,
'odds-type': 'DECIMAL',
'side': 'back'},
{'available-amount': 96.87754,
'currency': 'GBP',
'decimal-odds': 1.40322,
'exchange-type': 'back-lay',
'odds': 1.40322,
'odds-type': 'DECIMAL',
'side': 'back'},
{'available-amount': 710.74,
'currency': 'GBP',
'decimal-odds': 1.5814,
'exchange-type': 'back-lay',
'odds': 1.5814,
'odds-type': 'DECIMAL',
'side': 'lay'},
{'available-amount': 163.74192,
'currency': 'GBP',
'decimal-odds': 1.76336,
'exchange-type': 'back-lay',
'odds': 1.76336,
'odds-type': 'DECIMAL',
'side': 'lay'},
{'available-amount': 151.55072,
'currency': 'GBP',
'decimal-odds': 1.9434,
'exchange-type': 'back-lay',
'odds': 1.9434,
'odds-type': 'DECIMAL',
'side': 'lay'}],
'status': 'open',
'volume': 6955.72965},
{'event-id': 488243069570009,
'event-participant-id': 488243069660010,
'id': 488243069890010,
'market-id': 488243069830010,
'name': 'A Bedene',
'prices': [{'available-amount': 413.22093,
'currency': 'GBP',
'decimal-odds': 2.72,
'exchange-type': 'back-lay',
'odds': 2.72,
'odds-type': 'DECIMAL',
'side': 'back'},
{'available-amount': 124.99383,
'currency': 'GBP',
'decimal-odds': 2.31,
'exchange-type': 'back-lay',
'odds': 2.31,
'odds-type': 'DECIMAL',
'side': 'back'},
{'available-amount': 142.97238,
'currency': 'GBP',
'decimal-odds': 2.06,
'exchange-type': 'back-lay',
'odds': 2.06,
'odds-type': 'DECIMAL',
'side': 'back'},
{'available-amount': 25.75872,
'currency': 'GBP',
'decimal-odds': 2.88,
'exchange-type': 'back-lay',
'odds': 2.88,
'odds-type': 'DECIMAL',
'side': 'lay'},
{'available-amount': 244.07,
'currency': 'GBP',
'decimal-odds': 2.94,
'exchange-type': 'back-lay',
'odds': 2.94,
'odds-type': 'DECIMAL',
'side': 'lay'},
{'available-amount': 39.06352,
'currency': 'GBP',
'decimal-odds': 3.48,
'exchange-type': 'back-lay',
'odds': 3.48,
'odds-type': 'DECIMAL',
'side': 'lay'}],
'status': 'open',
'volume': 16033.12548}],
'start': '2017-05-15T11:05:00.000Z',
'status': 'open',
'type': 'binary',
'volume': 22988.85514}],
'meta-tags': [{'id': 9,
'name': 'Tennis',
'type': 'SPORT',
'url-name': 'tennis'},
{'id': 297063445660036,
'name': 'R1',
'type': 'DATE',
'url-name': 'r1'},
{'id': 410444698700010,
'name': 'Italy',
'type': 'COUNTRY',
'url-name': 'italy'},
{'id': 410468520880009,
'name': 'Live Betting',
'type': 'OTHER',
'url-name': 'live-betting'},
{'id': 477274740480010,
'name': 'May 15th 2017',
'type': 'DATE',
'url-name': 'may-15th-2017'},
{'id': 486941505790009,
'name': 'ATP Rome',
'type': 'COMPETITION',
'url-name': 'ATP-Rome'}],
'name': 'G Mager vs A Bedene',
'sport-id': 9,
'start': '2017-05-15T11:05:00.000Z',
'status': 'open',
'volume': 23098.72333},
'offset': 0,
'per-page': 20,
'total': 223}
I want to find the event-id within this data and print "'event-id': 488243069570009"
I managed to do this with:
r11 = s.get('https://api.matchbook.com/edge/rest/events')
r12 = r11.json()
for each in r12['events'][0]['markets']:
print(each['event-id'])
but it only prints the first event-id on the complete set of json data
You can simply create a new array using shorthand
data = {"offset":0,"per-page":20,"sports":[{"id":112,"name":"Australian Rules","type":"SPORT"},{"id":13,"name":"Auto Racing","type":"SPORT"},{"id":3,"name":"Baseball","type":"SPORT"},{"id":4,"name":"Basketball","type":"SPORT"},{"id":14,"name":"Boxing","type":"SPORT"},{"id":110,"name":"Cricket","type":"SPORT"},{"id":11,"name":"Current Events","type":"SPORT"},{"id":115,"name":"Cycling","type":"SPORT"},{"id":116,"name":"Darts","type":"SPORT"},{"id":117,"name":"Gaelic Football","type":"SPORT"},{"id":8,"name":"Golf","type":"SPORT"},{"id":241798357140019,"name":"Greyhound Racing","type":"SPORT"},{"id":24735152712200,"name":"Horse Racing","type":"SPORT"},{"id":222109340250019,"name":"Horse Racing (Ante Post)","type":"SPORT"},{"id":231138347942400,"name":"Horse Racing Beta","type":"SPORT"},{"id":118,"name":"Hurling","type":"SPORT"},{"id":6,"name":"Ice Hockey","type":"SPORT"},{"id":126,"name":"MMA","type":"SPORT"},{"id":5,"name":"NCAA Basketball","type":"SPORT"},{"id":2,"name":"NCAA Football","type":"SPORT"}],"total":28}
# print(data['sports'])
# for element in data['sports']:
# del element['id']
# del element['type']
# print(data['sports'])
new_arr = [_['name'] for _ in data['sports']];
print(new_arr)
UPDATE
Check the online demo
You can't do what you're asking the way you're asking. The matchbook API just doesn't work like that.
You could do something silly, like add a proxy server in between you and their API that does the filtering for you, so you aren't the one doing the filtering, it's the proxy, but that's just playing with the semantics.
Agam Banga's example does exactly what you stated you want:
how would I just show the names of those sports, filtering out the id and types?
Here's a full example that you can literally copy, paste, and run, and it will print out all the sports that matchbook supports:
import requests
r = requests.get('https://api.matchbook.com/edge/rest/lookups/sports')
names = [_['name'] for _ in r.json()['sports']]
for name in names:
print('Sport Name:', name)
I ran it, and here's what it output:
Sport Name: Australian Rules
Sport Name: Auto Racing
Sport Name: Baseball
Sport Name: Basketball
Sport Name: Boxing
Sport Name: Cricket
Sport Name: Current Events
Sport Name: Cycling
Sport Name: Darts
Sport Name: Gaelic Football
Sport Name: Golf
Sport Name: Greyhound Racing
Sport Name: Horse Racing
Sport Name: Horse Racing (Ante Post)
Sport Name: Horse Racing Beta
Sport Name: Hurling
Sport Name: Ice Hockey
Sport Name: MMA
Sport Name: NCAA Basketball
Sport Name: NCAA Football