JSON issue with Beautifulsoup

JSON issue with Beautifulsoup - python

I'm trying to extract consistently this portion of code (line 237 in page source) with python requests.
<script type="text/javascript">
window.classified = {"EverythingHere"
</script>
Yet I'm unable to parse it using the following code, while it works for every other "script" tags.
url = 'https://www.immoweb.be/en/classified/new-real-estate-project-apartments/for-sale/auderghem/1160/8950161'
soup = BeautifulSoup(requests.get(url).content, "html.parser")
data=[]
for p in soup.find_all():
for n in p:
if n.name == 'script':
data.append(str(n))
print(data[0])
Do I need to set up a time.sleep to let the parser loading the webpage or do I need to change something in my code ?
Any helps would be really appreciated ! Thanks.

Couple things:
You are unnecsarily using a nested loop to search for 'script' tags. Rather than find_all() tags, then iterate through all the tags to find the ones that are <script>, you can do that right off the bat with .find_all('script').
Secondly, it's the last <script> tag. You can either just call the [-1] index, or probably a better way incase for whatever reason it's not that last tag for another page, just get the one with 'window.classified'
import requests
from bs4 import BeautifulSoup
import re
import json
url = 'https://www.immoweb.be/en/classified/new-real-estate-project-apartments/for-sale/auderghem/1160/8950161'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36'}
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")
for p in soup.find_all('script'):
if 'window.classified' in str(p):
dataStr = str(p)
#print(str(p))
# Parse the json
data = re.search('{.*}', dataStr).group(0)
jsonData = json.loads(data)
Output:
print(jsonData)
{'id': 8950161, 'cluster': {'minPrice': 240500, 'maxPrice': 521500, 'minSurface': 44, 'maxSurface': 104, 'projectInfo': {'constructor': None, 'groupId': 8950161, 'phase': None, 'projectName': 'DELTA VIEW', 'deliveryDate': '2022-10-01T00:00:00.000+0000', 'soldPercentage': 47, 'unitsDisplayMode': 'DETAILED'}, 'units': [{'type': 'APARTMENT', 'maxPrice': 521500, 'minPrice': 240500, 'minRoom': 0, 'maxRoom': 2, 'minSurface': 44, 'maxSurface': 104, 'soldCount': 36, 'items': [{'id': 8950204, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950212, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950213, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950217, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950222, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950223, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950170, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950192, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950203, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950205, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950211, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950218, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950220, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950226, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950175, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950179, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950182, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950183, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950216, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950162, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950202, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950244, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950171, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950194, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 240500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950193, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 240500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950195, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 243500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950196, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 243500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950206, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 244500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950197, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 245500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950198, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950207, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 247500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950208, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 247500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950224, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 247500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950225, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950172, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 248500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950199, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 248500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950200, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 248500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950209, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 249500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950210, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 6, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950239, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950248, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950201, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950242, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950163, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950164, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950173, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950166, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 252500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950165, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 252500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950174, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 5, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950167, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 255500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950168, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 255500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950176, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 255500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950169, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 257500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950178, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950177, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 260500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 47}, {'id': 8950190, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 3, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950188, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950189, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 3, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950187, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 312000, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950191, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 4, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950181, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950180, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 335500, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950243, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 335500, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950185, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 5, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950241, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 5, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950184, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 338500, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950186, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 6, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950240, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 338500, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950227, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 501500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 102}, {'id': 8950228, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 504500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 102}, {'id': 8950229, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 506500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 102}, {'id': 8950231, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 7, 'price': 511500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950230, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 511500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 102}, {'id': 8950233, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 8, 'price': 513500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950234, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 9, 'price': 516500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950236, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 518500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950238, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 11, 'price': 521500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950247, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 11, 'price': 521500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}]}], 'bedroomRange': '1 - 2', 'surfaceRange': '44 - 104'}, 'customers': [{'id': 1364801, 'type': 'REAL_ESTATE_AGENCY', 'email': 'victoire_1490#importfrommedia.be', 'logoUrl': 'https://static.immoweb.be/logos/1364801.gif?cache=201721090326Z', 'phoneNumber': '+32 2 375 10 10', 'mobileNumber': None, 'name': 'Victoire (Immobilier Neuf)', 'website': None, 'location': {'country': 'Belgium', 'region': None, 'province': 'Brussels', 'district': 'Brussels', 'locality': 'Uccle', 'postalCode': '1180', 'street': 'Chaussée de Waterloo 1382', 'number': None, 'box': None, 'propertyName': None, 'floor': None, 'latitude': 50.8261604, 'longitude': 4.4461589, 'approximated': None, 'regionCode': None, 'type': None, 'hasSeaView': None, 'pointsOfInterest': None, 'placeName': None}, 'ipiNo': None, 'isOwner': True, 'contactHoursMobile': 'anytime', 'contactHoursLandline': 'anytime', 'salesRepresentative': None}], 'premiumProjectPage': {'medias': None, 'options': None, 'promoter': None, 'tabs': None}, 'flags': {'isPublicSale': False, 'isNewClassified': False, 'isNewPrice': False, 'isNewlyBuilt': True, 'isNotarySale': None, 'isLifeAnnuitySale': False, 'adQuality': None, 'date': None, 'priceSqm': None, 'price': None, 'default': None, 'isSoldOrRented': False, 'isLowEnergy': None, 'percentSold': 47, 'isPassiveHouse': None, 'isNewRealEstateProject': True, 'isAnInteractiveSale': None, 'isUnderOption': None}, 'media': {'pictures': [{'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_1.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_1.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_1.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_2.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_2.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_2.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_3.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_3.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_3.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_4.gif?cache=20210217040440', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_4.jpg?cache=20210217040440', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_4.jpg?cache=20210217040440', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_5.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_5.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_5.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_6.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_6.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_6.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_7.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_7.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_7.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_8.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_8.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_8.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_9.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_9.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_9.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_0.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_0.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_0.jpg?cache=20210217035932', 'isVertical': True}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_A.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_A.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_A.jpg?cache=20210217035933', 'isVertical': True}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_B.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_B.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_B.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_C.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_C.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_C.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_D.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_D.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_D.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_E.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_E.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_E.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_F.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_F.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_F.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_G.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_G.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_G.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_H.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_H.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_H.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_I.gif?cache=20200923031600', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_I.jpg?cache=20200923031600', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_I.jpg?cache=20200923031600', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_J.gif?cache=20200923031600', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_J.jpg?cache=20200923031600', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_J.jpg?cache=20200923031600', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_K.gif?cache=20200923031600', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_K.jpg?cache=20200923031600', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_K.jpg?cache=20200923031600', 'isVertical': False}], 'virtualTourUrl': None, 'floorPlans': None, 'specifications': None, 'virtualExperienceUrl': 'https://youtu.be/D8wua7Z_Pe8'}, 'property': {'type': 'APARTMENT_GROUP', 'subtype': 'APARTMENT_GROUP', 'title': 'BUY NOW AND PAY IN 2022', 'description': 'This new project (available at the end of 2022) is ideally located in the new district of Chirec / Delta, near the Université Libre de Bruxelles (ULB and VUB) and consists of 70 units, from studios to 2-bedroom apartments. Most units have a terrace and / or a view of the green surroundings and the green rooftops. The building has a passive energy label. PEB A+ A bicycle storage is available on the 1st floor. Parkings and cellars are also provided at the same floor, at an additional cost. Sale under the mixed system of registration on the land value and VAT on the construction value. Choose your apartment now and pay for it at the delivery, at no extra cost! Contact us for more information about this project: 02 3751010 / new#victoire.be', 'name': None, 'isHolidayProperty': None, 'bedroomCount': None, 'bedrooms': [], 'bathroomCount': None, 'bathrooms': [], 'location': {'country': 'Belgium', 'region': 'Brussels', 'province': 'Brussels', 'district': 'Brussels', 'locality': 'Auderghem', 'postalCode': '1160', 'street': 'Boulevard du Triomphe', 'number': '201', 'box': None, 'propertyName': None, 'floor': None, 'latitude': 50.816023, 'longitude': 4.3999304, 'approximated': None, 'regionCode': 'BRUSSELS', 'type': None, 'hasSeaView': None, 'pointsOfInterest': [{'type': 'SCHOOL', 'distance': 150}, {'type': 'SHOPS', 'distance': 200}, {'type': 'TRANSPORT', 'distance': 0}], 'placeName': 'Projets Bruxelles Est'}, 'netHabitableSurface': None, 'roomCount': None, 'attic': None, 'hasAttic': None, 'basement': None, 'hasBasement': None, 'hasDressingRoom': None, 'diningRoom': None, 'hasDiningRoom': None, 'building': None, 'propertyCertificates': None, 'hasCaretakerOrConcierge': None, 'hasDisabledAccess': None, 'hasLift': None, 'constructionPermit': {'constructionType': None, 'floodZoneType': 'NON_FLOOD_ZONE', 'isObtained': None, 'hasObligationToConstruct': None, 'hasPlotDivisionAuthorization': None, 'hasPossiblePriorityPurchaseRight': None, 'isBreachingUrbanPlanningRegulation': None, 'floodZoneIconUrl': None, 'totalBuildableGroundFloorSurface': 0, 'urbanPlanningInformation': None}, 'energy': None, 'kitchen': None, 'land': None, 'laundryRoom': None, 'hasLaundryRoom': None, 'livingRoom': None, 'hasLivingRoom': False, 'isFirstOccupation': None, 'hasBalcony': None, 'hasBarbecue': None, 'hasGarden': None, 'gardenSurface': None, 'gardenOrientation': None, 'parkingCountIndoor': 1, 'parkingCountOutdoor': 1, 'parkingCountClosedBox': None, 'hasAirConditioning': None, 'hasArmoredDoor': None, 'hasVisiophone': None, 'hasSecureAccessAlarm': None, 'hasTVCable': None, 'hasDoorPhone': None, 'hasInternet': None, 'showerRoomCount': None, 'showerRooms': [], 'specificities': None, 'toiletCount': None, 'toilets': [], 'hasFitnessRoom': None, 'hasTennisCourt': None, 'hasSwimmingPool': None, 'hasSauna': None, 'hasJacuzzi': None, 'hasHammam': None, 'bedroomSurface': None, 'alternativeDescriptions': {'fr': "Ce projet (disponible fin 2022), bien situé dans le nouveau quartier du Chirec/Delta, et des facultés universitaires ULB/VUB, comporte 70 logements allant principalement du studio à l’appartement 2 chambres. Très urbain, les logements bénéficient néanmoins pour la plupart de terrasses et/ou de vues sur la verdure et les toits verdurisés. Le bÃtiment est passif et profite de l’alimentation en chauffage co-génération du Chirec. PEB A+ Le local vélo est disponible au 1er et les parkings et les caves au © 1 sont en supplément. Vente sous droits d'enregistrement sur le terrain et sous régime TVA sur la construction, Réservez votre appartement maintenant et payez-le à la livraison, sans augmentation de prix ! Consultez-nous sans attendre pour tout détail sur ce projet : 02 3751010 / new#victoire.be", 'nl': "Dit nieuwbouwproject (beschikbaar eind 2022) is uitstekend gelegen in de nieuwe stadswijk Chirec/Delta, nabij de Vrije Universiteit Brussel en omvat 70 wooneenheden gaande van studio's tot appartementen met 2 slaapkamers. De meeste units hebben een terras en/of zicht op de groene omgeving en de groene daken. Het gebouw heeft een passief energielabel. EPC A+ Een fietsenberging is voorzien op de 1e verdieping. Hier zijn ook parkings en kelders voorzien, dit tegen een meerprijs. Verkoop onder het gemend stelsel van registratierechten op de grondwaarde en BTW op de constructiewaarde. Kies nu uw appartement en betaal het aan de levering Ã zonder meerprijs! Raadpleeg ons voor meer informatie over dit project: 02 3751010 / new#victoire.be"}, 'habitableUnitCount': None, 'fireplaceCount': None, 'fireplaceExists': False, 'hasTerrace': None, 'terraceSurface': None, 'terraceOrientation': None}, 'publication': {'creationDate': '2020-09-23T03:12:09.000+0000', 'expirationDate': '2021-04-30T21:59:59.000+0000', 'publisherId': None, 'visualisationOption': 'XL', 'lastModificationDate': '2021-04-22T02:36:15.000+0000'}, 'transaction': {'type': 'FOR_SALE', 'subtype': 'BUY_REGULAR', 'availabilityPeriodType': 'AT_DELIVERY', 'availabilityDate': '2022-10-01T00:00:00.000+0000', 'certificates': {'carbonEmission': 40, 'hasElectricalInstallationComplianceCertificate': None, 'primaryEnergyConsumptionPerSqm': 45, 'primaryEnergyConsumptionYearly': None, 'epcDescription': None, 'epcReference': 'A+', 'epcScore': 'A', 'epcUrl': 'https://static.immoweb.be/en/pics/bxlA3.gif'}, 'rental': None, 'sale': {'price': None, 'vatType': 'VAT_EXCLUDED', 'cadastralIncome': 0, 'publicSale': None, 'pricePerSqm': None, 'oldPrice': None, 'lifeAnnuity': None, 'hasStartingPrice': True, 'isFurnished': None, 'homeToBuild': None}, 'investor': {'isInvestmentProperty': False, 'currentMonthlyRentalIncome': None, 'currentReturnOnInvestment': None, 'expectedMonthlyRentalIncome': None, 'expectedMonthlyRentalIncomeDescription': None, 'expectedReturnOnInvestment': None, 'habitableUnitCount': None, 'occupancyRate': None}}, 'priceType': None, 'price': {'type': 'group_sale', 'mainValue': None, 'alternativeValue': None, 'additionalValue': None, 'oldValue': None, 'minRangeValue': 240500, 'maxRangeValue': 521500, 'mainDisplayPrice': '€240,500 - €521,500', 'HTMLDisplayPrice': '<span aria-hidden="true">€240,500 - €521,500</span>', 'alternativeDisplayPrice': '', 'oldDisplayPrice': None, 'shortDisplayPrice': '240.5K - 521.5K €', 'accessibilityPrice': 'From 240500€ To 521500€', 'label': 'Min price - Max price', 'language': 'en'}, 'externalReference': '4161726', 'isBookmarked': False, 'hasSectionsArray': {'hasGeneralSection': True, 'hasInteriorSection': False, 'hasExteriorSection': False, 'hasFacilitiesSection': False, 'hasEnergySection': True, 'hasPlanningSection': True, 'hasFinancialSection': True, 'hasPublicSaleSection': False}, 'unitGroupings': [], 'displayFlags': {'main': None, 'secondary': ['new_real_estate_project', 'percent_sold'], 'percentSold': 47}, 'statistics': {'bookmarkCount': 18, 'viewCount': 2604, 'alertPrice': None, 'creationDate': None, 'description': None, 'isAlertEmailSet': None, 'rating': None, 'wasOwnerContacted': None, 'wasPropertyVisited': None}}

Related

spark.read.json error: (java.lang.ArrayStoreException: java.util.HashMap)

I'm using the code below to pull in multiple json files into 1 using pagination. When I try to create a spark dataframe I get the error '
java.lang.ArrayStoreException: java.util.HashMap' during spark.read.json. Below the code block I've provided output from printing the 'issues' data set (minus proprietary info). I've done a bit of research and can't figure out what I can try to make this work. Any assistance would be greatly appreciated!!
import requests
import json
limit = 2
startat = 0
issues = []
for page_num in range(2):
startat = page_num*50
url = f"https://URL/rest/api/2/search?jql=TEST&startAt={startat}&maxResults={limit}"
req = requests.get(url, headers={'Accept': 'application/json', 'Authorization': 'Basic xxxxxxxxxxxxxxxxxxxxxxxxxx'})
data = req.json()
issues.extend(data['issues'])
jsonDF = spark.read.json(issues)
jsonDF.printSchema()
[{'expand': 'operations,versionedRepresentations,editmeta,changelog,customfield_10010.requestTypePractice,renderedFields', 'id': '11441', 'self': 'https://my.url.net/rest/api/2/issue/11441', 'key': 'TS-1401', 'fields': {'statuscategorychangedate': '2022-11-29T07:05:17.359-0800', 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10004', 'id': '10004', 'description': 'Functionality or a feature expressed as a user goal.', 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10315?size=medium', 'name': 'Story', 'subtask': False, 'avatarId': 10315, 'hierarchyLevel': 0}, 'parent': {'id': '11420', 'key': 'TS-1380', 'self': 'https://my.url.net/rest/api/2/issue/11420', 'fields': {'summary': 'Clone30 - Migration Epics', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10000', 'id': '10000', 'description': 'A big user story that needs to be broken down. Created by Jira Software - do not edit or delete.', 'iconUrl': 'https://my.url.net/images/icons/issuetypes/epic.svg', 'name': 'Epic', 'subtask': False, 'hierarchyLevel': 1}}}, 'timespent': None, 'project': {'self': 'https://my.url.net/rest/api/2/project/10001', 'id': '10001', 'key': 'TS', 'name': 'Project', 'projectTypeKey': 'software', 'simplified': False, 'avatarUrls': {'48x48': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556', '24x24': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=small', '16x16': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=xsmall', '32x32': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=medium'}}, 'customfield_10033': None, 'fixVersions': [], 'aggregatetimespent': None, 'customfield_10034': [], 'customfield_10035': None, 'resolution': None, 'customfield_10036': None, 'customfield_10037': None, 'customfield_10027': None, 'customfield_10028': None, 'customfield_10029': None, 'resolutiondate': None, 'workratio': -1, 'watches': {'self': 'https://my.url.net/rest/api/2/issue/TS-1401/watchers', 'watchCount': 1, 'isWatching': True}, 'lastViewed': '2022-12-08T10:06:57.022-0800', 'created': '2022-11-29T07:05:16.501-0800', 'customfield_10020': None, 'customfield_10021': None, 'customfield_10022': None, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'customfield_10023': None, 'customfield_10024': None, 'customfield_10025': None, 'customfield_10026': None, 'labels': [], 'customfield_10016': None, 'customfield_10017': None, 'customfield_10018': {'hasEpicLinkFieldDependency': False, 'showField': False, 'nonEditableReason': {'reason': 'EPIC_LINK_SHOULD_BE_USED', 'message': 'To set an epic as the parent, use the epic link instead'}}, 'customfield_10019': '0|i008a3:', 'timeestimate': None, 'aggregatetimeoriginalestimate': None, 'versions': [], 'issuelinks': [], 'assignee': None, 'updated': '2022-11-29T07:05:20.759-0800', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'components': [], 'timeoriginalestimate': None, 'description': 'Data owner completes template (understand scope of migration efforts)', 'customfield_10010': None, 'customfield_10014': 'TS-1380', 'customfield_10015': None, 'customfield_10005': None, 'customfield_10006': None, 'customfield_10007': None, 'security': None, 'customfield_10008': None, 'customfield_10009': None, 'aggregatetimeestimate': None, 'summary': 'Template', 'creator': {'self': 'https://my.url.net/rest/api/2/user?accountId=5d669f4bf81f2c0d99ee9e38', 'accountId': '5d669f4bf81f2c0d99ee9e38', 'emailAddress': 'test#aol.com', 'avatarUrls': {'48x48': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '24x24': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '16x16': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '32x32': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png'}, 'displayName': 'Joe Test', 'active': True, 'timeZone': 'America/Los_Angeles', 'accountType': 'atlassian'}, 'subtasks': [{'id': '11442', 'key': 'TS-1402', 'self': 'https://my.url.net/rest/api/2/issue/11442', 'fields': {'summary': 'Complete Template with table/views required (in) and produced (out)', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10006', 'id': '10006', 'description': "A small piece of work that's part of a larger task.", 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10316?size=medium', 'name': 'Sub-task', 'subtask': True, 'avatarId': 10316, 'hierarchyLevel': -1}}}], 'reporter': {'self': 'https://my.url.net/rest/api/2/user?accountId=5d669f4bf81f2c0d99ee9e38', 'accountId': '5d669f4bf81f2c0d99ee9e38', 'emailAddress': 'test#aol.com', 'avatarUrls': {'48x48': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '24x24': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '16x16': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '32x32': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png'}, 'displayName': 'Joe Test', 'active': True, 'timeZone': 'America/Los_Angeles', 'accountType': 'atlassian'}, 'aggregateprogress': {'progress': 0, 'total': 0}, 'customfield_10001': None, 'customfield_10002': None, 'customfield_10003': None, 'customfield_10004': None, 'customfield_10038': None, 'environment': None, 'duedate': None, 'progress': {'progress': 0, 'total': 0}, 'votes': {'self': 'https://my.url.net/rest/api/2/issue/TS-1401/votes', 'votes': 0, 'hasVoted': False}}}, {'expand': 'operations,versionedRepresentations,editmeta,changelog,customfield_10010.requestTypePractice,renderedFields', 'id': '11438', 'self': 'https://my.url.net/rest/api/2/issue/11438', 'key': 'TS-1398', 'fields': {'statuscategorychangedate': '2022-11-29T07:05:09.126-0800', 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10004', 'id': '10004', 'description': 'Functionality or a feature expressed as a user goal.', 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10315?size=medium', 'name': 'Story', 'subtask': False, 'avatarId': 10315, 'hierarchyLevel': 0}, 'parent': {'id': '11420', 'key': 'TS-1380', 'self': 'https://my.url.net/rest/api/2/issue/11420', 'fields': {'summary': 'Clone30 - Migration Epics', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10000', 'id': '10000', 'description': 'A big user story that needs to be broken down. Created by Jira Software - do not edit or delete.', 'iconUrl': 'https://my.url.net/images/icons/issuetypes/epic.svg', 'name': 'Epic', 'subtask': False, 'hierarchyLevel': 1}}}, 'timespent': None, 'project': {'self': 'https://my.url.net/rest/api/2/project/10001', 'id': '10001', 'key': 'TS', 'name': 'Project', 'projectTypeKey': 'software', 'simplified': False, 'avatarUrls': {'48x48': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556', '24x24': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=small', '16x16': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=xsmall', '32x32': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=medium'}}, 'fixVersions': [], 'customfield_10033': None, 'customfield_10034': [], 'aggregatetimespent': None, 'customfield_10035': None, 'resolution': None, 'customfield_10036': None, 'customfield_10037': None, 'customfield_10027': None, 'customfield_10028': None, 'customfield_10029': None, 'resolutiondate': None, 'workratio': -1, 'lastViewed': None, 'watches': {'self': 'https://my.url.net/rest/api/2/issue/TS-1398/watchers', 'watchCount': 1, 'isWatching': True}, 'created': '2022-11-29T07:05:08.312-0800', 'customfield_10020': None, 'customfield_10021': None, 'customfield_10022': None, 'customfield_10023': None, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'customfield_10024': None, 'customfield_10025': None, 'customfield_10026': None, 'labels': [], 'customfield_10016': None, 'customfield_10017': None, 'customfield_10018': {'hasEpicLinkFieldDependency': False, 'showField': False, 'nonEditableReason': {'reason': 'EPIC_LINK_SHOULD_BE_USED', 'message': 'To set an epic as the parent, use the epic link instead'}}, 'customfield_10019': '0|i008ae:y', 'timeestimate': None, 'aggregatetimeoriginalestimate': None, 'versions': [], 'issuelinks': [], 'assignee': None, 'updated': '2022-11-29T07:05:22.417-0800', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'components': [], 'timeoriginalestimate': None, 'description': 'Creating reports/reporting cubes; need to find out reports used', 'customfield_10010': None, 'customfield_10014': 'TS-1380', 'customfield_10015': None, 'customfield_10005': None, 'customfield_10006': None, 'security': None, 'customfield_10007': None, 'customfield_10008': None, 'customfield_10009': None, 'aggregatetimeestimate': None, 'summary': '\xa0create reports/cubes', 'creator': {'self': 'https://my.url.net/rest/api/2/user?accountId=5d669f4bf81f2c0d99ee9e38', 'accountId': '5d669f4bf81f2c0d99ee9e38', 'emailAddress': 'test#aol.com', 'avatarUrls': {'48x48': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '24x24': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '16x16': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '32x32': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png'}, 'displayName': 'Joe Test', 'active': True, 'timeZone': 'America/Los_Angeles', 'accountType': 'atlassian'}, 'subtasks': [{'id': '11439', 'key': 'TS-1399', 'self': 'https://my.url.net/rest/api/2/issue/11439', 'fields': {'summary': 'Confirm: any reporting cubes required using this data are created and in production?', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10006', 'id': '10006', 'description': "A small piece of work that's part of a larger task.", 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10316?size=medium', 'name': 'Sub-task', 'subtask': True, 'avatarId': 10316, 'hierarchyLevel': -1}}}, {'id': '11440', 'key': 'TS-1400', 'self': 'https://my.url.net/rest/api/2/issue/11440', 'fields': {'summary': 'Confirm: any structured reports using this data are created and in production?', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10006', 'id': '10006', 'description': "A small piece of work that's part of a larger task.", 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10316?size=medium', 'name': 'Sub-task', 'subtask': True, 'avatarId': 10316, 'hierarchyLevel': -1}}}], 'reporter': {'self': 'https://my.url.net/rest/api/2/user?accountId=5d669f4bf81f2c0d99ee9e38', 'accountId': '5d669f4bf81f2c0d99ee9e38', 'emailAddress': 'test#aol.com', 'avatarUrls': {'48x48': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '24x24': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '16x16': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '32x32': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png'}, 'displayName': 'Joe Test', 'active': True, 'timeZone': 'America/Los_Angeles', 'accountType': 'atlassian'}, 'aggregateprogress': {'progress': 0, 'total': 0}, 'customfield_10001': None, 'customfield_10002': None, 'customfield_10003': None, 'customfield_10004': None, 'customfield_10038': None, 'environment': None, 'duedate': None, 'progress': {'progress': 0, 'total': 0}, 'votes': {'self': 'https://my.url.net/rest/api/2/issue/TS-1398/votes', 'votes': 0, 'hasVoted': False}}}]

Python Request with Cookies - Content blocked by Cookie Banner

I'm trying to access a website via Python Requests. To avoid the iframe of the "Cookie Banner" I want to pass the cookie that handles the banner.
With Selenium I already managed to figure out which cookie that is and there it works fine with just passing the key/value" pair. I already found online, that it is necessary to "get" the page before passing and then refreshing it with "get" again after adding the cookies.
website = "https://www.myfitnesspal.com"
path = "path/to/your/chromedriver.exe"
service = Service(executable_path=path)
driver = webdriver.Chrome(service = service)
driver.get(website)
driver.add_cookie({'name': 'notice_preferences','value': '2:'})
driver.get(website)
So far so good. However, if I pass the same cookie that already worked in Selenium to a Python Request, the response.text that I receive still shows the content of the iframe and "Cookie Banner".
response = requests.get(website, cookies={"notice_preferences":"2:"})
Does anyone know why this is happening or if there is even a solution for this?

I don't think that page's content is blocked by cookie banner. It's rather blocked by the lack of a proper user-agent in header. The following code will return the page content as seen in browser:
import requests
import pandas as pd
s = requests.Session()
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
url = 'https://www.myfitnesspal.com'
s.headers.update(headers)
s.cookies.set("name", "notice_preferences", domain="www.myfitnesspal.com/")
s.cookies.set("value", "2:", domain="www.myfitnesspal.com/")
r = s.get(url)
# print(r.text)
print(s.cookies)
Result printed in terminal:
<RequestsCookieJar[<Cookie split-id=e28e4968-c2e3-4145-9226-0d9db15bcffe for www.myfitnesspal.com/>, <Cookie name=notice_preferences for www.myfitnesspal.com//>, <Cookie value=2: for www.myfitnesspal.com//>]>
You can then navigate to another page in that website, and requests' Session will preserve the headers and cookies. Also, print out the text response, see if the info you're looking for is there.
For more info on Requests, you can visit https://requests.readthedocs.io/en/latest/
EDIT: This is an a la carte XY problem - luckily the OP clarified it in comments.
That data is being pulled via an XHR call from an API endpoint. To get the info youi want, you need to scrape the endpoint. THis is how you do it (after inspecting Dev Tools - Network Tab and seeing that endpoint):
import requests
import pandas as pd
s = requests.Session()
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
url = 'https://www.myfitnesspal.com/api/nutrition?query=banane&page=1&offset=10'
r = s.get(url)
df = pd.DataFrame(r.json()['items'])
display(df)
This will print in terminal:
item tags type
0 {'country_code': 'US', 'deleted': False, 'description': 'Banane', 'id': 1873022840, 'nutritional_contents': {'calcium': 0.5, 'carbohydrates': 22.84, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 89}, 'fat': 0.33, 'fiber': 2.6, 'iron': 1.44444, 'monounsaturated_fat': 0.032, 'polyunsaturated_fat': 0.073, 'potassium': 358, 'protein': 1.09, 'saturated_fat': 0.112, 'sodium': 1, 'sugar': 12.23, 'trans_fat': 0, 'vitamin_a': 4.26667, 'vitamin_c': 14.5, 'vitamin_d': 0}, 'public': True, 'serving_sizes': [{'id': '67628178485117', 'index': 0, 'nutrition_multiplier': 1.18, 'unit': 'medium', 'value': 1}, {'id': '67078422671357', 'index': 1, 'nutrition_multiplier': 1.36, 'unit': 'large', 'value': 1}, {'id': '67628178485245', 'index': 2, 'nutrition_multiplier': 1.5, 'unit': 'cup, sliced', 'value': 1}, {'id': '67078414315389', 'index': 3, 'nutrition_multiplier': 2.25, 'unit': 'cup, mashed', 'value': 1}, {'id': '67628170129277', 'index': 4, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '67078414315517', 'index': 5, 'nutrition_multiplier': 0.283495, 'unit': 'oz', 'value': 1}, {'id': '67628170129405', 'index': 6, 'nutrition_multiplier': 0.81, 'unit': 'extra small', 'value': 1}, {'id': '67078422703997', 'index': 7, 'nutrition_multiplier': 1.52, 'unit': 'extra large', 'value': 1}, {'id': '67628178517885', 'index': 8, 'nutrition_multiplier': 4.53592, 'unit': 'lb(s)', 'value': 1}, {'id': '67078422704125', 'index': 9, 'nutrition_multiplier': 1e-05, 'unit': 'mg(s)', 'value': 1}, {'id': '67628178518013', 'index': 10, 'nutrition_multiplier': 10, 'unit': 'kg(s)', 'value': 1}, {'id': '67076304547197', 'index': 11, 'nutrition_multiplier': 0.00625, 'unit': 'mL, sliced ', 'value': 1}, {'id': '67626060361085', 'index': 12, 'nutrition_multiplier': 0.009375, 'unit': 'mL, mashed ', 'value': 1}, {'id': '67076304547325', 'index': 13, 'nutrition_multiplier': 6.25, 'unit': 'liter(s), sliced ', 'value': 1}, {'id': '67626060361213', 'index': 14, 'nutrition_multiplier': 9.375, 'unit': 'liter(s), mashed ', 'value': 1}], 'type': 'food', 'user_id': '133476501057389', 'verified': True, 'version': '199432263862133'} [canonical, best_match] food
1 {'brand_name': 'Banane', 'country_code': 'FR', 'deleted': False, 'description': 'Une banane', 'id': 2007191148, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 27, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 105}, 'fat': 0.4, 'fiber': 2.1, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 1.3, 'saturated_fat': 0, 'sodium': 0, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '93902759513197', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'fruit entier (120g)', 'value': 1}, {'id': '94452515327085', 'index': 1, 'nutrition_multiplier': 0.00833333, 'unit': 'gram', 'value': 1}], 'type': 'food', 'user_id': '160406080319149', 'verified': False, 'version': '198055450101605'} [] food
2 {'brand_name': 'Obst', 'country_code': 'DE', 'deleted': False, 'description': ' Banane ()', 'id': 1659839707, 'nutritional_contents': {'calcium': 0.625, 'carbohydrates': 22.84, 'energy': {'unit': 'calories', 'value': 90}, 'fat': 0.33, 'fiber': 2.6, 'iron': 1.857, 'potassium': 358, 'protein': 1.09, 'sodium': 1, 'sugar': 12.23, 'vitamin_a': 0.375, 'vitamin_c': 10.875}, 'public': True, 'serving_sizes': [{'id': '268297681372533', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '268297681372661', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '268847437186549', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}, {'id': '268297673016693', 'index': 3, 'nutrition_multiplier': 0.992232, 'unit': 'ounce', 'value': 3.5}], 'type': 'food', 'user_id': '163850601983789', 'verified': False, 'version': '129355447387317'} [] food
3 {'brand_name': 'Obst', 'country_code': 'DE', 'deleted': False, 'description': 'Banane 1 Stück', 'id': 1887842011, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 26.4, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 115}, 'fat': 0.2, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 1.2, 'saturated_fat': 0, 'sodium': 0, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '27521653231597', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 120}, {'id': '28071409045485', 'index': 1, 'nutrition_multiplier': 0.00833333, 'unit': 'g', 'value': 1}, {'id': '27521661620077', 'index': 2, 'nutrition_multiplier': 0.236246, 'unit': 'ounce', 'value': 1}], 'type': 'food', 'user_id': '234889390534445', 'verified': False, 'version': '53489009870261'} [] food
4 {'brand_name': 'Obst', 'country_code': 'DE', 'deleted': False, 'description': 'Banane', 'id': 227750309, 'nutritional_contents': {'calcium': 5, 'carbohydrates': 22.8, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 89}, 'fat': 0.3, 'fiber': 2.6, 'iron': 0.3, 'monounsaturated_fat': 0.1, 'polyunsaturated_fat': 0.1, 'potassium': 358, 'protein': 1.1, 'saturated_fat': 0.1, 'sodium': 1, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 64, 'vitamin_c': 9}, 'public': True, 'serving_sizes': [{'id': '88267487686061', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '88817243499949', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '88267496074541', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}], 'type': 'food', 'user_id': '134026256871405', 'verified': True, 'version': '230354056064301'} [] food
5 {'brand_name': 'Banane Ohne Schale', 'country_code': 'DE', 'deleted': False, 'description': 'Banane', 'id': 1889101676, 'nutritional_contents': {'calcium': 0.625, 'carbohydrates': 22.84, 'energy': {'unit': 'calories', 'value': 95}, 'fat': 0.33, 'fiber': 2.6, 'iron': 35.71, 'potassium': 358, 'protein': 1.09, 'sugar': 12.23, 'vitamin_a': 0.375, 'vitamin_c': 10.875}, 'public': True, 'serving_sizes': [{'id': '138151424970349', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '137601669156589', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '138151424970477', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}], 'type': 'food', 'user_id': '278614430748141', 'verified': False, 'version': '53074667210277'} [] food
6 {'brand_name': 'Banane', 'country_code': 'FR', 'deleted': False, 'description': 'Banane Gebacken', 'id': 1349524295, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 25, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 157}, 'fat': 4, 'fiber': 0, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 4, 'saturated_fat': 0, 'sodium': 1, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '59106944525429', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '58557188711669', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '59106944525557', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}, {'id': '58557197100149', 'index': 3, 'nutrition_multiplier': 0.992232, 'unit': 'ounce', 'value': 3.5}], 'type': 'food', 'user_id': '133324127170493', 'verified': False, 'version': '31634077001709'} [] food
7 {'brand_name': 'Banane', 'country_code': 'FR', 'deleted': False, 'description': 'Demi banane', 'id': 139561661, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 14, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 93}, 'fat': 2, 'fiber': 0, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 3, 'saturated_fat': 0, 'sodium': 0, 'sugar': 0, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '124397058706493', 'index': 1, 'nutrition_multiplier': 1, 'unit': 'yaourt', 'value': 1}], 'type': 'food', 'user_id': '133476501057517', 'verified': False, 'version': '63530949537133'} [] food
8 {'brand_name': 'Banane', 'country_code': 'CA', 'deleted': False, 'description': 'Banane (Santé Canada)', 'id': 1568891032, 'nutritional_contents': {'calcium': 1, 'carbohydrates': 27, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 105}, 'fat': 0, 'fiber': 3, 'iron': 2, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 487, 'protein': 1, 'saturated_fat': 0, 'sodium': 1, 'sugar': 14, 'trans_fat': 0, 'vitamin_a': 2, 'vitamin_c': 17}, 'public': True, 'serving_sizes': [{'id': '27092291822629', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'banana 118g', 'value': 1}, {'id': '26542536008869', 'index': 1, 'nutrition_multiplier': 1, 'unit': 'med bananna', 'value': 1}], 'type': 'food', 'user_id': '199722294733869', 'verified': False, 'version': '264028726224173'} [] food
9 {'brand_name': 'Banane', 'country_code': 'CA', 'deleted': False, 'description': ' Une banane moyenne', 'id': 1484522768, 'nutritional_contents': {'calcium': 1.53, 'carbohydrates': 22.8, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 89}, 'fat': 0.33, 'fiber': 2, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0.07, 'potassium': 0, 'protein': 1.1, 'saturated_fat': 0.11, 'sodium': 8, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '63099251926181', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '63649007740069', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '63099260314661', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}, {'id': '63649016128549', 'index': 3, 'nutrition_multiplier': 0.992232, 'unit': 'ounce', 'value': 3.5}], 'type': 'food', 'user_id': '128659968929645', 'verified': False, 'version': '136640012748413'} [] food
You can drill down further into that json object, (normalize it, etc) to get data in different shapes and forms.

BeautifulSoup find class with space

I am trying to get price info using BS4 with Python.
This is the <p> class from which I am trying to extract the value of '$3245'.
<div class="s1fqyqkq-3 bDmzjk">
<div class="s1fqyqkq-4 ckAXTq">Price Info</div>
<p class="s1fqyqkq-5 dekHBg">$3245</p>
</div>
And this is the code I am trying to use to get the data, but it's returning "None"
headers = {
"Connection": "keep-alive",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
}
req = requests.get(url, headers=headers)
soup = bs4.BeautifulSoup(req.text, 'html.parser')
result = soup.find("div", {"class": "s1fqyqkq-5 dekHBg"})
print(result)
I tried all suggestions, but I am still I receiving "None" as a result.
I tried the suggestion here to no avail. Any idea what I am missing?

While cooking your soup taste it to ensure it comes with all expected ingridients.
Main issue, content is created dynamically by javascript so you wont find your information that way, cause requests do not act like a browser and is not able to render all information.
To get informations take a look at the script variables and extract the JSON strings.
import json, requests
url = 'https://www.cimri.com/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-siyah-fiyatlari,337340793'
jsonData = json.loads(re.search(r"window.productDetails = (.*?);", requests.get(url).text).group(1))
jsonData
Output
{'id': '337340793', 'path': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-siyah-fiyatlari,337340793', 'itemType': 0, 'mainId': 337341638, 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah', 'offers': [{'id': '506482765', 'price': 13135.85, 'title': 'Yeni Apple iPhone 11 (64\xa0GB) - Siyah', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '10370', 'url': 'https://www.amazon.com.tr', 'seller': 'Amazon.com.tr', 'slogan': 'Amazon.com.tr uygulamasında ilk 150 TL ve üzeri alışverişe 40 TL indirim ', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495611798}, {'id': '894741295', 'price': 13149, 'title': 'iPhone 11 64 GB Aksesuarsız Kutu Siyah', 'shipping': {'fee': 0, 'day': 3}, 'merchant': {'id': '10109', 'url': 'https://www.trendyol.com/', 'seller': 'Trendyol', 'slogan': "Tüm İhtiyaçların Trendyol'da, Tek Tıkla Kapında!", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': True, 'feedLastProcess': 1655493405188}, {'id': '887498622', 'price': 13249, 'title': 'Apple iPhone 11 64 GB Cep Telefonu Siyah', 'shipping': {'fee': 0, 'day': 2}, 'merchant': {'id': '10416', 'url': 'https://www.a101.com.tr/', 'seller': None, 'slogan': 'A101|Harca Harca Bitmez', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655489830306}, {'id': '767094934', 'price': 13299, 'title': 'iPhone 11 64 GB Siyah', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '4093', 'url': 'http://www.turkcell.com.tr', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494357178}, {'id': '855428213', 'price': 13350, 'title': 'iPhone 11 64 GB Siyah (Apple Türkiye Garantili) - (Aksesuarsız Kutu)', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '12192', 'url': 'https://www.needion.com/', 'seller': None, 'slogan': 'Alan Memnun, Satan Memnun', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': True, 'feedLastProcess': 1655495321932}, {'id': '914643935', 'price': 13379, 'title': 'Apple iPhone 11 64GB Akıllı Cep Telefonu Siyah ( Apple Türkiye Garantili )', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '12665', 'url': 'https://wnt.com.tr/', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494393339}, {'id': '491596021', 'price': 13398.99, 'title': 'iPhone 11 64 GB - Siyah - Aksesuarsız Kutu', 'shipping': {'fee': None, 'day': 0}, 'merchant': {'id': '1497', 'url': 'http://www.hepsiburada.com', 'seller': 'Hepsiburada', 'slogan': 'Türkiye’de İlk Defa Alışveriş Kredisi Burada', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495861494}, {'id': '1027767764', 'price': 13459, 'title': 'Apple iPhone 11 64GB Siyah Cep Telefonu (Apple Türkiye Garantili) (Siyah)', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '5825', 'url': 'https://www.vodafone.com.tr/', 'seller': 'GENÇPA', 'slogan': 'Vodafone Her Şey Yanımda Alışverişlerinizde Kargo Bedava, Vodafone’lulara İlk Siparişe 15GB Hediye!', 'onlyApp': True}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495537182}, {'id': '952479181', 'price': 13499, 'title': 'iPhone 11 64 GB Aksesuarsız Kutu Siyah', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '10109', 'url': 'https://www.trendyol.com/', 'seller': 'Teknosa', 'slogan': "Tüm İhtiyaçların Trendyol'da, Tek Tıkla Kapında!", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655493405188}, {'id': '1032790438', 'price': 13499, 'title': 'Apple iPhone 11 64 GB Siyah (Apple Türkiye Garantili) (Siyah)', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '5825', 'url': 'https://www.vodafone.com.tr/', 'seller': 'CepHane Teknoloji', 'slogan': 'Vodafone Her Şey Yanımda Alışverişlerinizde Kargo Bedava, Vodafone’lulara İlk Siparişe 15GB Hediye!', 'onlyApp': True}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495537182}, {'id': '1063638766', 'price': 13509, 'title': 'Apple iPhone 11 64 GB (Apple Türkiye Garantili) siyah', 'shipping': {'fee': 0, 'day': 2}, 'merchant': {'id': '7000', 'url': 'http://www.n11.com', 'seller': 'cokteknolojik', 'slogan': 'Hayat Sana Gelir', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495952093}, {'id': '930481030', 'price': 13559, 'title': 'iPhone 11 64GB Black', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '13105', 'url': 'https://www.inbox.com.tr/', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655491323973}, {'id': '1007434732', 'price': 13633, 'title': 'Apple iPhone 11 64 GB (Apple Türkiye Garantili) siyah', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '7000', 'url': 'http://www.n11.com', 'seller': 'yukseliletisim', 'slogan': 'Hayat Sana Gelir', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495952093}, {'id': '912636803', 'price': 13699, 'title': 'Apple iPhone 11 64GB Siyah - MHDA3TU/A', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '11403', 'url': 'https://gurgencler.com.tr/', 'seller': None, 'slogan': '10.000TL üzeri alışverişinize 500TL Bankkart Lira hediye!', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655492821318}, {'id': '974603297', 'price': 13779, 'title': 'IPhone 11 64 GB Siyah', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '12938', 'url': 'https://www.cepux.com.tr/', 'seller': None, 'slogan': 'Cepux Kazandırır!', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495878822}, {'id': '960442769', 'price': 13779, 'title': 'Apple iPhone 11 64 GB Aksesuarsız Kutu Siyah (Siyah)', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '5825', 'url': 'https://www.vodafone.com.tr/', 'seller': 'Tokdemir Bilişim Teknolojileri', 'slogan': 'Vodafone Her Şey Yanımda Alışverişlerinizde Kargo Bedava, Vodafone’lulara İlk Siparişe 15GB Hediye!', 'onlyApp': True}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495537182}, {'id': '965671448', 'price': 13789, 'title': 'iPhone 11 64 GB Siyah', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '4093', 'url': 'http://www.turkcell.com.tr', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494357178}, {'id': '808307738', 'price': 13798, 'title': 'iPhone 11 64 GB Siyah', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '4093', 'url': 'http://www.turkcell.com.tr', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494357178}, {'id': '1059226040', 'price': 13799, 'title': 'Apple iPhone 11 64 GB (Apple Türkiye Garantili) siyah', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '7000', 'url': 'http://www.n11.com', 'seller': 'integrabilisim', 'slogan': 'Hayat Sana Gelir', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495952093}, {'id': '734101431', 'price': 13799, 'title': 'Iphone 11 64 Gb Siyah Aksesuarsız Kutu', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '11512', 'url': 'https://www.pttavm.com', 'seller': 'BittiBitiyor', 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495542557}, {'id': '765227763', 'price': 14484, 'title': 'Apple iPhone 11 64Gb Akıllı Telefon Siyah', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '46', 'url': 'http://www.teknosa.com/', 'seller': 'Bittibitiyor', 'slogan': "Sepette 2.500 TL'ye Varan İndirimler", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655489853593}, {'id': '765957525', 'price': 14499, 'title': 'iPhone 11 64 Gb Siyah', 'shipping': {'fee': 15, 'day': 1}, 'merchant': {'id': '11375', 'url': 'https://www.vatanbilgisayar.com/', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655489535355}, {'id': '765434969', 'price': 14499, 'title': 'APPLE iPhone 11 64GB Akıllı Telefon Siyah', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '4230', 'url': 'http://www.mediamarkt.com.tr/', 'seller': None, 'slogan': '250 TL ve üzeri alışverişlerde geçerli ücretsiz kargo', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655489226707}, {'id': '765500161', 'price': 14549.03, 'title': 'Apple İphone 11 64Gb Siyah - Mhda3tu/A 194252097335', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '10052', 'url': 'http://www.troyestore.com/', 'seller': None, 'slogan': 'Apple Yetkili Satıcı, Apple Yetkili Teknik Servis', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655493437121}, {'id': '765498940', 'price': 14999, 'title': 'Apple iPhone 11 64 Gb Siyah Cep Telefonu', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '10313', 'url': 'https://www.migros.com.tr/', 'seller': None, 'slogan': '250 TL ve Üzeri Ücretsiz Teslimat', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655488948137}, {'id': '765987003', 'price': 14999, 'title': 'Apple iPhone 11 64GB Siyah Yeni Cep Telefonu', 'shipping': {'fee': 0, 'day': 2}, 'merchant': {'id': '10604', 'url': 'https://www.beko.com.tr/', 'seller': None, 'slogan': 'Ücretsiz ve Hızlı Teslimat ile Beko Kapında!', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655492802285}, {'id': '765350098', 'price': 14999, 'title': 'Apple iPhone 11 64GB Siyah Yeni Cep Telefonu', 'shipping': {'fee': 0, 'day': 2}, 'merchant': {'id': '10591', 'url': 'https://www.arcelik.com.tr/', 'seller': None, 'slogan': 'Ücretsiz ve Hızlı Teslimat ile Arçelik Kapında!', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655492539972}, {'id': '765171664', 'price': 14999, 'title': 'Apple iPhone 11 64GB Siyah (New Edt)', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '9334', 'url': 'http://www.evkur.com.tr/', 'seller': None, 'slogan': 'Hiç Peşinatsız Kredi Kartsız 24 Aya Varan Taksitlerle', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655491644137}, {'id': '806067367', 'price': 15292.64, 'title': 'İphone 11 64 Gb', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '5164', 'url': 'http://www.gittigidiyor.com/', 'seller': 'gncpazar', 'slogan': "Yapı Kredi World'e Özel 50 TL'ye Varan Worldpuan", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495287185}, {'id': '1068272963', 'price': 15599, 'title': 'İphone 11 64 Gb Siyah(kvk Garantili)', 'shipping': {'fee': 0, 'day': 3}, 'merchant': {'id': '5164', 'url': 'http://www.gittigidiyor.com/', 'seller': 'AYAZGRUP', 'slogan': "Yapı Kredi World'e Özel 50 TL'ye Varan Worldpuan", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495287185}, {'id': '793747418', 'price': 17591.37, 'title': 'Apple IPHONE 11 64GB Apple IPHONE 11 64GB Cep Telefonu (Apple Türkiye Garantili)', 'shipping': {'fee': 25, 'day': None}, 'merchant': {'id': '11898', 'url': 'https://www.turkuaztrade.com/', 'seller': None, 'slogan': 'Simply The Best / Kısaca En İyisi', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494006222}], 'imageIds': [219571586, 219571585, 219571582, 219571577, 219575009], 'review': {'count': 95, 'rate': 4}, 'badge': {'discountRatio': 0, 'isNew': False}, 'metaData': {'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah Fiyatları ve Özellikleri', 'description': "Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah modellerini tüm detayları ile Cimri.com'da inceleyin! Siz de Cimri.com ile Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah fiyatlarına ve özelliklerine kolayca ulaşabilir, ürüne ait kullanıcı yorum ve değerlendirmelerini inceleyebilirsiniz. Dilerseniz fiyat ve özellik karşılaştıraması yapabilir ve son 1 yıla ait fiyat değişimlerini takip edebilirsiniz. Cimri.com ile siz de en ucuz Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah fırsatlarını kaçırmayın!"}, 'specs': [{'name': 'Model Bilgisi', 'specEntries': [{'name': 'Iphone Modelleri', 'value': 'iPhone 11', 'order': 1}], 'order': 1}, {'name': 'Ekran Özellikleri', 'specEntries': [{'name': 'Ekran Boyutu', 'value': '6.1 inç', 'order': 3}, {'name': 'Ekran Teknolojisi', 'value': 'IPS LCD', 'order': 5}, {'name': 'Yan Ekran', 'value': 'Yok', 'order': 14}, {'name': 'Mavi Işık Filtresi', 'value': 'Var', 'order': 15}, {'name': 'Çözünürlük Standartı', 'value': 'HD+', 'order': 16}, {'name': 'Karanlık Modu', 'value': 'Var', 'order': 16}, {'name': 'Ekran Çözünürlüğü', 'value': '828x1792 Piksel', 'order': 21}, {'name': 'Ekran Gövde Oranı', 'value': '79 %', 'order': 21}, {'name': 'Piksel Yoğunluğu', 'value': '326 PPI', 'order': 22}, {'name': 'Multi Touch', 'value': 'Var', 'order': 22}, {'name': 'Dokunmatik Türü', 'value': 'Kapasitif Ekran', 'order': 23}, {'name': 'Ekran Parlaklığı (cd-m²)', 'value': '625', 'order': 26}, {'name': 'Çizilmeye Karşı Dayanıklılık', 'value': 'Var', 'order': 28}, {'name': 'Ekran Kontrast Oranı', 'value': '1400:1', 'order': 29}, {'name': 'Sürekli Açık Ekran', 'value': 'Yok', 'order': 30}], 'order': 1}, {'name': 'Teknik Özellikler', 'specEntries': [{'name': 'İşlemci Modeli', 'value': 'Apple A13 Bionic', 'order': 1}, {'name': 'İşlemci Frekansı', 'value': '2.66 GHz', 'order': 2}, {'name': 'RAM Kapasitesi', 'value': '4 GB RAM', 'order': 6}, {'name': 'İşletim Sistemi Versiyonu', 'value': 'iOS 13', 'order': 8}, {'name': 'İşletim Sistemi', 'value': 'iOS', 'order': 19}, {'name': 'İşlemci Çekirdek Sayısı', 'value': '6 Çekirdek', 'order': 26}, {'name': 'CPU Üretim Süreci', 'value': '7 nm', 'order': 27}, {'name': 'İşlemci Mimarisi', 'value': '64 bit', 'order': 28}, {'name': 'Ram Tipi', 'value': 'LPDDR4x', 'order': 30}, {'name': 'Ram Kanal Sayısı', 'value': 'Çift Kanal', 'order': 31}], 'order': 2}, {'name': 'Kamera Özellikleri', 'specEntries': [{'name': 'Kamera Çözünürlüğü', 'value': '12 MP', 'order': 4}, {'name': 'Ön (Selfie) Kamera Diyafram Açıklığı', 'value': 'F2.2', 'order': 6}, {'name': 'Ön (Selfie) Kamera Çözünürlüğü', 'value': '12 MP', 'order': 8}, {'name': 'Diyafram Açıklığı', 'value': 'F1.8', 'order': 9}, {'name': 'İkinci Arka Kamera', 'value': 'Var', 'order': 10}, {'name': 'İkinci Arka Kamera Diyafram Açıklığı', 'value': 'F2.4', 'order': 10}, {'name': 'Video Kayıt Çözünürlüğü', 'value': '2160p', 'order': 11}, {'name': 'Video FPS Değeri', 'value': '60 FPS', 'order': 12}, {'name': 'İkinci Arka Kamera Çözünürlüğü', 'value': '12 MP', 'order': 22}, {'name': 'Görüntü Sabitleyici', 'value': 'Var', 'order': 23}, {'name': 'Yüz Tanıma', 'value': 'Var', 'order': 24}, {'name': 'Slow Motion Video Çekimi', 'value': 'Var', 'order': 25}, {'name': 'Kamera Yapay Zeka (AI) Desteği', 'value': 'Yok', 'order': 26}, {'name': 'Time Lapse Video Çekimi', 'value': 'Var', 'order': 30}, {'name': 'Flaş Türü', 'value': 'Çift LED', 'order': 31}, {'name': 'Phase Detection', 'value': 'Var', 'order': 32}, {'name': 'HDR', 'value': 'Var', 'order': 33}, {'name': 'Zamanlayıcı', 'value': 'Var', 'order': 35}, {'name': 'Otomatik Odaklama', 'value': 'Var', 'order': 36}, {'name': 'Panoroma', 'value': 'Var', 'order': 37}, {'name': 'Coğrafi Konum Ekleme', 'value': 'Var', 'order': 38}], 'order': 3}, {'name': 'Tasarım', 'specEntries': [{'name': 'En', 'value': '75.7 mm', 'order': 1}, {'name': 'Boy', 'value': '150.9 mm', 'order': 2}, {'name': 'Kalınlık', 'value': '8.3 mm', 'order': 3}, {'name': 'Ağırlık', 'value': '194 gr', 'order': 7}, {'name': 'Gövde Malzemesi (Kapak)', 'value': 'Cam', 'order': 15}, {'name': 'Gövde Malzemesi (Çerçeve)', 'value': 'Alüminyum', 'order': 16}, {'name': 'Renk', 'value': 'Siyah', 'order': 31}], 'order': 4}, {'name': 'Batarya Özellikleri', 'specEntries': [{'name': 'Şarj Tipi', 'value': 'Lightning', 'order': 2}, {'name': 'Batarya Tipi', 'value': 'Li-ion', 'order': 11}, {'name': 'Kablosuz Hızlı Şarj', 'value': 'Yok', 'order': 12}, {'name': 'Kablosuz Hızlı Şarj Güç', 'value': 'Yok', 'order': 13}, {'name': 'Çıkarılabilir Batarya', 'value': 'Yok', 'order': 14}, {'name': 'Kablosuz Şarj', 'value': 'Var', 'order': 26}, {'name': 'Hızlı Şarj', 'value': 'Var', 'order': 27}], 'order': 5}, {'name': 'Depolama', 'specEntries': [{'name': 'Dahili Depolama (Hafıza)', 'value': '64 GB', 'order': 5}, {'name': 'Maksimum Hafıza Kartı Kapasitesi', 'value': 'Yok', 'order': 8}, {'name': 'Hafıza Kartı Desteği', 'value': 'Yok', 'order': 31}], 'order': 6}, {'name': 'Bağlantılar', 'specEntries': [{'name': 'Bluetooth Versiyonu', 'value': '5.0', 'order': 3}, {'name': 'Ses Çıkışı', 'value': 'Lightning', 'order': 5}, {'name': '4.5G Desteği', 'value': 'Var', 'order': 11}, {'name': 'Wİ-Fİ', 'value': 'Var', 'order': 11}, {'name': 'Wİ-Fİ Frekansı', 'value': 'Wi-Fi 6', 'order': 12}, {'name': 'Bluetooth', 'value': 'Var', 'order': 13}, {'name': 'Wİ-Fİ Hotspot', 'value': 'Var', 'order': 14}, {'name': 'Navigasyon', 'value': 'Var', 'order': 16}, {'name': '3G İndirme Hızı', 'value': '42.2 Mbps', 'order': 23}, {'name': '3G Yükleme Hızı', 'value': '5.76 Mbps', 'order': 24}, {'name': 'NFC', 'value': 'Var', 'order': 25}, {'name': '5G Desteği', 'value': 'Yok', 'order': 28}, {'name': 'Kızılötesi', 'value': 'Yok', 'order': 29}, {'name': 'USB Type-C', 'value': 'Yok', 'order': 30}], 'order': 7}, {'name': 'Ek Özellikler', 'specEntries': [{'name': 'Kalem', 'value': 'Yok', 'order': 11}, {'name': 'Güvenli Klasör', 'value': 'Yok', 'order': 12}, {'name': 'Antutu Puanı', 'value': '456900', 'order': 13}, {'name': 'Suya Dayanıklılık Seviyesi', 'value': 'IPX8', 'order': 14}, {'name': 'Toza Dayanıklılık Seviyesi', 'value': 'IP6X', 'order': 15}, {'name': 'Parmak İzi Okuyucu', 'value': 'Yok', 'order': 21}, {'name': 'Suya Dayanıklılık', 'value': 'Var', 'order': 23}, {'name': 'Toza Dayanıklılık', 'value': 'Var', 'order': 24}, {'name': 'Çıkış Tarihi', 'value': '2019', 'order': 35}, {'name': 'Parmak İzi Okuyucu Tipi', 'value': 'Yok', 'order': 37}, {'name': 'FM Radyo', 'value': 'Yok', 'order': 38}, {'name': 'Sim Kart Türü', 'value': 'Nano-SIM (4FF)', 'order': 39}], 'order': 8}, {'name': 'Sensörler', 'specEntries': [{'name': 'RGB Işık Sensörü', 'value': 'Yok', 'order': 2}, {'name': 'Bildirim Işığı', 'value': 'Yok', 'order': 8}, {'name': 'Barometre', 'value': 'Var', 'order': 31}, {'name': 'Jiroskop', 'value': 'Var', 'order': 32}, {'name': 'Hall Sensörü', 'value': 'Yok', 'order': 33}, {'name': 'Pusula', 'value': 'Var', 'order': 34}, {'name': 'Yakınlık Sensörü', 'value': 'Var', 'order': 35}, {'name': 'İvme Ölçer', 'value': 'Var', 'order': 36}], 'order': 9}, {'name': 'Kamera Özellikleri(Arka)', 'specEntries': [{'name': 'Elle Odaklama', 'value': 'Var', 'order': 8}, {'name': 'Seri Çekim Modu', 'value': 'Var', 'order': 10}, {'name': 'Zamanlayıcı', 'value': 'Var', 'order': 11}, {'name': 'Portre Modu', 'value': 'Var', 'order': 13}, {'name': 'Geniş Açı', 'value': 'Var', 'order': 13}, {'name': 'Yapay Zeka', 'value': 'Yok', 'order': 14}, {'name': 'Yüz Algılama', 'value': 'Var', 'order': 15}, {'name': 'Görüntü Sabitleyici', 'value': 'Var', 'order': 16}, {'name': 'Manuel Kontrol', 'value': 'Var', 'order': 17}, {'name': 'Hızlı Odaklama', 'value': 'Var', 'order': 52}, {'name': 'Odak Takibi', 'value': 'Var', 'order': 52}, {'name': 'Sesli Komut', 'value': 'Var', 'order': 52}, {'name': 'QR Kod Okuyucu', 'value': 'Yok', 'order': 55}], 'order': 10}, {'name': 'Kamera Özellikleri(Ön)', 'specEntries': [{'name': 'Geniş Açı', 'value': 'Yok', 'order': 2}, {'name': 'Otomatik Odaklama', 'value': 'Var', 'order': 3}, {'name': 'Panorama Özçekim', 'value': 'Yok', 'order': 4}, {'name': 'Seçmeli Odaklama', 'value': 'Yok', 'order': 5}, {'name': 'Sanal Flaş', 'value': 'Var', 'order': 8}, {'name': 'Zamanlayıcı', 'value': 'Var', 'order': 9}, {'name': 'Portre', 'value': 'Var', 'order': 18}, {'name': 'Sesle Komut', 'value': 'Yok', 'order': 19}, {'name': 'Yapay Zeka', 'value': 'Yok', 'order': 20}], 'order': 11}, {'name': 'Kamera Özellikleri(Video)', 'specEntries': [{'name': 'FPS', 'value': '60', 'order': 1}, {'name': 'Çözünürlük Standardı', 'value': '2160p', 'order': 2}, {'name': 'HDR', 'value': 'Var', 'order': 33}, {'name': 'Portre', 'value': 'Yok', 'order': 52}, {'name': 'Görüntü Sabitleme', 'value': 'Var', 'order': 53}, {'name': 'Ağır Çekim', 'value': 'Var', 'order': 54}, {'name': 'Hızlı Çekim', 'value': 'Var', 'order': 55}], 'order': 12}, {'name': 'USB', 'specEntries': [{'name': 'OTG', 'value': 'Var', 'order': 38}, {'name': 'DisplayPort', 'value': 'Yok', 'order': 39}, {'name': 'E-Sim', 'value': 'Var', 'order': 40}, {'name': 'Micro USB', 'value': 'Yok', 'order': 50}, {'name': 'Lightning', 'value': 'Var', 'order': 52}], 'order': 13}], 'topSpecs': [{'name': 'Dahili Depolama (Hafıza)', 'value': '64 GB', 'order': 5}, {'name': 'Diyafram Açıklığı', 'value': 'F1.8', 'order': 9}, {'name': 'Ekran Boyutu', 'value': '6.1 inç', 'order': 3}, {'name': 'Kamera Çözünürlüğü', 'value': '12 MP', 'order': 4}, {'name': 'Ön (Selfie) Kamera Çözünürlüğü', 'value': '12 MP', 'order': 8}], 'unitType': None, 'variantsHeader': 'Renk', 'variants': [{'id': '337337465', 'imageId': 178955343, 'minPrice': 13229, 'shortTitle': 'Beyaz', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Beyaz', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-beyaz-fiyatlari,337337465', 'unitPrice': None}, {'id': '337339455', 'imageId': 178955504, 'minPrice': 13249, 'shortTitle': 'Sarı', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Sarı', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-sari-fiyatlari,337339455', 'unitPrice': None}, {'id': '337338130', 'imageId': 178955431, 'minPrice': 13299, 'shortTitle': 'Yeşil', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Yeşil', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-yesil-fiyatlari,337338130', 'unitPrice': None}, {'id': '337338937', 'imageId': 178955480, 'minPrice': 13299, 'shortTitle': 'Mor', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Mor', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-mor-fiyatlari,337338937', 'unitPrice': None}, {'id': '337340793', 'imageId': 219571586, 'minPrice': 12986, 'shortTitle': 'Siyah', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-siyah-fiyatlari,337340793', 'unitPrice': None}, {'id': '337340093', 'imageId': 178955538, 'minPrice': 13385.59, 'shortTitle': 'Kırmızı', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Kırmızı', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-kirmizi-fiyatlari,337340093', 'unitPrice': None}], 'mainVariantsHeader': 'Kapasite', 'mainVariants': [{'id': '337341638', 'imageId': 178955480, 'minPrice': 12986, 'shortTitle': '64 GB', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-fiyatlari,a337341638'}, {'id': '331840845', 'imageId': 178954771, 'minPrice': 13999, 'shortTitle': '128 GB', 'title': 'Apple iPhone 11 128GB Akıllı Cep Telefonu', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-128gb-akilli-cep-telefonu-fiyatlari,a331840845'}, {'id': '337357780', 'imageId': 178954858, 'minPrice': 20899, 'shortTitle': '256 GB', 'title': 'Apple iPhone 11 256GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-256gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-fiyatlari,a337357780'}], 'brand': {'id': 4, 'name': 'Apple', 'slug': 'apple'}, 'category': {'id': '130', 'name': 'Cep Telefonları', 'slug': 'cep-telefonlari'}}

How to scrap data when the site kind of doesn't allow it? [duplicate]

This question already has answers here:
Not able to scrape dynamic content using Selenium or BeautifulSoup
(2 answers)
Closed last year.
I have been trying to scrap data from https://gov.gitcoin.co/u/owocki/summary using python's BeautifulSoup.
image: https://i.stack.imgur.com/0EgUk.png
Inspecting the page with Dev tools gives an idea but with the following code, I'm not getting the full HTML code returned or as it seems the site isn't allowing scraping if I'm correct.
from bs4 import BeautifulSoup
import requests
content= requests.get("https://gov.gitcoin.co/u/owocki/summary").text
soup= BeautifulSoup(content, 'html.parser')
print(soup)
Output:
<html lang="en">
<head>
<meta charset="utf-8"/>
<title> Profile - owocki - Gitcoin Governance
</title>
<meta content="Build & Fund the Open Web" name="description"/>
<meta content="Discourse 2.9.0.beta2 - https://github.com/discourse/discourse version 45c404a6c7b72bb8cb5aa9559c716bf88ce04a24" name="generator"/>
<link href="https://aws1.discourse-cdn.com/standard11/uploads/gitcoin1/optimized/1X/d6503c3e8f849859f29987d8816cd515c34b04f0_2_32x32.png" rel="icon" type="image/png"/>
<link href="https://aws1.discourse-cdn.com/standard11/uploads/gitcoin1/optimized/1X/d6503c3e8f849859f29987d8816cd515c34b04f0_2_180x180.png" rel="apple-touch-icon" type="image/png"/>
<meta content="#ffffff" name="theme-color"/>
<meta content="width=device-width, initial-scale=1.0, minimum-scale=1.0, user-scalable=yes, viewport-fit=cover" name="viewport"/>
<link href="https://gov.gitcoin.co/u/owocki/summary" rel="canonical">
<script type="application/ld+json">{"#context":"http://schema.org","#type":"WebSite","url":"https://gov.gitcoin.co","potentialAction":{"#type":"SearchAction","target":"https://gov.gitcoin.co/search?q={search_term_string}","query-input":"required name=search_term_string"}}</script>
<link href="https://gov.gitcoin.co/opensearch.xml" rel="search" title="Gitcoin Governance Search" type="application/opensearchdescription+xml"/>
<link data-target="desktop" href="https://sjc6.discourse-cdn.com/standard11/stylesheets/desktop_0b58afe0a66d2b0df8d67fb4eb0aa990540726fa.css?__ws=gov.gitcoin.co" media="all" rel="stylesheet">
<link data-target="desktop_theme" data-theme-id="1" data-theme-name="default" href="https://sjc6.discourse-cdn.com/standard11/stylesheets/desktop_theme_1_11b2df797b4b6664ccc65d5971f471f5e0094d1f.css?__ws=gov.gitcoin.co" media="all" rel="stylesheet">
<meta content="Gitcoin Governance" property="og:site_name">
<meta content="website" property="og:type">
<meta content="summary" name="twitter:card">
<meta content="https://sjc6.discourse-cdn.com/standard11/user_avatar/gov.gitcoin.co/owocki/45/3597_2.png" name="twitter:image">
<meta content="https://sjc6.discourse-cdn.com/standard11/user_avatar/gov.gitcoin.co/owocki/45/3597_2.png" property="og:image"><meta content="https://gov.gitcoin.co/u/owocki/summary" property="og:url"/>
<meta content="https://gov.gitcoin.co/u/owocki/summary" name="twitter:url"/>
<meta content="owocki" property="og:title"/>
<meta content="owocki" name="twitter:title"/>
<meta content="i spin bits for fun & profit; watch out for my megabyte." property="og:description"/>
<meta content="i spin bits for fun & profit; watch out for my megabyte." name="twitter:description"/>
</meta></meta></meta></meta></meta></link></link></link></head>
<body class="crawler">
<header>
<a href="/">
<img alt="Gitcoin Governance" id="site-logo" src="https://aws1.discourse-cdn.com/standard11/uploads/gitcoin1/original/1X/7074a77ab7768030574e7b2aebeff2d491886f55.png" style="max-width: 150px;"/>
</a>
</header>
<div class="wrap" id="main-outlet">
<div class="user-crawler">
<img alt="owocki" src="https://sjc6.discourse-cdn.com/standard11/user_avatar/gov.gitcoin.co/owocki/45/3597_2.png" title="owocki">
<h2 class="username">owocki</h2>
</img></div>
<p><p>i spin bits for fun & profit; watch out for my megabyte.</p></p>
</div>
<footer class="container wrap">
<nav class="crawler-nav">
<ul>
<li itemscope="" itemtype="http://schema.org/SiteNavigationElement">
<span itemprop="name">
Home
</span>
</li>
<li itemscope="" itemtype="http://schema.org/SiteNavigationElement">
<span itemprop="name">
Categories
</span>
</li>
<li itemscope="" itemtype="http://schema.org/SiteNavigationElement">
<span itemprop="name">
FAQ/Guidelines
</span>
</li>
<li itemscope="" itemtype="http://schema.org/SiteNavigationElement">
<span itemprop="name">
Terms of Service
</span>
</li>
<li itemscope="" itemtype="http://schema.org/SiteNavigationElement">
<span itemprop="name">
Privacy Policy
</span>
</li>
</ul>
</nav>
<p class="powered-by-link">Powered by Discourse, best viewed with JavaScript enabled</p>
</footer>
</body>
</html>
How to solve this issue & get to scrap? If you suggest to use Selenium, we need to specify it's webdriver locally, I have to get this working in production, please do mention then how to achieve that.
Really thanks.

What happens?
As mentioned in the comments content of website is provided dynamically, so you won't get your information with requests on that specific ressource / url, cause it is not able to render the website like a browser would do.
How to fix?
It do not need beautifulsoup for that task, cause there are ressources that will give you structured json data:
requests.get('https://gov.gitcoin.co/u/owocki.json').json()
requests.get('https://gov.gitcoin.co/u/owocki/summary.json').json()
Output
{'user_badges': [{'id': 4, 'granted_at': '2021-03-15T21:55:51.766Z', 'created_at': '2021-03-15T21:55:51.766Z', 'count': 1, 'badge_id': 4, 'user_id': 1, 'granted_by_id': -1}, {'id': 7832, 'granted_at': '2022-02-08T00:12:50.573Z', 'created_at': '2022-02-08T00:12:50.573Z', 'count': 1, 'post_id': 21272, 'post_number': 1, 'badge_id': 20, 'user_id': 1, 'granted_by_id': -1, 'topic_id': 8694}, {'id': 7327, 'granted_at': '2021-12-10T15:38:35.995Z', 'created_at': '2021-12-10T15:38:35.995Z', 'count': 1, 'badge_id': 37, 'user_id': 1, 'granted_by_id': -1}], 'badges': [{'id': 4, 'name': 'Leader', 'description': 'Granted global edit, pin, close, archive, split and merge, more likes', 'grant_count': 6, 'allow_title': True, 'multiple_grant': False, 'icon': 'fa-user', 'image_url': None, 'listable': True, 'enabled': True, 'badge_grouping_id': 4, 'system': True, 'slug': 'leader', 'manually_grantable': False, 'badge_type_id': 1}, {'id': 20, 'name': 'Great Topic', 'description': 'Received 50 likes on a topic', 'grant_count': 13, 'allow_title': False, 'multiple_grant': True, 'icon': 'fa-certificate', 'image_url': None, 'listable': True, 'enabled': True, 'badge_grouping_id': 3, 'system': True, 'slug': 'great-topic', 'manually_grantable': False, 'badge_type_id': 1}, {'id': 37, 'name': 'Respected', 'description': 'Received 2 likes on 100 posts', 'grant_count': 1, 'allow_title': False, 'multiple_grant': False, 'icon': 'fa-heart', 'image_url': None, 'listable': True, 'enabled': True, 'badge_grouping_id': 2, 'system': True, 'slug': 'respected', 'manually_grantable': False, 'badge_type_id': 2}], 'badge_types': [{'id': 1, 'name': 'Gold', 'sort_order': 9}, {'id': 2, 'name': 'Silver', 'sort_order': 8}], 'users': [{'id': 1, 'username': 'owocki', 'name': 'Owocki', 'avatar_template': '/user_avatar/gov.gitcoin.co/owocki/{size}/3597_2.png', 'flair_name': None, 'admin': True, 'trust_level': 4}, {'id': -1, 'username': 'system', 'name': 'system', 'avatar_template': 'https://aws1.discourse-cdn.com/standard11/uploads/gitcoin1/original/1X/d6503c3e8f849859f29987d8816cd515c34b04f0.png', 'flair_name': None, 'admin': True, 'moderator': True, 'trust_level': 4}], 'topics': [{'id': 8694, 'title': 'TLDR - What is Gitcoin? 🤖 [Updated Feb 2022]', 'fancy_title': 'TLDR - What is Gitcoin? :robot: [Updated Feb 2022]', 'slug': 'tldr-what-is-gitcoin-updated-feb-2022', 'posts_count': 7}], 'user': {'id': 1, 'username': 'owocki', 'name': 'Owocki', 'avatar_template': '/user_avatar/gov.gitcoin.co/owocki/{size}/3597_2.png', 'last_posted_at': '2022-02-14T14:20:48.874Z', 'last_seen_at': '2022-02-19T02:31:38.067Z', 'created_at': '2021-03-15T21:55:10.808Z', 'ignored': False, 'muted': False, 'can_ignore_user': False, 'can_mute_user': False, 'can_send_private_messages': False, 'can_send_private_message_to_user': False, 'trust_level': 4, 'moderator': False, 'admin': True, 'title': '', 'badge_count': 27, 'custom_fields': {}, 'time_read': 71929, 'recent_time_read': 25520, 'primary_group_id': None, 'primary_group_name': None, 'flair_group_id': None, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'featured_topic': None, 'bio_excerpt': 'i spin bits for fun & profit; watch out for my megabyte.', 'website': 'http://gitcoin.co', 'website_name': 'gitcoin.co', 'location': 'Boulder, CO', 'bio_raw': 'i spin bits for fun & profit; watch out for my megabyte.', 'bio_cooked': '<p>i spin bits for fun & profit; watch out for my megabyte.</p>', 'can_edit': False, 'can_edit_username': False, 'can_edit_email': False, 'can_edit_name': False, 'uploaded_avatar_id': 3597, 'pending_count': 0, 'profile_view_count': 1192, 'can_upload_profile_header': False, 'can_upload_user_card_background': False, 'custom_avatar_upload_id': 3597, 'custom_avatar_template': '/user_avatar/gov.gitcoin.co/owocki/{size}/3597_2.png', 'user_notification_schedule': {'enabled': False, 'day_0_start_time': 480, 'day_0_end_time': 1020, 'day_1_start_time': 480, 'day_1_end_time': 1020, 'day_2_start_time': 480, 'day_2_end_time': 1020, 'day_3_start_time': 480, 'day_3_end_time': 1020, 'day_4_start_time': 480, 'day_4_end_time': 1020, 'day_5_start_time': 480, 'day_5_end_time': 1020, 'day_6_start_time': 480, 'day_6_end_time': 1020}, 'accepted_answers': 0, 'featured_user_badge_ids': [4, 7832, 7327], 'invited_by': None, 'groups': []}}
and
{'topics': [{'id': 8383, 'title': 'Proposal to adjust Code of Conduct WRT Airdrops', 'fancy_title': 'Proposal to adjust Code of Conduct WRT Airdrops', 'slug': 'proposal-to-adjust-code-of-conduct-wrt-airdrops', 'posts_count': 36, 'category_id': 6, 'like_count': 147, 'created_at': '2021-08-25T17:48:20.856Z', 'has_accepted_answer': False}, {'id': 9503, 'title': 'A Vision for A Pluralistic Civilizational Scale Infrastructure for funding Public Goods', 'fancy_title': 'A Vision for A Pluralistic Civilizational Scale Infrastructure for funding Public Goods', 'slug': 'a-vision-for-a-pluralistic-civilizational-scale-infrastructure-for-funding-public-goods', 'posts_count': 22, 'category_id': 6, 'like_count': 80, 'created_at': '2021-12-24T16:45:12.082Z', 'has_accepted_answer': False}, {'id': 8694, 'title': 'TLDR - What is Gitcoin? 🤖 [Updated Feb 2022]', 'fancy_title': 'TLDR - What is Gitcoin? :robot: [Updated Feb 2022]', 'slug': 'tldr-what-is-gitcoin-updated-feb-2022', 'posts_count': 7, 'category_id': 6, 'like_count': 72, 'created_at': '2021-09-27T16:54:37.143Z', 'has_accepted_answer': False}, {'id': 8902, 'title': 'Calling all GitcoinDAO Metaverse Coordinators!', 'fancy_title': 'Calling all GitcoinDAO Metaverse Coordinators!', 'slug': 'calling-all-gitcoindao-metaverse-coordinators', 'posts_count': 14, 'category_id': 9, 'like_count': 71, 'created_at': '2021-10-28T23:00:54.169Z', 'has_accepted_answer': False}, {'id': 8497, 'title': 'What are the KPIs that GitcoinDAO Optimizes for?', 'fancy_title': 'What are the KPIs that GitcoinDAO Optimizes for?', 'slug': 'what-are-the-kpis-that-gitcoindao-optimizes-for', 'posts_count': 13, 'category_id': 6, 'like_count': 66, 'created_at': '2021-09-06T17:06:57.566Z', 'has_accepted_answer': False}, {'id': 8200, 'title': 'The Gitcoin/GitcoinDAO Egregore is Emerging', 'fancy_title': 'The Gitcoin/GitcoinDAO Egregore is Emerging', 'slug': 'the-gitcoin-gitcoindao-egregore-is-emerging', 'posts_count': 10, 'category_id': 6, 'like_count': 65, 'created_at': '2021-08-10T17:20:45.933Z', 'has_accepted_answer': False}, {'id': 9613, 'title': '2022 Roadmap & 2022 Goals', 'fancy_title': '2022 Roadmap & 2022 Goals', 'slug': '2022-roadmap-2022-goals', 'posts_count': 19, 'category_id': 6, 'like_count': 46, 'created_at': '2022-01-03T16:43:05.418Z', 'has_accepted_answer': False}, {'id': 8425, 'title': 'How Do You Create A Ban on Airdrops?', 'fancy_title': 'How Do You Create A Ban on Airdrops?', 'slug': 'how-do-you-create-a-ban-on-airdrops', 'posts_count': 12, 'category_id': 6, 'like_count': 54, 'created_at': '2021-08-29T05:58:55.365Z', 'has_accepted_answer': False}, {'id': 7388, 'title': '[Proposal] Second Claim Window for $GTC', 'fancy_title': '[Proposal] Second Claim Window for $GTC', 'slug': 'proposal-second-claim-window-for-gtc', 'posts_count': 71, 'category_id': 5, 'like_count': 237, 'created_at': '2021-06-26T18:09:47.850Z', 'has_accepted_answer': False}, {'id': 7506, 'title': 'Establishing a New Process for Identify Verification Scoring (and removing troubled ID methods)', 'fancy_title': 'Establishing a New Process for Identify Verification Scoring (and removing troubled ID methods)', 'slug': 'establishing-a-new-process-for-identify-verification-scoring-and-removing-troubled-id-methods', 'posts_count': 17, 'category_id': 5, 'like_count': 47, 'created_at': '2021-06-29T17:03:02.910Z', 'has_accepted_answer': False}, {'id': 6372, 'title': '[Proposal] GR10 Matching Pool: Use Gitcoin to build GitcoinDAO', 'fancy_title': '[Proposal] GR10 Matching Pool: Use Gitcoin to build GitcoinDAO', 'slug': 'proposal-gr10-matching-pool-use-gitcoin-to-build-gitcoindao', 'posts_count': 26, 'category_id': 5, 'like_count': 111, 'created_at': '2021-06-14T21:32:52.025Z', 'has_accepted_answer': False}, {'id': 9046, 'title': 'Schelling Point Event at ETHDenver 2/17/2022 [REQUEST FOR COORDINATION]', 'fancy_title': 'Schelling Point Event at ETHDenver 2/17/2022 [REQUEST FOR COORDINATION]', 'slug': 'schelling-point-event-at-ethdenver-2-17-2022-request-for-coordination', 'posts_count': 6, 'category_id': 9, 'like_count': 20, 'created_at': '2021-11-16T03:32:07.901Z', 'has_accepted_answer': False}], 'badges': [{'id': 4, 'name': 'Leader', 'description': 'Granted global edit, pin, close, archive, split and merge, more likes', 'grant_count': 6, 'allow_title': True, 'multiple_grant': False, 'icon': 'fa-user', 'image_url': None, 'listable': True, 'enabled': True, 'badge_grouping_id': 4, 'system': True, 'slug': 'leader', 'manually_grantable': False, 'badge_type_id': 1}, {'id': 20, 'name': 'Great Topic', 'description': 'Received 50 likes on a topic', 'grant_count': 13, 'allow_title': False, 'multiple_grant': True, 'icon': 'fa-certificate', 'image_url': None, 'listable': True, 'enabled': True, 'badge_grouping_id': 3, 'system': True, 'slug': 'great-topic', 'manually_grantable': False, 'badge_type_id': 1}, {'id': 37, 'name': 'Respected', 'description': 'Received 2 likes on 100 posts', 'grant_count': 1, 'allow_title': False, 'multiple_grant': False, 'icon': 'fa-heart', 'image_url': None, 'listable': True, 'enabled': True, 'badge_grouping_id': 2, 'system': True, 'slug': 'respected', 'manually_grantable': False, 'badge_type_id': 2}, {'id': 26, 'name': 'Campaigner', 'description': 'Invited 3 basic users', 'grant_count': 1, 'allow_title': False, 'multiple_grant': False, 'icon': 'fa-user-plus', 'image_url': None, 'listable': True, 'enabled': True, 'badge_grouping_id': 2, 'system': True, 'slug': 'campaigner', 'manually_grantable': False, 'badge_type_id': 2}, {'id': 19, 'name': 'Good Topic', 'description': 'Received 25 likes on a topic', 'grant_count': 26, 'allow_title': False, 'multiple_grant': True, 'icon': 'fa-certificate', 'image_url': None, 'listable': True, 'enabled': True, 'badge_grouping_id': 3, 'system': True, 'slug': 'good-topic', 'manually_grantable': False, 'badge_type_id': 2}, {'id': 25, 'name': 'Promoter', 'description': 'Invited a user', 'grant_count': 3, 'allow_title': False, 'multiple_grant': False, 'icon': 'fa-user-plus', 'image_url': None, 'listable': True, 'enabled': True, 'badge_grouping_id': 2, 'system': True, 'slug': 'promoter', 'manually_grantable': False, 'badge_type_id': 3}], 'badge_types': [{'id': 1, 'name': 'Gold', 'sort_order': 9}, {'id': 2, 'name': 'Silver', 'sort_order': 8}, {'id': 3, 'name': 'Bronze', 'sort_order': 7}], 'users': [{'id': 1, 'username': 'owocki', 'name': 'Owocki', 'avatar_template': '/user_avatar/gov.gitcoin.co/owocki/{size}/3597_2.png', 'flair_name': None, 'admin': True, 'trust_level': 4}, {'id': -1, 'username': 'system', 'name': 'system', 'avatar_template': 'https://aws1.discourse-cdn.com/standard11/uploads/gitcoin1/original/1X/d6503c3e8f849859f29987d8816cd515c34b04f0.png', 'flair_name': None, 'admin': True, 'moderator': True, 'trust_level': 4}], 'user_summary': {'likes_given': 24, 'likes_received': 1036, 'topics_entered': 240, 'posts_read_count': 1555, 'days_visited': 216, 'topic_count': 66, 'post_count': 129, 'time_read': 71929, 'recent_time_read': 25520, 'can_see_summary_stats': True, 'solved_count': 0, 'topic_ids': [8383, 9503, 8694, 8902, 8497, 8200], 'replies': [{'post_number': 16, 'like_count': 12, 'created_at': '2021-08-26T18:41:25.210Z', 'topic_id': 8383}, {'post_number': 3, 'like_count': 10, 'created_at': '2022-01-03T16:49:50.145Z', 'topic_id': 9613}, {'post_number': 2, 'like_count': 10, 'created_at': '2021-08-29T13:41:42.503Z', 'topic_id': 8425}, {'post_number': 18, 'like_count': 10, 'created_at': '2021-06-28T01:53:13.183Z', 'topic_id': 7388}, {'post_number': 2, 'like_count': 9, 'created_at': '2021-06-29T17:45:32.686Z', 'topic_id': 7506}, {'post_number': 21, 'like_count': 9, 'created_at': '2021-06-18T19:32:54.151Z', 'topic_id': 6372}], 'links': [{'url': 'https://gitcoin.co/earn', 'title': 'Earn | Gitcoin', 'clicks': 105, 'post_number': 1, 'topic_id': 8694}, {'url': 'https://gitcoin.co/mission', 'title': 'Gitcoin - Support open web development. | Gitcoin', 'clicks': 90, 'post_number': 1, 'topic_id': 8694}, {'url': 'https://schellingpoint.gitcoin.co/', 'title': 'SCHELLING P0INT | 2/17/2022 # SportsCastle in Denver CO USA', 'clicks': 86, 'post_number': 1, 'topic_id': 9046}, {'url': 'https://t.me/joinchat/DRkmhTiIH6E0MjYx', 'title': 'Telegram: Join Group Chat', 'clicks': 72, 'post_number': 1, 'topic_id': 8902}, {'url': 'https://gitcoin.co/discord', 'title': 'Gitcoin', 'clicks': 57, 'post_number': 1, 'topic_id': 8694}, {'url': 'https://gitcoin.co/learn', 'title': 'Learn | Gitcoin', 'clicks': 56, 'post_number': 1, 'topic_id': 8694}], 'most_liked_by_users': [{'id': 2044, 'username': 'Sirlupinwatson', 'name': 'Armand Brunelle', 'count': 104, 'avatar_template': '/user_avatar/gov.gitcoin.co/sirlupinwatson/{size}/549_2.png', 'admin': False, 'moderator': False, 'trust_level': 3, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 5974, 'username': 'BrianMillsJr', 'name': 'BrianMillsJr.eth', 'count': 36, 'avatar_template': '/user_avatar/gov.gitcoin.co/brianmillsjr/{size}/4581_2.png', 'admin': False, 'moderator': False, 'trust_level': 1, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 95, 'username': 'Fred', 'name': 'Fred', 'count': 28, 'avatar_template': '/user_avatar/gov.gitcoin.co/fred/{size}/3799_2.png', 'admin': False, 'moderator': True, 'trust_level': 3, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 5441, 'username': 'Huxwell', 'name': 'Cali', 'count': 19, 'avatar_template': '/user_avatar/gov.gitcoin.co/huxwell/{size}/3808_2.png', 'admin': False, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 5463, 'username': 'lee0007', 'name': 'Renee Lee', 'count': 18, 'avatar_template': '/user_avatar/gov.gitcoin.co/lee0007/{size}/3839_2.png', 'admin': False, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 2203, 'username': 'ntquang22298', 'name': 'Quang Nguyen', 'count': 15, 'avatar_template': '/user_avatar/gov.gitcoin.co/ntquang22298/{size}/2027_2.png', 'admin': False, 'moderator': False, 'trust_level': 1, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}], 'most_liked_users': [{'id': 17, 'username': 'Pop', 'name': 'Simona Pop', 'count': 3, 'avatar_template': '/user_avatar/gov.gitcoin.co/pop/{size}/270_2.png', 'admin': False, 'moderator': False, 'trust_level': 4, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 14, 'username': 'ceresstation', 'name': 'Scott', 'count': 3, 'avatar_template': '/user_avatar/gov.gitcoin.co/ceresstation/{size}/13_2.png', 'admin': True, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 5441, 'username': 'Huxwell', 'name': 'Cali', 'count': 2, 'avatar_template': '/user_avatar/gov.gitcoin.co/huxwell/{size}/3808_2.png', 'admin': False, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 15, 'username': 'Anett', 'name': 'Anett Rolikova', 'count': 1, 'avatar_template': '/user_avatar/gov.gitcoin.co/anett/{size}/425_2.png', 'admin': False, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 86, 'username': 'griff', 'name': 'Griff Green', 'count': 1, 'avatar_template': '/user_avatar/gov.gitcoin.co/griff/{size}/82_2.png', 'admin': False, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 25, 'username': 'trent', 'name': '', 'count': 1, 'avatar_template': '/user_avatar/gov.gitcoin.co/trent/{size}/32_2.png', 'admin': False, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}], 'most_replied_to_users': [{'id': 5303, 'username': 'Developer-piyush', 'name': 'PIYUSH CHOUDHARY', 'count': 4, 'avatar_template': '/user_avatar/gov.gitcoin.co/developer-piyush/{size}/3571_2.png', 'admin': False, 'moderator': False, 'trust_level': 1, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 5441, 'username': 'Huxwell', 'name': 'Cali', 'count': 4, 'avatar_template': '/user_avatar/gov.gitcoin.co/huxwell/{size}/3808_2.png', 'admin': False, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 66, 'username': 'auryn', 'name': 'auryn-macmillan', 'count': 3, 'avatar_template': '/user_avatar/gov.gitcoin.co/auryn/{size}/66_2.png', 'admin': False, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 3, 'username': 'kyle', 'name': 'kyle', 'count': 3, 'avatar_template': '/user_avatar/gov.gitcoin.co/kyle/{size}/3892_2.png', 'admin': True, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 5353, 'username': 'captnseagraves', 'name': 'Kevin Seagraves', 'count': 3, 'avatar_template': '/user_avatar/gov.gitcoin.co/captnseagraves/{size}/3656_2.png', 'admin': False, 'moderator': False, 'trust_level': 1, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}, {'id': 14, 'username': 'ceresstation', 'name': 'Scott', 'count': 2, 'avatar_template': '/user_avatar/gov.gitcoin.co/ceresstation/{size}/13_2.png', 'admin': True, 'moderator': False, 'trust_level': 2, 'flair_name': None, 'flair_url': None, 'flair_bg_color': None, 'flair_color': None, 'primary_group_name': None}], 'badges': [{'id': 4, 'granted_at': '2021-03-15T21:55:51.766Z', 'created_at': '2021-03-15T21:55:51.766Z', 'count': 1, 'badge_id': 4, 'user_id': 1, 'granted_by_id': -1}, {'id': 7832, 'granted_at': '2022-02-08T00:12:50.573Z', 'created_at': '2022-02-08T00:12:50.573Z', 'count': 1, 'post_id': 21272, 'post_number': 1, 'badge_id': 20, 'user_id': 1, 'granted_by_id': -1, 'topic_id': 8694}, {'id': 7327, 'granted_at': '2021-12-10T15:38:35.995Z', 'created_at': '2021-12-10T15:38:35.995Z', 'count': 1, 'badge_id': 37, 'user_id': 1, 'granted_by_id': -1}, {'id': 15, 'granted_at': '2021-03-16T23:43:02.686Z', 'created_at': '2021-03-16T23:43:02.686Z', 'count': 1, 'badge_id': 26, 'user_id': 1, 'granted_by_id': -1}, {'id': 7068, 'granted_at': '2021-11-15T20:00:11.215Z', 'created_at': '2021-11-15T20:00:11.215Z', 'count': 3, 'post_id': 21272, 'post_number': 1, 'badge_id': 19, 'user_id': 1, 'granted_by_id': -1, 'topic_id': 8694}, {'id': 14, 'granted_at': '2021-03-16T23:43:02.646Z', 'created_at': '2021-03-16T23:43:02.646Z', 'count': 1, 'badge_id': 25, 'user_id': 1, 'granted_by_id': -1}], 'top_categories': [{'topic_count': 31, 'post_count': 40, 'id': 6, 'name': '🤖 Meta-Governance', 'color': '652D90', 'text_color': 'FFFFFF', 'slug': 'governance-meta', 'read_restricted': False, 'parent_category_id': None}, {'topic_count': 6, 'post_count': 42, 'id': 5, 'name': '📜 Proposal Discussion', 'color': '0E76BD', 'text_color': 'FFFFFF', 'slug': 'governance-proposals', 'read_restricted': False, 'parent_category_id': None}, {'topic_count': 13, 'post_count': 14, 'id': 9, 'name': '🧙 🧙\u200d♀️ Workstream Discussion', 'color': '12A89D', 'text_color': 'FFFFFF', 'slug': 'workstream-discussion', 'read_restricted': False, 'parent_category_id': None}, {'topic_count': 8, 'post_count': 10, 'id': 8, 'name': '👋 Community Hangout', 'color': '3AB54A', 'text_color': 'FFFFFF', 'slug': 'community-hangout', 'read_restricted': False, 'parent_category_id': None}, {'topic_count': 7, 'post_count': 11, 'id': 12, 'name': 'Progressive Decentralization (Technical)', 'color': '808281', 'text_color': 'FFFFFF', 'slug': 'progressive-decentralization-technical', 'read_restricted': False, 'parent_category_id': 9}, {'topic_count': 1, 'post_count': 8, 'id': 13, 'name': 'Public Goods Funding', 'color': '9EB83B', 'text_color': 'FFFFFF', 'slug': 'public-goods-funding', 'read_restricted': False, 'parent_category_id': 9}]}}

How can i append dictionary to my existing key "processed_data" in mongodb using pymongo

I am trying to append a dictionary to my already existing key "processed_data" where data is saved in the list of dictionaries. I tried several methods as shown in already asked questions but they did not work. This is my schema.
{'_id': ObjectId('5fe46a5b7468e3498124fcbe'), 'metadata': {'_id': ObjectId('5fe4500c7b2c03decd86334f'), 'type': 'VIDEO', 'id': 'o6st4ces9Wg"},"qoeUrl":{"baseUrl":"https://s.youtube.com/api/stats/qoe?cl=348521801', 'user_id': 'fc3240b2d7ef9d33bbb04fd7203e35ea9da54ffb', 'name': 'City Ak47', 'thumbnail': 'https://i.ytimg.com/vi/o6st4ces9Wg/hqdefault.jpg', 'title': 'Alex Bhatti ki Video Viral Ho Gie | How To Become Tiktok Star | City AK47 - YouTube', 'publication_date': 'Sep 17, 2020', 'channel_id': 'UCuo6tBl2MfkWvMPyCqph2LA', 'channel_name': 'City Ak47', 'scrape_date': '2020-12-24 08:23:17.390018', 'regions_allowed': 'AD,AE,AF,AG,AI,AL,AM,AO,AQ,AR,AS,AT,AU,AW,AX,AZ,BA,BB,BD,BE,BF,BG,BH,BI,BJ,BL,BM,BN,BO,BQ,BR,BS,BT,BV,BW,BY,BZ,CA,CC,CD,CF,CG,CH,CI,CK,CL,CM,CN,CO,CR,CU,CV,CW,CX,CY,CZ,DE,DJ,DK,DM,DO,DZ,EC,EE,EG,EH,ER,ES,ET,FI,FJ,FK,FM,FO,FR,GA,GB,GD,GE,GF,GG,GH,GI,GL,GM,GN,GP,GQ,GR,GS,GT,GU,GW,GY,HK,HM,HN,HR,HT,HU,ID,IE,IL,IM,IN,IO,IQ,IR,IS,IT,JE,JM,JO,JP,KE,KG,KH,KI,KM,KN,KP,KR,KW,KY,KZ,LA,LB,LC,LI,LK,LR,LS,LT,LU,LV,LY,MA,MC,MD,ME,MF,MG,MH,MK,ML,MM,MN,MO,MP,MQ,MR,MS,MT,MU,MV,MW,MX,MY,MZ,NA,NC,NE,NF,NG,NI,NL,NO,NP,NR,NU,NZ,OM,PA,PE,PF,PG,PH,PK,PL,PM,PN,PR,PS,PT,PW,PY,QA,RE,RO,RS,RU,RW,SA,SB,SC,SD,SE,SG,SH,SI,SJ,SK,SL,SM,SN,SO,SR,SS,ST,SV,SX,SY,SZ,TC,TD,TF,TG,TH,TJ,TK,TL,TM,TN,TO,TR,TT,TV,TW,TZ,UA,UG,UM,US,UY,UZ,VA,VC,VE,VG,VI,VN,VU,WF,WS,YE,YT,ZA,ZM,ZW', 'views': '663962', 'is_family_friendly': 'true', 'category': 'Entertainment', 'tags': ['AmirFilms', 'Alex Bhatti ki Video Viral Ho Gie | How To Become Tiktok Star | City AK47', 'Tiktok star', 'Tiktok', 'Alex tiktokr', 'Alex bhatti tiktok star', 'Alex bhatti', 'Ayesha bukhari', 'Viral video', 'New video', 'Leak vidro', 'Ayesha leak video', 'Alex bhatti leak video', 'News', 'Tiktik funny video'], 'language': 'en-US', 'width': '480', 'height': '360', 'job_id': '539f61c4183c46448a75cfb65dc40926'}, 'results': {'unique_word_freq': [{'text': 'hai', 'value': 6}, {'text': 'famous', 'value': 4}, {'text': 'allah', 'value': 3}, {'text': 'kar', 'value': 3}, {'text': 'gy', 'value': 3}, {'text': 'ye', 'value': 3}, {'text': 'yeh', 'value': 2}, {'text': 'ka', 'value': 2}, {'text': 'video', 'value': 2}, {'text': 'asee', 'value': 2}, {'text': 'nhi', 'value': 2}, {'text': 'ho', 'value': 2}, {'text': 'tum', 'value': 2}, {'text': 'jao', 'value': 2}, {'text': 'kitna', 'value': 1}, {'text': 'budsoor', 'value': 1}, {'text': 'gundgi', 'value': 1}, {'text': 'dher', 'value': 1}, {'text': 'khusra', 'value': 1}, {'text': 'tiktok', 'value': 1}, {'text': 'kunjuro', 'value': 1}, {'text': 'zanano', 'value': 1}, {'text': 'kaam', 'value': 1}, {'text': 'usko', 'value': 1}, {'text': 'hadyat', 'value': 1}, {'text': 'de', 'value': 1}, {'text': 'ameen', 'value': 1}, {'text': '😔', 'value': 1}, {'text': 'kahn', 'value': 1}, {'text': 'puri', 'value': 1}, {'text': 'kotta', 'value': 1}, {'text': 'ٹک', 'value': 1}, {'text': 'ٹاک', 'value': 1}, {'text': 'ایپ', 'value': 1}, {'text': 'پر', 'value': 1}, {'text': 'پاکستان', 'value': 1}, {'text': 'میں', 'value': 1}, {'text': 'مکمل', 'value': 1}, {'text': 'پابندی', 'value': 1}, {'text': 'لگنی', 'value': 1}, {'text': 'چاہیے', 'value': 1}, {'text': 'leaked', 'value': 1}, {'text': 'purpose', 'value': 1}, {'text': 'fame', 'value': 1}, {'text': 'views', 'value': 1}, {'text': 'mean', 'value': 1}, {'text': 'people', 'value': 1}, {'text': 'like', 'value': 1}, {'text': 'kinda', 'value': 1}, {'text': 'cheap', 'value': 1}, {'text': 'acts', 'value': 1}, {'text': 'inki', 'value': 1}, {'text': 'maa', 'value': 1}, {'text': 'bhano', 'value': 1}, {'text': 'sath', 'value': 1}, {'text': 'bhi', 'value': 1}, {'text': 'hoo', 'value': 1}, {'text': 'pak', 'value': 1}, {'text': 'ko', 'value': 1}, {'text': 'bohot', 'value': 1}, {'text': 'bari', 'value': 1}, {'text': 'sazaa', 'value': 1}, {'text': 'dee', 'value': 1}, {'text': 'duniyan', 'value': 1}, {'text': 'hee', 'value': 1}, {'text': 'dikhaee', 'value': 1}, {'text': 'pata', 'value': 1}, {'text': 'khha', 'value': 1}, {'text': 'jay', 'value': 1}, {'text': 'kiyamat', 'value': 1}, {'text': 'din', 'value': 1}, {'text': 'logo', 'value': 1}, {'text': 'hisab', 'value': 1}, {'text': 'lena', 'value': 1}, {'text': 'log', 'value': 1}, {'text': 'sidah', 'value': 1}, {'text': 'janat', 'value': 1}, {'text': 'chaly', 'value': 1}, {'text': 'baaz', 'value': 1}, {'text': 'ap', 'value': 1}, {'text': 'bakwas', 'value': 1}, {'text': 'band', 'value': 1}, {'text': 'kareen', 'value': 1}, {'text': 'larka', 'value': 1}, {'text': 'bharva', 'value': 1}, {'text': 'bs', 'value': 1}, {'text': 'pakar', 'value': 1}, {'text': 'gal', 'value': 1}, {'text': 'ma', 'value': 1}, {'text': 'dala', 'value': 1}, {'text': 'gaya', 'value': 1}, {'text': 'bahut', 'value': 1}, {'text': 'ghatiya', 'value': 1}, {'text': 'insan', 'value': 1}, {'text': 'tu', 'value': 1}, {'text': 'chakka', 'value': 1}, {'text': 'alex', 'value': 1}, {'text': 'bhatti', 'value': 1}], 'polarity_freq': [{'date': '2020-12-03', 'total': 4, 'positive': 3, 'negative': 1}, {'date': '2020-12-10', 'total': 9, 'positive': 8, 'negative': 1}, {'date': '2020-12-17', 'total': 2, 'positive': 2, 'negative': 0}, {'date': '2020-12-21', 'total': 1, 'positive': 1, 'negative': 0}, {'date': '2020-12-22', 'total': 2, 'positive': 1, 'negative': 1}], 'polarity_dist': [{'name': 'positive', 'value': '15'}, {'name': 'negative', 'value': '3'}], 'assoc': []}, 'processed_data': [{'index': 0, '_id': ObjectId('5fe4500c7b2c03decd863350'), 'channel_id': '/channel/UCg7rf8yXy8wqVxlbnErgdyg', 'clean_text': 'kitna budsoor hai yeh gundgi ka dher khusra', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniDMBIClPo0sPLX5RDOLPHTJhECMOub-fC0ZTVY6Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-22 08:23:17', 'id': 'UgxzkGuC2JpeaZD7El14AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 days ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Kitna budsoor hai yeh gundgi ka dher khusra.', 'tokens': ['kitna', 'budsoor', 'hai', 'yeh', 'gundgi', 'ka', 'dher', 'khusra'], 'tokens_no_swords': ['kitna', 'budsoor', 'hai', 'yeh', 'gundgi', 'ka', 'dher', 'khusra'], 'tran_text': 'kitna budsoor hai yeh gundgi ka dher khusra .', 'type': 'COMMENT', 'user_id': 'f7961259b974ba9fae934410fca2e939d3493038', 'user_name': 'jimmi khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '1', 'date': '2020-12-22'}, {'index': 1, '_id': ObjectId('5fe4500c7b2c03decd863351'), 'channel_id': '/channel/UCg7rf8yXy8wqVxlbnErgdyg', 'clean_text': 'tiktok kunjuro zanano ka kaam hai', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniDMBIClPo0sPLX5RDOLPHTJhECMOub-fC0ZTVY6Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-22 08:23:17', 'id': 'UgwntMkhi7J2l2N3MZJ4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 days ago (edited)', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Tiktok kunjuro r zanano ka kaam hai.', 'tokens': ['tiktok', 'kunjuro', 'zanano', 'ka', 'kaam', 'hai'], 'tokens_no_swords': ['tiktok', 'kunjuro', 'zanano', 'ka', 'kaam', 'hai'], 'tran_text': 'tiktok kunjuro r zanano ka kaam hai .', 'type': 'COMMENT', 'user_id': 'f7961259b974ba9fae934410fca2e939d3493038', 'user_name': 'jimmi khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-22'}, {'index': 2, '_id': ObjectId('5fe4500c7b2c03decd863352'), 'channel_id': '/channel/UCMDNByou1B62upgmnv-UQMw', 'clean_text': 'allah usko hadyat de ameen 😔', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnik2uW0mzYoagKEYX1_kGY3HDhYd3Ni6UlOxSEHOA=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-21 08:23:17', 'id': 'UgxZrbzomoOLyGEGAjp4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 days ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Allah usko hadyat de ameen 😔', 'tokens': ['allah', 'usko', 'hadyat', 'de', 'ameen', '😔'], 'tokens_no_swords': ['allah', 'usko', 'hadyat', 'de', 'ameen', '😔'], 'tran_text': 'allah usko hadyat de ameen 😔', 'type': 'COMMENT', 'user_id': 'da9fe12c7945488a70f56355f8c122d2f35231c5', 'user_name': 'neha Rajput', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-21'}, {'index': 3, '_id': ObjectId('5fe4500c7b2c03decd863353'), 'channel_id': '/channel/UCkl4U918shu8CroBno8-aJg', 'clean_text': '', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniom0S4ta4uSnNx7yD69NfR4TmOqXPpYxv6_Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-17 08:23:17', 'id': 'UgzAfvcluRdyX9yi-JJ4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '1 week ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': '420', 'tokens': [], 'tokens_no_swords': [], 'tran_text': '420', 'type': 'COMMENT', 'user_id': '04b4dd4534a4acf47ba876387d752eda8d3087f6', 'user_name': 'Shahid Khankarachi', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-17'}, {'index': 5, '_id': ObjectId('5fe4500c7b2c03decd863355'), 'channel_id': '/channel/UCoL0h9EyBTNSvKIWIxl6WIg', 'clean_text': 'kahn hai yeh puri video', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnjNJAzxyS9mOk-R7TF5ICxa0_EQbtgcL3z2Yg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-17 08:23:17', 'id': 'Ugxx-JvihbK7P8Y8u5x4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '1 week ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Kahn hai yeh puri video', 'tokens': ['kahn', 'hai', 'yeh', 'puri', 'video'], 'tokens_no_swords': ['kahn', 'hai', 'yeh', 'puri', 'video'], 'tran_text': 'kahn hai yeh puri video', 'type': 'COMMENT', 'user_id': '3bc410f7e5133b61e2f2cc790ce6ae2692397778', 'user_name': 'ALISHA ZOYA', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-17'}, {'index': 6, '_id': ObjectId('5fe4500c7b2c03decd863356'), 'channel_id': '/channel/UCsd6TX3yWpNYK55hawyi8qw', 'clean_text': 'kotta', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnj19uWVIJ75wx27KLjDGDcsVcGtzVtp8SRQ0w=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgyhHfBJDWHtL73E71N4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Kotta', 'tokens': ['kotta'], 'tokens_no_swords': ['kotta'], 'tran_text': 'kotta', 'type': 'COMMENT', 'user_id': 'e83b422e66c1bd722306aee6715c3846c32e506b', 'user_name': 'Shakeel Khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 7, '_id': ObjectId('5fe4500c7b2c03decd863357'), 'channel_id': '/channel/UCX6LjA5LbC7xMO19yyM7m0Q', 'clean_text': 'ٹک ٹاک ایپ پر پاکستان میں مکمل پابندی لگنی چاہیے', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwni4WpKnrXzHmw2VwT0z5aYnM0T5IhRN0DG3Pmsg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgwFILpwDAQKYA9ioMV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'en', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'ٹک ٹاک ایپ پر پاکستان میں مکمل پابندی لگنی چاہیے', 'tokens': ['ٹک', 'ٹاک', 'ایپ', 'پر', 'پاکستان', 'میں', 'مکمل', 'پابندی', 'لگنی', 'چاہیے'], 'tokens_no_swords': ['ٹک', 'ٹاک', 'ایپ', 'پر', 'پاکستان', 'میں', 'مکمل', 'پابندی', 'لگنی', 'چاہیے'], 'tran_text': 'ٹک ٹاک ایپ پر پاکستان میں مکمل پابندی لگنی چاہیے', 'type': 'COMMENT', 'user_id': '9ca6083ff6234bd94fc218ef27d12c8b91c2fa33', 'user_name': 'Wahab Mirza', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 8, '_id': ObjectId('5fe4500c7b2c03decd863358'), 'channel_id': '/channel/UCKMvpfSppW24ixWCOmJDu_g', 'clean_text': 'he leaked this video on purpose to get fame and views i mean people like them do these kinda cheap acts to get famous', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniL8sePcWPsqDg6AOaLsW4nf14XDW3132kC0Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'Ugz7IdhD6s8zCJ6vHNt4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'en', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'He leaked this video on purpose to get fame and views...I mean people like them do these kinda cheap acts to get famous.', 'tokens': ['he', 'leaked', 'this', 'video', 'on', 'purpose', 'to', 'get', 'fame', 'and', 'views', 'i', 'mean', 'people', 'like', 'them', 'do', 'these', 'kinda', 'cheap', 'acts', 'to', 'get', 'famous'], 'tokens_no_swords': ['leaked', 'video', 'purpose', 'fame', 'views', 'mean', 'people', 'like', 'kinda', 'cheap', 'acts', 'famous'], 'tran_text': 'he leaked this video on purpose to get fame and views ... i mean people like them do these kinda cheap acts to get famous .', 'type': 'COMMENT', 'user_id': '3f75585892685df3ae4b3d733d9795a719b2d528', 'user_name': 'Ana T', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 9, '_id': ObjectId('5fe4500c7b2c03decd863359'), 'channel_id': '/channel/UCSUGRfHKn5qCNN4TKG3MAkw', 'clean_text': '', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwng6eAHeRd7CcM8mmkCHCA8VI2tqmMNPb1q1MA=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgwMvbYR29V4ISyic_d4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'en', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': '03065455318', 'tokens': [], 'tokens_no_swords': [], 'tran_text': '03065455318', 'type': 'COMMENT', 'user_id': '23ac482ba9b36182915c502c13d4cd45b7f7bf1f', 'user_name': 'Ali Rizwan', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 10, '_id': ObjectId('5fe4500c7b2c03decd86335a'), 'channel_id': '/channel/UC-fWQ2vkmngdVliZDkRSiiQ', 'clean_text': 'inki maa bhano sath bhi asee hoo', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwngl0V2Zy_AUGUyIZpMbrBDxqL6pq5AcdF4hNg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgzEMbiRNk2ywQHlgxR4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Inki maa bhano k sath bhi Asee hoo', 'tokens': ['inki', 'maa', 'bhano', 'sath', 'bhi', 'asee', 'hoo'], 'tokens_no_swords': ['inki', 'maa', 'bhano', 'sath', 'bhi', 'asee', 'hoo'], 'tran_text': 'inki maa bhano k sath bhi asee hoo', 'type': 'COMMENT', 'user_id': '2c7e490aa5d0ceca9340c92c3577fa75d3e5a8d3', 'user_name': 'M wali Yousuf', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 11, '_id': ObjectId('5fe4500c7b2c03decd86335b'), 'channel_id': '/channel/UC-fWQ2vkmngdVliZDkRSiiQ', 'clean_text': 'allah pak asee ko bohot bari sazaa dee or duniyan me hee dikhaee', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwngl0V2Zy_AUGUyIZpMbrBDxqL6pq5AcdF4hNg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'Ugyo9YlNa7zuVsSQlZh4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Allah Pak Asee ko Bohot bari Sazaa Dee Or Duniyan me hee dikhaee', 'tokens': ['allah', 'pak', 'asee', 'ko', 'bohot', 'bari', 'sazaa', 'dee', 'or', 'duniyan', 'me', 'hee', 'dikhaee'], 'tokens_no_swords': ['allah', 'pak', 'asee', 'ko', 'bohot', 'bari', 'sazaa', 'dee', 'duniyan', 'hee', 'dikhaee'], 'tran_text': 'allah pak asee ko bohot bari sazaa dee or duniyan me hee dikhaee', 'type': 'COMMENT', 'user_id': '2c7e490aa5d0ceca9340c92c3577fa75d3e5a8d3', 'user_name': 'M wali Yousuf', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 12, '_id': ObjectId('5fe4500c7b2c03decd86335c'), 'channel_id': '/channel/UCphcNEEoxrp08DARCX7dSNQ', 'clean_text': 'pata nhi famous ho kar khha jay gy kiyamat din allah famous logo hisab lena hai ye nhi tum log famous ho gy or sidah janat chaly jao gy baaz a jao', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnibNbjcM0UMLW2aTnOD3jfJXlaq2Iq5_hMg3Q-O=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgzfXCaZHDMSCKPqlKB4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': '1', 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Pata nhi famous ho Kar khha jay gy. Kiyamat k din Allah n famous logo c b hisab lena hai ye nhi k tum log famous ho gy or sidah janat m chaly jao gy. Baaz a jao', 'tokens': ['pata', 'nhi', 'famous', 'ho', 'kar', 'khha', 'jay', 'gy', 'kiyamat', 'din', 'allah', 'famous', 'logo', 'hisab', 'lena', 'hai', 'ye', 'nhi', 'tum', 'log', 'famous', 'ho', 'gy', 'or', 'sidah', 'janat', 'chaly', 'jao', 'gy', 'baaz', 'a', 'jao'], 'tokens_no_swords': ['pata', 'nhi', 'famous', 'ho', 'kar', 'khha', 'jay', 'gy', 'kiyamat', 'din', 'allah', 'famous', 'logo', 'hisab', 'lena', 'hai', 'ye', 'nhi', 'tum', 'log', 'famous', 'ho', 'gy', 'sidah', 'janat', 'chaly', 'jao', 'gy', 'baaz', 'jao'], 'tran_text': 'pata nhi famous ho kar khha jay gy . kiyamat k din allah n famous logo c b hisab lena hai ye nhi k tum log famous ho gy or sidah janat m chaly jao gy . baaz a jao', 'type': 'COMMENT', 'user_id': 'da5c845fbd0a39db29a99a9d620bd8c266956065', 'user_name': 'Rida Khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '1', 'date': '2020-12-10'}, {'index': 13, '_id': ObjectId('5fe4500c7b2c03decd86335d'), 'channel_id': '/channel/UCnVVsV2fd3P0lS9QClU5DCA', 'clean_text': 'ap bakwas band kareen', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniJxcGaZeKzmvDSUGeX5vFZo3m_ZXQ_yC7-Kw=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgwzEHvn_WWTbPOzl7x4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Ap bakwas band kareen', 'tokens': ['ap', 'bakwas', 'band', 'kareen'], 'tokens_no_swords': ['ap', 'bakwas', 'band', 'kareen'], 'tran_text': 'ap bakwas band kareen', 'type': 'COMMENT', 'user_id': 'e51ff2c592a9ad2fe8f6f373c6a2dab117f2c2e9', 'user_name': 'ahmad muaaz', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 14, '_id': ObjectId('5fe4500c7b2c03decd86335e'), 'channel_id': '/channel/UCtbUvUvL0qrREfEzUvJUQKQ', 'clean_text': 'ye larka bharva', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnjKBfmUKuMzFCtwM-KuKAfq_5y0RA7iez5w9Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgzvW63udc6CEwnsb_J4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'ye larka bharva', 'tokens': ['ye', 'larka', 'bharva'], 'tokens_no_swords': ['ye', 'larka', 'bharva'], 'tran_text': 'ye larka bharva', 'type': 'COMMENT', 'user_id': '7d7c7165c38c05a96c335421faf6ca3eb9eb1722', 'user_name': 'Rana Waqas', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 15, '_id': ObjectId('5fe4500c7b2c03decd86335f'), 'channel_id': '/channel/UCieduNjSrF2DPawdZh_HesQ', 'clean_text': 'bs kar do tum', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniOEn-mQTkzQu5ybCc6gjFqSlK8eQF-4RsB6w=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'Ugx9gyNwYeVV5DDKsJV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'bs kar do tum', 'tokens': ['bs', 'kar', 'do', 'tum'], 'tokens_no_swords': ['bs', 'kar', 'tum'], 'tran_text': 'bs kar do tum', 'type': 'COMMENT', 'user_id': '7339de48d380c9efe854dc9b6660a8fe22c28448', 'user_name': 'sami ali ali', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}, {'index': 16, '_id': ObjectId('5fe4500c7b2c03decd863360'), 'channel_id': '/channel/UCaPZsZzHcOMiDgZ3rkALMFg', 'clean_text': 'is pakar kar gal ma dala gaya', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnhv4vXDX16Pi0veGMZVUtqiYiYq_XOUp2yTvQ=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'UgzC4DaDahEFvjUsJiN4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Is pakar kar gal ma dala gaya', 'tokens': ['is', 'pakar', 'kar', 'gal', 'ma', 'dala', 'gaya'], 'tokens_no_swords': ['pakar', 'kar', 'gal', 'ma', 'dala', 'gaya'], 'tran_text': 'is pakar kar gal ma dala gaya', 'type': 'COMMENT', 'user_id': 'ae974ed633daab66164b5dcee9340e2ed0b1c455', 'user_name': 'munir gill', 'video_id': 'o6st4ces9Wg', 'is_hate': '1', 'date': '2020-12-03'}, {'index': 17, '_id': ObjectId('5fe4500c7b2c03decd863361'), 'channel_id': '/channel/UCqGRwUGDEBY98v0PcA9BpUQ', 'clean_text': 'bahut ghatiya insan hai', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnhMts6KGq4VtnvbDuVVatNlFduO6jmHbIRX6A=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'UgyyCAdkym_IDptSbNZ4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Bahut ghatiya insan hai', 'tokens': ['bahut', 'ghatiya', 'insan', 'hai'], 'tokens_no_swords': ['bahut', 'ghatiya', 'insan', 'hai'], 'tran_text': 'bahut ghatiya insan hai', 'type': 'COMMENT', 'user_id': '331593ec91edb449953d775229e7a91727415976', 'user_name': 'Asif Bhatti', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}, {'index': 18, '_id': ObjectId('5fe4500c7b2c03decd863362'), 'channel_id': '/channel/UCBZ0mLPPioFWW1i-kvmZBnA', 'clean_text': 'ye tu chakka hai alex bhatti', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnggrksT4HvfysI9VkzPzsKIXkcJsPfmWvvNyg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'Ugwh57O9lzDJgzCvKJV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Ye tu chakka hai alex bhatti', 'tokens': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tokens_no_swords': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tran_text': 'ye tu chakka hai alex bhatti', 'type': 'COMMENT', 'user_id': 'ee99e20e5128b5fc14c1972d55625585cf4d0237', 'user_name': 'Khizar Rao', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}]}
I want to append the following dictionary against the "processed_data" key.
{'index': 19, '_id': ObjectId('5fe4500c7b2c03decd863362'), 'channel_id': '/channel/UCBZ0mLPPioFWW1i-kvmZBnA', 'clean_text': 'ye tu chakka hai alex bhatti', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnggrksT4HvfysI9VkzPzsKIXkcJsPfmWvvNyg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'Ugwh57O9lzDJgzCvKJV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Ye tu chakka hai alex bhatti', 'tokens': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tokens_no_swords': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tran_text': 'ye tu chakka hai alex bhatti', 'type': 'COMMENT', 'user_id': 'ee99e20e5128b5fc14c1972d55625585cf4d0237', 'user_name': 'Khizar Rao', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}
Thanks!

Use update_one() with $push:
from pymongo import MongoClient
from bson import ObjectId
db = MongoClient()['mydatabase']
db.mycollection.insert_one({'_id': ObjectId('5fe46a5b7468e3498124fcbe'),
'processed_data': []})
update = {'index': 19, '_id': ObjectId('5fe4500c7b2c03decd863362'), 'channel_id': 'etc.'}
db.mycollection.update_one({'_id': ObjectId('5fe46a5b7468e3498124fcbe')}, {'$push': {'processed_data': update}})

We Keep Coding

Python is a programming language that lets you work quickly and integrate systems more effectively.

JSON issue with Beautifulsoup - python

Related

spark.read.json error: (java.lang.ArrayStoreException: java.util.HashMap)

Python Request with Cookies - Content blocked by Cookie Banner

BeautifulSoup find class with space

How to scrap data when the site kind of doesn't allow it? [duplicate]

How can i append dictionary to my existing key "processed_data" in mongodb using pymongo

Categories

Resources