I am trying to get price info using BS4 with Python.
This is the <p> class from which I am trying to extract the value of '$3245'.
<div class="s1fqyqkq-3 bDmzjk">
<div class="s1fqyqkq-4 ckAXTq">Price Info</div>
<p class="s1fqyqkq-5 dekHBg">$3245</p>
</div>
And this is the code I am trying to use to get the data, but it's returning "None"
headers = {
"Connection": "keep-alive",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
}
req = requests.get(url, headers=headers)
soup = bs4.BeautifulSoup(req.text, 'html.parser')
result = soup.find("div", {"class": "s1fqyqkq-5 dekHBg"})
print(result)
I tried all suggestions, but I am still I receiving "None" as a result.
I tried the suggestion here to no avail. Any idea what I am missing?
While cooking your soup taste it to ensure it comes with all expected ingridients.
Main issue, content is created dynamically by javascript so you wont find your information that way, cause requests do not act like a browser and is not able to render all information.
To get informations take a look at the script variables and extract the JSON strings.
import json, requests
url = 'https://www.cimri.com/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-siyah-fiyatlari,337340793'
jsonData = json.loads(re.search(r"window.productDetails = (.*?);", requests.get(url).text).group(1))
jsonData
Output
{'id': '337340793', 'path': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-siyah-fiyatlari,337340793', 'itemType': 0, 'mainId': 337341638, 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah', 'offers': [{'id': '506482765', 'price': 13135.85, 'title': 'Yeni Apple iPhone 11 (64\xa0GB) - Siyah', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '10370', 'url': 'https://www.amazon.com.tr', 'seller': 'Amazon.com.tr', 'slogan': 'Amazon.com.tr uygulamasında ilk 150 TL ve üzeri alışverişe 40 TL indirim ', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495611798}, {'id': '894741295', 'price': 13149, 'title': 'iPhone 11 64 GB Aksesuarsız Kutu Siyah', 'shipping': {'fee': 0, 'day': 3}, 'merchant': {'id': '10109', 'url': 'https://www.trendyol.com/', 'seller': 'Trendyol', 'slogan': "Tüm İhtiyaçların Trendyol'da, Tek Tıkla Kapında!", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': True, 'feedLastProcess': 1655493405188}, {'id': '887498622', 'price': 13249, 'title': 'Apple iPhone 11 64 GB Cep Telefonu Siyah', 'shipping': {'fee': 0, 'day': 2}, 'merchant': {'id': '10416', 'url': 'https://www.a101.com.tr/', 'seller': None, 'slogan': 'A101|Harca Harca Bitmez', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655489830306}, {'id': '767094934', 'price': 13299, 'title': 'iPhone 11 64 GB Siyah', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '4093', 'url': 'http://www.turkcell.com.tr', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494357178}, {'id': '855428213', 'price': 13350, 'title': 'iPhone 11 64 GB Siyah (Apple Türkiye Garantili) - (Aksesuarsız Kutu)', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '12192', 'url': 'https://www.needion.com/', 'seller': None, 'slogan': 'Alan Memnun, Satan Memnun', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': True, 'feedLastProcess': 1655495321932}, {'id': '914643935', 'price': 13379, 'title': 'Apple iPhone 11 64GB Akıllı Cep Telefonu Siyah ( Apple Türkiye Garantili )', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '12665', 'url': 'https://wnt.com.tr/', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494393339}, {'id': '491596021', 'price': 13398.99, 'title': 'iPhone 11 64 GB - Siyah - Aksesuarsız Kutu', 'shipping': {'fee': None, 'day': 0}, 'merchant': {'id': '1497', 'url': 'http://www.hepsiburada.com', 'seller': 'Hepsiburada', 'slogan': 'Türkiye’de İlk Defa Alışveriş Kredisi Burada', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495861494}, {'id': '1027767764', 'price': 13459, 'title': 'Apple iPhone 11 64GB Siyah Cep Telefonu (Apple Türkiye Garantili) (Siyah)', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '5825', 'url': 'https://www.vodafone.com.tr/', 'seller': 'GENÇPA', 'slogan': 'Vodafone Her Şey Yanımda Alışverişlerinizde Kargo Bedava, Vodafone’lulara İlk Siparişe 15GB Hediye!', 'onlyApp': True}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495537182}, {'id': '952479181', 'price': 13499, 'title': 'iPhone 11 64 GB Aksesuarsız Kutu Siyah', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '10109', 'url': 'https://www.trendyol.com/', 'seller': 'Teknosa', 'slogan': "Tüm İhtiyaçların Trendyol'da, Tek Tıkla Kapında!", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655493405188}, {'id': '1032790438', 'price': 13499, 'title': 'Apple iPhone 11 64 GB Siyah (Apple Türkiye Garantili) (Siyah)', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '5825', 'url': 'https://www.vodafone.com.tr/', 'seller': 'CepHane Teknoloji', 'slogan': 'Vodafone Her Şey Yanımda Alışverişlerinizde Kargo Bedava, Vodafone’lulara İlk Siparişe 15GB Hediye!', 'onlyApp': True}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495537182}, {'id': '1063638766', 'price': 13509, 'title': 'Apple iPhone 11 64 GB (Apple Türkiye Garantili) siyah', 'shipping': {'fee': 0, 'day': 2}, 'merchant': {'id': '7000', 'url': 'http://www.n11.com', 'seller': 'cokteknolojik', 'slogan': 'Hayat Sana Gelir', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495952093}, {'id': '930481030', 'price': 13559, 'title': 'iPhone 11 64GB Black', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '13105', 'url': 'https://www.inbox.com.tr/', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655491323973}, {'id': '1007434732', 'price': 13633, 'title': 'Apple iPhone 11 64 GB (Apple Türkiye Garantili) siyah', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '7000', 'url': 'http://www.n11.com', 'seller': 'yukseliletisim', 'slogan': 'Hayat Sana Gelir', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495952093}, {'id': '912636803', 'price': 13699, 'title': 'Apple iPhone 11 64GB Siyah - MHDA3TU/A', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '11403', 'url': 'https://gurgencler.com.tr/', 'seller': None, 'slogan': '10.000TL üzeri alışverişinize 500TL Bankkart Lira hediye!', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655492821318}, {'id': '974603297', 'price': 13779, 'title': 'IPhone 11 64 GB Siyah', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '12938', 'url': 'https://www.cepux.com.tr/', 'seller': None, 'slogan': 'Cepux Kazandırır!', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495878822}, {'id': '960442769', 'price': 13779, 'title': 'Apple iPhone 11 64 GB Aksesuarsız Kutu Siyah (Siyah)', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '5825', 'url': 'https://www.vodafone.com.tr/', 'seller': 'Tokdemir Bilişim Teknolojileri', 'slogan': 'Vodafone Her Şey Yanımda Alışverişlerinizde Kargo Bedava, Vodafone’lulara İlk Siparişe 15GB Hediye!', 'onlyApp': True}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495537182}, {'id': '965671448', 'price': 13789, 'title': 'iPhone 11 64 GB Siyah', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '4093', 'url': 'http://www.turkcell.com.tr', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494357178}, {'id': '808307738', 'price': 13798, 'title': 'iPhone 11 64 GB Siyah', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '4093', 'url': 'http://www.turkcell.com.tr', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494357178}, {'id': '1059226040', 'price': 13799, 'title': 'Apple iPhone 11 64 GB (Apple Türkiye Garantili) siyah', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '7000', 'url': 'http://www.n11.com', 'seller': 'integrabilisim', 'slogan': 'Hayat Sana Gelir', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495952093}, {'id': '734101431', 'price': 13799, 'title': 'Iphone 11 64 Gb Siyah Aksesuarsız Kutu', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '11512', 'url': 'https://www.pttavm.com', 'seller': 'BittiBitiyor', 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495542557}, {'id': '765227763', 'price': 14484, 'title': 'Apple iPhone 11 64Gb Akıllı Telefon Siyah', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '46', 'url': 'http://www.teknosa.com/', 'seller': 'Bittibitiyor', 'slogan': "Sepette 2.500 TL'ye Varan İndirimler", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655489853593}, {'id': '765957525', 'price': 14499, 'title': 'iPhone 11 64 Gb Siyah', 'shipping': {'fee': 15, 'day': 1}, 'merchant': {'id': '11375', 'url': 'https://www.vatanbilgisayar.com/', 'seller': None, 'slogan': None, 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655489535355}, {'id': '765434969', 'price': 14499, 'title': 'APPLE iPhone 11 64GB Akıllı Telefon Siyah', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '4230', 'url': 'http://www.mediamarkt.com.tr/', 'seller': None, 'slogan': '250 TL ve üzeri alışverişlerde geçerli ücretsiz kargo', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655489226707}, {'id': '765500161', 'price': 14549.03, 'title': 'Apple İphone 11 64Gb Siyah - Mhda3tu/A 194252097335', 'shipping': {'fee': 0, 'day': 1}, 'merchant': {'id': '10052', 'url': 'http://www.troyestore.com/', 'seller': None, 'slogan': 'Apple Yetkili Satıcı, Apple Yetkili Teknik Servis', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655493437121}, {'id': '765498940', 'price': 14999, 'title': 'Apple iPhone 11 64 Gb Siyah Cep Telefonu', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '10313', 'url': 'https://www.migros.com.tr/', 'seller': None, 'slogan': '250 TL ve Üzeri Ücretsiz Teslimat', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655488948137}, {'id': '765987003', 'price': 14999, 'title': 'Apple iPhone 11 64GB Siyah Yeni Cep Telefonu', 'shipping': {'fee': 0, 'day': 2}, 'merchant': {'id': '10604', 'url': 'https://www.beko.com.tr/', 'seller': None, 'slogan': 'Ücretsiz ve Hızlı Teslimat ile Beko Kapında!', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655492802285}, {'id': '765350098', 'price': 14999, 'title': 'Apple iPhone 11 64GB Siyah Yeni Cep Telefonu', 'shipping': {'fee': 0, 'day': 2}, 'merchant': {'id': '10591', 'url': 'https://www.arcelik.com.tr/', 'seller': None, 'slogan': 'Ücretsiz ve Hızlı Teslimat ile Arçelik Kapında!', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655492539972}, {'id': '765171664', 'price': 14999, 'title': 'Apple iPhone 11 64GB Siyah (New Edt)', 'shipping': {'fee': 0, 'day': None}, 'merchant': {'id': '9334', 'url': 'http://www.evkur.com.tr/', 'seller': None, 'slogan': 'Hiç Peşinatsız Kredi Kartsız 24 Aya Varan Taksitlerle', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655491644137}, {'id': '806067367', 'price': 15292.64, 'title': 'İphone 11 64 Gb', 'shipping': {'fee': 0, 'day': 0}, 'merchant': {'id': '5164', 'url': 'http://www.gittigidiyor.com/', 'seller': 'gncpazar', 'slogan': "Yapı Kredi World'e Özel 50 TL'ye Varan Worldpuan", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495287185}, {'id': '1068272963', 'price': 15599, 'title': 'İphone 11 64 Gb Siyah(kvk Garantili)', 'shipping': {'fee': 0, 'day': 3}, 'merchant': {'id': '5164', 'url': 'http://www.gittigidiyor.com/', 'seller': 'AYAZGRUP', 'slogan': "Yapı Kredi World'e Özel 50 TL'ye Varan Worldpuan", 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655495287185}, {'id': '793747418', 'price': 17591.37, 'title': 'Apple IPHONE 11 64GB Apple IPHONE 11 64GB Cep Telefonu (Apple Türkiye Garantili)', 'shipping': {'fee': 25, 'day': None}, 'merchant': {'id': '11898', 'url': 'https://www.turkuaztrade.com/', 'seller': None, 'slogan': 'Simply The Best / Kısaca En İyisi', 'onlyApp': False}, 'unitPrice': None, 'numbers': [], 'sponsored': False, 'feedLastProcess': 1655494006222}], 'imageIds': [219571586, 219571585, 219571582, 219571577, 219575009], 'review': {'count': 95, 'rate': 4}, 'badge': {'discountRatio': 0, 'isNew': False}, 'metaData': {'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah Fiyatları ve Özellikleri', 'description': "Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah modellerini tüm detayları ile Cimri.com'da inceleyin! Siz de Cimri.com ile Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah fiyatlarına ve özelliklerine kolayca ulaşabilir, ürüne ait kullanıcı yorum ve değerlendirmelerini inceleyebilirsiniz. Dilerseniz fiyat ve özellik karşılaştıraması yapabilir ve son 1 yıla ait fiyat değişimlerini takip edebilirsiniz. Cimri.com ile siz de en ucuz Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah fırsatlarını kaçırmayın!"}, 'specs': [{'name': 'Model Bilgisi', 'specEntries': [{'name': 'Iphone Modelleri', 'value': 'iPhone 11', 'order': 1}], 'order': 1}, {'name': 'Ekran Özellikleri', 'specEntries': [{'name': 'Ekran Boyutu', 'value': '6.1 inç', 'order': 3}, {'name': 'Ekran Teknolojisi', 'value': 'IPS LCD', 'order': 5}, {'name': 'Yan Ekran', 'value': 'Yok', 'order': 14}, {'name': 'Mavi Işık Filtresi', 'value': 'Var', 'order': 15}, {'name': 'Çözünürlük Standartı', 'value': 'HD+', 'order': 16}, {'name': 'Karanlık Modu', 'value': 'Var', 'order': 16}, {'name': 'Ekran Çözünürlüğü', 'value': '828x1792 Piksel', 'order': 21}, {'name': 'Ekran Gövde Oranı', 'value': '79 %', 'order': 21}, {'name': 'Piksel Yoğunluğu', 'value': '326 PPI', 'order': 22}, {'name': 'Multi Touch', 'value': 'Var', 'order': 22}, {'name': 'Dokunmatik Türü', 'value': 'Kapasitif Ekran', 'order': 23}, {'name': 'Ekran Parlaklığı (cd-m²)', 'value': '625', 'order': 26}, {'name': 'Çizilmeye Karşı Dayanıklılık', 'value': 'Var', 'order': 28}, {'name': 'Ekran Kontrast Oranı', 'value': '1400:1', 'order': 29}, {'name': 'Sürekli Açık Ekran', 'value': 'Yok', 'order': 30}], 'order': 1}, {'name': 'Teknik Özellikler', 'specEntries': [{'name': 'İşlemci Modeli', 'value': 'Apple A13 Bionic', 'order': 1}, {'name': 'İşlemci Frekansı', 'value': '2.66 GHz', 'order': 2}, {'name': 'RAM Kapasitesi', 'value': '4 GB RAM', 'order': 6}, {'name': 'İşletim Sistemi Versiyonu', 'value': 'iOS 13', 'order': 8}, {'name': 'İşletim Sistemi', 'value': 'iOS', 'order': 19}, {'name': 'İşlemci Çekirdek Sayısı', 'value': '6 Çekirdek', 'order': 26}, {'name': 'CPU Üretim Süreci', 'value': '7 nm', 'order': 27}, {'name': 'İşlemci Mimarisi', 'value': '64 bit', 'order': 28}, {'name': 'Ram Tipi', 'value': 'LPDDR4x', 'order': 30}, {'name': 'Ram Kanal Sayısı', 'value': 'Çift Kanal', 'order': 31}], 'order': 2}, {'name': 'Kamera Özellikleri', 'specEntries': [{'name': 'Kamera Çözünürlüğü', 'value': '12 MP', 'order': 4}, {'name': 'Ön (Selfie) Kamera Diyafram Açıklığı', 'value': 'F2.2', 'order': 6}, {'name': 'Ön (Selfie) Kamera Çözünürlüğü', 'value': '12 MP', 'order': 8}, {'name': 'Diyafram Açıklığı', 'value': 'F1.8', 'order': 9}, {'name': 'İkinci Arka Kamera', 'value': 'Var', 'order': 10}, {'name': 'İkinci Arka Kamera Diyafram Açıklığı', 'value': 'F2.4', 'order': 10}, {'name': 'Video Kayıt Çözünürlüğü', 'value': '2160p', 'order': 11}, {'name': 'Video FPS Değeri', 'value': '60 FPS', 'order': 12}, {'name': 'İkinci Arka Kamera Çözünürlüğü', 'value': '12 MP', 'order': 22}, {'name': 'Görüntü Sabitleyici', 'value': 'Var', 'order': 23}, {'name': 'Yüz Tanıma', 'value': 'Var', 'order': 24}, {'name': 'Slow Motion Video Çekimi', 'value': 'Var', 'order': 25}, {'name': 'Kamera Yapay Zeka (AI) Desteği', 'value': 'Yok', 'order': 26}, {'name': 'Time Lapse Video Çekimi', 'value': 'Var', 'order': 30}, {'name': 'Flaş Türü', 'value': 'Çift LED', 'order': 31}, {'name': 'Phase Detection', 'value': 'Var', 'order': 32}, {'name': 'HDR', 'value': 'Var', 'order': 33}, {'name': 'Zamanlayıcı', 'value': 'Var', 'order': 35}, {'name': 'Otomatik Odaklama', 'value': 'Var', 'order': 36}, {'name': 'Panoroma', 'value': 'Var', 'order': 37}, {'name': 'Coğrafi Konum Ekleme', 'value': 'Var', 'order': 38}], 'order': 3}, {'name': 'Tasarım', 'specEntries': [{'name': 'En', 'value': '75.7 mm', 'order': 1}, {'name': 'Boy', 'value': '150.9 mm', 'order': 2}, {'name': 'Kalınlık', 'value': '8.3 mm', 'order': 3}, {'name': 'Ağırlık', 'value': '194 gr', 'order': 7}, {'name': 'Gövde Malzemesi (Kapak)', 'value': 'Cam', 'order': 15}, {'name': 'Gövde Malzemesi (Çerçeve)', 'value': 'Alüminyum', 'order': 16}, {'name': 'Renk', 'value': 'Siyah', 'order': 31}], 'order': 4}, {'name': 'Batarya Özellikleri', 'specEntries': [{'name': 'Şarj Tipi', 'value': 'Lightning', 'order': 2}, {'name': 'Batarya Tipi', 'value': 'Li-ion', 'order': 11}, {'name': 'Kablosuz Hızlı Şarj', 'value': 'Yok', 'order': 12}, {'name': 'Kablosuz Hızlı Şarj Güç', 'value': 'Yok', 'order': 13}, {'name': 'Çıkarılabilir Batarya', 'value': 'Yok', 'order': 14}, {'name': 'Kablosuz Şarj', 'value': 'Var', 'order': 26}, {'name': 'Hızlı Şarj', 'value': 'Var', 'order': 27}], 'order': 5}, {'name': 'Depolama', 'specEntries': [{'name': 'Dahili Depolama (Hafıza)', 'value': '64 GB', 'order': 5}, {'name': 'Maksimum Hafıza Kartı Kapasitesi', 'value': 'Yok', 'order': 8}, {'name': 'Hafıza Kartı Desteği', 'value': 'Yok', 'order': 31}], 'order': 6}, {'name': 'Bağlantılar', 'specEntries': [{'name': 'Bluetooth Versiyonu', 'value': '5.0', 'order': 3}, {'name': 'Ses Çıkışı', 'value': 'Lightning', 'order': 5}, {'name': '4.5G Desteği', 'value': 'Var', 'order': 11}, {'name': 'Wİ-Fİ', 'value': 'Var', 'order': 11}, {'name': 'Wİ-Fİ Frekansı', 'value': 'Wi-Fi 6', 'order': 12}, {'name': 'Bluetooth', 'value': 'Var', 'order': 13}, {'name': 'Wİ-Fİ Hotspot', 'value': 'Var', 'order': 14}, {'name': 'Navigasyon', 'value': 'Var', 'order': 16}, {'name': '3G İndirme Hızı', 'value': '42.2 Mbps', 'order': 23}, {'name': '3G Yükleme Hızı', 'value': '5.76 Mbps', 'order': 24}, {'name': 'NFC', 'value': 'Var', 'order': 25}, {'name': '5G Desteği', 'value': 'Yok', 'order': 28}, {'name': 'Kızılötesi', 'value': 'Yok', 'order': 29}, {'name': 'USB Type-C', 'value': 'Yok', 'order': 30}], 'order': 7}, {'name': 'Ek Özellikler', 'specEntries': [{'name': 'Kalem', 'value': 'Yok', 'order': 11}, {'name': 'Güvenli Klasör', 'value': 'Yok', 'order': 12}, {'name': 'Antutu Puanı', 'value': '456900', 'order': 13}, {'name': 'Suya Dayanıklılık Seviyesi', 'value': 'IPX8', 'order': 14}, {'name': 'Toza Dayanıklılık Seviyesi', 'value': 'IP6X', 'order': 15}, {'name': 'Parmak İzi Okuyucu', 'value': 'Yok', 'order': 21}, {'name': 'Suya Dayanıklılık', 'value': 'Var', 'order': 23}, {'name': 'Toza Dayanıklılık', 'value': 'Var', 'order': 24}, {'name': 'Çıkış Tarihi', 'value': '2019', 'order': 35}, {'name': 'Parmak İzi Okuyucu Tipi', 'value': 'Yok', 'order': 37}, {'name': 'FM Radyo', 'value': 'Yok', 'order': 38}, {'name': 'Sim Kart Türü', 'value': 'Nano-SIM (4FF)', 'order': 39}], 'order': 8}, {'name': 'Sensörler', 'specEntries': [{'name': 'RGB Işık Sensörü', 'value': 'Yok', 'order': 2}, {'name': 'Bildirim Işığı', 'value': 'Yok', 'order': 8}, {'name': 'Barometre', 'value': 'Var', 'order': 31}, {'name': 'Jiroskop', 'value': 'Var', 'order': 32}, {'name': 'Hall Sensörü', 'value': 'Yok', 'order': 33}, {'name': 'Pusula', 'value': 'Var', 'order': 34}, {'name': 'Yakınlık Sensörü', 'value': 'Var', 'order': 35}, {'name': 'İvme Ölçer', 'value': 'Var', 'order': 36}], 'order': 9}, {'name': 'Kamera Özellikleri(Arka)', 'specEntries': [{'name': 'Elle Odaklama', 'value': 'Var', 'order': 8}, {'name': 'Seri Çekim Modu', 'value': 'Var', 'order': 10}, {'name': 'Zamanlayıcı', 'value': 'Var', 'order': 11}, {'name': 'Portre Modu', 'value': 'Var', 'order': 13}, {'name': 'Geniş Açı', 'value': 'Var', 'order': 13}, {'name': 'Yapay Zeka', 'value': 'Yok', 'order': 14}, {'name': 'Yüz Algılama', 'value': 'Var', 'order': 15}, {'name': 'Görüntü Sabitleyici', 'value': 'Var', 'order': 16}, {'name': 'Manuel Kontrol', 'value': 'Var', 'order': 17}, {'name': 'Hızlı Odaklama', 'value': 'Var', 'order': 52}, {'name': 'Odak Takibi', 'value': 'Var', 'order': 52}, {'name': 'Sesli Komut', 'value': 'Var', 'order': 52}, {'name': 'QR Kod Okuyucu', 'value': 'Yok', 'order': 55}], 'order': 10}, {'name': 'Kamera Özellikleri(Ön)', 'specEntries': [{'name': 'Geniş Açı', 'value': 'Yok', 'order': 2}, {'name': 'Otomatik Odaklama', 'value': 'Var', 'order': 3}, {'name': 'Panorama Özçekim', 'value': 'Yok', 'order': 4}, {'name': 'Seçmeli Odaklama', 'value': 'Yok', 'order': 5}, {'name': 'Sanal Flaş', 'value': 'Var', 'order': 8}, {'name': 'Zamanlayıcı', 'value': 'Var', 'order': 9}, {'name': 'Portre', 'value': 'Var', 'order': 18}, {'name': 'Sesle Komut', 'value': 'Yok', 'order': 19}, {'name': 'Yapay Zeka', 'value': 'Yok', 'order': 20}], 'order': 11}, {'name': 'Kamera Özellikleri(Video)', 'specEntries': [{'name': 'FPS', 'value': '60', 'order': 1}, {'name': 'Çözünürlük Standardı', 'value': '2160p', 'order': 2}, {'name': 'HDR', 'value': 'Var', 'order': 33}, {'name': 'Portre', 'value': 'Yok', 'order': 52}, {'name': 'Görüntü Sabitleme', 'value': 'Var', 'order': 53}, {'name': 'Ağır Çekim', 'value': 'Var', 'order': 54}, {'name': 'Hızlı Çekim', 'value': 'Var', 'order': 55}], 'order': 12}, {'name': 'USB', 'specEntries': [{'name': 'OTG', 'value': 'Var', 'order': 38}, {'name': 'DisplayPort', 'value': 'Yok', 'order': 39}, {'name': 'E-Sim', 'value': 'Var', 'order': 40}, {'name': 'Micro USB', 'value': 'Yok', 'order': 50}, {'name': 'Lightning', 'value': 'Var', 'order': 52}], 'order': 13}], 'topSpecs': [{'name': 'Dahili Depolama (Hafıza)', 'value': '64 GB', 'order': 5}, {'name': 'Diyafram Açıklığı', 'value': 'F1.8', 'order': 9}, {'name': 'Ekran Boyutu', 'value': '6.1 inç', 'order': 3}, {'name': 'Kamera Çözünürlüğü', 'value': '12 MP', 'order': 4}, {'name': 'Ön (Selfie) Kamera Çözünürlüğü', 'value': '12 MP', 'order': 8}], 'unitType': None, 'variantsHeader': 'Renk', 'variants': [{'id': '337337465', 'imageId': 178955343, 'minPrice': 13229, 'shortTitle': 'Beyaz', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Beyaz', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-beyaz-fiyatlari,337337465', 'unitPrice': None}, {'id': '337339455', 'imageId': 178955504, 'minPrice': 13249, 'shortTitle': 'Sarı', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Sarı', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-sari-fiyatlari,337339455', 'unitPrice': None}, {'id': '337338130', 'imageId': 178955431, 'minPrice': 13299, 'shortTitle': 'Yeşil', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Yeşil', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-yesil-fiyatlari,337338130', 'unitPrice': None}, {'id': '337338937', 'imageId': 178955480, 'minPrice': 13299, 'shortTitle': 'Mor', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Mor', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-mor-fiyatlari,337338937', 'unitPrice': None}, {'id': '337340793', 'imageId': 219571586, 'minPrice': 12986, 'shortTitle': 'Siyah', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Siyah', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-siyah-fiyatlari,337340793', 'unitPrice': None}, {'id': '337340093', 'imageId': 178955538, 'minPrice': 13385.59, 'shortTitle': 'Kırmızı', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu Kırmızı', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-kirmizi-fiyatlari,337340093', 'unitPrice': None}], 'mainVariantsHeader': 'Kapasite', 'mainVariants': [{'id': '337341638', 'imageId': 178955480, 'minPrice': 12986, 'shortTitle': '64 GB', 'title': 'Apple iPhone 11 64GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-64gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-fiyatlari,a337341638'}, {'id': '331840845', 'imageId': 178954771, 'minPrice': 13999, 'shortTitle': '128 GB', 'title': 'Apple iPhone 11 128GB Akıllı Cep Telefonu', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-128gb-akilli-cep-telefonu-fiyatlari,a331840845'}, {'id': '337357780', 'imageId': 178954858, 'minPrice': 20899, 'shortTitle': '256 GB', 'title': 'Apple iPhone 11 256GB 4GB Ram 6.1 inç 12MP Akıllı Cep Telefonu', 'url': '/cep-telefonlari/en-ucuz-apple-iphone-11-256gb-4gb-ram-6-1-inc-12mp-akilli-cep-telefonu-fiyatlari,a337357780'}], 'brand': {'id': 4, 'name': 'Apple', 'slug': 'apple'}, 'category': {'id': '130', 'name': 'Cep Telefonları', 'slug': 'cep-telefonlari'}}
Related
I'm using the code below to pull in multiple json files into 1 using pagination. When I try to create a spark dataframe I get the error '
java.lang.ArrayStoreException: java.util.HashMap' during spark.read.json. Below the code block I've provided output from printing the 'issues' data set (minus proprietary info). I've done a bit of research and can't figure out what I can try to make this work. Any assistance would be greatly appreciated!!
import requests
import json
limit = 2
startat = 0
issues = []
for page_num in range(2):
startat = page_num*50
url = f"https://URL/rest/api/2/search?jql=TEST&startAt={startat}&maxResults={limit}"
req = requests.get(url, headers={'Accept': 'application/json', 'Authorization': 'Basic xxxxxxxxxxxxxxxxxxxxxxxxxx'})
data = req.json()
issues.extend(data['issues'])
jsonDF = spark.read.json(issues)
jsonDF.printSchema()
[{'expand': 'operations,versionedRepresentations,editmeta,changelog,customfield_10010.requestTypePractice,renderedFields', 'id': '11441', 'self': 'https://my.url.net/rest/api/2/issue/11441', 'key': 'TS-1401', 'fields': {'statuscategorychangedate': '2022-11-29T07:05:17.359-0800', 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10004', 'id': '10004', 'description': 'Functionality or a feature expressed as a user goal.', 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10315?size=medium', 'name': 'Story', 'subtask': False, 'avatarId': 10315, 'hierarchyLevel': 0}, 'parent': {'id': '11420', 'key': 'TS-1380', 'self': 'https://my.url.net/rest/api/2/issue/11420', 'fields': {'summary': 'Clone30 - Migration Epics', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10000', 'id': '10000', 'description': 'A big user story that needs to be broken down. Created by Jira Software - do not edit or delete.', 'iconUrl': 'https://my.url.net/images/icons/issuetypes/epic.svg', 'name': 'Epic', 'subtask': False, 'hierarchyLevel': 1}}}, 'timespent': None, 'project': {'self': 'https://my.url.net/rest/api/2/project/10001', 'id': '10001', 'key': 'TS', 'name': 'Project', 'projectTypeKey': 'software', 'simplified': False, 'avatarUrls': {'48x48': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556', '24x24': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=small', '16x16': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=xsmall', '32x32': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=medium'}}, 'customfield_10033': None, 'fixVersions': [], 'aggregatetimespent': None, 'customfield_10034': [], 'customfield_10035': None, 'resolution': None, 'customfield_10036': None, 'customfield_10037': None, 'customfield_10027': None, 'customfield_10028': None, 'customfield_10029': None, 'resolutiondate': None, 'workratio': -1, 'watches': {'self': 'https://my.url.net/rest/api/2/issue/TS-1401/watchers', 'watchCount': 1, 'isWatching': True}, 'lastViewed': '2022-12-08T10:06:57.022-0800', 'created': '2022-11-29T07:05:16.501-0800', 'customfield_10020': None, 'customfield_10021': None, 'customfield_10022': None, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'customfield_10023': None, 'customfield_10024': None, 'customfield_10025': None, 'customfield_10026': None, 'labels': [], 'customfield_10016': None, 'customfield_10017': None, 'customfield_10018': {'hasEpicLinkFieldDependency': False, 'showField': False, 'nonEditableReason': {'reason': 'EPIC_LINK_SHOULD_BE_USED', 'message': 'To set an epic as the parent, use the epic link instead'}}, 'customfield_10019': '0|i008a3:', 'timeestimate': None, 'aggregatetimeoriginalestimate': None, 'versions': [], 'issuelinks': [], 'assignee': None, 'updated': '2022-11-29T07:05:20.759-0800', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'components': [], 'timeoriginalestimate': None, 'description': 'Data owner completes template (understand scope of migration efforts)', 'customfield_10010': None, 'customfield_10014': 'TS-1380', 'customfield_10015': None, 'customfield_10005': None, 'customfield_10006': None, 'customfield_10007': None, 'security': None, 'customfield_10008': None, 'customfield_10009': None, 'aggregatetimeestimate': None, 'summary': 'Template', 'creator': {'self': 'https://my.url.net/rest/api/2/user?accountId=5d669f4bf81f2c0d99ee9e38', 'accountId': '5d669f4bf81f2c0d99ee9e38', 'emailAddress': 'test#aol.com', 'avatarUrls': {'48x48': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '24x24': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '16x16': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '32x32': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png'}, 'displayName': 'Joe Test', 'active': True, 'timeZone': 'America/Los_Angeles', 'accountType': 'atlassian'}, 'subtasks': [{'id': '11442', 'key': 'TS-1402', 'self': 'https://my.url.net/rest/api/2/issue/11442', 'fields': {'summary': 'Complete Template with table/views required (in) and produced (out)', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10006', 'id': '10006', 'description': "A small piece of work that's part of a larger task.", 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10316?size=medium', 'name': 'Sub-task', 'subtask': True, 'avatarId': 10316, 'hierarchyLevel': -1}}}], 'reporter': {'self': 'https://my.url.net/rest/api/2/user?accountId=5d669f4bf81f2c0d99ee9e38', 'accountId': '5d669f4bf81f2c0d99ee9e38', 'emailAddress': 'test#aol.com', 'avatarUrls': {'48x48': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '24x24': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '16x16': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '32x32': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png'}, 'displayName': 'Joe Test', 'active': True, 'timeZone': 'America/Los_Angeles', 'accountType': 'atlassian'}, 'aggregateprogress': {'progress': 0, 'total': 0}, 'customfield_10001': None, 'customfield_10002': None, 'customfield_10003': None, 'customfield_10004': None, 'customfield_10038': None, 'environment': None, 'duedate': None, 'progress': {'progress': 0, 'total': 0}, 'votes': {'self': 'https://my.url.net/rest/api/2/issue/TS-1401/votes', 'votes': 0, 'hasVoted': False}}}, {'expand': 'operations,versionedRepresentations,editmeta,changelog,customfield_10010.requestTypePractice,renderedFields', 'id': '11438', 'self': 'https://my.url.net/rest/api/2/issue/11438', 'key': 'TS-1398', 'fields': {'statuscategorychangedate': '2022-11-29T07:05:09.126-0800', 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10004', 'id': '10004', 'description': 'Functionality or a feature expressed as a user goal.', 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10315?size=medium', 'name': 'Story', 'subtask': False, 'avatarId': 10315, 'hierarchyLevel': 0}, 'parent': {'id': '11420', 'key': 'TS-1380', 'self': 'https://my.url.net/rest/api/2/issue/11420', 'fields': {'summary': 'Clone30 - Migration Epics', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10000', 'id': '10000', 'description': 'A big user story that needs to be broken down. Created by Jira Software - do not edit or delete.', 'iconUrl': 'https://my.url.net/images/icons/issuetypes/epic.svg', 'name': 'Epic', 'subtask': False, 'hierarchyLevel': 1}}}, 'timespent': None, 'project': {'self': 'https://my.url.net/rest/api/2/project/10001', 'id': '10001', 'key': 'TS', 'name': 'Project', 'projectTypeKey': 'software', 'simplified': False, 'avatarUrls': {'48x48': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556', '24x24': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=small', '16x16': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=xsmall', '32x32': 'https://my.url.net/rest/api/2/universal_avatar/view/type/project/avatar/10556?size=medium'}}, 'fixVersions': [], 'customfield_10033': None, 'customfield_10034': [], 'aggregatetimespent': None, 'customfield_10035': None, 'resolution': None, 'customfield_10036': None, 'customfield_10037': None, 'customfield_10027': None, 'customfield_10028': None, 'customfield_10029': None, 'resolutiondate': None, 'workratio': -1, 'lastViewed': None, 'watches': {'self': 'https://my.url.net/rest/api/2/issue/TS-1398/watchers', 'watchCount': 1, 'isWatching': True}, 'created': '2022-11-29T07:05:08.312-0800', 'customfield_10020': None, 'customfield_10021': None, 'customfield_10022': None, 'customfield_10023': None, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'customfield_10024': None, 'customfield_10025': None, 'customfield_10026': None, 'labels': [], 'customfield_10016': None, 'customfield_10017': None, 'customfield_10018': {'hasEpicLinkFieldDependency': False, 'showField': False, 'nonEditableReason': {'reason': 'EPIC_LINK_SHOULD_BE_USED', 'message': 'To set an epic as the parent, use the epic link instead'}}, 'customfield_10019': '0|i008ae:y', 'timeestimate': None, 'aggregatetimeoriginalestimate': None, 'versions': [], 'issuelinks': [], 'assignee': None, 'updated': '2022-11-29T07:05:22.417-0800', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'components': [], 'timeoriginalestimate': None, 'description': 'Creating reports/reporting cubes; need to find out reports used', 'customfield_10010': None, 'customfield_10014': 'TS-1380', 'customfield_10015': None, 'customfield_10005': None, 'customfield_10006': None, 'security': None, 'customfield_10007': None, 'customfield_10008': None, 'customfield_10009': None, 'aggregatetimeestimate': None, 'summary': '\xa0create reports/cubes', 'creator': {'self': 'https://my.url.net/rest/api/2/user?accountId=5d669f4bf81f2c0d99ee9e38', 'accountId': '5d669f4bf81f2c0d99ee9e38', 'emailAddress': 'test#aol.com', 'avatarUrls': {'48x48': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '24x24': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '16x16': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '32x32': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png'}, 'displayName': 'Joe Test', 'active': True, 'timeZone': 'America/Los_Angeles', 'accountType': 'atlassian'}, 'subtasks': [{'id': '11439', 'key': 'TS-1399', 'self': 'https://my.url.net/rest/api/2/issue/11439', 'fields': {'summary': 'Confirm: any reporting cubes required using this data are created and in production?', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10006', 'id': '10006', 'description': "A small piece of work that's part of a larger task.", 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10316?size=medium', 'name': 'Sub-task', 'subtask': True, 'avatarId': 10316, 'hierarchyLevel': -1}}}, {'id': '11440', 'key': 'TS-1400', 'self': 'https://my.url.net/rest/api/2/issue/11440', 'fields': {'summary': 'Confirm: any structured reports using this data are created and in production?', 'status': {'self': 'https://my.url.net/rest/api/2/status/10003', 'description': '', 'iconUrl': 'https://my.url.net/', 'name': 'Backlog', 'id': '10003', 'statusCategory': {'self': 'https://my.url.net/rest/api/2/statuscategory/2', 'id': 2, 'key': 'new', 'colorName': 'blue-gray', 'name': 'To Do'}}, 'priority': {'self': 'https://my.url.net/rest/api/2/priority/3', 'iconUrl': 'https://my.url.net/images/icons/priorities/medium.svg', 'name': 'Medium', 'id': '3'}, 'issuetype': {'self': 'https://my.url.net/rest/api/2/issuetype/10006', 'id': '10006', 'description': "A small piece of work that's part of a larger task.", 'iconUrl': 'https://my.url.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10316?size=medium', 'name': 'Sub-task', 'subtask': True, 'avatarId': 10316, 'hierarchyLevel': -1}}}], 'reporter': {'self': 'https://my.url.net/rest/api/2/user?accountId=5d669f4bf81f2c0d99ee9e38', 'accountId': '5d669f4bf81f2c0d99ee9e38', 'emailAddress': 'test#aol.com', 'avatarUrls': {'48x48': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '24x24': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '16x16': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png', '32x32': 'https://secure.gravatar.com/avatar/69b7db33e65c274c27a07b28b356e329?d=https%3A%2F%2Favatar-management--avatars.us-west-2.test.png'}, 'displayName': 'Joe Test', 'active': True, 'timeZone': 'America/Los_Angeles', 'accountType': 'atlassian'}, 'aggregateprogress': {'progress': 0, 'total': 0}, 'customfield_10001': None, 'customfield_10002': None, 'customfield_10003': None, 'customfield_10004': None, 'customfield_10038': None, 'environment': None, 'duedate': None, 'progress': {'progress': 0, 'total': 0}, 'votes': {'self': 'https://my.url.net/rest/api/2/issue/TS-1398/votes', 'votes': 0, 'hasVoted': False}}}]
I'm trying to extract consistently this portion of code (line 237 in page source) with python requests.
<script type="text/javascript">
window.classified = {"EverythingHere"
</script>
Yet I'm unable to parse it using the following code, while it works for every other "script" tags.
url = 'https://www.immoweb.be/en/classified/new-real-estate-project-apartments/for-sale/auderghem/1160/8950161'
soup = BeautifulSoup(requests.get(url).content, "html.parser")
data=[]
for p in soup.find_all():
for n in p:
if n.name == 'script':
data.append(str(n))
print(data[0])
Do I need to set up a time.sleep to let the parser loading the webpage or do I need to change something in my code ?
Any helps would be really appreciated ! Thanks.
Couple things:
You are unnecsarily using a nested loop to search for 'script' tags. Rather than find_all() tags, then iterate through all the tags to find the ones that are <script>, you can do that right off the bat with .find_all('script').
Secondly, it's the last <script> tag. You can either just call the [-1] index, or probably a better way incase for whatever reason it's not that last tag for another page, just get the one with 'window.classified'
import requests
from bs4 import BeautifulSoup
import re
import json
url = 'https://www.immoweb.be/en/classified/new-real-estate-project-apartments/for-sale/auderghem/1160/8950161'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36'}
soup = BeautifulSoup(requests.get(url, headers=headers).content, "html.parser")
for p in soup.find_all('script'):
if 'window.classified' in str(p):
dataStr = str(p)
#print(str(p))
# Parse the json
data = re.search('{.*}', dataStr).group(0)
jsonData = json.loads(data)
Output:
print(jsonData)
{'id': 8950161, 'cluster': {'minPrice': 240500, 'maxPrice': 521500, 'minSurface': 44, 'maxSurface': 104, 'projectInfo': {'constructor': None, 'groupId': 8950161, 'phase': None, 'projectName': 'DELTA VIEW', 'deliveryDate': '2022-10-01T00:00:00.000+0000', 'soldPercentage': 47, 'unitsDisplayMode': 'DETAILED'}, 'units': [{'type': 'APARTMENT', 'maxPrice': 521500, 'minPrice': 240500, 'minRoom': 0, 'maxRoom': 2, 'minSurface': 44, 'maxSurface': 104, 'soldCount': 36, 'items': [{'id': 8950204, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950212, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950213, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950217, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950222, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950223, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950170, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950192, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950203, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950205, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950211, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950218, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950220, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950226, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950175, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950179, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950182, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950183, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950216, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 35}, {'id': 8950162, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950202, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950244, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950171, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950194, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 240500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950193, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 240500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950195, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 243500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950196, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 243500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950206, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 244500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950197, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 245500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950198, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950207, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 247500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950208, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 247500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950224, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 247500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950225, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950172, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 248500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950199, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 248500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950200, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 248500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950209, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 249500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950210, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 6, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950239, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950248, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950201, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950242, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950163, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950164, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950173, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 250500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950166, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 252500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950165, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 252500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950174, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 5, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 44}, {'id': 8950167, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 255500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950168, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 5, 'price': 255500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950176, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 255500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950169, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 6, 'price': 257500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 48}, {'id': 8950178, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'SOLD', 'floor': 0, 'price': None, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 46}, {'id': 8950177, 'subtype': 'FLAT_STUDIO', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 260500, 'bedroomCount': 0, 'realEstateProjectPhase': None, 'surface': 47}, {'id': 8950190, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 3, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950188, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950189, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 3, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950187, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 2, 'price': 312000, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950191, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 4, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 66}, {'id': 8950181, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 2, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950180, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 335500, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950243, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 3, 'price': 335500, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950185, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 5, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950241, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 5, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950184, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 338500, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950186, 'subtype': 'APARTMENT', 'saleStatus': 'SOLD', 'floor': 6, 'price': None, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950240, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 4, 'price': 338500, 'bedroomCount': 1, 'realEstateProjectPhase': None, 'surface': 70}, {'id': 8950227, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 501500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 102}, {'id': 8950228, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 504500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 102}, {'id': 8950229, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 506500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 102}, {'id': 8950231, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 7, 'price': 511500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950230, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 511500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 102}, {'id': 8950233, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 8, 'price': 513500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950234, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 9, 'price': 516500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950236, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 0, 'price': 518500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950238, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 11, 'price': 521500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}, {'id': 8950247, 'subtype': 'APARTMENT', 'saleStatus': 'AVAILABLE', 'floor': 11, 'price': 521500, 'bedroomCount': 2, 'realEstateProjectPhase': None, 'surface': 104}]}], 'bedroomRange': '1 - 2', 'surfaceRange': '44 - 104'}, 'customers': [{'id': 1364801, 'type': 'REAL_ESTATE_AGENCY', 'email': 'victoire_1490#importfrommedia.be', 'logoUrl': 'https://static.immoweb.be/logos/1364801.gif?cache=201721090326Z', 'phoneNumber': '+32 2 375 10 10', 'mobileNumber': None, 'name': 'Victoire (Immobilier Neuf)', 'website': None, 'location': {'country': 'Belgium', 'region': None, 'province': 'Brussels', 'district': 'Brussels', 'locality': 'Uccle', 'postalCode': '1180', 'street': 'Chaussée de Waterloo 1382', 'number': None, 'box': None, 'propertyName': None, 'floor': None, 'latitude': 50.8261604, 'longitude': 4.4461589, 'approximated': None, 'regionCode': None, 'type': None, 'hasSeaView': None, 'pointsOfInterest': None, 'placeName': None}, 'ipiNo': None, 'isOwner': True, 'contactHoursMobile': 'anytime', 'contactHoursLandline': 'anytime', 'salesRepresentative': None}], 'premiumProjectPage': {'medias': None, 'options': None, 'promoter': None, 'tabs': None}, 'flags': {'isPublicSale': False, 'isNewClassified': False, 'isNewPrice': False, 'isNewlyBuilt': True, 'isNotarySale': None, 'isLifeAnnuitySale': False, 'adQuality': None, 'date': None, 'priceSqm': None, 'price': None, 'default': None, 'isSoldOrRented': False, 'isLowEnergy': None, 'percentSold': 47, 'isPassiveHouse': None, 'isNewRealEstateProject': True, 'isAnInteractiveSale': None, 'isUnderOption': None}, 'media': {'pictures': [{'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_1.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_1.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_1.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_2.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_2.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_2.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_3.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_3.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_3.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_4.gif?cache=20210217040440', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_4.jpg?cache=20210217040440', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_4.jpg?cache=20210217040440', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_5.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_5.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_5.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_6.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_6.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_6.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_7.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_7.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_7.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_8.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_8.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_8.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_9.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_9.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_9.jpg?cache=20210217035932', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_0.gif?cache=20210217035932', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_0.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_0.jpg?cache=20210217035932', 'isVertical': True}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_A.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_A.jpg?cache=20210217035932', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_A.jpg?cache=20210217035933', 'isVertical': True}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_B.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_B.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_B.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_C.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_C.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_C.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_D.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_D.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_D.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_E.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_E.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_E.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_F.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_F.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_F.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_G.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_G.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_G.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_H.gif?cache=20210217035933', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_H.jpg?cache=20210217035933', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_H.jpg?cache=20210217035933', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_I.gif?cache=20200923031600', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_I.jpg?cache=20200923031600', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_I.jpg?cache=20200923031600', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_J.gif?cache=20200923031600', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_J.jpg?cache=20200923031600', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_J.jpg?cache=20200923031600', 'isVertical': False}, {'smallUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_K.gif?cache=20200923031600', 'mediumUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/M_8950161_K.jpg?cache=20200923031600', 'largeUrl': 'https://static.immoweb.be/photos/0/8/9/5/0/1/6/1/8950161_K.jpg?cache=20200923031600', 'isVertical': False}], 'virtualTourUrl': None, 'floorPlans': None, 'specifications': None, 'virtualExperienceUrl': 'https://youtu.be/D8wua7Z_Pe8'}, 'property': {'type': 'APARTMENT_GROUP', 'subtype': 'APARTMENT_GROUP', 'title': 'BUY NOW AND PAY IN 2022', 'description': 'This new project (available at the end of 2022) is ideally located in the new district of Chirec / Delta, near the Université Libre de Bruxelles (ULB and VUB) and consists of 70 units, from studios to 2-bedroom apartments. Most units have a terrace and / or a view of the green surroundings and the green rooftops. The building has a passive energy label. PEB A+ A bicycle storage is available on the 1st floor. Parkings and cellars are also provided at the same floor, at an additional cost. Sale under the mixed system of registration on the land value and VAT on the construction value. Choose your apartment now and pay for it at the delivery, at no extra cost! Contact us for more information about this project: 02 3751010 / new#victoire.be', 'name': None, 'isHolidayProperty': None, 'bedroomCount': None, 'bedrooms': [], 'bathroomCount': None, 'bathrooms': [], 'location': {'country': 'Belgium', 'region': 'Brussels', 'province': 'Brussels', 'district': 'Brussels', 'locality': 'Auderghem', 'postalCode': '1160', 'street': 'Boulevard du Triomphe', 'number': '201', 'box': None, 'propertyName': None, 'floor': None, 'latitude': 50.816023, 'longitude': 4.3999304, 'approximated': None, 'regionCode': 'BRUSSELS', 'type': None, 'hasSeaView': None, 'pointsOfInterest': [{'type': 'SCHOOL', 'distance': 150}, {'type': 'SHOPS', 'distance': 200}, {'type': 'TRANSPORT', 'distance': 0}], 'placeName': 'Projets Bruxelles Est'}, 'netHabitableSurface': None, 'roomCount': None, 'attic': None, 'hasAttic': None, 'basement': None, 'hasBasement': None, 'hasDressingRoom': None, 'diningRoom': None, 'hasDiningRoom': None, 'building': None, 'propertyCertificates': None, 'hasCaretakerOrConcierge': None, 'hasDisabledAccess': None, 'hasLift': None, 'constructionPermit': {'constructionType': None, 'floodZoneType': 'NON_FLOOD_ZONE', 'isObtained': None, 'hasObligationToConstruct': None, 'hasPlotDivisionAuthorization': None, 'hasPossiblePriorityPurchaseRight': None, 'isBreachingUrbanPlanningRegulation': None, 'floodZoneIconUrl': None, 'totalBuildableGroundFloorSurface': 0, 'urbanPlanningInformation': None}, 'energy': None, 'kitchen': None, 'land': None, 'laundryRoom': None, 'hasLaundryRoom': None, 'livingRoom': None, 'hasLivingRoom': False, 'isFirstOccupation': None, 'hasBalcony': None, 'hasBarbecue': None, 'hasGarden': None, 'gardenSurface': None, 'gardenOrientation': None, 'parkingCountIndoor': 1, 'parkingCountOutdoor': 1, 'parkingCountClosedBox': None, 'hasAirConditioning': None, 'hasArmoredDoor': None, 'hasVisiophone': None, 'hasSecureAccessAlarm': None, 'hasTVCable': None, 'hasDoorPhone': None, 'hasInternet': None, 'showerRoomCount': None, 'showerRooms': [], 'specificities': None, 'toiletCount': None, 'toilets': [], 'hasFitnessRoom': None, 'hasTennisCourt': None, 'hasSwimmingPool': None, 'hasSauna': None, 'hasJacuzzi': None, 'hasHammam': None, 'bedroomSurface': None, 'alternativeDescriptions': {'fr': "Ce projet (disponible fin 2022), bien situé dans le nouveau quartier du Chirec/Delta, et des facultés universitaires ULB/VUB, comporte 70 logements allant principalement du studio à l’appartement 2 chambres. Très urbain, les logements bénéficient néanmoins pour la plupart de terrasses et/ou de vues sur la verdure et les toits verdurisés. Le bÃtiment est passif et profite de l’alimentation en chauffage co-génération du Chirec. PEB A+ Le local vélo est disponible au 1er et les parkings et les caves au © 1 sont en supplément. Vente sous droits d'enregistrement sur le terrain et sous régime TVA sur la construction, Réservez votre appartement maintenant et payez-le à la livraison, sans augmentation de prix ! Consultez-nous sans attendre pour tout détail sur ce projet : 02 3751010 / new#victoire.be", 'nl': "Dit nieuwbouwproject (beschikbaar eind 2022) is uitstekend gelegen in de nieuwe stadswijk Chirec/Delta, nabij de Vrije Universiteit Brussel en omvat 70 wooneenheden gaande van studio's tot appartementen met 2 slaapkamers. De meeste units hebben een terras en/of zicht op de groene omgeving en de groene daken. Het gebouw heeft een passief energielabel. EPC A+ Een fietsenberging is voorzien op de 1e verdieping. Hier zijn ook parkings en kelders voorzien, dit tegen een meerprijs. Verkoop onder het gemend stelsel van registratierechten op de grondwaarde en BTW op de constructiewaarde. Kies nu uw appartement en betaal het aan de levering à zonder meerprijs! Raadpleeg ons voor meer informatie over dit project: 02 3751010 / new#victoire.be"}, 'habitableUnitCount': None, 'fireplaceCount': None, 'fireplaceExists': False, 'hasTerrace': None, 'terraceSurface': None, 'terraceOrientation': None}, 'publication': {'creationDate': '2020-09-23T03:12:09.000+0000', 'expirationDate': '2021-04-30T21:59:59.000+0000', 'publisherId': None, 'visualisationOption': 'XL', 'lastModificationDate': '2021-04-22T02:36:15.000+0000'}, 'transaction': {'type': 'FOR_SALE', 'subtype': 'BUY_REGULAR', 'availabilityPeriodType': 'AT_DELIVERY', 'availabilityDate': '2022-10-01T00:00:00.000+0000', 'certificates': {'carbonEmission': 40, 'hasElectricalInstallationComplianceCertificate': None, 'primaryEnergyConsumptionPerSqm': 45, 'primaryEnergyConsumptionYearly': None, 'epcDescription': None, 'epcReference': 'A+', 'epcScore': 'A', 'epcUrl': 'https://static.immoweb.be/en/pics/bxlA3.gif'}, 'rental': None, 'sale': {'price': None, 'vatType': 'VAT_EXCLUDED', 'cadastralIncome': 0, 'publicSale': None, 'pricePerSqm': None, 'oldPrice': None, 'lifeAnnuity': None, 'hasStartingPrice': True, 'isFurnished': None, 'homeToBuild': None}, 'investor': {'isInvestmentProperty': False, 'currentMonthlyRentalIncome': None, 'currentReturnOnInvestment': None, 'expectedMonthlyRentalIncome': None, 'expectedMonthlyRentalIncomeDescription': None, 'expectedReturnOnInvestment': None, 'habitableUnitCount': None, 'occupancyRate': None}}, 'priceType': None, 'price': {'type': 'group_sale', 'mainValue': None, 'alternativeValue': None, 'additionalValue': None, 'oldValue': None, 'minRangeValue': 240500, 'maxRangeValue': 521500, 'mainDisplayPrice': '€240,500 - €521,500', 'HTMLDisplayPrice': '<span aria-hidden="true">€240,500 - €521,500</span>', 'alternativeDisplayPrice': '', 'oldDisplayPrice': None, 'shortDisplayPrice': '240.5K - 521.5K €', 'accessibilityPrice': 'From 240500€ To 521500€', 'label': 'Min price - Max price', 'language': 'en'}, 'externalReference': '4161726', 'isBookmarked': False, 'hasSectionsArray': {'hasGeneralSection': True, 'hasInteriorSection': False, 'hasExteriorSection': False, 'hasFacilitiesSection': False, 'hasEnergySection': True, 'hasPlanningSection': True, 'hasFinancialSection': True, 'hasPublicSaleSection': False}, 'unitGroupings': [], 'displayFlags': {'main': None, 'secondary': ['new_real_estate_project', 'percent_sold'], 'percentSold': 47}, 'statistics': {'bookmarkCount': 18, 'viewCount': 2604, 'alertPrice': None, 'creationDate': None, 'description': None, 'isAlertEmailSet': None, 'rating': None, 'wasOwnerContacted': None, 'wasPropertyVisited': None}}
I am trying to append a dictionary to my already existing key "processed_data" where data is saved in the list of dictionaries. I tried several methods as shown in already asked questions but they did not work. This is my schema.
{'_id': ObjectId('5fe46a5b7468e3498124fcbe'), 'metadata': {'_id': ObjectId('5fe4500c7b2c03decd86334f'), 'type': 'VIDEO', 'id': 'o6st4ces9Wg"},"qoeUrl":{"baseUrl":"https://s.youtube.com/api/stats/qoe?cl=348521801', 'user_id': 'fc3240b2d7ef9d33bbb04fd7203e35ea9da54ffb', 'name': 'City Ak47', 'thumbnail': 'https://i.ytimg.com/vi/o6st4ces9Wg/hqdefault.jpg', 'title': 'Alex Bhatti ki Video Viral Ho Gie | How To Become Tiktok Star | City AK47 - YouTube', 'publication_date': 'Sep 17, 2020', 'channel_id': 'UCuo6tBl2MfkWvMPyCqph2LA', 'channel_name': 'City Ak47', 'scrape_date': '2020-12-24 08:23:17.390018', 'regions_allowed': 'AD,AE,AF,AG,AI,AL,AM,AO,AQ,AR,AS,AT,AU,AW,AX,AZ,BA,BB,BD,BE,BF,BG,BH,BI,BJ,BL,BM,BN,BO,BQ,BR,BS,BT,BV,BW,BY,BZ,CA,CC,CD,CF,CG,CH,CI,CK,CL,CM,CN,CO,CR,CU,CV,CW,CX,CY,CZ,DE,DJ,DK,DM,DO,DZ,EC,EE,EG,EH,ER,ES,ET,FI,FJ,FK,FM,FO,FR,GA,GB,GD,GE,GF,GG,GH,GI,GL,GM,GN,GP,GQ,GR,GS,GT,GU,GW,GY,HK,HM,HN,HR,HT,HU,ID,IE,IL,IM,IN,IO,IQ,IR,IS,IT,JE,JM,JO,JP,KE,KG,KH,KI,KM,KN,KP,KR,KW,KY,KZ,LA,LB,LC,LI,LK,LR,LS,LT,LU,LV,LY,MA,MC,MD,ME,MF,MG,MH,MK,ML,MM,MN,MO,MP,MQ,MR,MS,MT,MU,MV,MW,MX,MY,MZ,NA,NC,NE,NF,NG,NI,NL,NO,NP,NR,NU,NZ,OM,PA,PE,PF,PG,PH,PK,PL,PM,PN,PR,PS,PT,PW,PY,QA,RE,RO,RS,RU,RW,SA,SB,SC,SD,SE,SG,SH,SI,SJ,SK,SL,SM,SN,SO,SR,SS,ST,SV,SX,SY,SZ,TC,TD,TF,TG,TH,TJ,TK,TL,TM,TN,TO,TR,TT,TV,TW,TZ,UA,UG,UM,US,UY,UZ,VA,VC,VE,VG,VI,VN,VU,WF,WS,YE,YT,ZA,ZM,ZW', 'views': '663962', 'is_family_friendly': 'true', 'category': 'Entertainment', 'tags': ['AmirFilms', 'Alex Bhatti ki Video Viral Ho Gie | How To Become Tiktok Star | City AK47', 'Tiktok star', 'Tiktok', 'Alex tiktokr', 'Alex bhatti tiktok star', 'Alex bhatti', 'Ayesha bukhari', 'Viral video', 'New video', 'Leak vidro', 'Ayesha leak video', 'Alex bhatti leak video', 'News', 'Tiktik funny video'], 'language': 'en-US', 'width': '480', 'height': '360', 'job_id': '539f61c4183c46448a75cfb65dc40926'}, 'results': {'unique_word_freq': [{'text': 'hai', 'value': 6}, {'text': 'famous', 'value': 4}, {'text': 'allah', 'value': 3}, {'text': 'kar', 'value': 3}, {'text': 'gy', 'value': 3}, {'text': 'ye', 'value': 3}, {'text': 'yeh', 'value': 2}, {'text': 'ka', 'value': 2}, {'text': 'video', 'value': 2}, {'text': 'asee', 'value': 2}, {'text': 'nhi', 'value': 2}, {'text': 'ho', 'value': 2}, {'text': 'tum', 'value': 2}, {'text': 'jao', 'value': 2}, {'text': 'kitna', 'value': 1}, {'text': 'budsoor', 'value': 1}, {'text': 'gundgi', 'value': 1}, {'text': 'dher', 'value': 1}, {'text': 'khusra', 'value': 1}, {'text': 'tiktok', 'value': 1}, {'text': 'kunjuro', 'value': 1}, {'text': 'zanano', 'value': 1}, {'text': 'kaam', 'value': 1}, {'text': 'usko', 'value': 1}, {'text': 'hadyat', 'value': 1}, {'text': 'de', 'value': 1}, {'text': 'ameen', 'value': 1}, {'text': '😔', 'value': 1}, {'text': 'kahn', 'value': 1}, {'text': 'puri', 'value': 1}, {'text': 'kotta', 'value': 1}, {'text': 'ٹک', 'value': 1}, {'text': 'ٹاک', 'value': 1}, {'text': 'ایپ', 'value': 1}, {'text': 'پر', 'value': 1}, {'text': 'پاکستان', 'value': 1}, {'text': 'میں', 'value': 1}, {'text': 'مکمل', 'value': 1}, {'text': 'پابندی', 'value': 1}, {'text': 'لگنی', 'value': 1}, {'text': 'چاہیے', 'value': 1}, {'text': 'leaked', 'value': 1}, {'text': 'purpose', 'value': 1}, {'text': 'fame', 'value': 1}, {'text': 'views', 'value': 1}, {'text': 'mean', 'value': 1}, {'text': 'people', 'value': 1}, {'text': 'like', 'value': 1}, {'text': 'kinda', 'value': 1}, {'text': 'cheap', 'value': 1}, {'text': 'acts', 'value': 1}, {'text': 'inki', 'value': 1}, {'text': 'maa', 'value': 1}, {'text': 'bhano', 'value': 1}, {'text': 'sath', 'value': 1}, {'text': 'bhi', 'value': 1}, {'text': 'hoo', 'value': 1}, {'text': 'pak', 'value': 1}, {'text': 'ko', 'value': 1}, {'text': 'bohot', 'value': 1}, {'text': 'bari', 'value': 1}, {'text': 'sazaa', 'value': 1}, {'text': 'dee', 'value': 1}, {'text': 'duniyan', 'value': 1}, {'text': 'hee', 'value': 1}, {'text': 'dikhaee', 'value': 1}, {'text': 'pata', 'value': 1}, {'text': 'khha', 'value': 1}, {'text': 'jay', 'value': 1}, {'text': 'kiyamat', 'value': 1}, {'text': 'din', 'value': 1}, {'text': 'logo', 'value': 1}, {'text': 'hisab', 'value': 1}, {'text': 'lena', 'value': 1}, {'text': 'log', 'value': 1}, {'text': 'sidah', 'value': 1}, {'text': 'janat', 'value': 1}, {'text': 'chaly', 'value': 1}, {'text': 'baaz', 'value': 1}, {'text': 'ap', 'value': 1}, {'text': 'bakwas', 'value': 1}, {'text': 'band', 'value': 1}, {'text': 'kareen', 'value': 1}, {'text': 'larka', 'value': 1}, {'text': 'bharva', 'value': 1}, {'text': 'bs', 'value': 1}, {'text': 'pakar', 'value': 1}, {'text': 'gal', 'value': 1}, {'text': 'ma', 'value': 1}, {'text': 'dala', 'value': 1}, {'text': 'gaya', 'value': 1}, {'text': 'bahut', 'value': 1}, {'text': 'ghatiya', 'value': 1}, {'text': 'insan', 'value': 1}, {'text': 'tu', 'value': 1}, {'text': 'chakka', 'value': 1}, {'text': 'alex', 'value': 1}, {'text': 'bhatti', 'value': 1}], 'polarity_freq': [{'date': '2020-12-03', 'total': 4, 'positive': 3, 'negative': 1}, {'date': '2020-12-10', 'total': 9, 'positive': 8, 'negative': 1}, {'date': '2020-12-17', 'total': 2, 'positive': 2, 'negative': 0}, {'date': '2020-12-21', 'total': 1, 'positive': 1, 'negative': 0}, {'date': '2020-12-22', 'total': 2, 'positive': 1, 'negative': 1}], 'polarity_dist': [{'name': 'positive', 'value': '15'}, {'name': 'negative', 'value': '3'}], 'assoc': []}, 'processed_data': [{'index': 0, '_id': ObjectId('5fe4500c7b2c03decd863350'), 'channel_id': '/channel/UCg7rf8yXy8wqVxlbnErgdyg', 'clean_text': 'kitna budsoor hai yeh gundgi ka dher khusra', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniDMBIClPo0sPLX5RDOLPHTJhECMOub-fC0ZTVY6Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-22 08:23:17', 'id': 'UgxzkGuC2JpeaZD7El14AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 days ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Kitna budsoor hai yeh gundgi ka dher khusra.', 'tokens': ['kitna', 'budsoor', 'hai', 'yeh', 'gundgi', 'ka', 'dher', 'khusra'], 'tokens_no_swords': ['kitna', 'budsoor', 'hai', 'yeh', 'gundgi', 'ka', 'dher', 'khusra'], 'tran_text': 'kitna budsoor hai yeh gundgi ka dher khusra .', 'type': 'COMMENT', 'user_id': 'f7961259b974ba9fae934410fca2e939d3493038', 'user_name': 'jimmi khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '1', 'date': '2020-12-22'}, {'index': 1, '_id': ObjectId('5fe4500c7b2c03decd863351'), 'channel_id': '/channel/UCg7rf8yXy8wqVxlbnErgdyg', 'clean_text': 'tiktok kunjuro zanano ka kaam hai', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniDMBIClPo0sPLX5RDOLPHTJhECMOub-fC0ZTVY6Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-22 08:23:17', 'id': 'UgwntMkhi7J2l2N3MZJ4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 days ago (edited)', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Tiktok kunjuro r zanano ka kaam hai.', 'tokens': ['tiktok', 'kunjuro', 'zanano', 'ka', 'kaam', 'hai'], 'tokens_no_swords': ['tiktok', 'kunjuro', 'zanano', 'ka', 'kaam', 'hai'], 'tran_text': 'tiktok kunjuro r zanano ka kaam hai .', 'type': 'COMMENT', 'user_id': 'f7961259b974ba9fae934410fca2e939d3493038', 'user_name': 'jimmi khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-22'}, {'index': 2, '_id': ObjectId('5fe4500c7b2c03decd863352'), 'channel_id': '/channel/UCMDNByou1B62upgmnv-UQMw', 'clean_text': 'allah usko hadyat de ameen 😔', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnik2uW0mzYoagKEYX1_kGY3HDhYd3Ni6UlOxSEHOA=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-21 08:23:17', 'id': 'UgxZrbzomoOLyGEGAjp4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 days ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Allah usko hadyat de ameen 😔', 'tokens': ['allah', 'usko', 'hadyat', 'de', 'ameen', '😔'], 'tokens_no_swords': ['allah', 'usko', 'hadyat', 'de', 'ameen', '😔'], 'tran_text': 'allah usko hadyat de ameen 😔', 'type': 'COMMENT', 'user_id': 'da9fe12c7945488a70f56355f8c122d2f35231c5', 'user_name': 'neha Rajput', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-21'}, {'index': 3, '_id': ObjectId('5fe4500c7b2c03decd863353'), 'channel_id': '/channel/UCkl4U918shu8CroBno8-aJg', 'clean_text': '', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniom0S4ta4uSnNx7yD69NfR4TmOqXPpYxv6_Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-17 08:23:17', 'id': 'UgzAfvcluRdyX9yi-JJ4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '1 week ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': '420', 'tokens': [], 'tokens_no_swords': [], 'tran_text': '420', 'type': 'COMMENT', 'user_id': '04b4dd4534a4acf47ba876387d752eda8d3087f6', 'user_name': 'Shahid Khankarachi', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-17'}, {'index': 5, '_id': ObjectId('5fe4500c7b2c03decd863355'), 'channel_id': '/channel/UCoL0h9EyBTNSvKIWIxl6WIg', 'clean_text': 'kahn hai yeh puri video', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnjNJAzxyS9mOk-R7TF5ICxa0_EQbtgcL3z2Yg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-17 08:23:17', 'id': 'Ugxx-JvihbK7P8Y8u5x4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '1 week ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Kahn hai yeh puri video', 'tokens': ['kahn', 'hai', 'yeh', 'puri', 'video'], 'tokens_no_swords': ['kahn', 'hai', 'yeh', 'puri', 'video'], 'tran_text': 'kahn hai yeh puri video', 'type': 'COMMENT', 'user_id': '3bc410f7e5133b61e2f2cc790ce6ae2692397778', 'user_name': 'ALISHA ZOYA', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-17'}, {'index': 6, '_id': ObjectId('5fe4500c7b2c03decd863356'), 'channel_id': '/channel/UCsd6TX3yWpNYK55hawyi8qw', 'clean_text': 'kotta', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnj19uWVIJ75wx27KLjDGDcsVcGtzVtp8SRQ0w=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgyhHfBJDWHtL73E71N4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Kotta', 'tokens': ['kotta'], 'tokens_no_swords': ['kotta'], 'tran_text': 'kotta', 'type': 'COMMENT', 'user_id': 'e83b422e66c1bd722306aee6715c3846c32e506b', 'user_name': 'Shakeel Khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 7, '_id': ObjectId('5fe4500c7b2c03decd863357'), 'channel_id': '/channel/UCX6LjA5LbC7xMO19yyM7m0Q', 'clean_text': 'ٹک ٹاک ایپ پر پاکستان میں مکمل پابندی لگنی چاہیے', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwni4WpKnrXzHmw2VwT0z5aYnM0T5IhRN0DG3Pmsg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgwFILpwDAQKYA9ioMV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'en', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'ٹک ٹاک ایپ پر پاکستان میں مکمل پابندی لگنی چاہیے', 'tokens': ['ٹک', 'ٹاک', 'ایپ', 'پر', 'پاکستان', 'میں', 'مکمل', 'پابندی', 'لگنی', 'چاہیے'], 'tokens_no_swords': ['ٹک', 'ٹاک', 'ایپ', 'پر', 'پاکستان', 'میں', 'مکمل', 'پابندی', 'لگنی', 'چاہیے'], 'tran_text': 'ٹک ٹاک ایپ پر پاکستان میں مکمل پابندی لگنی چاہیے', 'type': 'COMMENT', 'user_id': '9ca6083ff6234bd94fc218ef27d12c8b91c2fa33', 'user_name': 'Wahab Mirza', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 8, '_id': ObjectId('5fe4500c7b2c03decd863358'), 'channel_id': '/channel/UCKMvpfSppW24ixWCOmJDu_g', 'clean_text': 'he leaked this video on purpose to get fame and views i mean people like them do these kinda cheap acts to get famous', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniL8sePcWPsqDg6AOaLsW4nf14XDW3132kC0Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'Ugz7IdhD6s8zCJ6vHNt4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'en', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'He leaked this video on purpose to get fame and views...I mean people like them do these kinda cheap acts to get famous.', 'tokens': ['he', 'leaked', 'this', 'video', 'on', 'purpose', 'to', 'get', 'fame', 'and', 'views', 'i', 'mean', 'people', 'like', 'them', 'do', 'these', 'kinda', 'cheap', 'acts', 'to', 'get', 'famous'], 'tokens_no_swords': ['leaked', 'video', 'purpose', 'fame', 'views', 'mean', 'people', 'like', 'kinda', 'cheap', 'acts', 'famous'], 'tran_text': 'he leaked this video on purpose to get fame and views ... i mean people like them do these kinda cheap acts to get famous .', 'type': 'COMMENT', 'user_id': '3f75585892685df3ae4b3d733d9795a719b2d528', 'user_name': 'Ana T', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 9, '_id': ObjectId('5fe4500c7b2c03decd863359'), 'channel_id': '/channel/UCSUGRfHKn5qCNN4TKG3MAkw', 'clean_text': '', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwng6eAHeRd7CcM8mmkCHCA8VI2tqmMNPb1q1MA=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgwMvbYR29V4ISyic_d4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'en', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': '03065455318', 'tokens': [], 'tokens_no_swords': [], 'tran_text': '03065455318', 'type': 'COMMENT', 'user_id': '23ac482ba9b36182915c502c13d4cd45b7f7bf1f', 'user_name': 'Ali Rizwan', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 10, '_id': ObjectId('5fe4500c7b2c03decd86335a'), 'channel_id': '/channel/UC-fWQ2vkmngdVliZDkRSiiQ', 'clean_text': 'inki maa bhano sath bhi asee hoo', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwngl0V2Zy_AUGUyIZpMbrBDxqL6pq5AcdF4hNg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgzEMbiRNk2ywQHlgxR4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Inki maa bhano k sath bhi Asee hoo', 'tokens': ['inki', 'maa', 'bhano', 'sath', 'bhi', 'asee', 'hoo'], 'tokens_no_swords': ['inki', 'maa', 'bhano', 'sath', 'bhi', 'asee', 'hoo'], 'tran_text': 'inki maa bhano k sath bhi asee hoo', 'type': 'COMMENT', 'user_id': '2c7e490aa5d0ceca9340c92c3577fa75d3e5a8d3', 'user_name': 'M wali Yousuf', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 11, '_id': ObjectId('5fe4500c7b2c03decd86335b'), 'channel_id': '/channel/UC-fWQ2vkmngdVliZDkRSiiQ', 'clean_text': 'allah pak asee ko bohot bari sazaa dee or duniyan me hee dikhaee', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwngl0V2Zy_AUGUyIZpMbrBDxqL6pq5AcdF4hNg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'Ugyo9YlNa7zuVsSQlZh4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Allah Pak Asee ko Bohot bari Sazaa Dee Or Duniyan me hee dikhaee', 'tokens': ['allah', 'pak', 'asee', 'ko', 'bohot', 'bari', 'sazaa', 'dee', 'or', 'duniyan', 'me', 'hee', 'dikhaee'], 'tokens_no_swords': ['allah', 'pak', 'asee', 'ko', 'bohot', 'bari', 'sazaa', 'dee', 'duniyan', 'hee', 'dikhaee'], 'tran_text': 'allah pak asee ko bohot bari sazaa dee or duniyan me hee dikhaee', 'type': 'COMMENT', 'user_id': '2c7e490aa5d0ceca9340c92c3577fa75d3e5a8d3', 'user_name': 'M wali Yousuf', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 12, '_id': ObjectId('5fe4500c7b2c03decd86335c'), 'channel_id': '/channel/UCphcNEEoxrp08DARCX7dSNQ', 'clean_text': 'pata nhi famous ho kar khha jay gy kiyamat din allah famous logo hisab lena hai ye nhi tum log famous ho gy or sidah janat chaly jao gy baaz a jao', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnibNbjcM0UMLW2aTnOD3jfJXlaq2Iq5_hMg3Q-O=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgzfXCaZHDMSCKPqlKB4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': '1', 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Pata nhi famous ho Kar khha jay gy. Kiyamat k din Allah n famous logo c b hisab lena hai ye nhi k tum log famous ho gy or sidah janat m chaly jao gy. Baaz a jao', 'tokens': ['pata', 'nhi', 'famous', 'ho', 'kar', 'khha', 'jay', 'gy', 'kiyamat', 'din', 'allah', 'famous', 'logo', 'hisab', 'lena', 'hai', 'ye', 'nhi', 'tum', 'log', 'famous', 'ho', 'gy', 'or', 'sidah', 'janat', 'chaly', 'jao', 'gy', 'baaz', 'a', 'jao'], 'tokens_no_swords': ['pata', 'nhi', 'famous', 'ho', 'kar', 'khha', 'jay', 'gy', 'kiyamat', 'din', 'allah', 'famous', 'logo', 'hisab', 'lena', 'hai', 'ye', 'nhi', 'tum', 'log', 'famous', 'ho', 'gy', 'sidah', 'janat', 'chaly', 'jao', 'gy', 'baaz', 'jao'], 'tran_text': 'pata nhi famous ho kar khha jay gy . kiyamat k din allah n famous logo c b hisab lena hai ye nhi k tum log famous ho gy or sidah janat m chaly jao gy . baaz a jao', 'type': 'COMMENT', 'user_id': 'da5c845fbd0a39db29a99a9d620bd8c266956065', 'user_name': 'Rida Khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '1', 'date': '2020-12-10'}, {'index': 13, '_id': ObjectId('5fe4500c7b2c03decd86335d'), 'channel_id': '/channel/UCnVVsV2fd3P0lS9QClU5DCA', 'clean_text': 'ap bakwas band kareen', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniJxcGaZeKzmvDSUGeX5vFZo3m_ZXQ_yC7-Kw=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgwzEHvn_WWTbPOzl7x4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Ap bakwas band kareen', 'tokens': ['ap', 'bakwas', 'band', 'kareen'], 'tokens_no_swords': ['ap', 'bakwas', 'band', 'kareen'], 'tran_text': 'ap bakwas band kareen', 'type': 'COMMENT', 'user_id': 'e51ff2c592a9ad2fe8f6f373c6a2dab117f2c2e9', 'user_name': 'ahmad muaaz', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 14, '_id': ObjectId('5fe4500c7b2c03decd86335e'), 'channel_id': '/channel/UCtbUvUvL0qrREfEzUvJUQKQ', 'clean_text': 'ye larka bharva', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnjKBfmUKuMzFCtwM-KuKAfq_5y0RA7iez5w9Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgzvW63udc6CEwnsb_J4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'ye larka bharva', 'tokens': ['ye', 'larka', 'bharva'], 'tokens_no_swords': ['ye', 'larka', 'bharva'], 'tran_text': 'ye larka bharva', 'type': 'COMMENT', 'user_id': '7d7c7165c38c05a96c335421faf6ca3eb9eb1722', 'user_name': 'Rana Waqas', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 15, '_id': ObjectId('5fe4500c7b2c03decd86335f'), 'channel_id': '/channel/UCieduNjSrF2DPawdZh_HesQ', 'clean_text': 'bs kar do tum', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniOEn-mQTkzQu5ybCc6gjFqSlK8eQF-4RsB6w=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'Ugx9gyNwYeVV5DDKsJV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'bs kar do tum', 'tokens': ['bs', 'kar', 'do', 'tum'], 'tokens_no_swords': ['bs', 'kar', 'tum'], 'tran_text': 'bs kar do tum', 'type': 'COMMENT', 'user_id': '7339de48d380c9efe854dc9b6660a8fe22c28448', 'user_name': 'sami ali ali', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}, {'index': 16, '_id': ObjectId('5fe4500c7b2c03decd863360'), 'channel_id': '/channel/UCaPZsZzHcOMiDgZ3rkALMFg', 'clean_text': 'is pakar kar gal ma dala gaya', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnhv4vXDX16Pi0veGMZVUtqiYiYq_XOUp2yTvQ=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'UgzC4DaDahEFvjUsJiN4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Is pakar kar gal ma dala gaya', 'tokens': ['is', 'pakar', 'kar', 'gal', 'ma', 'dala', 'gaya'], 'tokens_no_swords': ['pakar', 'kar', 'gal', 'ma', 'dala', 'gaya'], 'tran_text': 'is pakar kar gal ma dala gaya', 'type': 'COMMENT', 'user_id': 'ae974ed633daab66164b5dcee9340e2ed0b1c455', 'user_name': 'munir gill', 'video_id': 'o6st4ces9Wg', 'is_hate': '1', 'date': '2020-12-03'}, {'index': 17, '_id': ObjectId('5fe4500c7b2c03decd863361'), 'channel_id': '/channel/UCqGRwUGDEBY98v0PcA9BpUQ', 'clean_text': 'bahut ghatiya insan hai', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnhMts6KGq4VtnvbDuVVatNlFduO6jmHbIRX6A=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'UgyyCAdkym_IDptSbNZ4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Bahut ghatiya insan hai', 'tokens': ['bahut', 'ghatiya', 'insan', 'hai'], 'tokens_no_swords': ['bahut', 'ghatiya', 'insan', 'hai'], 'tran_text': 'bahut ghatiya insan hai', 'type': 'COMMENT', 'user_id': '331593ec91edb449953d775229e7a91727415976', 'user_name': 'Asif Bhatti', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}, {'index': 18, '_id': ObjectId('5fe4500c7b2c03decd863362'), 'channel_id': '/channel/UCBZ0mLPPioFWW1i-kvmZBnA', 'clean_text': 'ye tu chakka hai alex bhatti', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnggrksT4HvfysI9VkzPzsKIXkcJsPfmWvvNyg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'Ugwh57O9lzDJgzCvKJV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Ye tu chakka hai alex bhatti', 'tokens': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tokens_no_swords': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tran_text': 'ye tu chakka hai alex bhatti', 'type': 'COMMENT', 'user_id': 'ee99e20e5128b5fc14c1972d55625585cf4d0237', 'user_name': 'Khizar Rao', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}]}
I want to append the following dictionary against the "processed_data" key.
{'index': 19, '_id': ObjectId('5fe4500c7b2c03decd863362'), 'channel_id': '/channel/UCBZ0mLPPioFWW1i-kvmZBnA', 'clean_text': 'ye tu chakka hai alex bhatti', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnggrksT4HvfysI9VkzPzsKIXkcJsPfmWvvNyg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'Ugwh57O9lzDJgzCvKJV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Ye tu chakka hai alex bhatti', 'tokens': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tokens_no_swords': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tran_text': 'ye tu chakka hai alex bhatti', 'type': 'COMMENT', 'user_id': 'ee99e20e5128b5fc14c1972d55625585cf4d0237', 'user_name': 'Khizar Rao', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}
Thanks!
Use update_one() with $push:
from pymongo import MongoClient
from bson import ObjectId
db = MongoClient()['mydatabase']
db.mycollection.insert_one({'_id': ObjectId('5fe46a5b7468e3498124fcbe'),
'processed_data': []})
update = {'index': 19, '_id': ObjectId('5fe4500c7b2c03decd863362'), 'channel_id': 'etc.'}
db.mycollection.update_one({'_id': ObjectId('5fe46a5b7468e3498124fcbe')}, {'$push': {'processed_data': update}})
I am working on a nested dictionary and not able to put it in a dataframe.
This is what it looks like:
{'props': {'initialProps': {'statusCode': 0, '$isMobile': False, '$isIOS': None, '$isAndroid': False, '$host': 'www.tiktok.com', '$pageUrl': '/#alanwalkermusic/video/6816594007829859589', '$language': 'en', '$originalLanguage': 'en', '$languageList': [{'value': 'id', 'alias': 'id-ID', 'label': 'Bahasa Indonesia', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'de', 'alias': 'de-DE', 'label': 'Deutsch', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'en', 'alias': 'en', 'label': 'English', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'es', 'alias': 'es', 'label': 'Español', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'fr', 'alias': 'fr', 'label': 'Français', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'it', 'alias': 'it-IT', 'label': 'Italiano', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'pl', 'alias': 'pl-PL', 'label': 'Polski', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'pt_BR', 'alias': 'pt-BR', 'label': 'Português', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'vi', 'alias': 'vi-VN', 'label': 'Tiếng Việt', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'tr', 'alias': 'tr-TR', 'label': 'Türkçe', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'ru', 'alias': 'ru-RU', 'label': 'Русский', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'hi', 'alias': 'hi-IN', 'label': 'हिन्दी', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'ko', 'alias': 'ko-KR', 'label': '한국어', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'ja', 'alias': 'ja-JP', 'label': '日本語', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'zh_Hant', 'alias': 'zh-Hant-TW', 'label': '繁體中文', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'ar', 'alias': 'ar', 'label': 'العربية', 'children': [{'value': 'default', 'label': ''}]}], '$region': 'DE', '$appId': 1233, '$os': 'windows', '$baseURL': 'm.tiktok.com', '$downloadLink': {'amazon': {'visible': True, 'normal': 'https://www.amazon.com/dp/B0117U0G3M/'}, 'google': {'visible': True, 'normal': 'https://www.tiktok.com/download-link/af/com.zhiliaoapp.musically'}, 'apple': {'visible': True, 'normal': 'https://www.tiktok.com/download-link/af/id835599320'}}, '$abTestVersion': {'clientParameters': '{}', 'clientVersionName': '', 'versionName': '1618717', 'parameters': '{}', 'startTime': '', 'endTime': ''}, '$appType': 'm', '$gray': {'upload': False, 'tea': False}, '$reflowType': 't', '$legalList': [{'title': 'TikTok.com Cookies Policy', 'key': 'tiktok-website-cookies-policy', 'href': 'https://www.tiktok.com/legal/tiktok-website-cookies-policy?lang=en'}, {'title': 'Open Source', 'key': 'open-source', 'href': 'https://www.tiktok.com/legal/open-source?lang=en'}, {'title': 'Virtual Items', 'key': 'virtual-items', 'href': 'https://www.tiktok.com/legal/virtual-items?lang=en'}, {'title': 'Intellectual Property Policy', 'key': 'copyright-policy', 'href': 'https://www.tiktok.com/legal/copyright-policy?lang=en'}, {'title': 'Law Enforcement', 'key': 'law-enforcement', 'href': 'https://www.tiktok.com/legal/law-enforcement?lang=en'}, {'title': 'Privacy Policy', 'key': 'privacy-policy', 'href': 'https://www.tiktok.com/legal/privacy-policy?lang=en'}, {'title': 'Terms of Service', 'key': 'terms-of-use', 'href': 'https://www.tiktok.com/legal/terms-of-use?lang=en'}], '$botType': 'others', '$config': {'covidBanner': {'open': True, 'url': 'https://www.tiktok.com/safety/resources/covid-19', 'background': 'rgba(125,136,227,1)'}}, '$wid': '6819614602332063233'}, 'pageProps': {'serverCode': 200, 'pageState': {'regionAppId': 1233, 'os': 'windows', 'region': 'DE', 'baseURL': 'm.tiktok.com', 'appType': 't', 'fullUrl': 'https://www.tiktok.com/node/share/video/#alanwalkermusic/6816594007829859589'}, 'videoData': {'itemInfos': {'id': '6816594007829859589', 'video': {'urls': ['https://v19.muscdn.com/afefda4362dccd13b93a4d8e79ab5338/5ea477c7/video/tos/useast2a/tos-useast2a-ve-0068c004/7f922f16b9534b80be6523cbb8ba2ce2/?a=1233&br=2652&bt=1326&cr=0&cs=0&dr=0&ds=3&er=&l=202004251147400101150770371E5160EE&lr=tiktok_m&qs=0&rc=MzY3cXc1OjxwdDMzNjczM0ApNDtnNTY1Ojw2N2k8NDU4ZWdoZXBmLjE0ZGBfLS0vMTZzczZjYDEvL2BgMDAzXy0uMC86Yw%3D%3D&vl=&vr='], 'videoMeta': {'width': 540, 'height': 960, 'ratio': 11, 'duration': 11}}, 'covers': ['https://p16-va-default.akamaized.net/obj/tos-maliva-p-0068/8187e5fe9a0f684529910363f5dbe428'], 'authorId': '84453035275386880', 'coversOrigin': ['https://p16-va-default.akamaized.net/obj/tos-maliva-p-0068/7dd18717d8aa4916abc5699c25c09b3e_1587111976'], 'text': 'I’m that friend who can’t stand still when taking photos', 'commentCount': 300, 'diggCount': 14145, 'playCount': 91447, 'shareCount': 29, 'createTime': '1587111974', 'isActivityItem': False, 'warnInfo': [], 'liked': False, 'commentStatus': 0, 'showNotPass': False}, 'authorInfos': {'verified': True, 'secUid': 'MS4wLjABAAAAhS4j-WG_Zq5WJfdV35QEqorNPzc_kkFVNPQmGM6mQIWEg6FOgjS-Dl9eISlYvPDc', 'uniqueId': 'alanwalkermusic', 'userId': '84453035275386880', 'nickName': 'Alan Walker', 'covers': ['https://p16.muscdn.com/img/musically-maliva-obj/1649466056705029~c5_100x100.jpeg']}, 'musicInfos': {'musicId': '6805465027164768258', 'musicName': 'Heading Home', 'authorName': 'Alan Walker, Ruben', 'covers': ['https://p16.muscdn.com/img/musically-maliva-obj/1649466056705029~c5_100x100.jpeg']}, 'authorStats': {'followerCount': 2147815, 'heartCount': '10903873'}, 'challengeInfoList': [], 'duetInfo': '0', 'textExtra': []}, 'shareUser': {'secUid': '', 'userId': '', 'uniqueId': '', 'nickName': '', 'signature': '', 'covers': [], 'coversMedium': [], 'coversLarger': [], 'isSecret': False}, 'shareMeta': {'title': 'Alan Walker on TikTok', 'desc': 'I’m that friend who can’t stand still when taking photos', 'image': {'url': 'http://p16-va-default.akamaized.net/img/tos-maliva-p-0068/8187e5fe9a0f684529910363f5dbe428~tplv-tiktok-play.image', 'width': 540, 'height': 960}}, 'statusCode': 0, 'langList': [{'value': 'id', 'alias': 'id-ID', 'label': 'Bahasa Indonesia', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'de', 'alias': 'de-DE', 'label': 'Deutsch', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'en', 'alias': 'en', 'label': 'English', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'es', 'alias': 'es', 'label': 'Español', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'fr', 'alias': 'fr', 'label': 'Français', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'it', 'alias': 'it-IT', 'label': 'Italiano', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'pl', 'alias': 'pl-PL', 'label': 'Polski', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'pt_BR', 'alias': 'pt-BR', 'label': 'Português', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'vi', 'alias': 'vi-VN', 'label': 'Tiếng Việt', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'tr', 'alias': 'tr-TR', 'label': 'Türkçe', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'ru', 'alias': 'ru-RU', 'label': 'Русский', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'hi', 'alias': 'hi-IN', 'label': 'हिन्दी', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'ko', 'alias': 'ko-KR', 'label': '한국어', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'ja', 'alias': 'ja-JP', 'label': '日本語', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'zh_Hant', 'alias': 'zh-Hant-TW', 'label': '繁體中文', 'children': [{'value': 'default', 'label': ''}]}, {'value': 'ar', 'alias': 'ar', 'label': 'العربية', 'children': [{'value': 'default', 'label': ''}]}], 'webId': '6819614602168567298', 'requestId': '23892479587815260433', 'videoObjectPageProps': {'videoProps': {'url': 'https://www.tiktok.com/#alanwalkermusic/video/6816594007829859589', 'name': 'Alan Walker(#alanwalkermusic) on TikTok I’m that friend who can’t stand still when taking photos', 'description': 'Alan Walker(#alanwalkermusic) has created a short video on TikTok with music Heading Home. I’m that friend who can’t stand still when taking photos', 'keywords': 'alanwalkermusic, Alan Walker,', 'thumbnailUrl': ['https://p16-va-default.akamaized.net/obj/tos-maliva-p-0068/8187e5fe9a0f684529910363f5dbe428', 'https://p16-va-default.akamaized.net/obj/tos-maliva-p-0068/7dd18717d8aa4916abc5699c25c09b3e_1587111976'], 'uploadDate': '2020-04-17T08:26:14.000Z', 'contentUrl': 'https://v19.muscdn.com/afefda4362dccd13b93a4d8e79ab5338/5ea477c7/video/tos/useast2a/tos-useast2a-ve-0068c004/7f922f16b9534b80be6523cbb8ba2ce2/?a=1233&br=2652&bt=1326&cr=0&cs=0&dr=0&ds=3&er=&l=202004251147400101150770371E5160EE&lr=tiktok_m&qs=0&rc=MzY3cXc1OjxwdDMzNjczM0ApNDtnNTY1Ojw2N2k8NDU4ZWdoZXBmLjE0ZGBfLS0vMTZzczZjYDEvL2BgMDAzXy0uMC86Yw%3D%3D&vl=&vr=', 'embedUrl': 'https://www.tiktok.com/embed/v2/6816594007829859589', 'commentCount': '300', 'duration': 'PT11S', 'audio': {'name': 'Heading Home - Alan Walker, Ruben', 'author': 'Alan Walker, Ruben', 'mainEntityOfPage': {'#type': 'ItemPage', '#id': 'https://www.tiktok.com/music/Heading-Home-6805465027164768258'}}, 'creator': {'#type': 'Person', 'name': 'Alan Walker', 'alternateName': 'alanwalkermusic', 'url': 'https://www.tiktok.com/#alanwalkermusic', 'interactionStatistic': [{'#type': 'InteractionCounter', 'interactionType': {'#type': 'http://schema.org/LikeAction'}, 'userInteractionCount': '10903873'}, {'#type': 'InteractionCounter', 'interactionType': {'#type': 'http://schema.org/FollowAction'}, 'userInteractionCount': 2147815}]}, 'width': 540, 'height': 960, 'interactionStatistic': [{'#type': 'InteractionCounter', 'interactionType': {'#type': 'http://schema.org/WatchAction'}, 'userInteractionCount': 91447}, {'#type': 'InteractionCounter', 'interactionType': {'#type': 'http://schema.org/LikeAction'}, 'userInteractionCount': 14145}, {'#type': 'InteractionCounter', 'interactionType': {'#type': 'http://schema.org/ShareAction'}, 'userInteractionCount': 29}]}, 'pageProps': {'type': 'ItemPage', 'id': 'https://www.tiktok.com/#alanwalkermusic/video/6816594007829859589'}, 'breadcrumbProps': {'urlList': [{'name': 'TikTok', 'url': 'https://www.tiktok.com'}, {'name': 'Alan Walker(#alanwalkermusic) Official | TikTok', 'url': 'https://www.tiktok.com/#alanwalkermusic'}, {'name': 'Alan Walker(#alanwalkermusic) on TikTok I’m that friend who can’t stand still when taking photos', 'url': 'https://www.tiktok.com/#alanwalkermusic/video/6816594007829859589'}]}}, 'metaParams': {'title': 'Alan Walker(#alanwalkermusic) on TikTok I’m that friend who can’t stand still when taking photos', 'keywords': 'alanwalkermusic, Alan Walker,', 'description': 'Alan Walker(#alanwalkermusic) has created a short video on TikTok with music Heading Home. I’m that friend who can’t stand still when taking photos', 'robotsContent': 'index, follow'}, 'isSSR': True, 'pageOptions': {'header': {'showUpload': True, 'type': 'webapp'}, 'headOptions': None}}, 'pathname': '/share/video'}, 'page': '/share/video', 'query': {'uniqueId': 'alanwalkermusic', 'id': '6816594007829859589', '$initialProps': {'$wid': '6819614602332063233'}, 'webtoken': 'tac=\'i+2gv2ay5eehis!i#piis"yZl!%s"l"u&kLs#l l#vr*charCodeAtx0[!cb^i$1em7b*0d#>>>s j\\uffeel s#0y\\xe8g,&qnfme|ms l vk.}l ,(dfijxdaamvk5}l ,(dfijxdaam,$lwcamk\\xadl ,(dfijxdaam,$lwcams!l!v,\\\'nfmosCkmx,*~bgyad>r}~[!c0b<k$t jql!v,\\\'nfmosCkmx,%cokm3[!c0d#===k$t jMl!v,\\\'nfmosCkmx,0xefc.:9&*.4+2-0.[!c0d#===k"t f z[ cb|1d<y\\u01ea,(|fY\\x7f~d`hs g,(lfi~ah`{ms!g,&qnfme|ms"g,)gk}ejo{\\x7fcms#,)|doikgauus$ul"d\\\',typeofl$d#===v!kA}l"vl mx[ c,/T\\x7fsxvwa6#qw~tk#d#!==v!k\\\\}gr&Object,)yxdxbzv`tml mvr$callxl"[!c,/T\\x7fsxvwa6#qw~tk#d#!==v!k4}ul!d\\\',typeofl$d#===v!kG}l!vl mx[ cv,\\\'nfmosCkmx,(Lfi~ah`{[!c0b<v!k4}ul#d\\\',typeofl$d#===v!kD}l#vl mx[ c,2I|v\\x7fstl9Tzjty~TNP~d#!==v!k=}ugr(locationd\\\',typeofl$d#===k"t fv!k4}l!,,hbmz}t|gYzrrm!!!kjg,\\\'oaz~d~tms%ul%d\\\',typeofl$d#===v!kB}l%vl mx[ c,0K~pyqvb7PpiosogBd#!==k"t f z[ cb|1d<y\\xe2g,&qnfme|ms l vk-}l ,\\\'dggyd`hmvk7}l ,\\\'dggyd`hm,\\\'aa{oiyjmk"t ugr.InstallTriggerd\\\',typeof,)|doikgauud#!==kwl vkn}l ,*e~xh|Xyuf{ml ,*cebh|Xyuf{mb-\\x94b>v!kF}l ,+dyyk}Xt{t|aml ,+bbck}Xt{t|amb-\\x94b>k"t f z[ cb|1d<y\\xa0g,&akgkkgms g,&Iebli\\x7fms!ul d\\\',typeof,)|doikgauud#!==vkh}l!,)yxdxbzv`tm,(|fY\\x7f~d`hmvr$callxl ,\\\'wzfin\\x7f~m[!c,0K~pyqvb7hkuxynmBd#=== z[ cb|1d<y\\u011eg,(lfi~ah`{ms g,)gk}ejo{\\x7fcms!g,&qnfme|ms"fv!k4}l ,,hbmz}t|gYzrrm!!!k\\xd7,\\\'wd|mbb~l!d"in!v!kI}l!,\\\'wd|mbb~mg,+[`xif~P`aulmd*instanceof!v!k1},(Wybjbyabl"d"inv!k4},+hmab_xp|g{xl"d"inv!k4},+TScghxe\\x7frfpl"d"inv!kP},%Dscafl"d"in,8[xtm}nLzNEGQMKAdGG^NTY\\x1ckl"d"inb< f z[ cb|1d<y\\u02a1g,&qnfme|ms g,)gk}ejo{\\x7fcms!g,&Iebli\\x7fms"l!,)~oih\\x7fgyucmk"t (\\x80,.jjvx|vDgyg}knbl"d"inkfl"v,.jjvx|vDgyg}knbmxl!,)~oih\\x7fgyucgr&Objectn vuq%valuevfq(writable[#c}) %{s#t ,4KJarz}hrjxl#EWCOQDRB,3LKfs{}wsnqB{iAMWBP#,;DCj{}DSKUAWyTK[C[XrHZ^RFZ[[,7HGn\\x7fyxowiES}PGWOW\\\\vL^BN,5JI`}{~iuk{m\\x7fRAQMURxNG,3LKsnsjpl~nB{iAMWBP#,2MLpg\\x7fa}kEnrjl~PQGG,5JI`}{~iuk{m\\x7fTLTVDVWMM,1NMwf|`rjF\\x7fm}qk~TD,4KJert|tripAjNVPBTUCC,4KJpo|ksmyoAjNVPBTUCC[+s#,)Vyn`h`fe|,,olbcCt~vz|cz,6ID}u\\x7fuuhs#ieg|v#EHZMOY[#s$l$*%s%l%u&k4s&l$l&ms\\\'l l\\\'mk"t j\\x06l#*%s%l%u&k?s&l#l&ms\\\'l ,(lfi~ah`{ml\\\'mk"t j\\ufffbl ,(lfi~ah`{m*%s%l%u&kls&l&vr%matchxgr&RegExp$*\\\\$[a-z]dc_$ n"[!cvk:}l ,(lfi~ah`{ml&m,&efkaoTmk"t j\\uffcef z[ cb|1d<y\\u024fg,&qnfme|ms gr&RegExp$+constructor$!in"vr$testxl ,+CX#BJ|t\\x7fvzam[!cv!k\\xb2}yZl vr(toStringx[ c,AzMAN#ES\\x08zKMM_G}U\\\\]GQ{YCQ_SX]IWP.\\x1cd#=== l ,&ufnhxbm!v!kd}ugr&safarid\\\',typeof,)|doikgauud#!==vk=}gr&safari,0`da{Zzb~~pyzhtqqm&k\\u010e(=l v,,c}kaTpfrvtermxzzzz[$c}) %{s!t (\\x85l ,.}jcb{|zFbxjx}~mvr\\\'setItemx,+xc`kDuhZvfp, ["c}l ,.}jcb{|zFbxjx}~mvr*removeItemx,+xc`kDuhZvfp[!c}) \\x7f{s!l!,$gjbbmgr,DOMException,2CF[AWH]AY^YY[[\\x7fdpqmd#===vkC}l ,.}jcb{|zFbxjx}~m,&jbfn~cm0d#===k"t f fv!k>}g,(lfi~ah`{m,,hbmz}t|gYzrrm!!k`l ,)`doiukkTSm!vkJ}l ,,\\\\bgadt`Vbpxcmv!k4}l ,.C\\\\#~{}`pdRn|tomk"t f z[ cb|1d<y\\u0165g,(lfi~ah`{ms g,)gk}ejo{\\x7fcms!,(|fY\\x7f~d`hs",\\\'nfmosCks#fv!k4}l ,,hbmz}t|gYzrrm!!!k\\u0113l v,-n|jqewVxp{rvmmx,&eff\\x7fkx[!cs$l$,)}eOmyoZB]mvl"mx[ cv,\\\'umyfjohmxgr&RegExp$#\\\\s*$!gn"$ ["cvl#mx,*djxdxjs~vv[!c0b<v!kh}l!l"mvl"mx[ cvr\\\'replacexgr&RegExp$#\\\\s*$!gn"$ ["cvl#mx,*djxdxjs~vv[!c0b<v!kP}l!,\\\'wd|mbb~mvl"mx[ c,4Ozt}}zn;LqkxIOcQVD_zd#!== f z[ cb|1d<1b|\'', '0fea6a13c52b4d4725368f24b045ca84': {}, 'aa59d67c2123f094d0d6798ffe651c4d': {}}, 'buildId': '1.0.1.1363', 'assetPrefix': '//s16.tiktokcdn.com/tiktok/falcon', 'isFallback': False, 'customServer': True}
I can reach any values by writing so:
import json
parsed = json.loads(source_code)
props = parsed['props']['pageProps']['videoData']
follower = props['authorStats']['followerCount']
heartcount = props['authorStats']['heartCount']
AuthorID = props['itemInfos']['authorId']
Commentcount = pandas.DataFrame(props['itemInfos']['commentCount']
CreateTime = props['itemInfos']['createTime']
But I was not able to put all of them into a dataframe with one row and all of those follower,heartcount etc as columns.
Thank you for your help!
This question already has answers here:
How can I convert JSON to CSV?
(26 answers)
Closed 3 years ago.
I am trying to write my JSON output to CSV, but I'm not sure how to separate my values into individual columns
This is my current code
with open('dict.csv', 'w') as csv_file:
writer = csv.writer(csv_file)
for key, value in response.json().items():
writer.writerow([value])
print(value)
This is the csv file I am getting:
current csv file
This is the desired csv file/output I want to get:
desired output
This is an example of my JSON Output
[{'id': '123', 'custom_id': '12', 'company': 28, 'company_name': 'Sunshine'}, {'id': '224', 'custom_id': '14', 'company': 38, 'company_name': 'Flowers'},
{'id': '888', 'custom_id': '10', 'company': 99, 'company_name': 'Fields'}]
how about this JSON format? (a more complicated one)
[{'id': '777', 'custom_id': '000112', 'company': 28, 'company_name':
'Weddings Inc', 'delivery_address': '25 olive park terrace, 61234', 'delivery_timeslot': {'lower': '2019-12-06T10:00:00Z', 'upper': '2019-12-06T13:00:00Z', 'bounds': '[)'}, 'sender_name': 'Joline', 'sender_email': '', 'sender_contact': '91234567', 'removed': None, 'recipient_name': 'Joline', 'recipient_contact': '91866655', 'notes': '', 'items': [{'id': 21668, 'name': 'Loose hair flowers', 'quantity': 1, 'metadata': {}, 'removed': None}, {'id': 21667, 'name': "Groom's Boutonniere", 'quantity': 1, 'metadata': {}, 'removed': None}, {'id': 21666, 'name': 'Bridal Bouquet', 'quantity': 1, 'metadata': {}, 'removed': None}], 'latitude': '1.1234550920764211111', 'longitude': '103.864352476201000000', 'created': '2019-08-15T05:40:30.385467Z', 'updated': '2019-08-15T05:41:27.930110Z', 'status': 'pending', 'verbose_status': 'Pending', 'logs': [{'id': 56363, 'order': '50c402', 'order_custom_id': '000112', 'order_delivery_address': '25 olive park terrace, 61234', 'order_delivery_timeslot': {'lower': '2019-12-06T10:00:00Z', 'upper': '2019-12-06T13:00:00Z', 'bounds': '[)'}, 'message': 'Order was created.', 'failure_reason': None, 'success_code': None, 'success_description': None, 'created': '2019-08-15T05:40:30.431790Z', 'removed': None}, {'id': 56364, 'order': '50c402d8-7c76-45b5-b883-e2fb887a507e', 'order_custom_id': 'INV-000112', 'order_delivery_address': '25 olive park terrace, 61234', 'order_delivery_timeslot': {'lower': '2019-12-06T10:00:00Z', 'upper': '2019-12-06T13:00:00Z', 'bounds': '[)'}, 'message': 'Order is pending.', 'failure_reason': None, 'success_code': None, 'success_description': None, 'created': '2019-08-15T05:40:30.433139Z', 'removed': None}], 'reschedule_requests': [], 'signature': None},
{'id': '241', 'custom_id': '000123', 'company': 22, 'company_name': 'Pearl Pte Ltd', 'delivery_address': '90 Merchant Road, Hotel Royal, 223344', 'delivery_timeslot': {'lower': '2019-11-29T10:00:00Z', 'upper': '2019-11-29T13:00:00Z', 'bounds': '[)'}, 'sender_name': 'Vera Smith', 'sender_email': '', 'sender_contact': '81234567', 'removed': None, 'recipient_name': 'Vera Smith', 'recipient_contact': '81234561', 'notes': '', 'items': [{'id': 22975, 'name': 'Custom wrapped bouquet', 'quantity': 2, 'metadata': {}, 'removed': None}, {'id': 22974, 'name': "Parents' boutonniere x 3", 'quantity': 1, 'metadata': {}, 'removed': None}, {'id': 22973, 'name': "Groom's boutonniere", 'quantity': 1, 'metadata': {}, 'removed': None}, {'id': 22972, 'name': 'Loose hair flowers', 'quantity': 1, 'metadata': {}, 'removed': None}, {'id': 22971, 'name': 'Bridal Bouquet', 'quantity': 1, 'metadata': {}, 'removed': None}], 'latitude': '1.28821802835873000000', 'longitude': '103.84569230314800000000', 'created': '2019-08-30T03:20:17.477528Z', 'updated': '2019-08-30T03:29:25.307856Z', 'status': 'pending', 'verbose_status': 'Pending', 'logs': [{'id': 59847, 'order': '24117085-9104-4442-841b-4a734f801d39', 'order_custom_id': 'INV-000123', 'order_delivery_address': '90 Merchant Road, Hotel Royal, 223344', 'order_delivery_timeslot': {'lower': '2019-11-29T10:00:00Z', 'upper': '2019-11-29T13:00:00Z', 'bounds': '[)'}, 'message': 'Order was created.', 'failure_reason': None, 'success_code': None, 'success_description': None, 'created': '2019-08-30T03:20:17.511250Z', 'removed': None}, {'id': 59848, 'order': '24117085-9104-4442-841b-4a734f801d39', 'order_custom_id': 'INV-000123', 'order_delivery_address': '90 Merchant Road, Hotel Royal, 223344', 'order_delivery_timeslot': {'lower': '2019-11-29T10:00:00Z', 'upper': '2019-11-29T13:00:00Z', 'bounds': '[)'}, 'message': 'Order is pending.', 'failure_reason': None, 'success_code': None, 'success_description': None, 'created': '2019-08-30T03:20:17.513132Z', 'removed': None}], 'reschedule_requests': [], 'signature': None}]
Use pandas library:
df.to_csv() - Write object to a comma-separated values (csv) file.
Ex.
import pandas as pd
data = [{'id': '123', 'custom_id': '12', 'company': 28, 'company_name': 'Sunshine'},
{'id': '224', 'custom_id': '14', 'company': 38, 'company_name': 'Flowers'},
{'id': '888', 'custom_id': '10', 'company': 99, 'company_name': 'Fields'}]
df = pd.DataFrame(data)
df.to_csv('sample.csv')
Try:
import csv
csv_file = 'my_file.csv'
csv_columns = ['id', 'custom_id', 'company', 'company_name']
dict_data = [{'id': '123', 'custom_id': '12', 'company': 28, 'company_name': 'Sunshine'}, {'id': '224', 'custom_id': '14', 'company': 38, 'company_name': 'Flowers'}, {'id': '888', 'custom_id': '10', 'company': 99, 'company_name': 'Fields'}]
try:
with open(csv_file, 'w') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=csv_columns)
writer.writeheader()
for data in dict_data:
writer.writerow(data)
except IOError:
print("I/O error")
Given your response data in json format
response = [{'id': '123', 'custom_id': '12', 'company': 28, 'company_name': 'Sunshine'},
{'id': '224', 'custom_id': '14', 'company': 38, 'company_name': 'Flowers'},
{'id': '888', 'custom_id': '10', 'company': 99, 'company_name': 'Fields'}]
You can convert it to a list of lists using
header = [response[0].keys()]
data = [row.values() for row in response]
csv_list = header + data
And then save it to csv using
with open('dict.csv', "w") as f:
for row in csv_list:
f.write("%s\n" % ','.join(str(col) for col in row))
This should yield your desired output