I'm trying to access a website via Python Requests. To avoid the iframe of the "Cookie Banner" I want to pass the cookie that handles the banner.
With Selenium I already managed to figure out which cookie that is and there it works fine with just passing the key/value" pair. I already found online, that it is necessary to "get" the page before passing and then refreshing it with "get" again after adding the cookies.
website = "https://www.myfitnesspal.com"
path = "path/to/your/chromedriver.exe"
service = Service(executable_path=path)
driver = webdriver.Chrome(service = service)
driver.get(website)
driver.add_cookie({'name': 'notice_preferences','value': '2:'})
driver.get(website)
So far so good. However, if I pass the same cookie that already worked in Selenium to a Python Request, the response.text that I receive still shows the content of the iframe and "Cookie Banner".
response = requests.get(website, cookies={"notice_preferences":"2:"})
Does anyone know why this is happening or if there is even a solution for this?
I don't think that page's content is blocked by cookie banner. It's rather blocked by the lack of a proper user-agent in header. The following code will return the page content as seen in browser:
import requests
import pandas as pd
s = requests.Session()
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
url = 'https://www.myfitnesspal.com'
s.headers.update(headers)
s.cookies.set("name", "notice_preferences", domain="www.myfitnesspal.com/")
s.cookies.set("value", "2:", domain="www.myfitnesspal.com/")
r = s.get(url)
# print(r.text)
print(s.cookies)
Result printed in terminal:
<RequestsCookieJar[<Cookie split-id=e28e4968-c2e3-4145-9226-0d9db15bcffe for www.myfitnesspal.com/>, <Cookie name=notice_preferences for www.myfitnesspal.com//>, <Cookie value=2: for www.myfitnesspal.com//>]>
You can then navigate to another page in that website, and requests' Session will preserve the headers and cookies. Also, print out the text response, see if the info you're looking for is there.
For more info on Requests, you can visit https://requests.readthedocs.io/en/latest/
EDIT: This is an a la carte XY problem - luckily the OP clarified it in comments.
That data is being pulled via an XHR call from an API endpoint. To get the info youi want, you need to scrape the endpoint. THis is how you do it (after inspecting Dev Tools - Network Tab and seeing that endpoint):
import requests
import pandas as pd
s = requests.Session()
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
url = 'https://www.myfitnesspal.com/api/nutrition?query=banane&page=1&offset=10'
r = s.get(url)
df = pd.DataFrame(r.json()['items'])
display(df)
This will print in terminal:
item tags type
0 {'country_code': 'US', 'deleted': False, 'description': 'Banane', 'id': 1873022840, 'nutritional_contents': {'calcium': 0.5, 'carbohydrates': 22.84, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 89}, 'fat': 0.33, 'fiber': 2.6, 'iron': 1.44444, 'monounsaturated_fat': 0.032, 'polyunsaturated_fat': 0.073, 'potassium': 358, 'protein': 1.09, 'saturated_fat': 0.112, 'sodium': 1, 'sugar': 12.23, 'trans_fat': 0, 'vitamin_a': 4.26667, 'vitamin_c': 14.5, 'vitamin_d': 0}, 'public': True, 'serving_sizes': [{'id': '67628178485117', 'index': 0, 'nutrition_multiplier': 1.18, 'unit': 'medium', 'value': 1}, {'id': '67078422671357', 'index': 1, 'nutrition_multiplier': 1.36, 'unit': 'large', 'value': 1}, {'id': '67628178485245', 'index': 2, 'nutrition_multiplier': 1.5, 'unit': 'cup, sliced', 'value': 1}, {'id': '67078414315389', 'index': 3, 'nutrition_multiplier': 2.25, 'unit': 'cup, mashed', 'value': 1}, {'id': '67628170129277', 'index': 4, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '67078414315517', 'index': 5, 'nutrition_multiplier': 0.283495, 'unit': 'oz', 'value': 1}, {'id': '67628170129405', 'index': 6, 'nutrition_multiplier': 0.81, 'unit': 'extra small', 'value': 1}, {'id': '67078422703997', 'index': 7, 'nutrition_multiplier': 1.52, 'unit': 'extra large', 'value': 1}, {'id': '67628178517885', 'index': 8, 'nutrition_multiplier': 4.53592, 'unit': 'lb(s)', 'value': 1}, {'id': '67078422704125', 'index': 9, 'nutrition_multiplier': 1e-05, 'unit': 'mg(s)', 'value': 1}, {'id': '67628178518013', 'index': 10, 'nutrition_multiplier': 10, 'unit': 'kg(s)', 'value': 1}, {'id': '67076304547197', 'index': 11, 'nutrition_multiplier': 0.00625, 'unit': 'mL, sliced ', 'value': 1}, {'id': '67626060361085', 'index': 12, 'nutrition_multiplier': 0.009375, 'unit': 'mL, mashed ', 'value': 1}, {'id': '67076304547325', 'index': 13, 'nutrition_multiplier': 6.25, 'unit': 'liter(s), sliced ', 'value': 1}, {'id': '67626060361213', 'index': 14, 'nutrition_multiplier': 9.375, 'unit': 'liter(s), mashed ', 'value': 1}], 'type': 'food', 'user_id': '133476501057389', 'verified': True, 'version': '199432263862133'} [canonical, best_match] food
1 {'brand_name': 'Banane', 'country_code': 'FR', 'deleted': False, 'description': 'Une banane', 'id': 2007191148, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 27, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 105}, 'fat': 0.4, 'fiber': 2.1, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 1.3, 'saturated_fat': 0, 'sodium': 0, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '93902759513197', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'fruit entier (120g)', 'value': 1}, {'id': '94452515327085', 'index': 1, 'nutrition_multiplier': 0.00833333, 'unit': 'gram', 'value': 1}], 'type': 'food', 'user_id': '160406080319149', 'verified': False, 'version': '198055450101605'} [] food
2 {'brand_name': 'Obst', 'country_code': 'DE', 'deleted': False, 'description': ' Banane ()', 'id': 1659839707, 'nutritional_contents': {'calcium': 0.625, 'carbohydrates': 22.84, 'energy': {'unit': 'calories', 'value': 90}, 'fat': 0.33, 'fiber': 2.6, 'iron': 1.857, 'potassium': 358, 'protein': 1.09, 'sodium': 1, 'sugar': 12.23, 'vitamin_a': 0.375, 'vitamin_c': 10.875}, 'public': True, 'serving_sizes': [{'id': '268297681372533', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '268297681372661', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '268847437186549', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}, {'id': '268297673016693', 'index': 3, 'nutrition_multiplier': 0.992232, 'unit': 'ounce', 'value': 3.5}], 'type': 'food', 'user_id': '163850601983789', 'verified': False, 'version': '129355447387317'} [] food
3 {'brand_name': 'Obst', 'country_code': 'DE', 'deleted': False, 'description': 'Banane 1 Stück', 'id': 1887842011, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 26.4, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 115}, 'fat': 0.2, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 1.2, 'saturated_fat': 0, 'sodium': 0, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '27521653231597', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 120}, {'id': '28071409045485', 'index': 1, 'nutrition_multiplier': 0.00833333, 'unit': 'g', 'value': 1}, {'id': '27521661620077', 'index': 2, 'nutrition_multiplier': 0.236246, 'unit': 'ounce', 'value': 1}], 'type': 'food', 'user_id': '234889390534445', 'verified': False, 'version': '53489009870261'} [] food
4 {'brand_name': 'Obst', 'country_code': 'DE', 'deleted': False, 'description': 'Banane', 'id': 227750309, 'nutritional_contents': {'calcium': 5, 'carbohydrates': 22.8, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 89}, 'fat': 0.3, 'fiber': 2.6, 'iron': 0.3, 'monounsaturated_fat': 0.1, 'polyunsaturated_fat': 0.1, 'potassium': 358, 'protein': 1.1, 'saturated_fat': 0.1, 'sodium': 1, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 64, 'vitamin_c': 9}, 'public': True, 'serving_sizes': [{'id': '88267487686061', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '88817243499949', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '88267496074541', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}], 'type': 'food', 'user_id': '134026256871405', 'verified': True, 'version': '230354056064301'} [] food
5 {'brand_name': 'Banane Ohne Schale', 'country_code': 'DE', 'deleted': False, 'description': 'Banane', 'id': 1889101676, 'nutritional_contents': {'calcium': 0.625, 'carbohydrates': 22.84, 'energy': {'unit': 'calories', 'value': 95}, 'fat': 0.33, 'fiber': 2.6, 'iron': 35.71, 'potassium': 358, 'protein': 1.09, 'sugar': 12.23, 'vitamin_a': 0.375, 'vitamin_c': 10.875}, 'public': True, 'serving_sizes': [{'id': '138151424970349', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '137601669156589', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '138151424970477', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}], 'type': 'food', 'user_id': '278614430748141', 'verified': False, 'version': '53074667210277'} [] food
6 {'brand_name': 'Banane', 'country_code': 'FR', 'deleted': False, 'description': 'Banane Gebacken', 'id': 1349524295, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 25, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 157}, 'fat': 4, 'fiber': 0, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 4, 'saturated_fat': 0, 'sodium': 1, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '59106944525429', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '58557188711669', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '59106944525557', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}, {'id': '58557197100149', 'index': 3, 'nutrition_multiplier': 0.992232, 'unit': 'ounce', 'value': 3.5}], 'type': 'food', 'user_id': '133324127170493', 'verified': False, 'version': '31634077001709'} [] food
7 {'brand_name': 'Banane', 'country_code': 'FR', 'deleted': False, 'description': 'Demi banane', 'id': 139561661, 'nutritional_contents': {'calcium': 0, 'carbohydrates': 14, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 93}, 'fat': 2, 'fiber': 0, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 0, 'protein': 3, 'saturated_fat': 0, 'sodium': 0, 'sugar': 0, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '124397058706493', 'index': 1, 'nutrition_multiplier': 1, 'unit': 'yaourt', 'value': 1}], 'type': 'food', 'user_id': '133476501057517', 'verified': False, 'version': '63530949537133'} [] food
8 {'brand_name': 'Banane', 'country_code': 'CA', 'deleted': False, 'description': 'Banane (Santé Canada)', 'id': 1568891032, 'nutritional_contents': {'calcium': 1, 'carbohydrates': 27, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 105}, 'fat': 0, 'fiber': 3, 'iron': 2, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0, 'potassium': 487, 'protein': 1, 'saturated_fat': 0, 'sodium': 1, 'sugar': 14, 'trans_fat': 0, 'vitamin_a': 2, 'vitamin_c': 17}, 'public': True, 'serving_sizes': [{'id': '27092291822629', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'banana 118g', 'value': 1}, {'id': '26542536008869', 'index': 1, 'nutrition_multiplier': 1, 'unit': 'med bananna', 'value': 1}], 'type': 'food', 'user_id': '199722294733869', 'verified': False, 'version': '264028726224173'} [] food
9 {'brand_name': 'Banane', 'country_code': 'CA', 'deleted': False, 'description': ' Une banane moyenne', 'id': 1484522768, 'nutritional_contents': {'calcium': 1.53, 'carbohydrates': 22.8, 'cholesterol': 0, 'energy': {'unit': 'calories', 'value': 89}, 'fat': 0.33, 'fiber': 2, 'iron': 0, 'monounsaturated_fat': 0, 'polyunsaturated_fat': 0.07, 'potassium': 0, 'protein': 1.1, 'saturated_fat': 0.11, 'sodium': 8, 'sugar': 12, 'trans_fat': 0, 'vitamin_a': 0, 'vitamin_c': 0}, 'public': True, 'serving_sizes': [{'id': '63099251926181', 'index': 0, 'nutrition_multiplier': 1, 'unit': 'g', 'value': 100}, {'id': '63649007740069', 'index': 1, 'nutrition_multiplier': 0.01, 'unit': 'g', 'value': 1}, {'id': '63099260314661', 'index': 2, 'nutrition_multiplier': 0.283495, 'unit': 'ounce', 'value': 1}, {'id': '63649016128549', 'index': 3, 'nutrition_multiplier': 0.992232, 'unit': 'ounce', 'value': 3.5}], 'type': 'food', 'user_id': '128659968929645', 'verified': False, 'version': '136640012748413'} [] food
You can drill down further into that json object, (normalize it, etc) to get data in different shapes and forms.
I am trying to append a dictionary to my already existing key "processed_data" where data is saved in the list of dictionaries. I tried several methods as shown in already asked questions but they did not work. This is my schema.
{'_id': ObjectId('5fe46a5b7468e3498124fcbe'), 'metadata': {'_id': ObjectId('5fe4500c7b2c03decd86334f'), 'type': 'VIDEO', 'id': 'o6st4ces9Wg"},"qoeUrl":{"baseUrl":"https://s.youtube.com/api/stats/qoe?cl=348521801', 'user_id': 'fc3240b2d7ef9d33bbb04fd7203e35ea9da54ffb', 'name': 'City Ak47', 'thumbnail': 'https://i.ytimg.com/vi/o6st4ces9Wg/hqdefault.jpg', 'title': 'Alex Bhatti ki Video Viral Ho Gie | How To Become Tiktok Star | City AK47 - YouTube', 'publication_date': 'Sep 17, 2020', 'channel_id': 'UCuo6tBl2MfkWvMPyCqph2LA', 'channel_name': 'City Ak47', 'scrape_date': '2020-12-24 08:23:17.390018', 'regions_allowed': 'AD,AE,AF,AG,AI,AL,AM,AO,AQ,AR,AS,AT,AU,AW,AX,AZ,BA,BB,BD,BE,BF,BG,BH,BI,BJ,BL,BM,BN,BO,BQ,BR,BS,BT,BV,BW,BY,BZ,CA,CC,CD,CF,CG,CH,CI,CK,CL,CM,CN,CO,CR,CU,CV,CW,CX,CY,CZ,DE,DJ,DK,DM,DO,DZ,EC,EE,EG,EH,ER,ES,ET,FI,FJ,FK,FM,FO,FR,GA,GB,GD,GE,GF,GG,GH,GI,GL,GM,GN,GP,GQ,GR,GS,GT,GU,GW,GY,HK,HM,HN,HR,HT,HU,ID,IE,IL,IM,IN,IO,IQ,IR,IS,IT,JE,JM,JO,JP,KE,KG,KH,KI,KM,KN,KP,KR,KW,KY,KZ,LA,LB,LC,LI,LK,LR,LS,LT,LU,LV,LY,MA,MC,MD,ME,MF,MG,MH,MK,ML,MM,MN,MO,MP,MQ,MR,MS,MT,MU,MV,MW,MX,MY,MZ,NA,NC,NE,NF,NG,NI,NL,NO,NP,NR,NU,NZ,OM,PA,PE,PF,PG,PH,PK,PL,PM,PN,PR,PS,PT,PW,PY,QA,RE,RO,RS,RU,RW,SA,SB,SC,SD,SE,SG,SH,SI,SJ,SK,SL,SM,SN,SO,SR,SS,ST,SV,SX,SY,SZ,TC,TD,TF,TG,TH,TJ,TK,TL,TM,TN,TO,TR,TT,TV,TW,TZ,UA,UG,UM,US,UY,UZ,VA,VC,VE,VG,VI,VN,VU,WF,WS,YE,YT,ZA,ZM,ZW', 'views': '663962', 'is_family_friendly': 'true', 'category': 'Entertainment', 'tags': ['AmirFilms', 'Alex Bhatti ki Video Viral Ho Gie | How To Become Tiktok Star | City AK47', 'Tiktok star', 'Tiktok', 'Alex tiktokr', 'Alex bhatti tiktok star', 'Alex bhatti', 'Ayesha bukhari', 'Viral video', 'New video', 'Leak vidro', 'Ayesha leak video', 'Alex bhatti leak video', 'News', 'Tiktik funny video'], 'language': 'en-US', 'width': '480', 'height': '360', 'job_id': '539f61c4183c46448a75cfb65dc40926'}, 'results': {'unique_word_freq': [{'text': 'hai', 'value': 6}, {'text': 'famous', 'value': 4}, {'text': 'allah', 'value': 3}, {'text': 'kar', 'value': 3}, {'text': 'gy', 'value': 3}, {'text': 'ye', 'value': 3}, {'text': 'yeh', 'value': 2}, {'text': 'ka', 'value': 2}, {'text': 'video', 'value': 2}, {'text': 'asee', 'value': 2}, {'text': 'nhi', 'value': 2}, {'text': 'ho', 'value': 2}, {'text': 'tum', 'value': 2}, {'text': 'jao', 'value': 2}, {'text': 'kitna', 'value': 1}, {'text': 'budsoor', 'value': 1}, {'text': 'gundgi', 'value': 1}, {'text': 'dher', 'value': 1}, {'text': 'khusra', 'value': 1}, {'text': 'tiktok', 'value': 1}, {'text': 'kunjuro', 'value': 1}, {'text': 'zanano', 'value': 1}, {'text': 'kaam', 'value': 1}, {'text': 'usko', 'value': 1}, {'text': 'hadyat', 'value': 1}, {'text': 'de', 'value': 1}, {'text': 'ameen', 'value': 1}, {'text': '😔', 'value': 1}, {'text': 'kahn', 'value': 1}, {'text': 'puri', 'value': 1}, {'text': 'kotta', 'value': 1}, {'text': 'ٹک', 'value': 1}, {'text': 'ٹاک', 'value': 1}, {'text': 'ایپ', 'value': 1}, {'text': 'پر', 'value': 1}, {'text': 'پاکستان', 'value': 1}, {'text': 'میں', 'value': 1}, {'text': 'مکمل', 'value': 1}, {'text': 'پابندی', 'value': 1}, {'text': 'لگنی', 'value': 1}, {'text': 'چاہیے', 'value': 1}, {'text': 'leaked', 'value': 1}, {'text': 'purpose', 'value': 1}, {'text': 'fame', 'value': 1}, {'text': 'views', 'value': 1}, {'text': 'mean', 'value': 1}, {'text': 'people', 'value': 1}, {'text': 'like', 'value': 1}, {'text': 'kinda', 'value': 1}, {'text': 'cheap', 'value': 1}, {'text': 'acts', 'value': 1}, {'text': 'inki', 'value': 1}, {'text': 'maa', 'value': 1}, {'text': 'bhano', 'value': 1}, {'text': 'sath', 'value': 1}, {'text': 'bhi', 'value': 1}, {'text': 'hoo', 'value': 1}, {'text': 'pak', 'value': 1}, {'text': 'ko', 'value': 1}, {'text': 'bohot', 'value': 1}, {'text': 'bari', 'value': 1}, {'text': 'sazaa', 'value': 1}, {'text': 'dee', 'value': 1}, {'text': 'duniyan', 'value': 1}, {'text': 'hee', 'value': 1}, {'text': 'dikhaee', 'value': 1}, {'text': 'pata', 'value': 1}, {'text': 'khha', 'value': 1}, {'text': 'jay', 'value': 1}, {'text': 'kiyamat', 'value': 1}, {'text': 'din', 'value': 1}, {'text': 'logo', 'value': 1}, {'text': 'hisab', 'value': 1}, {'text': 'lena', 'value': 1}, {'text': 'log', 'value': 1}, {'text': 'sidah', 'value': 1}, {'text': 'janat', 'value': 1}, {'text': 'chaly', 'value': 1}, {'text': 'baaz', 'value': 1}, {'text': 'ap', 'value': 1}, {'text': 'bakwas', 'value': 1}, {'text': 'band', 'value': 1}, {'text': 'kareen', 'value': 1}, {'text': 'larka', 'value': 1}, {'text': 'bharva', 'value': 1}, {'text': 'bs', 'value': 1}, {'text': 'pakar', 'value': 1}, {'text': 'gal', 'value': 1}, {'text': 'ma', 'value': 1}, {'text': 'dala', 'value': 1}, {'text': 'gaya', 'value': 1}, {'text': 'bahut', 'value': 1}, {'text': 'ghatiya', 'value': 1}, {'text': 'insan', 'value': 1}, {'text': 'tu', 'value': 1}, {'text': 'chakka', 'value': 1}, {'text': 'alex', 'value': 1}, {'text': 'bhatti', 'value': 1}], 'polarity_freq': [{'date': '2020-12-03', 'total': 4, 'positive': 3, 'negative': 1}, {'date': '2020-12-10', 'total': 9, 'positive': 8, 'negative': 1}, {'date': '2020-12-17', 'total': 2, 'positive': 2, 'negative': 0}, {'date': '2020-12-21', 'total': 1, 'positive': 1, 'negative': 0}, {'date': '2020-12-22', 'total': 2, 'positive': 1, 'negative': 1}], 'polarity_dist': [{'name': 'positive', 'value': '15'}, {'name': 'negative', 'value': '3'}], 'assoc': []}, 'processed_data': [{'index': 0, '_id': ObjectId('5fe4500c7b2c03decd863350'), 'channel_id': '/channel/UCg7rf8yXy8wqVxlbnErgdyg', 'clean_text': 'kitna budsoor hai yeh gundgi ka dher khusra', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniDMBIClPo0sPLX5RDOLPHTJhECMOub-fC0ZTVY6Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-22 08:23:17', 'id': 'UgxzkGuC2JpeaZD7El14AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 days ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Kitna budsoor hai yeh gundgi ka dher khusra.', 'tokens': ['kitna', 'budsoor', 'hai', 'yeh', 'gundgi', 'ka', 'dher', 'khusra'], 'tokens_no_swords': ['kitna', 'budsoor', 'hai', 'yeh', 'gundgi', 'ka', 'dher', 'khusra'], 'tran_text': 'kitna budsoor hai yeh gundgi ka dher khusra .', 'type': 'COMMENT', 'user_id': 'f7961259b974ba9fae934410fca2e939d3493038', 'user_name': 'jimmi khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '1', 'date': '2020-12-22'}, {'index': 1, '_id': ObjectId('5fe4500c7b2c03decd863351'), 'channel_id': '/channel/UCg7rf8yXy8wqVxlbnErgdyg', 'clean_text': 'tiktok kunjuro zanano ka kaam hai', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniDMBIClPo0sPLX5RDOLPHTJhECMOub-fC0ZTVY6Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-22 08:23:17', 'id': 'UgwntMkhi7J2l2N3MZJ4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 days ago (edited)', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Tiktok kunjuro r zanano ka kaam hai.', 'tokens': ['tiktok', 'kunjuro', 'zanano', 'ka', 'kaam', 'hai'], 'tokens_no_swords': ['tiktok', 'kunjuro', 'zanano', 'ka', 'kaam', 'hai'], 'tran_text': 'tiktok kunjuro r zanano ka kaam hai .', 'type': 'COMMENT', 'user_id': 'f7961259b974ba9fae934410fca2e939d3493038', 'user_name': 'jimmi khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-22'}, {'index': 2, '_id': ObjectId('5fe4500c7b2c03decd863352'), 'channel_id': '/channel/UCMDNByou1B62upgmnv-UQMw', 'clean_text': 'allah usko hadyat de ameen 😔', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnik2uW0mzYoagKEYX1_kGY3HDhYd3Ni6UlOxSEHOA=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-21 08:23:17', 'id': 'UgxZrbzomoOLyGEGAjp4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 days ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Allah usko hadyat de ameen 😔', 'tokens': ['allah', 'usko', 'hadyat', 'de', 'ameen', '😔'], 'tokens_no_swords': ['allah', 'usko', 'hadyat', 'de', 'ameen', '😔'], 'tran_text': 'allah usko hadyat de ameen 😔', 'type': 'COMMENT', 'user_id': 'da9fe12c7945488a70f56355f8c122d2f35231c5', 'user_name': 'neha Rajput', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-21'}, {'index': 3, '_id': ObjectId('5fe4500c7b2c03decd863353'), 'channel_id': '/channel/UCkl4U918shu8CroBno8-aJg', 'clean_text': '', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniom0S4ta4uSnNx7yD69NfR4TmOqXPpYxv6_Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-17 08:23:17', 'id': 'UgzAfvcluRdyX9yi-JJ4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '1 week ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': '420', 'tokens': [], 'tokens_no_swords': [], 'tran_text': '420', 'type': 'COMMENT', 'user_id': '04b4dd4534a4acf47ba876387d752eda8d3087f6', 'user_name': 'Shahid Khankarachi', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-17'}, {'index': 5, '_id': ObjectId('5fe4500c7b2c03decd863355'), 'channel_id': '/channel/UCoL0h9EyBTNSvKIWIxl6WIg', 'clean_text': 'kahn hai yeh puri video', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnjNJAzxyS9mOk-R7TF5ICxa0_EQbtgcL3z2Yg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-17 08:23:17', 'id': 'Ugxx-JvihbK7P8Y8u5x4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '1 week ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Kahn hai yeh puri video', 'tokens': ['kahn', 'hai', 'yeh', 'puri', 'video'], 'tokens_no_swords': ['kahn', 'hai', 'yeh', 'puri', 'video'], 'tran_text': 'kahn hai yeh puri video', 'type': 'COMMENT', 'user_id': '3bc410f7e5133b61e2f2cc790ce6ae2692397778', 'user_name': 'ALISHA ZOYA', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-17'}, {'index': 6, '_id': ObjectId('5fe4500c7b2c03decd863356'), 'channel_id': '/channel/UCsd6TX3yWpNYK55hawyi8qw', 'clean_text': 'kotta', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnj19uWVIJ75wx27KLjDGDcsVcGtzVtp8SRQ0w=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgyhHfBJDWHtL73E71N4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Kotta', 'tokens': ['kotta'], 'tokens_no_swords': ['kotta'], 'tran_text': 'kotta', 'type': 'COMMENT', 'user_id': 'e83b422e66c1bd722306aee6715c3846c32e506b', 'user_name': 'Shakeel Khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 7, '_id': ObjectId('5fe4500c7b2c03decd863357'), 'channel_id': '/channel/UCX6LjA5LbC7xMO19yyM7m0Q', 'clean_text': 'ٹک ٹاک ایپ پر پاکستان میں مکمل پابندی لگنی چاہیے', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwni4WpKnrXzHmw2VwT0z5aYnM0T5IhRN0DG3Pmsg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgwFILpwDAQKYA9ioMV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'en', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'ٹک ٹاک ایپ پر پاکستان میں مکمل پابندی لگنی چاہیے', 'tokens': ['ٹک', 'ٹاک', 'ایپ', 'پر', 'پاکستان', 'میں', 'مکمل', 'پابندی', 'لگنی', 'چاہیے'], 'tokens_no_swords': ['ٹک', 'ٹاک', 'ایپ', 'پر', 'پاکستان', 'میں', 'مکمل', 'پابندی', 'لگنی', 'چاہیے'], 'tran_text': 'ٹک ٹاک ایپ پر پاکستان میں مکمل پابندی لگنی چاہیے', 'type': 'COMMENT', 'user_id': '9ca6083ff6234bd94fc218ef27d12c8b91c2fa33', 'user_name': 'Wahab Mirza', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 8, '_id': ObjectId('5fe4500c7b2c03decd863358'), 'channel_id': '/channel/UCKMvpfSppW24ixWCOmJDu_g', 'clean_text': 'he leaked this video on purpose to get fame and views i mean people like them do these kinda cheap acts to get famous', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniL8sePcWPsqDg6AOaLsW4nf14XDW3132kC0Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'Ugz7IdhD6s8zCJ6vHNt4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'en', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'He leaked this video on purpose to get fame and views...I mean people like them do these kinda cheap acts to get famous.', 'tokens': ['he', 'leaked', 'this', 'video', 'on', 'purpose', 'to', 'get', 'fame', 'and', 'views', 'i', 'mean', 'people', 'like', 'them', 'do', 'these', 'kinda', 'cheap', 'acts', 'to', 'get', 'famous'], 'tokens_no_swords': ['leaked', 'video', 'purpose', 'fame', 'views', 'mean', 'people', 'like', 'kinda', 'cheap', 'acts', 'famous'], 'tran_text': 'he leaked this video on purpose to get fame and views ... i mean people like them do these kinda cheap acts to get famous .', 'type': 'COMMENT', 'user_id': '3f75585892685df3ae4b3d733d9795a719b2d528', 'user_name': 'Ana T', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 9, '_id': ObjectId('5fe4500c7b2c03decd863359'), 'channel_id': '/channel/UCSUGRfHKn5qCNN4TKG3MAkw', 'clean_text': '', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwng6eAHeRd7CcM8mmkCHCA8VI2tqmMNPb1q1MA=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgwMvbYR29V4ISyic_d4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'en', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': '03065455318', 'tokens': [], 'tokens_no_swords': [], 'tran_text': '03065455318', 'type': 'COMMENT', 'user_id': '23ac482ba9b36182915c502c13d4cd45b7f7bf1f', 'user_name': 'Ali Rizwan', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 10, '_id': ObjectId('5fe4500c7b2c03decd86335a'), 'channel_id': '/channel/UC-fWQ2vkmngdVliZDkRSiiQ', 'clean_text': 'inki maa bhano sath bhi asee hoo', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwngl0V2Zy_AUGUyIZpMbrBDxqL6pq5AcdF4hNg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgzEMbiRNk2ywQHlgxR4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Inki maa bhano k sath bhi Asee hoo', 'tokens': ['inki', 'maa', 'bhano', 'sath', 'bhi', 'asee', 'hoo'], 'tokens_no_swords': ['inki', 'maa', 'bhano', 'sath', 'bhi', 'asee', 'hoo'], 'tran_text': 'inki maa bhano k sath bhi asee hoo', 'type': 'COMMENT', 'user_id': '2c7e490aa5d0ceca9340c92c3577fa75d3e5a8d3', 'user_name': 'M wali Yousuf', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 11, '_id': ObjectId('5fe4500c7b2c03decd86335b'), 'channel_id': '/channel/UC-fWQ2vkmngdVliZDkRSiiQ', 'clean_text': 'allah pak asee ko bohot bari sazaa dee or duniyan me hee dikhaee', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwngl0V2Zy_AUGUyIZpMbrBDxqL6pq5AcdF4hNg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'Ugyo9YlNa7zuVsSQlZh4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Allah Pak Asee ko Bohot bari Sazaa Dee Or Duniyan me hee dikhaee', 'tokens': ['allah', 'pak', 'asee', 'ko', 'bohot', 'bari', 'sazaa', 'dee', 'or', 'duniyan', 'me', 'hee', 'dikhaee'], 'tokens_no_swords': ['allah', 'pak', 'asee', 'ko', 'bohot', 'bari', 'sazaa', 'dee', 'duniyan', 'hee', 'dikhaee'], 'tran_text': 'allah pak asee ko bohot bari sazaa dee or duniyan me hee dikhaee', 'type': 'COMMENT', 'user_id': '2c7e490aa5d0ceca9340c92c3577fa75d3e5a8d3', 'user_name': 'M wali Yousuf', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 12, '_id': ObjectId('5fe4500c7b2c03decd86335c'), 'channel_id': '/channel/UCphcNEEoxrp08DARCX7dSNQ', 'clean_text': 'pata nhi famous ho kar khha jay gy kiyamat din allah famous logo hisab lena hai ye nhi tum log famous ho gy or sidah janat chaly jao gy baaz a jao', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnibNbjcM0UMLW2aTnOD3jfJXlaq2Iq5_hMg3Q-O=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgzfXCaZHDMSCKPqlKB4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': '1', 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Pata nhi famous ho Kar khha jay gy. Kiyamat k din Allah n famous logo c b hisab lena hai ye nhi k tum log famous ho gy or sidah janat m chaly jao gy. Baaz a jao', 'tokens': ['pata', 'nhi', 'famous', 'ho', 'kar', 'khha', 'jay', 'gy', 'kiyamat', 'din', 'allah', 'famous', 'logo', 'hisab', 'lena', 'hai', 'ye', 'nhi', 'tum', 'log', 'famous', 'ho', 'gy', 'or', 'sidah', 'janat', 'chaly', 'jao', 'gy', 'baaz', 'a', 'jao'], 'tokens_no_swords': ['pata', 'nhi', 'famous', 'ho', 'kar', 'khha', 'jay', 'gy', 'kiyamat', 'din', 'allah', 'famous', 'logo', 'hisab', 'lena', 'hai', 'ye', 'nhi', 'tum', 'log', 'famous', 'ho', 'gy', 'sidah', 'janat', 'chaly', 'jao', 'gy', 'baaz', 'jao'], 'tran_text': 'pata nhi famous ho kar khha jay gy . kiyamat k din allah n famous logo c b hisab lena hai ye nhi k tum log famous ho gy or sidah janat m chaly jao gy . baaz a jao', 'type': 'COMMENT', 'user_id': 'da5c845fbd0a39db29a99a9d620bd8c266956065', 'user_name': 'Rida Khan', 'video_id': 'o6st4ces9Wg', 'is_hate': '1', 'date': '2020-12-10'}, {'index': 13, '_id': ObjectId('5fe4500c7b2c03decd86335d'), 'channel_id': '/channel/UCnVVsV2fd3P0lS9QClU5DCA', 'clean_text': 'ap bakwas band kareen', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniJxcGaZeKzmvDSUGeX5vFZo3m_ZXQ_yC7-Kw=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgwzEHvn_WWTbPOzl7x4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Ap bakwas band kareen', 'tokens': ['ap', 'bakwas', 'band', 'kareen'], 'tokens_no_swords': ['ap', 'bakwas', 'band', 'kareen'], 'tran_text': 'ap bakwas band kareen', 'type': 'COMMENT', 'user_id': 'e51ff2c592a9ad2fe8f6f373c6a2dab117f2c2e9', 'user_name': 'ahmad muaaz', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 14, '_id': ObjectId('5fe4500c7b2c03decd86335e'), 'channel_id': '/channel/UCtbUvUvL0qrREfEzUvJUQKQ', 'clean_text': 'ye larka bharva', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnjKBfmUKuMzFCtwM-KuKAfq_5y0RA7iez5w9Q=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-10 08:23:17', 'id': 'UgzvW63udc6CEwnsb_J4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '2 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'ye larka bharva', 'tokens': ['ye', 'larka', 'bharva'], 'tokens_no_swords': ['ye', 'larka', 'bharva'], 'tran_text': 'ye larka bharva', 'type': 'COMMENT', 'user_id': '7d7c7165c38c05a96c335421faf6ca3eb9eb1722', 'user_name': 'Rana Waqas', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-10'}, {'index': 15, '_id': ObjectId('5fe4500c7b2c03decd86335f'), 'channel_id': '/channel/UCieduNjSrF2DPawdZh_HesQ', 'clean_text': 'bs kar do tum', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwniOEn-mQTkzQu5ybCc6gjFqSlK8eQF-4RsB6w=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'Ugx9gyNwYeVV5DDKsJV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'bs kar do tum', 'tokens': ['bs', 'kar', 'do', 'tum'], 'tokens_no_swords': ['bs', 'kar', 'tum'], 'tran_text': 'bs kar do tum', 'type': 'COMMENT', 'user_id': '7339de48d380c9efe854dc9b6660a8fe22c28448', 'user_name': 'sami ali ali', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}, {'index': 16, '_id': ObjectId('5fe4500c7b2c03decd863360'), 'channel_id': '/channel/UCaPZsZzHcOMiDgZ3rkALMFg', 'clean_text': 'is pakar kar gal ma dala gaya', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnhv4vXDX16Pi0veGMZVUtqiYiYq_XOUp2yTvQ=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'UgzC4DaDahEFvjUsJiN4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Is pakar kar gal ma dala gaya', 'tokens': ['is', 'pakar', 'kar', 'gal', 'ma', 'dala', 'gaya'], 'tokens_no_swords': ['pakar', 'kar', 'gal', 'ma', 'dala', 'gaya'], 'tran_text': 'is pakar kar gal ma dala gaya', 'type': 'COMMENT', 'user_id': 'ae974ed633daab66164b5dcee9340e2ed0b1c455', 'user_name': 'munir gill', 'video_id': 'o6st4ces9Wg', 'is_hate': '1', 'date': '2020-12-03'}, {'index': 17, '_id': ObjectId('5fe4500c7b2c03decd863361'), 'channel_id': '/channel/UCqGRwUGDEBY98v0PcA9BpUQ', 'clean_text': 'bahut ghatiya insan hai', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnhMts6KGq4VtnvbDuVVatNlFduO6jmHbIRX6A=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'UgyyCAdkym_IDptSbNZ4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Bahut ghatiya insan hai', 'tokens': ['bahut', 'ghatiya', 'insan', 'hai'], 'tokens_no_swords': ['bahut', 'ghatiya', 'insan', 'hai'], 'tran_text': 'bahut ghatiya insan hai', 'type': 'COMMENT', 'user_id': '331593ec91edb449953d775229e7a91727415976', 'user_name': 'Asif Bhatti', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}, {'index': 18, '_id': ObjectId('5fe4500c7b2c03decd863362'), 'channel_id': '/channel/UCBZ0mLPPioFWW1i-kvmZBnA', 'clean_text': 'ye tu chakka hai alex bhatti', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnggrksT4HvfysI9VkzPzsKIXkcJsPfmWvvNyg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'Ugwh57O9lzDJgzCvKJV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Ye tu chakka hai alex bhatti', 'tokens': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tokens_no_swords': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tran_text': 'ye tu chakka hai alex bhatti', 'type': 'COMMENT', 'user_id': 'ee99e20e5128b5fc14c1972d55625585cf4d0237', 'user_name': 'Khizar Rao', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}]}
I want to append the following dictionary against the "processed_data" key.
{'index': 19, '_id': ObjectId('5fe4500c7b2c03decd863362'), 'channel_id': '/channel/UCBZ0mLPPioFWW1i-kvmZBnA', 'clean_text': 'ye tu chakka hai alex bhatti', 'comment_user_image': 'https://yt3.ggpht.com/ytc/AAUvwnggrksT4HvfysI9VkzPzsKIXkcJsPfmWvvNyg=s48-c-k-c0xffffffff-no-rj-mo', 'datetime': '2020-12-03 08:23:17', 'id': 'Ugwh57O9lzDJgzCvKJV4AaABAg', 'job_id': '539f61c4183c46448a75cfb65dc40926', 'lang': 'ro-ur', 'likes': 0, 'orig_lang': 'unknown', 'published_time_display': '3 weeks ago', 'replies': None, 'reply_to': None, 'scrape_date': '2020-12-24 08:23:17.955821', 'text': 'Ye tu chakka hai alex bhatti', 'tokens': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tokens_no_swords': ['ye', 'tu', 'chakka', 'hai', 'alex', 'bhatti'], 'tran_text': 'ye tu chakka hai alex bhatti', 'type': 'COMMENT', 'user_id': 'ee99e20e5128b5fc14c1972d55625585cf4d0237', 'user_name': 'Khizar Rao', 'video_id': 'o6st4ces9Wg', 'is_hate': '0', 'date': '2020-12-03'}
Thanks!
Use update_one() with $push:
from pymongo import MongoClient
from bson import ObjectId
db = MongoClient()['mydatabase']
db.mycollection.insert_one({'_id': ObjectId('5fe46a5b7468e3498124fcbe'),
'processed_data': []})
update = {'index': 19, '_id': ObjectId('5fe4500c7b2c03decd863362'), 'channel_id': 'etc.'}
db.mycollection.update_one({'_id': ObjectId('5fe46a5b7468e3498124fcbe')}, {'$push': {'processed_data': update}})
I'm trying to make an algorithmic trading program, and
open_positions = trader.open_positions
for position in open_positions:
print(position)
outputs two dictionaries (?)
{'t': 1, 'ratePrecision': 5, 'tradeId': '32572646', 'accountName': '05654022', 'accountId': '5654022', 'roll': 0, 'com': 0, 'open': 0.71538, 'valueDate': '', 'grossPL': 1298.32056, 'close': 0.71631, 'visiblePL': 9.3, 'isDisabled': False, 'currency': 'AUD/USD', 'isBuy': True, 'amountK': 1000, 'currencyPoint': 139.59559, 'time': '10022020065344', 'usedMargin': 2500, 'OpenOrderRequestTXT': 'FXTC', 'stop': 0, 'stopMove': 0, 'limit': 0}
{'t': 1, 'ratePrecision': 0, 'tradeId': '', 'accountName': '', 'accountId': '', 'roll': 0, 'com': 0, 'open': 0, 'valueDate': '', 'grossPL': 1298.32056, 'close': 0, 'visiblePL': 9.3, 'isDisabled': False, 'currency': 'AUD/USD', 'isBuy': False, 'amountK': 1000, 'currencyPoint': 0, 'time': None, 'usedMargin': 0, 'stop': 0, 'stopMove': 0, 'limit': 0, 'isTotal': True}
The thing is, when I put the above block of code in my actual program, it falls into a weird loop and never comes out. So, I'm trying to find a way to isolate the real dictionary (the first one) from the list without using a loop. Any help would be greatly appreciated. Thank you very much!
*the list
[{'t': 1, 'ratePrecision': 5, 'tradeId': '32572646', 'accountName': '05654022', 'accountId': '5654022', 'roll': 0, 'com': 0, 'open': 0.71538, 'valueDate': '', 'grossPL': 433.14843, 'close': 0.71569, 'visiblePL': 3.1, 'isDisabled': False, 'currency': 'AUD/USD', 'isBuy': True, 'amountK': 1000, 'currencyPoint': 139.71652, 'time': '10022020065344', 'usedMargin': 2500, 'OpenOrderRequestTXT': 'FXTC', 'stop': 0, 'stopMove': 0, 'limit': 0}, {'t': 1, 'ratePrecision': 0, 'tradeId': '', 'accountName': '', 'accountId': '', 'roll': 0, 'com': 0, 'open': 0, 'valueDate': '', 'grossPL': 433.14843, 'close': 0, 'visiblePL': 3.1, 'isDisabled': False, 'currency': 'AUD/USD', 'isBuy': False, 'amountK': 1000, 'currencyPoint': 0, 'time': None, 'usedMargin': 0, 'stop': 0, 'stopMove': 0, 'limit': 0, 'isTotal': True}]
You can do something like this
dict1, dict2 = [{'t': 1, 'ratePrecision': 5, 'tradeId': '32572646', 'accountName': '05654022', 'accountId': '5654022', 'roll': 0, 'com': 0, 'open': 0.71538, 'valueDate': '', 'grossPL': 433.14843, 'close': 0.71569, 'visiblePL': 3.1, 'isDisabled': False, 'currency': 'AUD/USD', 'isBuy': True, 'amountK': 1000, 'currencyPoint': 139.71652, 'time': '10022020065344', 'usedMargin': 2500, 'OpenOrderRequestTXT': 'FXTC', 'stop': 0, 'stopMove': 0, 'limit': 0}, {'t': 1, 'ratePrecision': 0, 'tradeId': '', 'accountName': '', 'accountId': '', 'roll': 0, 'com': 0, 'open': 0, 'valueDate': '', 'grossPL': 433.14843, 'close': 0, 'visiblePL': 3.1, 'isDisabled': False, 'currency': 'AUD/USD', 'isBuy': False, 'amountK': 1000, 'currencyPoint': 0, 'time': None, 'usedMargin': 0, 'stop': 0, 'stopMove': 0, 'limit': 0, 'isTotal': True}]
dict1 would then give you:
{'t': 1,
'ratePrecision': 5,
'tradeId': '32572646',
'accountName': '05654022',
'accountId': '5654022',
'roll': 0,
'com': 0,
'open': 0.71538,
'valueDate': '',
'grossPL': 433.14843,
'close': 0.71569,
'visiblePL': 3.1,
'isDisabled': False,
'currency': 'AUD/USD',
'isBuy': True,
'amountK': 1000,
'currencyPoint': 139.71652,
'time': '10022020065344',
'usedMargin': 2500,
'OpenOrderRequestTXT': 'FXTC',
'stop': 0,
'stopMove': 0,
'limit': 0}
OR you can do something like this:
L = [{'t': 1, 'ratePrecision': 5, 'tradeId': '32572646', 'accountName': '05654022', 'accountId': '5654022', 'roll': 0, 'com': 0, 'open': 0.71538, 'valueDate': '', 'grossPL': 433.14843, 'close': 0.71569, 'visiblePL': 3.1, 'isDisabled': False, 'currency': 'AUD/USD', 'isBuy': True, 'amountK': 1000, 'currencyPoint': 139.71652, 'time': '10022020065344', 'usedMargin': 2500, 'OpenOrderRequestTXT': 'FXTC', 'stop': 0, 'stopMove': 0, 'limit': 0}, {'t': 1, 'ratePrecision': 0, 'tradeId': '', 'accountName': '', 'accountId': '', 'roll': 0, 'com': 0, 'open': 0, 'valueDate': '', 'grossPL': 433.14843, 'close': 0, 'visiblePL': 3.1, 'isDisabled': False, 'currency': 'AUD/USD', 'isBuy': False, 'amountK': 1000, 'currencyPoint': 0, 'time': None, 'usedMargin': 0, 'stop': 0, 'stopMove': 0, 'limit': 0, 'isTotal': True}]
L[0]
{'t': 1,
'ratePrecision': 5,
'tradeId': '32572646',
'accountName': '05654022',
'accountId': '5654022',
'roll': 0,
'com': 0,
'open': 0.71538,
'valueDate': '',
'grossPL': 433.14843,
'close': 0.71569,
'visiblePL': 3.1,
'isDisabled': False,
'currency': 'AUD/USD',
'isBuy': True,
'amountK': 1000,
'currencyPoint': 139.71652,
'time': '10022020065344',
'usedMargin': 2500,
'OpenOrderRequestTXT': 'FXTC',
'stop': 0,
'stopMove': 0,
'limit': 0}