This didnt work for me. I am scraping the website slider.kz. But when i scrape the website it pulls the html of the main page instead of the specified url.
My code
import fake_useragent
from bs4 import BeautifulSoup
from requests import get
headers = {'user-agent': fake_useragent.UserAgent().random}
soup = BeautifulSoup(get('https://slider.kz/#unravel', headers=headers).text, 'html.parser')
for i in soup.find_all('a'):
print(i.get('href'))
The actual output returned:
https://twitter.com/x_slider
https://paypal.me/xslider
/
What it should return extracted using selenium :
Selenium code used to extract the desired output
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://slider.kz/#unravel')
for i in driver.find_elements(By.TAG_NAME, 'a'):
if i.text != '':
print(i.get_attribute('href'))
output
https://slider.kz/download/371745468_456554892/238/cs3-3v4/s/v1/acmp/cOE3Il1BHU9ESEFr9Hz4pL7WfYMASX1jY3IxqYGfVbDA7njmXutAbM3kHNfpwOzSBQWa2IZQ7huupcnHjAdwPgptzXqreGX0tmqzvcVCEsQn7_VVdO2neH6DlOB61uNtIBCcnHAMlApXwBfbMccaKZDif1_ZPf1e-4rkVEyHP9VQBiCGZQ/TK%20from%20Ling%20tosite%20sigure%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499200_456477539/242/cs3-2v4/s/v1/acmp/68YuXV09VzsZx8eNd_eKHNOjhjlx7Ae93T9h6eA4wjSJruGLjWdJqvGRkKfvEla9GYPHVopebj2BMYIsTf9R2a5-ztCHImKELMaUZQBuM1_OtyVbt2_AxQqxfs9ziB-1F1_vSJ2wJQO8EfhsuQi8Ouz-qH5nyqJPMAaPhuKuqeJ7XDdIew/Anata%20No%20Senin%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001620886_66620886/237/cs3-1v4/s/v1/acmp/9F7u-H32JUQGkAjlBPuZTbwaM2ZkjLchrhJxmqDZjIODWGIF6ClRnA49X4_rMHf5E8b9SPQUJqKtEwS_4cq1EkwDf97msqqps6gnKzAwJRNU7lhLkaJ6hqd0yZg27uBAvK6wMpizrt3HpGUIuURFmYDMWX7ggp802aWW51gkn_rQOAE0Gg/FalKKonE%20-%20unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499141_456425057/270/cs3-1v4/s/v1/acmp/sD64r4KNC9Ye4CfOMggOFmSA-ric7pUSKLaN_bfGIArMK7MSwjhu4lHpICSxWKCDniNKGTOOiGcg_6b6VUfLRuvSzn-Cakp4YU2BFw-5DJ2KAlFf4Duz6txtUAen3mslUB-vOSV8mT48PYf4VuoxFkltIb-kYqzkXgvsr56GGrXFzYo54w/Dima%20Lancaster%20-%20unravel%20(Tokyo%20Ghoul%20OP).mp3?extra=null
https://slider.kz/download/474499133_456468907/240/psv4/s/v1/amp/I4gkKu5n8-ePABgx_5blRV5izpZRPdm6o3_5C619b-f0tDciBsojh2EkLvtQ3xref_TYPVL7vBktamf1mdxDmxi3xZz1XC_HKAwsjUqU03dJpvytOBXsiCZ6kgP2FQ/Yung%20Anime%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001476389_94476389/229/cs3-6v4/s/v1/acmp/A6FBXBqjCCmhlytjFG4rAKFNfD78JTsqzgNAZToGubFqSZ_t1OQ_4gXhqO9qxFs3nPX2V6u745hB34NMrpGSvAEEKhfV0u5jYjHKeAdqBTbp05hvkdg2aJ6lJMFO3gjA9DZjanfjDL1fjhDC02e50HYr-K-ILhQkjdrixmzi8Ed5PQJfXA/Nika%20Lenina%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001355458_97355458/244/cs3-1v4/s/v1/acmp/ZRd1eY7d7KVAi23V-pkv91B0rBxyH6TBawtgTp0u_agc_1WbMS1061fhyFRFye9GHrUcaKX2KI1HIEzV0SXmJbSgy3q86HHZq08hTiwHdE27YVaxwn2HqW0r_AmOLtPMt6EcSNjcRyKUc-8sPnJ-PYKIPGeHoF67WUBsDbBteD1LjaG8fQ/TK%20from%20Ling%20tosite%20sigure%20-%20unravel.mp3?extra=null
https://slider.kz/download/474499285_456610507/236/cs3-5v4/s/v1/acmp/yQ75TttqYQXX7TBSoFeRrACN7JTw4pDVbn7gQ0uYHgKyUtvarViLiz47siDjWti8wr-Udt9pYektb5tp-VY68B-nFSq8SQ1Amqo3YCROcOFMhfA-xxT__aZOT2eLjyC_8KqHeLGi6Dzc-Jo52P-30wiWCh09ScW4t8rmQGjspFzDuWMkqQ/Yung%20Anime%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499192_456485073/91/cs3-5v4/s/v1/acmp/CnNkh81L0geRARmPlCx6HDtwarMIshv51_2gNlOHnFHKfJ3sTWbOXjzT7-SBqMmIMIusJ8lRN9tsjx4djCww8g3zd5n-x8SxDVVcMxhXKx-XqtOdT1q88mABUQhKY4ZcLos_XBYe7lx7C9CzcBabYFpBYZRZ-9x9M-HX52DfvajtnggW5w/Yendorami%20-%20Unravel%20(Opening%201)%20%5BFrom%20%22Tokyo%20Ghoul%22%5D.mp3?extra=null
https://slider.kz/download/-2001941575_98941575/185/cs3-3v4/s/v1/acmp/fmdPR0KKIvzX-evg6umhtRemEihc16Hc0s0W6fanPFxDNR8vTeCaTGdId8L20cp4r43dNQO3XYNbdp4ftt0mwIxb89SFMmcn5bBMZCymCay9cegxcXx4dDhc2329j0Evv_KSB3wpIzA5IR9cBf34DyOdDkd6L1b_ygIpDGF5E1_yp4acPA/Collosia%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz/download/474499243_456360717/225/cs3-4v4/s/v1/acmp/i3ZW2FQYrSPIkVmjzhKxasRpNU2sHQA24dwNyYtNb7SpQTg9bTqGlKbpeknyi2-z74quCFgDKR6Vox0JFmyJJVHQmQkGrPxTgnngADz-Iaxa4q3MVUAJqzTZJgu32Ur-FRa3n-lcN3wy/Trinthepianist%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22)%20%5BPiano%20Version%5D.mp3?extra=null
https://slider.kz/download/-2001187532_103187532/144/cs3-5v4/s/v1/acmp/NuVJyRij62WtsZrXRq60T7I1G8mfLu-Q01ITWVXxnn9DvIa9hiRGNUfMX2HF7qJMNv6nLC8GzobY8nmh71W9ulIpnXI44tGkyBKcy3lyAq1rTpBjLXssQiC94zA07P238siyirG99-EQsSAFL29rMcG8UYN9hXqC5g9-Xc4hyhT5vuNsHA/KinovariSquad%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499138_456497179/330/cs3-1v4/s/v1/acmp/OClerb_S3_yMpf46bD6zx0ObnCHcC6OQTPnK5Pry3qn8XVTOSLd_xRzCzUodFJoaLYbrjnKNHktd_J6PZu2m1hkZM3N--UNOqZkPIMubcOCJfcBBloN5BoCve2WvgftZ3yWubFzzWHiVpLnosbeogAzf_bMpblrewEeJ0-xC7toQeGTJ6Q/SLSMusic%20-%20unravel%20-%20Slow%20Version%20(From%3A%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001343575_62343575/235/cs3-4v4/s/v1/acmp/O6pRAZYWRIaUHjj-3d-I7wDamPE2M1MziJsmpF1XKCPGPbOKR3UV_A5vqtpbBqPSPochs34hpYKTTsHS5eAL_FiKJwwKvbUFqD3exRAAbDouMGxfW7YEpmMkN8NcpVz5QmH0lEJypGWO84b9cRnPSB8ZAkWNyKpGue26FP9Qc-jFLAj6Kw/Geek%20Music%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499226_456421440/235/cs3-4v4/s/v1/acmp/ErptzsuuaoawWhx_finD4Pb5GRbG05r7PrMIQy5LYmfw9QfJqjPBRGK-vRTrZnbsfSKoCA8K0odPCNvb_3rC3Mv-97bmbyv188pzC2PIKET0DntUvYYE__cD-KCVTnTyUpx3M5fe5hNdYjsb9UkYVngGlSBMNrA4OQhrp3-15Fybbvrfsw/Geek%20Music%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499206_456447563/243/cs3-6v4/s/v1/acmp/Ll3SqtO1lm7C2W7j7T6-bCEZ03fOS_MAFwvXUqzizClbi3AJS4LYDddyPM_iI531Tv-t1hUYKQFeJCUsF9MCM_ox93UUSaP_M9Zp0uVdB4PSpBXcu6xgqoKCDkbQ69CcnGpARuXxeZ5XPo6hJox-8xvMuZnfoT_b76BasOFB7rcYjFPDsA/%E6%AD%8C%E3%81%A3%E3%81%A1%E3%82%83%E7%8E%8B%20-%20unravel(%E3%82%AA%E3%83%AA%E3%82%B8%E3%83%8A%E3%83%AB%E3%82%A2%E3%83%BC%E3%83%86%E3%82%A3%E3%82%B9%E3%83%88%3ATK%20from%20%E5%87%9B%E3%81%A8%E3%81%97%E3%81%A6%E6%99%82%E9%9B%A8)%5B%E3%82%AC%E3%82%A4%E3%83%89%E3%83%A1%E3%83%AD%E3%83%87%E3%82%A3%E7%84%A1%E3%81%97%E3%82%AB%E3%83%A9%E3%82%AA%E3%82%B1%5D.mp3?extra=null
https://slider.kz/download/474499222_456568196/98/psv4/s/v1/amp2/G5Qoz-PWlD8w86QFGNvVLmXfps93Kv7CVsbbVFpo4e5LjDQRJ_yuY4Ha3Nrp10xVXf5hur40alIvROP88ncZTLhPICCjkBn3mM7QGpaFgBQf3SEib-JCMELaZoMg36gpsc6vdsRKzSqjecjnhDAvjkixFTc05wTyL3b7/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Winter).mp3?extra=null
https://slider.kz/download/474499266_456496174/238/cs3-2v4/s/v1/acmp/ZSnEUsWBGhnXj5XDpV5_RYtliMXOCl747j9MfPXEJut-jDHGl9lI3z-Wyrmo1fnrb7QYMdmCyCFGChtLb_Jlnhj47w9llLDSfOeT-ejt6HrEFCVeG9pSJfb1x2Orv9ls1AWiprtX9TBylxyJYkTTMCT8lXPxyT_1Lhh63I_Vclx__QWHAg/The%20Unknown%20Songbird%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz/download/-2001091207_65091207/243/cs3-4v4/s/v1/acmp/KQtj35XBV_qATTaJDm3ULd5piLxC8bhqrobCvIFfHM35M5YDIBlSZQyfTvcmGktv5upmL40wFiQuX65il7CSkTyvGgCubXJXch30BTk_8VgSnOQitmFNebJzW_ADxbaBwFpJsZRZ1La5UKahLdafzttj-qiWjk7B9uuiI2PejUViDS7oGQ/Bibi%20Gamino%20-%20Unravel%20(Opening%20de%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001815387_86815387/189/cs3-5v4/s/v1/acmp/ao_2H9LXIsXS3wGH47_DjS8MTmTGiH8wBvOjh7ZTyXXT6gGCzwqO9j3fW940H4MSUnTlarmjLDf4kFN9izLPboP6opybIVo1zv0efcOASrk03xIjwIFbsZuB7Ub1vpl_U9jevxKne9xT7-8ecNhrm8_MrrcqEt2yUw6ApNIIDiFIjnPm9Q/Baur%20Karbon%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001343001_91343001/91/cs9-18v4/s/v1/acmp/zwfryFVsrFggm-hEVOd6RiXeyEUooW9tZMLBrntNjiPJOxatNavKMSQ6tdcBDb5DfN-5oz89p6eXNd61Td00hMgQXQ3pfU_uRhQXPtSNRXH26a_vH5x1z6VVndmwxWikty-kmyuctXkSLygSr-__9qnxcZYaNplAvE79Hl5maQC14zIq3A/Ren%20Avel%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz/download/-2001459950_102459950/222/cs3-5v4/s/v1/acmp/dXsMP_kZ98lEdBVsmwcSr0zvYGcr5Spyn8RNNVYxfMDehWA3Jbcq7BpGMLoaKcgCMpQ4PpHszAAC3bkW4wNbKiFD6LumIhx0_T2C2xekVJsfR1xb8cVtOgwLUMFnffOWz3GULiHXBKoPS2xjpWf1TmkQXdDhcpmKqokCzhx0yvKBagDO0Q/Jonathan%20Young%20-%20Unravel%20(Full%20Version).mp3?extra=null
https://slider.kz/download/474499249_456500220/95/cs9-5v4/s/v1/acmp/kI7HhI6sOV-jdnMnoifzOJwbZwidKai2LQ6nQOuVeZ73W2eyOcQUDIDLEuDWL6qhNZM4ATnhITDeeI1W34s3kaY03Bf9CGzRm2RTJHtIbF7U1ZHL5GJ_lKj8DR27Ulz_4whswvN7p4sUAuL8LHEyYCFjLrgQVlLTb_xfVL9j7q5_4ndF5w/Tony%20Gu%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22)%20%5BOrchestral%20Arrangement%5D.mp3?extra=null
https://slider.kz/download/-2001472007_90472007/95/cs9-2v4/s/v1/acmp/D25FbaAzhvubfYuHGk82DiPeG6gHBuhK6YwgRt3UV2gnf-s7ynvbhcNomwhgm7kqj6RZ5cqVRgRhzUIVCLvLPjnuChQiPRefKOp8IH35ZjCBLRB5-CWFnNKMc2Z3Z_jQzeyxzikPI9PI5u9fErheiCWVb0HnR5k8FSg_guhCQaOBeYKgbA/Radio%20Control%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499153_456452837/238/cs3-6v4/s/v1/acmp/QOuii-Czus3bM2CyRJILUYMWsCNb-6TEgDNKyBO_qJYdxeoLM5YEqyW03di_3mSuFpbTNqCD8cEN8ijmtjMRESRZCKya9Wj89Hyo8A6s7oQFURgmc0OAQw1ifU6FNPgNuLt0cM8UuVu5ONMZcXLxQB_X-vyVga9enPIDvzadwIPzMLrkKA/Kevin%20Remisch%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499122_456591314/97/cs3-7v4/s/v1/acmp/lOsM3wtU0cPI-POYId2zCQKMuFocZYxke-zM9kGXjLh_HWnKmpt9FT_BoUqdG_QMSQyXfnUdjzt11bJW4Z6hkc8izzQ_xo7UA70AhMVu2Ik-d5INvNV2W7xAbavVROqzFEPxMEmmYpazITlYQEy07UapD3KiFWUY1WRDbr_-1v30o018GA/Lil%20Bubb1egum%20-%20unravel%20but%20is%20it%20okay%20if%20it's%20lofi%3F.mp3?extra=null
https://slider.kz/download/-2001077985_82077985/238/psv4/s/v1/amp2/VllxAjiDhUD2kGGN_1QYYlEi6ljH-VawyGBR5t7gRwmBd4bnyoy7Q9TP42q83gu6qs0Q3xjHT7HEgi-HkPQAs3w417-ESpSQGMQEySHxg8O43I_0gt0dJ6edsjnIsWzRIrOtSJwBSeqtoUKitAfdgVRjP5-6egt8gLqr/%E6%AD%8C%E3%81%A3%E3%81%A1%E3%82%83%E7%8E%8B%20-%20unravel%20(%E5%8E%9F%E6%9B%B2%E6%AD%8C%E6%89%8B%3ATK%20from%20%E5%87%9B%E3%81%A8%E3%81%97%E3%81%A6%E6%99%82%E9%9B%A8)%5B%E3%82%AC%E3%82%A4%E3%83%89%E7%84%A1%E3%81%97%E3%82%AB%E3%83%A9%E3%82%AA%E3%82%B1%5D.mp3?extra=null
https://slider.kz/download/474499138_456433311/238/cs3-6v4/s/v1/acmp/VNdbMoXjNPrw1jhq0upwNnZfbolEFriZNKeUEHSf1ncOJUmACaSbyYEZiu1N8JghIVspFPzHWO3STGmE0lGuFvgo0bF6Y74ykyvwwSiDgcRqMbUBbn2ySHSA5RSLN8nCinRGvDvJ-5YSbYmvgZbwaNVWVFVJ0R7D4gwA5RPvt7BQj7eaeQ/PelleK%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz/download/-2001778472_74778472/238/cs9-2v4/s/v1/acmp/z1oe_35fnQwSwR-yAjBnQ4MlCd2PNIEnRSLz_ZCKF_5PldV40_lerssvNhZazYLx7YpdpVjqm_HNA8PIuAqrlvKioKNyrOu3GL048jLLCz_xJL38tDuU22qmn8mhf5YJ7_XzQygG4tmjHFh88hASl-wYpTowxw4kmOAcS0perZf4inwKbw/senakisaragi%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499220_456599167/246/psv4/s/v1/amp2/ZyCoku2jGfwD1BaUlyq7-q6TDH-IwTCPHDVoyJSz9TRj4hLvzyeFs3j5um5duLh8mru1sc22DqkxA7Ajara2-Y9GKl7PomX_F88WidLCB1MARvkw2iYrizN-q4Et4QsAWJm2VaSamp1vRDksmw5SUgN4a74Qh58sRx2Y/Matteo%20Leonetti%20-%20Unravel%20(Japanese%20Version)%20%5BTokyo%20Ghoul%5D.mp3?extra=null
https://slider.kz/download/-2001804556_66804556/238/cs3-5v4/s/v1/acmp/gIyB4Vft-_97FyJAE9_0hQlav32398RrusRMOaXgEkSsoBNW93224d76-9tGbFvAT07TSJEwetYGewbka3fjyJfSui8-paQTAJMVI5lMtwgMuTU9e6fB_cRtlIsXuN6vZhQORVl47To-JGxYgfN1-zRNimSOEBj4Ve_p121ne1UwhTnDLQ/V0RA%20-%20Unravel%20(Tokyo%20Ghoul%20OP).mp3?extra=null
https://slider.kz/download/474499291_456568508/102/cs9-6v4/s/v1/acmp/fKvZmSOyvLXOxRXgOatLErO-EzVCD2cn-tB1Dys0NFpugNtd7A4hZlhBPUZMXAKZIevqS49l2ZAscFh-k3RRHX02kgrbPM6H6AhcAZ_jbvNstSTtxywJvz68KeUWJ5y0Bp8ll3fL7g-rNSYGQEzIxoTjQ6LXqfB1eolI4ZUsqC8vUfoRcg/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Fall).mp3?extra=null
https://slider.kz/download/-2001227120_98227120/186/psv4/s/v1/amp2/cLi6IzWrhQhwJVpx-nbHAVgRnIxbVTGQQZfcqBf1JyjYIYi55BhiY6bc_fFMdWxKV5lMiYXFcAbUTzrketBr5NXGs3a4B682hU58z4nXP-E3AET0RRFi2-zBVQkK8zeHEOfBGk4-RH8HbJvY4f_oJHfeSSfhVPyxOKdg/Olga%20Scheps%20-%20Start%20Anew%20(from%20%22Unravel%20Two%22).mp3?extra=null
https://slider.kz/download/474499232_456549884/90/psv4/s/v1/amp2/G_D-kC0LEQYYW7ekPdSMg06ozpZssPpm4_4ke1P8RhdHN2a8EH0t27UjRhyBw98-veAur6NiLRcicS9vp4nzOPQIsqaPr_-v-x9h8m2L3yIzpDcVKTftL2Th9rB_H4g4tHTUl8orsWQU6npEA7sTWysshIwpFs_nqWQq/Mary%20Antoine%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001667855_68667855/186/psv4/s/v1/amp2/xznk6i9L8LRfJar58xq3PjlC8dSayqHAQnXP8JIbQTSUFF6dtmf4AG1_Sr7dR-6gYxW05dzwebqzJJU-KkNJvqu6ieBiK_trxotPhLkBA3mmBcskTm8z1q6VnpE3YPOYyiJIOFl_NLvxbdT7GqlNW1UYnKScRVumta8m/Track%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499146_456450407/186/psv4/s/v1/amp2/wEP-qGrwzxqR21c80CCBheBmiu2fIZVI4VsGd7CuyGxTkVI1boxILGBdheTs6tX0N6XCuLO1j4bAs5B8XGAmAEdD9ijXYGTW5cOYDQ7psn31PqmHWKubTdiefBdP9ztaYS6szPgmhtQ9FN9OoKKOH3-WWs3qW_IvYAFM/Track%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499318_456592131/96/psv4/s/v1/amp2/1Ngs7lG1WBh6TJ1IhzauBhSNBeazCty_k25JatXLvLqpadvgBgoQ0lY9ISeBhfts6PV2hs5QtYk9EZFXczQq7GVshKDrg8ovPv0OnPTolvNdOhhJjLiKPoF2dUewl09LW4yvJmX5ogCX7q_Pq-F3bWeLpa20NankBa_c/Vangakuz%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499173_456432829/137/cs3-1v4/s/v1/acmp/rT3ABEaA8kmBYfjWSUgBN1i_wGr6c7BO5YQXfOdt474aADbZ0Tjg4OWB08dgPLHIcx6o3-rRRELc59NKPIhWhfVQq8jatvyFzxR6k6bo-wp5dJFVptg21nj7NTtWPOMcgvjXxblmVfZrXtZKGKCfFoqb-sO2sDKOM3prAMESLmk_HQieCg/Kenzie%20Smith%20Piano%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001400485_76400485/187/psv4/s/v1/amp2/wQrrOm95FEb1EpWPN05KTxkNgjEfDh_faLEMRAA1Qk4F4cCJSO_9sd8XjDlc714dR_OK9JyY_9AQCe6moPsLeNnM6iCbXzFZrdWObICKCxggwhxuCtyaM4liHWk5TgNcFC1GLQp8Lu7Hh8v2yiC062n2nfsvZHjenrho/Simpsonill%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001521828_57521828/551/cs3-6v4/s/v1/acmp/dZXnCT1fUfJ6PdA2mzGe8yHpA9s-DY_9UYw3SUYt3z_QoxPkXt6m4epU3ilm_7hieMGGJIe-HQeq6UYmj1QZYvJPezdSZyd6wBiy288Zdv2zwMEqQsWrKziLnBIDpenTzJfh8Lht97PQ4hqkQ7mOGO8ucyZm1evH5kQFymrzeFQcstVf5A/Alveol%20-%20Unravel.mp3?extra=null
https://slider.kz/download/474499237_456436594/88/psv4/s/v1/amp2/8Bo41ZWBvHso221zqJhGjyABsxjNh8o27U_XLhVWL93AEuMhcNxyhU2Ur43TuL48o6nFZQDF5HpQj43ra3EM0ddi_ptqUCJRgVr1vF5ADTqFXYlzqdIO_Jg64Vku2kEfVPwd5KYOHoJD_VCsRTFN83sW5epkBJm1c798/MattxAJ%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001915683_101915683/90/psv4/s/v1/amp2/QaH-avVk5DSEc-4Xw50z0JX3TZLUGZ0nk4PblwUkmscM8QYZh2u6t8rFGB7D_GzbyTwxltM4Rp0kE1x0WfCO_QHO5IHZe047S93z7D2a_-DEnr8v_bwDtLv_10NQRabdX77HE8h9iWlSs_doa76LQMm3Gm6fYtCutX29/Music%20Legends%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499272_456394915/236/cs3-4v4/s/v1/acmp/WDQwU05V9M89alMuCTi786gxWCQDoeFFwYlNzojOyQ5G5oeCmgknrMPfQGlX1dE1nn-JHFfaJOOhgXQJWptsR070hN8OufTiDgs4dzg43TzgDhi_Cn6mXxnfAq3qK5xDWSY4Kamlydi4hVH6xcC5uRrpy70lK5xEce7eq01yornEDvVkcA/Shiroku%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001529426_101529426/255/cs3-2v4/s/v1/acmp/yz7am2kIPxn76UUaQEQFAeOfeTQciBwOu8g151LzAMMKMQ5NRg6nI92UZBjPePHnH9rV7svs2udN-fiGIpg2s8QqIbP9lWCyG-J1qUB9CywlXwu8tfg6CzSTN2GgbRp8iQM-QKlGrRgLxi56Up3r0U7oWubLYL0hEOKr3ysemG_9tB_Q8w/HalcyonMusic%20-%20Unravel%20(from%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499176_456433059/241/psv4/s/v1/amp2/EZVr_Le7Wnc33wId8JajbbOo03wZBaoT8f8_ZyHVTiL5h8hs8HtGSVdm3BLNaW_AH5znzjmROfyAYYPoYnLQ-pY789RfVtTwX034IVdl_tM097Dube-bzIGaa_7COKBlo3Dp_1lHWfn1EjrgHEI3YhYPhnVyxOjkRsrd/I%20am%20Justice%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/-2001621142_58621142/68/psv4/s/v1/amp2/Oc2oMY7m2oW3hfpZDQIW24g3lWZIUDD1idJn8Wn5ZN5LJy29qZTBNwcHYCcxvtLPz5YsK4hWSvi71bW03O1eKw4PhvRXcqWreiPMJI9BJsHU6vwjXk1OieqMk3zC-stJLo5ZRIMxUpuefWK8HQlv7HfCAqrba3PI1_yG/Gianni%20Tallone%20Della%20Cavallotta%20-%20Unravel.mp3?extra=null
https://slider.kz/download/-2001052300_100052300/89/cs9-11v4/s/v1/acmp/jENDq0_xjfWlA1Fau3Cd16w9dQT3Won_n8YIMUI3daFMPE4P3PEuYPxpnn37TYSDXtZsXjyDSPLFO2HbnzGi3yDkvfePGfIrY2lK4y19tTEY9Y6aRUZIbAN5eS-IWG91t19G8soWuPs8B09mV82_4lj6lW3900E11hWClLA-Xl7jCoN3bQ/Fonzi%20M%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz/download/474499157_456624310/212/psv4/s/v1/amp2/WpHC4MCzE7o7pBv5lzq2K9K3pf4fZPUgW_GycJUP-92L5fd1yNx9fOLzJol5M9F5Mkwv8BApcjZkjLw4nC75G1rX7bnR-oZNOBP1pSw0EgJZi1o_a2soIKK3YLTMmcHdcdxXGNTJ3IG0YasN9w9n_8pzlc8mYoQYkOf1/Love%2C%20Lies%20and%20Fiction%20-%20Unravel.mp3?extra=null
https://slider.kz/download/371745435_456449924/271/cs3-2v4/s/v1/acmp/2vGBvFG1ejsCoSv88vOr1JwaXPjPCw6aFRXBB_8fqa1j_LxgWXH7bzy_9JbcymQd5ACAr1bYoLnwBwfthYgqYHMWaL2DGTCu5Lp67BHZXWHRyGFeD3iKM5OeKLh4642GtJSGxVkgApp_9cRCvvIj-sRUAI1a0jgd_zjfAafM8eE8pf8WTA/The%20Blue%20Notes%20-%20Unravel%20from%20%22tokyo%20Ghoul%22.mp3?extra=null
https://slider.kz/download/-2001260329_90260329/94/cs9-15v4/s/v1/acmp/9QTEytbysHCNmBriy3cvC2OOMqLUBoFMaUgslyo1b_QU4KGSKxWTJ-1x9bmqlxOIaCKJKTrsStWyE01pTlyCUU-tImpXxgFKyqcdpwQOvNAtLT_jiCrxmVpKGGryMeOgDp7xqcsbwWrDTqUYh685cY_Gc4avJ7va_QDg935M6qq_TxcxIg/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Fall).mp3?extra=null
https://slider.kz/#Contaminated
https://slider.kz/#Excarnate
https://slider.kz/#Extremity
https://slider.kz/#Serpent%20Of%20Gnosis
https://slider.kz/#Petrification
https://slider.kz/#Skullsmasher
https://slider.kz/#Hissing
https://slider.kz/#Mortuous
https://slider.kz/#Skelethal
https://slider.kz/#stabbed
https://slider.kz/#Winds%20of%20Leng
https://slider.kz/#Usurpress
https://slider.kz/#Encoffinized
https://slider.kz/#Glacial%20Tomb
https://slider.kz/#Overlord
https://slider.kz/#RESIN%20TOMB
https://slider.kz/#Descent
https://slider.kz/#Hurricane%20Death
https://slider.kz/#Gutter%20Instinct
https://slider.kz/#Crypt%20Crawler
https://slider.kz/#This%20Game
https://slider.kz/#LIFETAKER
https://slider.kz/#Autokrator
https://slider.kz/#Vanhelgd
https://slider.kz/#ATARAXY
How do i fix this.I tried randomizing my user-agent but the same result.The website doesnt have any log/sign in.
Chrome version 98
Python 3.8.8
This page uses JavaScript to download data and generate links. But requests/BeautifulSoup can't run JavaScript.
You may try to find url which use JavaScript to get data and later use it with requests. Usually JavaScript sends JSON data which doesn't need BeautifulSoup.
Using DevTools in Firefox/Chrome (tab: Network, filter: XHR) you can see URL which it use to get data and try to use with requests.
It gets JSON data from http://slider.kz/vk_auth.php?q=unravel but it gets numbers, names, etc, and you would have to compare with real url to see how to use them to create real url.
At this moment server sends only messasge error so maybe too many users try to get data
import requests
response = requests.get('http://slider.kz/vk_auth.php?q=unravel')
print('text:', response.text[:100])
data = response.json()
for item in data['audios']:
print(item)
It gets also other JSON data - similar artists - from url https://slider.kz/similar/artist/unravel - and it sends data without problem so I show how code can looks like after getting data.
response = requests.get('https://slider.kz/similar/artist/unravel')
print('text:', response.text[:1000])
data = response.json()
print('key1:', data.keys())
print('key2:', data['similarartists'].keys())
print('key2:', data['similarartists']['artist'][0].keys())
for item in data['similarartists']['artist']:
print('name:', item['name'])
print('url:', item['url'])
print('image:', item['image'][0]['#text'])
print('---')
Result:
key1: dict_keys(['similarartists'])
key2: dict_keys(['artist', '#attr'])
key2: dict_keys(['name', 'mbid', 'match', 'url', 'image', 'streamable'])
name: Contaminated
url: https://www.last.fm/music/Contaminated
image: https://lastfm.freetls.fastly.net/i/u/34s/2a96cbd8b46e442fc41c2b86b821562f.png
---
name: Excarnate
url: https://www.last.fm/music/Excarnate
image: https://lastfm.freetls.fastly.net/i/u/34s/2a96cbd8b46e442fc41c2b86b821562f.png
---
name: Extremity
url: https://www.last.fm/music/Extremity
image: https://lastfm.freetls.fastly.net/i/u/34s/2a96cbd8b46e442fc41c2b86b821562f.png
---
name: Serpent Of Gnosis
url: https://www.last.fm/music/Serpent+Of+Gnosis
image: https://lastfm.freetls.fastly.net/i/u/34s/2a96cbd8b46e442fc41c2b86b821562f.png
---
If you use the browser inspector (dev tools - network tab) you can see that, every time you hit "Search" it makes a get request to this kind of url: https://slider.kz/vk_auth.php?q=unravel
The response is a JSON with the results.
So, if you want to use requests you should get this url and then extract the information from the response JSON.
Now it's working.Please just run the code.
from bs4 import BeautifulSoup
import time
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
url = 'https://slider.kz/#unravel'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.get(url)
time.sleep(4)
soup = BeautifulSoup(driver.page_source, 'html.parser')
#driver.close()
for link in soup.select('div.sm2-360ui + a'):
url=link.get('href')
abs_url='https://slider.kz'+ url
print(abs_url)
Output:
https://slider.kz//download/371745468_456554892/238/cs3-3v4/s/v1/acmp/QRhl3TT5g1GEHPv-bkQJXT4UPbfj2OAAOI01kg3IFGV3PQUawPjulWC4U1oquAFZgADZF4QaOxk8CrEfBzLih4EpJf2a8qTr3WYT0yJQiCNDV78tnhuZAGmi1p2I4wwdw_743Rfe6Y9lvMXcVHwnKd7NdRyUlKLz5XxjVAUTGxP17pY_qA/TK%20from%20Ling%20tosite%20sigure%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499200_456477539/242/cs3-2v4/s/v1/acmp/AsZGD44ViRPS8iviU4fbOjb7I095Guqb8LMt4EBRZEwgTYR2BFlFAI4HtLEu_JTeohwHwakRgcJ9pnUurkL98oL6co9QTzepES6zqJ_tDYp4oUN5D3E5pm-m2LZGkqeCNcLsZmA6PeuxNSkpEvbfnq7W-in6XtIpASiP9qQ1UdSb1wXIgA/Anata%20No%20Senin%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001620886_66620886/237/cs3-1v4/s/v1/acmp/gOSXhnJZc_7qVqSRt7soH679KeuB1qEa_Odx7P2YY9BwEAbmB9L2N-j8Qcm2roEWAxA-45rEGOQ1KT3r6CxnNYFZKC6t0Gg0J-wIjnC0sfDNhJrQpogT_kHrV7rhUSyFPLq6mrP3A7Tj1Dgl_KzZdjB-REw-21XuVyxlrLIgQRJiApQ1vQ/FalKKonE%20-%20unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499141_456425057/270/cs3-1v4/s/v1/acmp/9l2EX9ERaANRDGCKW8RS4-_VZR2MZYDlMRLy-cTAlKW4SP7QsodWeBObK7VaJa3vQiD7DOC6YEdOxTZHUUUksY6Xj7l1OgMyaJ1uzDLiVxtTy7NfJ98tdknxtnDhvNk3gXnoTZI5Fhs18fSDusmQkFAcDYkC8A7Czpng3gLvsvtoHQeQjQ/Dima%20Lancaster%20-%20unravel%20(Tokyo%20Ghoul%20OP).mp3?extra=null
https://slider.kz//download/474499133_456468907/240/psv4/s/v1/amp/84PrmRmfu7EBaFB3FMcMuLy9CrIgecvqzu74ZHTGdBVyhe1rQRRhSBfVA56cHNnwSPP4GNXVvQ88IZJq0exhcN8Fvn5k9rIdI-PJWu8QviAWxX4aYZqon1Sflf1Cbg/Yung%20Anime%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001476389_94476389/229/cs3-6v4/s/v1/acmp/YRjjyO0TiBxN0blkh9T8e-6g6SepkCtdIMtBqNLYvHbfqOCYLBqpvwS6RmWXLeYNUKMy_FutPbQ9LLRPSHFO-TYBQwAM4AoT6Z5_vvSepusrUJu33dDVYMAIWntUc8ABhjmnLRfVPeBbHYzGnx22n2Je_U15qYFD1we3aWiaw2qV9CxbTg/Nika%20Lenina%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001355458_97355458/244/cs3-1v4/s/v1/acmp/f-WX05d925zpwPtfj0RS6oUftizqO337PogbUWwnM1M8hsXNV26bPGdtEgze2yNpsEQygL5qeip-xc5s7LBFOtY5X_4n0fQo-oWz9vkdRvD3sUl9XEPgibbUob0VFJ4itJXtm1l1D1LRHTyDcNeA3xDj2S2n1s_7Bi59L8RpZgIGWmvtcg/TK%20from%20Ling%20tosite%20sigure%20-%20unravel.mp3?extra=null
https://slider.kz//download/474499285_456610507/236/cs3-5v4/s/v1/acmp/QMrplAB509ufV1tZgQn756q7IsLtpEmITRyaCLzQBrMEFmK-Zc-IkEWdOI_eE5D4-Nz1i2Y97wu7cmlF69HxYWPqjcp-2pWLuUzg4h1YeOUtTGh-nteiMEdfnpJYfYkyP6ZwayqLFLOh7OH0Ki9n3lNYRm1A_U47zVi-lsRS2IGOCZQAOw/Yung%20Anime%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499192_456485073/91/cs3-5v4/s/v1/acmp/ukuxjLP1ByDVop0pyoGvFwd4eQTFJnLH_6Dod2uT9uR9VCLHcTk9zoKqes1bAAb7bM8wDj1VTxi_ooy8I_AcNjW91hjWhb2uKMBYow8RR2WciYlyJ2ndYfqcBHwTOH_bj4fd1nPng0AFGG7phKNleCZW0tg3tlvzMjTHPKfsOI3w-Rypug/Yendorami%20-%20Unravel%20(Opening%201)%20%5BFrom%20%22Tokyo%20Ghoul%22%5D.mp3?extra=null
https://slider.kz//download/-2001941575_98941575/185/cs3-3v4/s/v1/acmp/kLoq7G6WCnnVNV4gIOaTAA0sZWVTO34BChRHQ-SLhB47bp0Fu3ogijAaADtrwsqaZlKDLkY6RwLlAf8f-hgIGM9TCJ11mwzLsnIOc9YJRkDeUYH74GT2WF0tIO7nlPGg9iv2dVFDld089JM6-XcbT849iJCegdNGQ3qSkcaFjrQ1NaJFbA/Collosia%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz//download/474499243_456360717/225/cs3-4v4/s/v1/acmp/N0jZrOAfPu-5bJlwP0HzVU_8yG96Xddn12wE1QMmBOH_3aza1-ig-cHZHrY3dx76Ahm8ppqKO-uYXOCNAYUEK_mPy2Uy6npKbUlnRKn-YHIze3gP0IRvLHaaVMCACfEx3W-srnsSQTiN/Trinthepianist%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22)%20%5BPiano%20Version%5D.mp3?extra=null
https://slider.kz//download/-2001187532_103187532/144/cs3-5v4/s/v1/acmp/TYPoa8_kCxJuIGNUL1Wl_5vYYnxcl0CrUNYlv6FzMCLCgRyI5TrNjhX7xCuuRFBsSxGeTDX8rqI8acAvhNSGX0u__Hzr84NWTzMp2rtgee6NyN793lVcVF7r499W_34RTTC6exIIGcYEIhwUxDMC-BBwIQr6pYnBctsAbVJYd4ZcD-2JKg/KinovariSquad%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499138_456497179/330/cs3-1v4/s/v1/acmp/_WbB2KC7ZDs22xTYAFfwRyk0p3GfbW8L-q9exF4raO5H_YnoX1rdRmAGXSW9xQCk_7sxBLfA-WO0UT0ilQVPRVwo9GR6upY8VLlMWyRHtntEaNOnUakzaiUBcTIE5mGRYs3ti5Ek1JGmhtioUBPEr94GVtrOMeKGRPe4s56oYHWlO3xXrg/SLSMusic%20-%20unravel%20-%20Slow%20Version%20(From%3A%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001343575_62343575/235/cs3-4v4/s/v1/acmp/6pYoGLam7Gpf2Urd4PDhHInGLa0NZXgC4MiBNBCE_PG-YBTqjPZiIJNdgJty3HEEA42N4htrAN0TyudLT_l0rASk3yvwH8HSSl1GEA7xTxy8ZuQ1BfLrpHwIZ6g9MY2saaMmwaz51K32mUtBQHKXltyWuCvWzwYteRs3ZwuOhyEES2AJFQ/Geek%20Music%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499226_456421440/235/cs3-4v4/s/v1/acmp/ZHspQ1RQdnss3kdW6LREudZe1CVOFo2wItlxVeppU8SDGGPmfoVUH1p4451LnaaNG61RtRDrZD46eBfoTGwdmsiXmZ12Fzyhg_A5UMGplsTPHxOcL9Pw8CJ-VTQlIEwNiLJKwoWXxKIZ0e4h2A7rq3hRs49qnAzIc3lRE93yzrLCymc0ng/Geek%20Music%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499206_456447563/243/cs3-6v4/s/v1/acmp/26YcSjZs2EOKlDmgk6YspYb5imv6RXR1NsTkMBi6jgJMvdf5nNX3vJLWO-z9YAxl97b8JTxF3HK_u5wT7TcQfiIcFHOoLvXM7jg_Y1YMW1Vlc-sj-kqadoXFbGw_x9aI0NzZIT-VpYncsYC6N0N3cQsi_UjPPgHT5iiuIvFW6JHFBm5ROg/%E6%AD%8C%E3%81%A3%E3%81%A1%E3%82%83%E7%8E%8B%20-%20unravel(%E3%82%AA%E3%83%AA%E3%82%B8%E3%83%8A%E3%83%AB%E3%82%A2%E3%83%BC%E3%83%86%E3%82%A3%E3%82%B9%E3%83%88%3ATK%20from%20%E5%87%9B%E3%81%A8%E3%81%97%E3%81%A6%E6%99%82%E9%9B%A8)%5B%E3%82%AC%E3%82%A4%E3%83%89%E3%83%A1%E3%83%AD%E3%83%87%E3%82%A3%E7%84%A1%E3%81%97%E3%82%AB%E3%83%A9%E3%82%AA%E3%82%B1%5D.mp3?extra=null
https://slider.kz//download/474499222_456568196/98/psv4/s/v1/amp2/5MdAXyMQXscyhnNEfxrNMz4rZAhA3Ckw9tV0tLA94BDtvVCeDVz_Ycc62dFPtyJvjDaucoWBQZAguZHmkHV81InGjY22Am3TtoTYWH4rGOoiaM8coOi8DpIUQd21gda6fFcTZMkuLf0t33P5it1dH86nAmDGduot9AYu/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Winter).mp3?extra=null
https://slider.kz//download/474499266_456496174/238/cs3-2v4/s/v1/acmp/GkaHnOLSSRydiMt--oHvxPF2ZrdnAK-NO5Lei7J48RoxS_io3XwVnMFTQJN54MLZ5oagc9cSb4COMi_yxoCuAEm4FMnA3Kqetj5cj5D_alBQkHJEtNHS4IL1-kgE_QQ9RDsAfu5Qq2GrdyXj06rpvRr1EyKs1ITAkPez6uvlHAozli0mWQ/The%20Unknown%20Songbird%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz//download/-2001091207_65091207/243/cs3-4v4/s/v1/acmp/izDJAIBhtw7Gbo37AL4HCeFiLWypB9BgtdIqUnziUZ06PBBWt6OVqGY5BJIjaY4E3fG1szeoZcQeU7946ltUPsX86cPyq9jMfaPTbNS2rk4qaDYoJtK70qlGrA_OwvHUS3Ec_BQTFNq3-fqv2tJuyElVIaUesQOo4MMH7BpaTUI1T3n-BA/Bibi%20Gamino%20-%20Unravel%20(Opening%20de%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001815387_86815387/189/cs3-5v4/s/v1/acmp/1q3ZcWs-DyWh-iYHwleI-jAD5A1WVgP8sN0zolnoNmBaj55OQqmHopLzZPd7AWmYP2Xs6E7Tv7BDtGn3TU1_-cDD13sbCywSWwQB2ckEsW84uZKn38qj0vcbYCN7YqDp7L4-drmoSdzIyHRE6xXrixB9cJuXgcKlTbSwkFh7-HRoU1lA9g/Baur%20Karbon%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001343001_91343001/91/cs9-18v4/s/v1/acmp/pk4LPfaKgfokG8pHZAzxFPGhpemNTcIw-Ey_8BN1RPX_QNtv9m7H5HzsrKVY-b2p1_AKiSm_jJOdWV1iUXGsjtOF-mkjo4er3adDmZSFB6BqlCGQgXkK7F1Kn8XQDc5dc41KESB5p-JTSPKOCzitje8M8wcsdELenOJANNsvKtByjHRL6A/Ren%20Avel%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz//download/-2001459950_102459950/222/cs3-5v4/s/v1/acmp/dOeaiW4FJ3Q0jfOOTinST8c8Pn59ODRoRT5BgYrmn-DJw7m-u3x--lLPiUeECWRcHc7kKLGuAKoVTMUt32QfU0RDeGRCBGV2F8V8MPzNNczL3Qk6GQP9Bpj4ZL7SUUWlINmeRZyDChuD1APnUZtAzQ31DhAkQgpCH70MMHmtl6SNLNCVCQ/Jonathan%20Young%20-%20Unravel%20(Full%20Version).mp3?extra=null
https://slider.kz//download/474499249_456500220/95/cs9-5v4/s/v1/acmp/Cd6m3qhcIfjSesLIbaLc8IPOVc4E0H9ljKh_VIlj5LcIDifka-qMXBDquBCwd16bV2an-ZGX9X_NFFyV32q5DOg6AEKAMrSOdWQl9EC_mkvCLX1RLBF8HVarOoMZ9Y9oaV-gDNO9iTstvL0vBGQoxYz-CXIKok5FOfC5ptoG3dfSsRwkmA/Tony%20Gu%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22)%20%5BOrchestral%20Arrangement%5D.mp3?extra=null
https://slider.kz//download/-2001472007_90472007/95/cs9-2v4/s/v1/acmp/sqXJVOoZTIZvRB2CxjUHJZdvYHeaQkVffFqfpn4SW39S1-lImykJzWyENkb2tWUMO9eZNnOwNR0a6eZWPKvrBKdpLzLQE7JJFvV9kxKu8h2inNFyhSazlAljMtC3cn8R5Wk1B0uQdQY-17wb6-oBb7qoGSeiIBT-7C_Ob4G-JXcJ_6EUJw/Radio%20Control%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499153_456452837/238/cs3-6v4/s/v1/acmp/IQdnnyJQZtLM-cx27JQiYpqw3hrvc7Y4LGbUy5ruMBZNzC-7Bb4W6Wlw3M5CLykxFM2ALSU-aPR2xyMvggKW4gtqeXIhyfOq4yQbFNCYgG7mXXwwHp18FeyDTkXdYbdCbOZ05gQpO26e54LFLEky3FaAwvy48gQdHSUBAlCDNFN7gNUoMA/Kevin%20Remisch%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499122_456591314/97/cs3-7v4/s/v1/acmp/05S842jqQXODQOr-pq-LCi3z3CKmZR-6f9sqWNjGrXOaJDf-Ga0F42wMYDQmLTtChJaCKz-4kIdgGTXL-unYs4sJd3cg7ZwbywsPL2lMS5CiseljaXJ5I960mN1frRnZV-jVMiNqASMWfF9m5COgxgS_IAEvVDW5_tiXQxMxDfp_JSygOA/Lil%20Bubb1egum%20-%20unravel%20but%20is%20it%20okay%20if%20it's%20lofi%3F.mp3?extra=null
https://slider.kz//download/-2001077985_82077985/238/psv4/s/v1/amp2/TJu-2VmMhrc6VA7SF1OM2092kq003sX40joJxtIzXgAulLrdmWnzcVDMjn7L-dp5oXZVnqh-tXxyEbY-20YIvqOOWOuYlnldJHceTYDII9aJM-JhsA_Y9jkdCSOKZ0lwp_OhCWGFKEbfHu3NBJanFbms4EdBFUhNQH7K/%E6%AD%8C%E3%81%A3%E3%81%A1%E3%82%83%E7%8E%8B%20-%20unravel%20(%E5%8E%9F%E6%9B%B2%E6%AD%8C%E6%89%8B%3ATK%20from%20%E5%87%9B%E3%81%A8%E3%81%97%E3%81%A6%E6%99%82%E9%9B%A8)%5B%E3%82%AC%E3%82%A4%E3%83%89%E7%84%A1%E3%81%97%E3%82%AB%E3%83%A9%E3%82%AA%E3%82%B1%5D.mp3?extra=null
https://slider.kz//download/474499138_456433311/238/cs3-6v4/s/v1/acmp/CqmMRJm5B03c3so1ex1GshI705Z472EzHgAWm-M9eznXzjVF_6G-7lYlhkHdfPpHsWcVU6hWhLRqVr6WrYKBXcyvZ-ri0q5iV9Jv8Iwyd91t9cLu0sWVP1W3016tZKK0CSoXyfiL_oOfFhHbF9DbqmjmyUVO9ZB-McrQY6PlmOk4M-MBHg/PelleK%20-%20Unravel%20(Tokyo%20Ghoul).mp3?extra=null
https://slider.kz//download/-2001778472_74778472/238/cs9-2v4/s/v1/acmp/26oh4w3DDMsrZziue-jJNLoL_DKv7JMHYjXTixs0K4KjVjP0V6qTG2t5nmh6H9e8DVf3PH6Xoeqq1nzN718lXcKkSuutTFbWpgvA1nmV8M3DlQvm_WXyWRMReTYLwSwmwpALZ60-VbutNMFN3l8vPJ9F7lJCyi7V2ncQu9KxyWMCRD5C1Q/senakisaragi%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499220_456599167/246/psv4/s/v1/amp2/SX9vu47zGi9Uos8gQWm7XbfU1l0EKA1jqYblOj3ePK0pl5yPWLNFqITGYWgpzA1sRw6DzP_RAzgnu0C9-b2Ej5NDF4gU0px6dVSeOvO0DlKeW6XGcy2jFMJJCjHB_ag1VXEJgRb3jUYZLYRItNORbxOd45tRox8MGKBm/Matteo%20Leonetti%20-%20Unravel%20(Japanese%20Version)%20%5BTokyo%20Ghoul%5D.mp3?extra=null
https://slider.kz//download/-2001804556_66804556/238/cs3-5v4/s/v1/acmp/SGrWKb-wUFGqKV704k_bMGD7EJHh_A6sw2C9D8sWzWUA0R956jql_ZlVKEMv0MsJQ8gFsacMt_tywv3ZeXisu-rGTn4j_E8T3HS-o5RaqLzLOJauSlPoidthk8eIcx5xy4tGURb4KQkIfwIFJhriTy9HlnR7vCbaPpj_PfVNTBdlo7CU2g/V0RA%20-%20Unravel%20(Tokyo%20Ghoul%20OP).mp3?extra=null
https://slider.kz//download/474499291_456568508/102/cs9-6v4/s/v1/acmp/44hsI7dzI30x7Ho7N20vnYC06nWK2Sx-iXDPJXyN--Coele4vZUdekMfyNazmGRQ3IYMnLr1XBMPRWLVqZFkzHLk4fa_H5JN85IDDtft8EGPjsBRpSJOmbiAVAodQwQw3JcF6LW88ajmqKtJh3eUYARn9vrfQVN_J-ODVFY-dYm_akXUmg/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Fall).mp3?extra=null
https://slider.kz//download/-2001227120_98227120/186/psv4/s/v1/amp2/s57ppy-s9vm4n3qy-y4h8p4l0f5XyxK38-d91I3odmc9gvabMD_t42h8zd2oRL49tQkR1CD_vsPwIRA6DHsRUuW2WAkGJVwcef_VBmXUGsw7xEJSJG8kx-9AIXAUlj-8Ot32vMWg4TCiy/Olga%20Scheps%20-%20Start%20Anew%20(from%20%22Unravel%20Two%22).mp3?extra=null
https://slider.kz//download/474499232_456549884/90/psv4/s/v1/amp2/-eUtSpHonW9PPES9zCRzzNYbTSu7i_AEVujmMuifOKCuYme-q6vqFR5fXVHik2ARsaGX9_x4Za45VNNZohqsdkdCToDbne2W_6P7zgqnPJSuosGi8Z4ol0h0V_ViLH1ZIw_meOnJOkFDOTjKKyF-Ol0IvevzfvqPJtC0/Mary%20Antoine%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001667855_68667855/186/psv4/s/v1/amp2/IwKjMHOHdlAXWq1z1B5qxNBGhN8JVmJy-pJEl_b3RrTdZrGTPB_U0KIkNnnq7NcP5e1wa5ms9jSB1LIhEdiZTN6c7PHTMyJaLxNM8TDA46JpfBIXy73o11mf_FyBbAjkOIevfweKBRPukNSdConEqZr2ZUQoVaFeT7ZK/Track%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499146_456450407/186/psv4/s/v1/amp2/NMuGmiwJOxvdnOKQLkH0gIvMLbj_I_b5UIivo2Q2qVIxnI6kkJ5ITkC9rtL3H-28k3P6SzP-SEiegRZp_CyuXtuxPOA-qcx8Jo7VJEu817oIipyKTLFF9TBnMa6_E7w4Fa9uWM7RhR_J0VBAdPCDE4hi58gKNRBJ3bl5/Track%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499318_456592131/96/psv4/s/v1/amp2/zVQkmtvUwnxkcZxoMYAgcJ-RBoo8WvzKV6PyKAif3fw1C7fr-rzaWQbimUXK7_xSaKfsZpokZ7WRNdLOzxWm8QP8rlG2NNHYNQv6b2OmOiqOpP-WqeQ3VRvHN-0FtzcWnjYsIJZKcLcBA_q89U2c9Otw2Bqcvbt4lxRq/Vangakuz%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499173_456432829/137/cs3-1v4/s/v1/acmp/_LgsM2-WTNOvaV85Hy6ObbzZSKnJNfppdaWt9A_NyTv3XYhfSpyJ3QXFyQDb8nISAoRbTgNEt4yXtvvbWDli7gClVqaNCdiuTpEIwDl0EUZqyUF_aJrueZAjvd1DdUJR0YhvL6Fmv5C-8KoGEKotgiL1Eq1O4hU1wFDxayNVb4TkOzwwEQ/Kenzie%20Smith%20Piano%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001400485_76400485/187/psv4/s/v1/amp2/TFqk4JfH34feLtswrYZZXKfFUtIfXRTq8bOmXxLIHX99vXnkmPTp_A6-Rjj8fS9qiAQ6ao9F-VQvLZSH0bAnjBiUCM8pZbtBsL6z1v38vZzsc3kpP21MYG6kBCtfqg_jJhlj4CqDmfvQ0nW1NoUSNmutLvFy_7KTyjVG/Simpsonill%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001521828_57521828/551/cs3-6v4/s/v1/acmp/kXLXCQe33bHi2WDA_-0y9UGNCgbK7Xhca4q6pig_COLFf5bNaN11Z-4Kzd3A9tanob-U9_Jni_Ix5beoPUaJurH1ozqI6XQiKuHPmkxkLKRSXZpd_tj6SQhkniNDwxBxyRANOsxVfbHe80fcXahF0OWTKc4R5mjB1L1XVLsOhILcdpyS1g/Alveol%20-%20Unravel.mp3?extra=null
https://slider.kz//download/474499237_456436594/88/psv4/s/v1/amp2/yha2bLmL0XLvXvjfVKdqa25-JSAeZQYu8QQKRZKNwVAINm3912ZetkeVDrvSt5MeOyyto6x-PvbwC7hiBAXKC7g8_PwbcXKcqmku8hiHr7xJKl9ca34WIzeqNLyayp9RcFxGjO_Gk_AuaQ0ZBgXTaChq2nE4OTl1bqbX/MattxAJ%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001915683_101915683/90/psv4/s/v1/amp2/E79KiQTMA5b8mthMlpzYU5TRFcns8aqBNzqvKDFGJ14IR9Cz_C5yChPQUPQW-VFAm9_8sbo51D01NENhaegDJJIyui36C5ECVDwGaLhOcoLrhB8RKPLBGDLN01fcrb8mIxb7t0MhWr_YefPgj4Zt13xlufjor89n84eg/Music%20Legends%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499272_456394915/236/cs3-4v4/s/v1/acmp/2LiycWJnJntKnyikeZn_7TLKdSb8v343jFSIXZ9dbhcnsbSsjQZvlUI0jBkPWRGwGp5cK7UdprZarl3TUh_e_j_0vPrQs1SRHyogfbJlfQuM54Tr34iFNBasQLHCmZv1ktctdpEmuwNm4Yj2yi_1wLs9Iwqc9lBaZ-WmbDsfeeayQNwOLg/Shiroku%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001529426_101529426/255/cs3-2v4/s/v1/acmp/em6rSRlSNNcJHpGpiuSV4-VjdhJoAsemcgDXdS5VPu3XYPF1y-45PleMXDNAZ8jNEkf-5V9h7AInOtxaLBiP8KcWDvKW5d6EVy_hhh_qqK4PgpSPLS5Kv6yhdh7gUy2vo0t-ZVt-mZcQ1-8k48Q58_JE2U4JRD2ZgV8yVVbXKAK92bDnqg/HalcyonMusic%20-%20Unravel%20(from%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499176_456433059/241/psv4/s/v1/amp2/ATyS1Qxr7uBxM2KN4G0dMynLla7gPAaTh9nkDH5fMGYBEycOzy3sfM8LmLIjM2JP5yO3Vml0jBuTpeLoY9BBFCcWaD4qZ81dHjXsWFddJdyXqK0lkV4aKOo8EbMdyCJwRzeCesqn3r4aa_Pvxp3f6oPzLQgubvIXUF-s/I%20am%20Justice%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/-2001621142_58621142/68/psv4/s/v1/amp2/CL5J0kD4Xm1iSLFGqD7zrODdTqubYTFwGCzC5r1n3Z_Tu6G_68RtjH3HaEY_D8qHuckrW0C0bCfUD5fXz0gxfoJD5PtEGwPpi_wQUdGi-D7dJITBjyMxx6dWz5duvvqZ7dmmW7ydKBGGaxHPI9_tofxRB_qqQmop_I11/Gianni%20Tallone%20Della%20Cavallotta%20-%20Unravel.mp3?extra=null
https://slider.kz//download/-2001052300_100052300/89/cs9-11v4/s/v1/acmp/IvCOhCCclMK3tCeswQwVtNhtyYHX5lgjJUZfQsCfskiHciXLjBpj9SzjZMop3u-kPRSuyeyFvJoJEiP-2z_owq5MPdx4uNgrbl5Mqm0xbC_JWlpl4VH8Sc_vdNn28SHpU6EJiWDKgN7kf_sRozvO5OCEq7YmcfQe2lC_awFHsJV0KU7rHg/Fonzi%20M%20-%20Unravel%20(From%20%22Tokyo%20Ghoul%22).mp3?extra=null
https://slider.kz//download/474499157_456624310/212/psv4/s/v1/amp2/-oLu8sAswnvZ-I4RRXNqDu59CLZGoMUsv6qNVAL9QTVO9TcP4SWmGmUJSgfiXAw-_MgjRO1Y1y90YBuAirIcJgFenZe0XxlsqWrtbr6CEyQypX3sfmidTTBcH91G2x-4dXiDLI6JUmSXR9vZ4KQ_Z4vl-9o8SbiPmdD8/Love%2C%20Lies%20and%20Fiction%20-%20Unravel.mp3?extra=null
https://slider.kz//download/371745435_456449924/271/cs3-2v4/s/v1/acmp/ased-v3q4z-i_oiBH_OoVrZntdBkQGYKVQre0wr9DyNB7KVvbXTnpHh6G0_EyuBf90j3bW9SbB6aUeTt8NgAKKiuezgzETKKXzUhtpacQuKQEo-upEFNHLq5VzSrFAz2IjSSA3lm-p9plN7U4DQOPnCzn1_klu8Mje4uY3TCcwoTAuPckg/The%20Blue%20Notes%20-%20Unravel%20from%20%22tokyo%20Ghoul%22.mp3?extra=null
https://slider.kz//download/-2001260329_90260329/94/cs9-15v4/s/v1/acmp/qI5QjSZzHW-fYGXdVihZndt972zUCvmUN2Qzjk9gfUgyTjh_XPeOh9ctff9Ob2ezMEbFLEL7oNkrnvgoh4EJy9JlUjJPhTQS2Fh3L1cLzqJELD-waEz9vDaZwS_IuPXr0nH2PO5img_a4xkkqRLIy-VDm64EVAdGelBPmwLf7rIj_qjF9w/Piano%20Cat%2C%20Anime%20Cat%20-%20Unravel%20Tokyo%20Ghoul%20(Fall).mp3?extra=null
Using requests module from api
import requests
import json
headers={
'X-Requested-With': 'XMLHttpRequest'}
api_url='https://slider.kz/similar/artist/unravel'
req=requests.get(api_url,headers=headers).json()
for url in req['similarartists']['artist']:
print(url['url'])
Output:
https://www.last.fm/music/Contaminated
https://www.last.fm/music/Excarnate
https://www.last.fm/music/Extremity
https://www.last.fm/music/Serpent+Of+Gnosis
https://www.last.fm/music/Petrification
https://www.last.fm/music/Skullsmasher
https://www.last.fm/music/Hissing
https://www.last.fm/music/Mortuous
https://www.last.fm/music/Skelethal
https://www.last.fm/music/stabbed
https://www.last.fm/music/Winds+of+Leng
https://www.last.fm/music/Usurpress
https://www.last.fm/music/Encoffinized
https://www.last.fm/music/Glacial+Tomb
https://www.last.fm/music/Overlord
https://www.last.fm/music/RESIN+TOMB
https://www.last.fm/music/Descent
https://www.last.fm/music/Hurricane+Death
https://www.last.fm/music/Gutter+Instinct
https://www.last.fm/music/Crypt+Crawler
https://www.last.fm/music/This+Game
https://www.last.fm/music/LIFETAKER
https://www.last.fm/music/Autokrator
https://www.last.fm/music/Vanhelgd
https://www.last.fm/music/ATARAXY
Related
Looking to Pass Python a list then using a combinate of Beautiful soup and requests, pull the corresponding peice of information for each web page.
So i have a list of around 7000 barcodes that i want to pass to this site 'https://www.barcodelookup.com/' (you just add the barcode after the backslash), then pull back the manufacturer of that product which is in the span "product-text".I'm currently trying to get it to run with the below;
from bs4 import BeautifulSoup
import requests
source = requests.get('https://www.barcodelookup.com/194398882321')
soup = BeautifulSoup(source, 'lxml')
#print(soup.prettify())
price = soup.find('span', {'class' : 'product-text'})
print(price.text)
This gives an error as below;
TypeError: object of type 'Response' has no len()
Any help would be greatly appreciated, thanks
If you inspect the source, you will see that the response status is 403 and the overall source.text reveals that the website is protected by Cloudflare. This means that using requests is not really helpful for you. You need the means to overcome the 'antibot' protection from Cloudflare. Here are two options:
1. Use a third party service
I am an engineer at WebScrapingAPI and I can recommend you our web scraping API. We're preventing detection by using various proxies, IP rotations, captcha solvers and other advanced features. A basic example of using our API for your scenarios is:
import requests
API_KEY = '<YOUR_API_KEY>'
SCRAPER_URL = 'https://api.webscrapingapi.com/v1'
TARGET_URL = 'https://www.barcodelookup.com/194398882321'
PARAMS = {
"api_key":API_KEY,
"url": TARGET_URL,
"render_js":1,
"timeout":"20000",
"proxy_type": "residential",
"extract_rules":'{"elements":{"selector":"span.product-text","output":"text"}}',
}
response = requests.get(SCRAPER_URL, params=PARAMS )
print(response.text)
Response:
{"elements":["\nUPC-A 194398882321, EAN-13 0194398882321\n","Media ","Sony Uk ","\n1-2-3: The 80s CD.\n"]}
2. Build an undetectable web scraper
You can also try building a more 'undetectable' web scraper on your end. For example, try using a real browser for your scraper, instead of requests. Selenium would be a good place to start. Here is an implementation example:
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
BASE_URL = 'https://www.barcodelookup.com/194398882321'
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 5000)
driver.get(BASE_URL)
html = driver.page_source
soup = BeautifulSoup(html, 'html.parser')
price = soup.find('span', {'class' : 'product-text'})
print(price)
driver.quit()
In time though, Cloudflare might flag your 'fingerprint' and block your requests. Some more things you could add to your project are:
Residential proxies
Advanced Selenium Evasions
I am trying to download multiple csv files from the below url and hoping to use selenium or any other method for it. The url requires filling up of a form that includes selecting options from multiple dropdowns. Then, an 'image' button needs to be clicked for the download link to appear.
If I run selenium chrome driver from python and click on the button, nothing appears. I am also unable to figure out the url of the csv files so they could be downloaded using 'requests' or 'urllib'.
Here's the url I need to download from:
https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm
Here's my code so far:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
url = 'https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm'
driver.get(url)
instr_type = Select(driver.find_element_by_id('instrumentType'))
symbol = Select(driver.find_element_by_id('symbol'))
opt_type = Select(driver.find_element_by_id('optionType'))
date_range = Select(driver.find_element_by_id('dateRange'))
button = driver.find_element_by_xpath("//input[#src='/common/images/btn-get-data.gif' and #type='image']")
instr_type.select_by_visible_text('Index Options')
symbol.select_by_visible_text('NIFTY 50')
opt_type.select_by_visible_text('CE')
date_range.select_by_visible_text('90 Days')
button.click()
And this is what happens in the selenium driver -
Any thoughts on how to download the csv files from above link? Doesn't necessarily have to be using selenium.
I don't know how to resolve problem with Selenium but I now how to get it with requests and BeautifulSoup
This page sends your options from form to this page as values directly in this URL (not as POST)
https://www1.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp
and server sends back HTML with table and also all data in <div id="csvContentDiv">.
This tag has all data as text already formatted to save in csv - it need only to replace : with \n
EDIT:
Sometimes server gives me Status 405 Method Not Allowed so I added requests.Session() to get cookies and maybe it will work better.
import requests
from bs4 import BeautifulSoup
session = requests.Session()
session.headers.update({'User-Agent': 'Mozilla/5.0'})
# --- main page ---
url = 'https://www1.nseindia.com/products/content/derivatives/equities/historical_fo.htm'
r = session.get(url)
#print(r.status_code)
# --- table ---
url = 'https://www1.nseindia.com/products/dynaContent/common/productsSymbolMapping.jsp'
payload = {
'instrumentType': 'OPTIDX',
'symbol': 'NIFTY',
'expiryDate': 'select',
'optionType': 'CE',
'strikePrice': '',
'dateRange': '3month',
'fromDate': '',
'toDate': '',
'segmentLink': '9',
'symbolCount':'',
}
r = session.get(url, params=payload)
#print(r.text)
soup = BeautifulSoup(r.text, 'html.parser')
data = soup.find('div', {'id': 'csvContentDiv'})
#print(data.text)
data = data.text.replace(':', '\n')
with open('output.csv', 'w') as fh:
fh.write(data)
print(data)
Server doesn't send data if you don't use real User-Agent - at least short 'Mozilla/5.0'
I found this url using DevTools in Firefox/Chrome, tab Network. And later I get response from this url and manually check what I get and I found csv data. But I expected that I would have to scrape data from HTML table in this response.
from bs4 import BeautifulSoup
import requests
import time
from selenium import webdriver
driver = webdriver.Chrome(r'C:\chromedriver.exe')
url ='https://www.sambav.com/hyderabad/doctors'
driver.get(url)
soup = BeautifulSoup(driver.page_source,'html.parser')
for links in soup.find_all('div',class_='sambavdoctorname'):
link = links.find('a')
print(link['href'])
driver.close()
I am trying to scrape this page, the link is same in all pages. I am trying to extract the links from all mutiple pages but it's not giving any output nor showing any error just the program gets end.
If you check that website by developer tools in browser ( chrome or mozilla or whatever), before loading website, the website fetch data from few sources. One of this sources is "https://www.sambav.com/api/search/DoctorSearch?searchText=&city=Hyderabad&location=" . Your code could be simplified (and there is no need to use selenium):
import requests
r = requests.get('https://www.sambav.com/api/search/DoctorSearch?searchText=&city=Hyderabad&location=')
BASE_URL_DOCTOR = 'https://www.sambav.com/hyderabad/doctor/'
for item in r.json():
print(BASE_URL_DOCTOR + item['uniqueName'])
I have a page that has a table (table id= "ctl00_ContentPlaceHolder_ctl00_ctl00_GV" class="GridListings" )i need to scrape.
I usually use BeautifulSoup & urllib for it,but in this case the problem is that the table takes some time to load ,so it isnt captured when i try to fetch it using BS.
I cannot use PyQt4,drysracpe or windmill because of some installation issues,so the only possible way is to use Selenium/PhantomJS
I tried the following,still no success:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.PhantomJS()
driver.get(url)
wait = WebDriverWait(driver, 10)
table = wait.until(EC.presence_of_element_located(By.CSS_SELECTOR, 'table#ctl00_ContentPlaceHolder_ctl00_ctl00_GV'))
The above code doesnt give me the desired contents of the table.
How do i go about achieveing this???
You can get the data using requests and bs4,, with almost if not all asp sites there are a few post params that always need to be provided like __EVENTTARGET, __EVENTVALIDATION etc.. :
from bs4 import BeautifulSoup
import requests
data = {"__EVENTTARGET": "ctl00$ContentPlaceHolder$ctl00$ctl00$RadAjaxPanel_GV",
"__EVENTARGUMENT": "LISTINGS;0",
"ctl00$ContentPlaceHolder$ctl00$ctl00$ctl00$hdnProductID": "139",
"ctl00$ContentPlaceHolder$ctl00$ctl00$hdnProductID": "139",
"ctl00$ContentPlaceHolder$ctl00$ctl00$drpSortField": "Listing Number",
"ctl00$ContentPlaceHolder$ctl00$ctl00$drpSortDirection": "A-Z, Low-High",
"__ASYNCPOST": "true"}
And for the actual post, we need to add a few more values to out post data:
post = "https://seahawks.strmarketplace.com/Charter-Seat-Licenses/Charter-Seat-Licenses.aspx"
with requests.Session() as s:
s.headers.update({"User-Agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0"})
soup = BeautifulSoup(s.get(post).content)
data["__VIEWSTATEGENERATOR"] = soup.select_one("#__VIEWSTATEGENERATOR")["value"]
data["__EVENTVALIDATION"] = soup.select_one("#__EVENTVALIDATION")["value"]
data["__VIEWSTATE"] = soup.select_one("#__VIEWSTATE")["value"]
r = s.post(post, data=data)
soup2 = BeautifulSoup(r.content)
table = soup2.select_one("div.GridListings")
print(table)
You will see the table printed when you run the code.
If you want to scrap something, it will be nice first to install a web debugger ( Firebug for Mozilla Firefox for example) to watch how the website you want to scrap is working.
Next, you need to copy the process of how the website is connecting to backoffice
As you said, the content that you want to scrap is being loaded asynchronously (only when the document is ready)
Assuming the debugger is running and also you have refreshed the page, you will see on the network tab the following request:
POST https://seahawks.strmarketplace.com/Charter-Seat-Licenses/Charter-Seat-Licenses.aspx
The final process flow to reach your goal will be:
1/ Use requests python module
2/ Open a requests session to the index page website site (with cookies handling)
3/ Scrap all the input for the specific POST form request
4/ Build a POST parameter DICT containing all inputs & value fields scrapped in the previous step + adding some specific fixed params.
5/ POST the request (with required data)
6/ Use finally BS4 module (as usual) to soup the answered html to scrap your data
Please see bellow a working code:
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
from bs4 import BeautifulSoup
import requests
base_url="https://seahawks.strmarketplace.com/Charter-Seat-Licenses/Charter-Seat-Licenses.aspx"
#create requests session
s = requests.session()
#get index page
r=s.get(base_url)
#soup page
bs=BeautifulSoup(r.text)
#extract FORM html
form_soup= bs.find('form',{'name':'aspnetForm'})
#extracting all inputs
input_div = form_soup.findAll("input")
#build the data parameters for POST request
#we add some required <fixed> data parameters for post
data={
'__EVENTARGUMENT':'LISTINGS;0',
'__EVENTTARGET':'ctl00$ContentPlaceHolder$ctl00$ctl00$RadAjaxPanel_GV',
'__EVENTVALIDATION':'/wEWGwKis6fzCQLDnJnSDwLq4+CbDwK9jryHBQLrmcucCgL56enHAwLRrPHhCgKDk6P+CwL1/aWtDQLm0q+gCALRvI2QDAKch7HjBAKWqJHWBAKil5XsDQK58IbPAwLO3dKwCwL6uJOtBgLYnd3qBgKyp7zmBAKQyTBQK9qYAXAoieq54JAuG/rDkC1djKyQMC1qnUtgoC0OjaygUCv4b7sAhfkEODRvsa3noPfz2kMsxhAwlX3Q=='
}
#we add some <dynamic> data parameters
for input_d in input_div:
try:
data[ input_d['name'] ] =input_d['value']
except:
pass #skip unused input field
#post request
r2=s.post(base_url,data=data)
#write the result
with open("post_result.html","w") as f:
f.write(r2.text.encode('utf8'))
Now, please get a look at "post_result.html" content and you will find the data !
Regards
I'm trying to collect data from the SumofUs website; specifically the number of signatures on the petition. The datum is presented like this: <div class="percent">256,485 </div> (this is the only item of this class on the Page.)
So I tried this:
import requests
from bs4 import BeautifulSoup
user_agent = {'User-agent': 'Mozilla/5.0'}
url = 'http://action.sumofus.org/a/nhs-patient-corporations/'
raw = requests.get(url, headers = user_agent)
html = BeautifulSoup(raw.text)
# get the item we're seeking
number = html.find("div", class_="percent")
print number
It seems that the number isn't rendered (I've tried a couple of user agent strings.) What else could be causing this? How can I work around this in future?
In the general case you should use a headless browser. Ghost.py is written in python so its probably a good choice to try first.
In this specific case a little research reveals that there's a much simpler method. By using the network tab in chrome you can see that the site makes an ajax call to populate the value. So you can just get it directly:
url = "http://action.sumofus.org/api/ak_action_count_by_action/?action=nhs-patient-corporations&additional="
number = int(requests.get(url).text)
You could use Selenium:
from selenium import webdriver
from bs4 import BeautifulSoup
url = 'http://action.sumofus.org/a/nhs-patient-corporations/'
driver = webdriver.Firefox()
driver.get(url)
driver.set_window_position(0, 0)
driver.set_window_size(100000, 200000)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(5) # wait to load
# then load BeautifulSoup with browsers content
html = BeautifulSoup(driver.page_source)
...