Hello I have a selenium script in python which extract data with login on webpage. It take around 50 sec to execute and I want to deploy that script as an api. But API is getting timeout.
There we can also do one thing that we save that data in some google sheet using that script.
Please anyone can suggest how can i do this or any relevant content ?
Could you provide us a screenshot of API timeout or logs? Showing Python code with requests will be also helpful (sorry for answering instead of commenting because I don't have enough reputation points)
Related
I have a problem getting javascript content into HTML to use it for scripting. I used multiple methods as phantomjs or python QT library and they all get most of the content in nicely but the problem is that there are javascript buttons inside the page like this:
Pls see screenshot here
Now when I load this page from a script these buttons won't default to any value so I am getting back 0 for all SELL/NEUTRAL/BUY values below. Is there a way to set these values when you load the page from a script?
Example page with all the values is: https://www.tradingview.com/symbols/NEBLBTC/technicals/
Any help would be greatly appreciated.
If you are trying to achieve this with scrapy or with derivation of cURL or urrlib I am afraid that you can't do this. Python has another external packages such selenium that allow you to interact with the javascript of the page, but the problem with selenium is too slow, if you want something similar to scrapy you could check how the site works (as i can see it works through ajax or websockets) and fetch the info that you want through urllib, like you would do with an API.
Please let me know if you understand me or i misunderstood your question
I used seleneum which was perfect for this job, it is indeed slow but fits my purpose. I also used the seleneum firefox plugin to generate the python script as it was very challenging to find where exactly in the code as the button I had to press.
I want to webscraping from a website, where i have to log in first. The problem is that, there is a "robotprotection" too (so I have to verify that i am not a robot + a recaptcha-security.), but it's chances of success (passing the captcha) is ~30% and this is horrible for me.
There is another possibility maybe which one i am log in with my browser (for example chrome or firefox), and after im going to use this session ID in my python script to webscraping dataes automatically?
So, more simplier: I want to webscraping tables from a website, so i have to log in first. This 30% succes rate is not enough good for me, so i hope there is another possibilty : log in manually, and after use this session in python?!
After that, there is a textbox in this page, where i want to write what i want to search, and after it is navigate to the page, where i'll found the table and dataes.
Any ideas, or it is possible?
(now i have only a script which one i have to download the html code to this datapage, and after change some name in the code manually..it is a very big waste time, i hope i can automate it more.) - Python 2.7
I have been trying to download past broadcasts for a streamer on twitch using python. I found this python code online:
https://gist.github.com/baderj/8340312
However, when I try to call the functions I am getting errors giving me a status 400 message.
Unsure if this is the code I want to download the video (as an mp4) or how to use it properly.
And by video I mean something like this as an example: www(dot)twitch.tv/imaqtpie/v/108909385 //note cant put more than 3 links since I don't have 10 reputation
Any tips on how i should go about doing this?
Here's an example of running it in cmd:
python twitch_past_broadcast_downloader.py 108909385
After running it, it gave me this:
Exception API returned 400
This is where i got the information on running it:
https://www.johannesbader.ch/2014/01/find-video-url-of-twitch-tv-live-streams-or-past-broadcasts/
Huh it's not as easy at it seems ... The code you found on this gist is quite old and Twitch has completely changed its API. Now you will need a Client ID to download videos, in order to limit the amount of video you're downloading.
If you want to correct this gist, here are simple steps you can do :
Register an application : Everything is explained here ! Register you app and keep closely your client id.
Change API route : It is no longer '{base}/api/videos/a{id_}' but {base}/kraken/videos/{id_} (not sure about the last one). You will need to change it inside the python code. The doc is here.
Add the client id to the url : As said in the doc, you need to give a header to the request you make, so add a Client-ID: <client_id> header in the request.
And now I think you will need to start debugging a bit, because it is old code :/
I will try myself to do it and I will edit this answer when I'm finished, but try yourself :)
See ya !
EDIT : Mhhh ... It doesn't seem to be possible anyway to download a video with the API :/ I was thinking only links to API changed, but the chunks section of the response from the video url disappeared and Twitch is no longer giving access to raw videos :/
Really sorry I told you to do that, even with the API I think is no longer possible :/
You can download past broadcasts of Twitch videos with Python library streamlink.
You will need OAuth token which you can generate with the command
streamlink --twitch-oauth-authenticate
Download VODs with:
streamlink --twitch-oauth-token <your-oauth-token> https://www.twitch.tv/videos/<VideoID> best -o <your-output-folder>
I have written my scripts in Selenium Python. In the Jenkins, I have added Publish JUnit test results report(http://nose.readthedocs.org/en/latest/plugins/xunit.html). However, I want to add the screenshot in that report as well. I feel like my tests are not useful without screenshots. How can I achieve this? I also would like receive error screenshots as attachments in my email.
Can I achieve this just by using a Jenkins plugin? or I need to add something in my selenium script? I am not able to find anything useful in selenium python.
I went over the following question, but it does explain the full process
Adding a png image and table html while sending email in Jenkins
Please help.....
I was trying to make an image uploading script for Postimage.org.
I tried searching for an API but it seems that there is not any available. Can anyone help me how to make this script ? I don't have any idea how to make the uploading proccess? I think that something that i should do is open the image file in read binary mode ("rb").
Anyway i am waiting for your suggestions and ideas.
Firstly you should think about what happens when you press the upload button on the website. What your script could do is mimic this functionality, because essentially all it's triggering is a POST request to the web server with the specified information in the form and the image file data. You can initiate HTTP requests (e.g. GET, POST, etc.) using a library such as Requests (http://docs.python-requests.org/en/latest/index.html).
However, as this seems to have been discussed before, I will instead point you in the right direction: Send file using POST from a Python script