Is there a possibility to retrieve access token just from script in aws lambda without user interaction in the next way? =>
url = f'https://{user}:{password}#api.stocktwits.com/api/2/oauth/authorize?client_id={client_id}&response_type=token&redirect_uri={redirect_uri}&scope=publish_messages'
response = requests.get(
url, allow_redirects=True
)
For this moment I only retrieve redirect to => https://api.stocktwits.com/api/2/oauth/signin?
There is a plethora of solutions for this question. I'm assuming you are redirecting to an oauth page where it's requesting a sign in form? If that is the case one way you could do it is drive that page with selenium and have it fill in the credentials for you then you will be able to retrieve the access token.
Related
I started recently working on API's and few API providers are not requesting redirect URL while some others are requesting. I have written an algorithmic strategy for trading using python. When I requested for API to Fyers(stockbroker), the team said me to provide a redirect URL. what is a redirect URL? and how to create it?
I have attached image for reference. In the above image, there is a text box for Redirect URL. Can you please explain what exactly is Redirect URL and how to create one for calling API for authentication if my code is on heroku?
The Redirect URL is required by the oAuth workflow: basically the authorisation server will redirect the user back to the URL registered as "Redirect URL" including an authorization code or a token.
If you register a URL like https://myapp.herokuapp.com you will be redirected to
https://myapp.herokuapp.com?access_code=XXX&app_id=YYY
The Redirect URL needs to be a valid accessible page: if the process is manual you just copy the access_code from the browser and use it accordingly.
If it is an application you need to receive the redirect above (the URL is basically your app), fetch the required information (parameters) and implement your logic.
Default Fyers Redirect URL for Testing
Use the default url from fyers
https://trade.fyers.in/api-login/redirect-uri/index.html
Copy the auth key value
Use it in your python app in the second run
You can also use google collab, to run part of code only (authentication) without restarting the whole project
I am having trouble creating and keeping new sessions when I am scraping my page. I am initiating a session within my script using the Requests library and then parsing values to a web form. However, it's is returning a "Your session has timed out" page.
Here is my source:
import requests
session = requests.Session()
params = {'Rctl00$ContentPlaceHolder1$txtName': 'Andrew'}
r = session.post("https://www.searchiqs.com/NYALB/SearchResultsMP.aspx", data=params)
print(r.text)
The url I want to search from is this https://www.searchiqs.com/NYALB/SearchAdvancedMP.aspx
I am searching for a Party 1 name called "Andrew". I have identified the form element holding this search box as 'Rctl00$ContentPlaceHolder1$txtName'. The action url is SearchResultsMP.aspx.
When i do it from a browser, it gives the first page of results. When i do it in the terminal it gives me the session expired page. Any ideas?
First, I would refer you to the advanced documentation related to use of sessions within the requests Python module.
http://docs.python-requests.org/en/master/user/advanced/
I also notice that navigating to the base URL in your invocation of sessions.post redirects to:
https://www.searchiqs.com/NYALB/InvalidLogin.aspx?InvLogInCode=OldSession%2007/24/2016%2004:19:37%20AM
I "hacked" the URL to navigate to:
https://www.searchiqs.com/NYALB/
...and notice that if I click on the Show Login Fields link on that page, I am prompted a form appears with prompts for User ID and Password. Your attempts to programmatically do your searches are likely failing because you have not done any sorts of authentication. It likely works in your browser because you have been permitted to access this, either by some previous authentication you have completed and may have forgotten about, or some sort of server side access rules that don't ask for this based upon some criteria.
Running those commands in a local interpreter, I can see that the site owner did not bother to return a status code indicative of failed auth. If you check, the r.status_code is 200 but your r.text will be the Invalid Login page. I know nada about ASP, but am guessing that HTTP status codes should be indicative of what actually happened.
Here is some code, that does not really work, but may illustrate how you may want to interact with the site and sessions.
import requests
# Create dicts with our login and search data
login_params = {'btnGuestLogin': 'Log+In+as+GUEST'}
search_params = {'ctl00$ContentPlaceHolder1$txtName': 'Andrew'}
full_params = {'btnGuestLogin': 'Log+In+as+GUEST', 'ctl00$ContentPlaceHolder1$txtName': 'Andrew'}
# Create session and add login params
albany_session = requests.session()
albany_session.params = login_params
# Login and confirm login via searching for the 'ASP.NET_SessionId' cookie.
# Use the login page, not the search page first.
albany_session.post('https://www.searchiqs.com/NYALB/LogIn.aspx')
print(albany_session.cookies)
# Prepare a your search request
search_req = requests.Request('POST', 'https://www.searchiqs.com/NYALB/SearchAdvancedMP.aspx',data=search_params)
prepped_search_req = albany_session.prepare_request(search_req)
# Probably should work but does not seem to, for "reasons" unknown to me.
search_response = albany_session.send(prepped_search_req)
print(search_response.text)
An alternative may be for you to consider is Selenium browser automation with Python bindings.
http://selenium-python.readthedocs.io/
i'm trying to retrieve the page content from https://www.awesomebox.io/scan
But before I can do that need to be logged in. At the moment I still get the login page content. Thats because it redirects because im not logged in.
Anybody know how to get the scan page content with python-requests?
I tried multiple requests authentication methods.
My code so far:
import requests
session = requests.session()
loginURL = 'http://www.awesomebox.io/login'
payload = {'username': '******','password': '******'}
session.post(loginURL, data=payload)
scanURL = "http://awesomebox.io/scan"
scanpage = session.get(scanURL)
print scanpage.content
I don't have an account with awesomebox, so therefore don't know exactly. But nowadays a login on websites is more sophisticated and secure than a simple post of username and password.
To find out, you can do a manual login and trace the web traffic in the developer mode of the browser (e.g. F12 for MSIE or Edge) and store it in a .har file. There you can (hopefully) see, how the Login procedure is implemented and build the same sequence in your requests session.
Sometimes there is a hidden field in the form (e.g. "lt" for login ticket) that has been populated via js by the page before. Sometimes it's even more complex, if a secret login in run via Ajax in the Background. In this case you even see nothing in the F12 view and have to dig into the js scripts.
Thank you, I noticed i forgot a hidden parameter.
I added the csrfmiddlewaretoken.
I want to use facebook api for which I will be needing oauth token, so when the program starts the python program will open the the authentication url by webbrowser.open() method after this the user will will give permission and then facebook will generate access token and redirect to a different link. I need to grab this redirected link and retrieve the access token. How do I grab this redirected url.
Afaik there is no simple way to do this (see Python - Getting url a browser was redirected to)
If you have a WebServer set the redirect page to your server and the get Tokens from there.
Otherwise use the python oauth module.
If you have a fb-app (with id and token) you can do it locally, althrough its not a very good/safe way..
I am trying to automate files download via a webserver. I plan on using wget or curl or python urllib / urllib2.
Most solutions use wget and urllib and urllib2. They all talk of HHTP based authentication and cookie based authentication. My problem is I dont know which one is used in the website that stores my data.
Here is the interaction with the site:
Normally I login to site http://www.anysite.com/index.cgi?
I get a form with a login and password. I type in both and hit return.
The url stays as http://www.anysite.com/index.cgi? during the entire interaction. But now I have a list of folders and files
If I click on a folder or file the URL changes to http://shamrockstructures.com/cgi-bin/index.cgi?page=download&file=%2Fhome%2Fjanysite%2Fpublic_html%2Fuser_data%2Fuserareas%2Ffile.tar.bz2
And the browser offers me a chance to save the file
I want to know how to figure out whether the site is using HTTP or cookie based authentication. After which I am assuming I can use cookielib or urllib2 in python to connect to it, get the list of files and folders and recursively download everything while staying connected.
p.S: I have tried the cookie cutter ways to connect via wget and wget --http-user "uname" --http-password "passwd" http://www.anysite.com/index.cgi? , but they only return the web form back to me.
If you log in using a Web page, the site is probably using cookie-based authentication. (It could technically use HTTP basic auth, by embedding your credentials in the URI, but this would be a dumb thing to do in most cases.) If you get a separate, smallish dialog with a user name and password field (like this one), it is using HTTP basic authentication.
If you try to log in using HTTP basic auth, and get back the login page, as is happening to you, this is a certain indication that the site is not using HTTP basic auth.
Most sites use cookie-based authentication these days. To do this with an HTTP cilent such as urllib2, you will need to do an HTTP POST of the fields in the login form. (You may need to actually request the login form first, as a site could include a cookie that you need to even log in, but usually this is not necessary.) This should return a "successfully logged in" page that you can test for. Save the cookies you get back from this request. When making the next request, include these cookies. Each request you make may respond with cookies, and you need to save those and send them again with the next request.
urllib2 has a function called a "cookie jar" which will automatically handle the cookies for you as you send requests and receive Web pages. That's what you want.
You can use pycurl like this:
import pycurl
COOKIE_JAR = 'cookiejar' # file to store the cookies
LOGIN_URL = 'http://www.yoursite.com/login.cgi'
USER_FIELD = 'user' # Name of the element in the HTML form
USER = 'joe'
PASSWD_FIELD = 'passwd' # Name of the element in the HTML form
PASSWD = 'MySecretPassword'
def read(html):
"""Read the body of the response, with posible
future html parsing and re-requesting"""
print html
com = pycurl.Curl()
com.setopt(pycurl.WRITEFUNCTION, read)
com.setopt(pycurl.COOKIEJAR, COOKIE_JAR)
com.setopt(pycurl.FOLLOWLOCATION, 1) # follow redirects
com.setopt(pycurl.POST, 1)
com.setopt(pycurl.POSTFIELDS, '%s=%s;%s=%s'%(USER_FIELD, USER,
PASSWD_FIELD, PASSWD))
com.setopt(pycurl.URL, LOGIN_URL )
com.perform()
Plain pycurl it may seam very "primitive" (with the limited setopt approach),
but it gets the job done, and handle pretty well the cookies with the cookie jar option.
AFAIK cookie based authentication is only used once you have logged in successfully atleast ONCE. You can try disabling storing cookies from that domain by changing your browser settings, if you are still able to download files that it should be a HTTP based authentication.
Try doing a equivalent GET request for the (possibly POST) login request that is probably happening right now for login. Use firebug or fiddler to see the login request that is sent.
Also note if there is some javascript code which is returning you a different output, based on your useragent string or some other parameter.
See if httplib, mechanize helps.