We Keep Coding

Authorization error 403 with Twitter API using python [duplicate]

I attempted to run the code below and am getting an error that states:
HTTP Error code: 403: Forbidden: Authentication succeeded but account is not authorized to access this resource.
from searchtweets import ResultStream, gen_rule_payload, load_credentials, collect_results
import requests
premium_search_args = load_credentials("/home/dirname/twitter_keys.yaml",
yaml_key="search_tweets_premium",
env_overwrite=False)
rule = gen_rule_payload("basketball", results_per_call=100) # testing with a sandbox account
print(rule)
from searchtweets import collect_results
tweets = collect_results(rule,
max_results=100,
result_stream_args=premium_search_args)
# print(tweets.all_text)
[print(tweet.all_text, end='\n\n') for tweet in tweets[0:10]];
My YAML file looks like this:
search_tweets_premium:
account_type: premium
endpoint: https://api.twitter.com/1.1/tweets/search/fullarchive/dev.json
consumer_key: AAAAAAAAAAAAAAAAAAAAA
consumer_secret: BBBBBBBBBBBBBBBBBBBBBBBBBBB
Only other thing to note is that I am using the free/sandbox service.
Any ideas if I am doing anything wrong in the code, the YAML, and/or within my Twitter developer account?

You'll need to go to https://developer.twitter.com/en/account/environments
There you should be able to see the various development environments that you have. You can create one should they not have been created.
The dev environment label would then be the thing you use to replace in your endpoint.
In my example, it would be:
https://api.twitter.com/1.1/tweets/search/fullarchive/development.json
If that still doesn't work, you might need to include a bearer token in your YAML file.

How to serialize an OAuth1Session?

I currently have a monolithic Python script which performs an OAuth authentication, returning an OAuth1Session, and then proceeds to perform some business logic using that OAuth1Session to gain authorization to a third-party service.
I need to split this up into two separate scripts, one which performs the OAuth authentication and will run on one machine, and the other which will run on a remote machine to perform the business logic authorized against the third-party service.
How can I serialize the OAuth1Session object so that the authenticated tokens can be handed off seamlessly from the authentication script on machine A to the processing script on machine B?
I tried the obvious:
print(json.dumps(session))
But I got this error:
TypeError: Object of type OAuth1Session is not JSON serializable
Is there a canonical solution for this simple requirement?
UPDATE
Here's the entire source code. Please note this is not my code, I downloaded it from the author and now I'm trying to modify it to work a bit differently.
"""This Python script provides examples on using the E*TRADE API endpoints"""
from __future__ import print_function
import webbrowser
import json
import logging
import configparser
import sys
import requests
from rauth import OAuth1Service
def oauth():
"""Allows user authorization for the sample application with OAuth 1"""
etrade = OAuth1Service(
name="etrade",
consumer_key=config["DEFAULT"]["CONSUMER_KEY"],
consumer_secret=config["DEFAULT"]["CONSUMER_SECRET"],
request_token_url="https://api.etrade.com/oauth/request_token",
access_token_url="https://api.etrade.com/oauth/access_token",
authorize_url="https://us.etrade.com/e/t/etws/authorize?key={}&token={}",
base_url="https://api.etrade.com")
base_url = config["DEFAULT"]["PROD_BASE_URL"]
# Step 1: Get OAuth 1 request token and secret
request_token, request_token_secret = etrade.get_request_token(
params={"oauth_callback": "oob", "format": "json"})
# Step 2: Go through the authentication flow. Login to E*TRADE.
# After you login, the page will provide a text code to enter.
authorize_url = etrade.authorize_url.format(etrade.consumer_key, request_token)
webbrowser.open(authorize_url)
text_code = input("Please accept agreement and enter text code from browser: ")
# Step 3: Exchange the authorized request token for an authenticated OAuth 1 session
session = etrade.get_auth_session(request_token,
request_token_secret,
params={"oauth_verifier": text_code})
return(session, base_url)
# loading configuration file
config = configparser.ConfigParser()
config.read(sys.argv[1])
(session, base_url) = oauth()
print(base_url)
print(json.dumps(session))
#original code
#market = Market(session, base_url)
#quotes = market.quotes(sys.argv[2])
Please note the last two commented-out lines. That is the original code: Immediate after the oauth is performed, the code invokes some business functionality. I want to break this up into two separate scripts running as isolated processes: Script 1 performs the oauth and persists the session, Script 2 reads the session from a file and performs the business functionality.
Unfortunately it fails at the last line, print(json.dumps(session)).
"XY Problem" Alert
My goal is to split up the script into two so that the business logic can run in a separate machine from the authentication code. I believe that the way to do this is to serialize the session object and then parse it back in the second script. Printing out the session using json.dumps() is an intermediate step, 'Y', in my journey to solving problem 'X'. If you can think of a better way to achieve the goal, that could be a valid answer.

From the comments available in the source code here: https://github.com/litl/rauth/blob/a6d887d7737cf21ec896a8104f25c2754c694011/rauth/session.py
You only need to serialize some attributes of your object to reinstantiate it:
Line 103
def __init__(self,
consumer_key,
consumer_secret,
access_token=None,
access_token_secret=None,
signature=None,
service=None):
...
Thus I would suggest serializing the following dict on the first machine:
info_to_serialize = {
'consumer_key': session.consumer_key,
'consumer_secret': session.consumer_secret,
'access_token': session.access_token,
'access_token_secret': session.access_token_secret
}
serialized_data = json.dumps(info_to_serialize)
And on the second machine reinstantiate your session like that:
from rauth.session import OAuth1Session
info_deserialized = json.loads(serialized_data)
session = OAuth1Session(**info_deserialized)
Hope this helped

Trying to Scrape Reddit with praw.Reddit

Im trying to scrape Reddit with the praw.reddit command and I keep getting the following:
prawcore.exceptions.OAuthException: unauthorized_client error processing
request (Only script apps may use password auth)
Heres the top of my code:(I removed the sensitive items)
import praw
import pandas as pd
import datetime as dt
reddit = praw.Reddit(client_id='zlpcoz08aNK8Bw', \
client_secret='', \
user_agent='comment_scraper 1.0 by /u/bullybear77777',
\
username='', \
password='')
I think it's because of my user_agent ID? I looked online and found that this appears to be the structure but im not sure. Any helps here would be greatly appreciated

This sort of error is caused by the type of app associated with that client id. Logging in with a password is restricted to script type apps.
When you create a new application, there are three types of apps to choose from:
web app: A web based application
installed app: An app intended for installation, such as on a mobile phone
script: Script for personal use. Will only have access to the developers accounts
If the application has the web app or installed app types then this form of authentication can't be used. You can't change the app type once it's created, but you can simply create a new one with a script type.

Accesing ASANA data using python requests

First of all, I'm not a Python guru as you can probably tell... So here we go.
I'm trying to use Asana's API to pull data with Python requests (Projects, tasks, etc) and doing the authentication using Oauth 2.0... I've been trying to find a simple python script to have something to begin with but I haven't had any luck and I can't find a decent and simple example!
I already created the app and got my client_secret and client_secret. But I don't really know where or how to start... Could anybody help me please?
import sys, os, requests
sys.path.append(os.path.dirname(os.path.dirname(__file__)))
import asana
import json
from six import print_
import requests_oauthlib
from requests_oauthlib import OAuth2Session
client_id=os.environ['ASANA_CLIENT_ID'],
client_secret=os.environ['ASANA_CLIENT_SECRET'],
# this special redirect URI will prompt the user to copy/paste the code.
# useful for command line scripts and other non-web apps
redirect_uri='urn:ietf:wg:oauth:2.0:oob'
if 'ASANA_CLIENT_ID' in os.environ:
#Creates a client with previously obtained Oauth credentials#
client = asana.Client.oauth(
#Asana Client ID and Secret, set as a Windows environments to avoid hardcoding variables into the script#
client_id=os.environ['ASANA_CLIENT_ID'],
client_secret=os.environ['ASANA_CLIENT_SECRET'],
# this special redirect URI will prompt the user to copy/paste the code.
# useful for command line scripts and other non-web apps
redirect_uri='urn:ietf:wg:oauth:2.0:oob'
)
print ("authorized=", client.session.authorized)
# get an authorization URL:
(url, state) = client.session.authorization_url()
try:
# in a web app you'd redirect the user to this URL when they take action to
# login with Asana or connect their account to Asana
import webbrowser
webbrowser.open(url)
except Exception as e:
print_("Open the following URL in a browser to authorize:")
print_(url)
print_("Copy and paste the returned code from the browser and press enter:")
code = sys.stdin.readline().strip()
# exchange the code for a bearer token
token = client.session.fetch_token(code=code)
#print_("token=", json.dumps(token))
print_("authorized=", client.session.authorized)
me = client.users.me()
print "Hello " + me['name'] + "\n"
params = {'client_id' : client_id, 'redirect_uri' : redirect_uri, 'response_type' : token,}
print_("*************** Request begings *******************"+"\n")
print_("r = requests.get('https://app.asana.com/api/1.0/users/me)" + "\n")
r = requests.get('https://app.asana.com/api/1.0/users/me', params)
print_(r)
print_(r.json)
print_(r.encoding)
workspace_id = me['workspaces'][0]['id']
print_("My workspace ID is" + "\n")
print_(workspace_id)
print_(client.options)
I'm not sure how to use the requests lib with Asana. Their python doc did not help me. I'm trying to pull the available projects and their code colours so I can later plot them into a web browser (For a high-level view of the different projects and their respective colours - Green, yellow or red)
When I introduce the url (https://app.asana.com/api/1.0/users/me) into a browser, it gives me back a json response with the data, but when I try to do the same with the script, it gives me back a 401 (not authorized) response.
Does anybody know what I'm missing / doing wrong?
Thank you!!!

I believe the issue is that the Requests library is a lower level library. You would need to pass all of the parameters to your requests.
Is there a reason you are not exclusively using the Asana Python client library to make requests? All of the data you are looking to fetch from Asana (projects, tasks, etc.) are accessible using the Asana Python library. You will want to look in the library to find the methods you need. For example, the methods for the tasks resource can be found here. I think this approach will be easier (and less error-prone) than switching between the Asana lib and the Requests lib. The Asana lib is actually built on top of Requests (as seen here).

Basic authentication with jira-python

I'm new to Python, new to the jira-python library, and new to network programming, though I do have quite a bit of experience with application and integration programming and database queries (though it's been a while).
Using Python 2.7 and requests 1.0.3
I'm trying to use this library - http://jira-python.readthedocs.org/en/latest/ to query Jira 5.1 using Python. I successfully connected using an unauthenticated query, though I had to make a change to a line in client.py, changing
I changed
self._session = requests.session(verify=verify, hooks={'args': self._add_content_type})
to
self._session = requests.session()
I didn't know what I was doing exactly but before the change I got an error and after the change I got a successful list of project names returned.
Then I tried basic authentication so I can take advantage of my Jira permissions and do reporting. That failed initially too. And I made the same change to
def _create_http_basic_session
in client.py , but now I just get another error. So problem not solved. Now I get a different error:
HTTP Status 415 - Unsupported Media Type
type Status report
message Unsupported Media Type
description The server refused this request because the request entity is in
a format not` `supported by the requested resource for the requested method
(Unsupported Media Type).
So then I decided to do a super simple test just using the requests module, which I believe is being used by the jira-python module and this code seemed to log me in. I got a good response:
import requests
r = requests.get(the_url, auth=(my username , password))
print r.text
Any suggestions?

Here's how I use the jira module with authentication in a Python script:
from jira.client import JIRA
import logging
# Defines a function for connecting to Jira
def connect_jira(log, jira_server, jira_user, jira_password):
'''
Connect to JIRA. Return None on error
'''
try:
log.info("Connecting to JIRA: %s" % jira_server)
jira_options = {'server': jira_server}
jira = JIRA(options=jira_options, basic_auth=(jira_user, jira_password))
# ^--- Note the tuple
return jira
except Exception,e:
log.error("Failed to connect to JIRA: %s" % e)
return None
# create logger
log = logging.getLogger(__name__)
# NOTE: You put your login details in the function call connect_jira(..) below!
# create a connection object, jc
jc = connect_jira(log, "https://myjira.mydom.com", "myusername", "mypassword")
# print names of all projects
projects = jc.projects()
for v in projects:
print v

Below Python script connects to Jira and does basic authentication and lists all projects.
from jira.client import JIRA
options = {'server': 'Jira-URL'}
jira = JIRA(options, basic_auth=('username', 'password'))
projects = jira.projects()
for v in projects:
print v
It prints a list of all the project's available within your instance of Jira.

Problem:
As of June 2019, Atlassian Cloud users who are using a REST endpoint in Jira or Confluence Cloud with basic or cookie-based authentication will need to update their app or integration processes to use an API token, OAuth, or Atlassian Connect.
After June 5th, 2019 attempts to authenticate via basic auth with an Atlassian account password will return an invalid credentials error.
Reference: Deprecation of basic authentication with passwords for Jira and Confluence APIs
Solution to the Above-mentioned Problem:
You can use an API token to authenticate a script or other process with an Atlassian cloud product. You generate the token from your Atlassian account, then copy and paste it to the script.
If you use two-step verification to authenticate, your script will need to use a REST API token to authenticate.
Steps to Create an API Token from your Atlassian Account:
Log in to https://id.atlassian.com/manage/api-tokens
Click Create API token.
From the dialog that appears, enter a memorable and concise Label for your token and click Create.
Click Copy to clipboard, then paste the token to your script.
Reference: API tokens
Python 3.8 Code Reference
from jira.client import JIRA
jira_client = JIRA(options={'server': JIRA_URL}, basic_auth=(JIRA_USERNAME, JIRA_TOKEN))
issue = jira_client.issue('PLAT-8742')
print(issue.fields.summary)

Don't change the library, instead put your credentials inside the ~/.netrc file.
If you put them there you will also be able to test your calls using curl or wget.
I am not sure anymore about compatibility with Jira 5.x, only 7.x and 6.4 are currently tested. If you setup an instance for testing I could modify the integration tests to run against it, too.
My lucky guess is that you broke it with that change.

As of 2019 Atlassian has deprecated authorizing with passwords.
You can easily replace the password with an API Token created here.
Here's a minimalistic example:
pip install jira
from jira import JIRA
jira = JIRA("YOUR-JIRA-URL", basic_auth=("YOUR-EMAIL", "YOUR-API-TOKEN"))
issue = jira.issue("YOUR-ISSUE-KEY (e.g. ABC-13)")
print(issue.fields.summary)
I recommend storing your API Token as an environment variable and accessing it with os.environ[key].