We Keep Coding

What is the best way to fetch all data (posts and their comments) from Reddit? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I have a requirement to analyse all the comments about a subreddit, (e.g. r/dogs, say from 2015 onward) using the Python Reddit API Wrapper (PRAW). I'd like the data to be stored in a JSON.
I have found a way to get all of the top, new, hot, etc. submissions as a JSON using PRAW:
new_submissions = reddit.subreddit("dogs").new()
top_submissions = reddit.subreddit("dogs").top("all")
However, these are only the submissions for the subreddit, and do not include comments. How can I get the comments for these posts as well?

You can use the Python Pushshift.io API Wrapper (PSAW) to get all the most recent submissions and comments from a specific subreddit, and can even do more complex queries (such as searching for specific text inside a comment). The docs are available here.
For example, you can use the get_submissions() function to get the top 1000 submissions from r/dogs from 2015:
import datetime as dt
import praw
from psaw import PushshiftAPI
r = praw.Reddit(...)
api = PushshiftAPI(r)
start_epoch=int(dt.datetime(2015, 1, 1).timestamp()) # Could be any date
submissions_generator = api.search_submissions(after=start_epoch, subreddit='dogs', limit=1000) # Returns a generator object
submissions = list(submissions_generator) # You can then use this, store it in mongoDB, etc.
Alternatively, to get the first 1000 comments from r/dogs in 2015, you can use the search_comments() function:
start_epoch=int(dt.datetime(2015, 1, 1).timestamp()) # Could be any date
comments_generator = api.search_comments(after=start_epoch, subreddit='dogs', limit=1000) # Returns a generator object
comments = list(comments_generator)
As you can see, PSAW still uses PRAW, and so returns PRAW objects for submissions and comments, which may be handy.

Key Error even though key IS in dictionary? [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 1 year ago.
Improve this question
Dictionary:
section of dictionary
My Code:
code
Error says:
3
So how come the date key works fine but for freq it fails?
ps. my first time posting, so am very sorry for the sloppy structure of the post

This can only happen when one of your day is missing the freq parameter.
Try catching the day in which the error is happening. Then manually check that particular entry.
date_list = []
frequency_list = []
try:
for i in obj:
date = obj[i]["date"]
frequency = obj[i]["freq"]
date_list.append(date)
frequency_list.append(frequency)
except:
print(i)

Sending a variable in python payload - http client [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
I'm trying to send a variable in python using a payload created by postman, I tried using {{}} and it
didn't work for me, the place where I need the variable is indicated as VARIABLE_SPOT, please let me know what I did wrong in order to send a variable and not a text.
payload = "{\r\n \"some_key\": \"VARIABLE_SPOT\",\r\n }

Just Declare Your Payload Like Below
payload = {"some_key": VARIABLE_SPOT}
if you want to stringfy it run json.dumps(payload)

You can create a dict and covert it to string:
import json
some_variable = 'some_variable'
payload_dict = {"some_key": some_variable}
payload_text = json.dumps(payload_dict)
payload_text is your payload now.
Result:
> print(payload_text)
{"some_key": "some_variable"}
> print(type(payload_text))
<class 'str'>

How do I make a REST query in python with a comparison statement? [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 4 years ago.
Improve this question
How do I make a REST query in python with a comparison statement? Like (in quasi code):
result = foo where bar > 10
I want to make a rest api query from python 2.7 using requests. What I want is to get all articles that have been updated during the last 24 hours.
In javascript it looks like this and it works great:
http://myDatabase.domain.io/api/v1/article/q={"update_date":{"$gt":"2018-08-27 13:44"}}
I just can't recreate this with python. Does anyone know how?

Assuming
that's actually ?q=, i.e. a query string
and the query should be JSON (it looks like it)
and the endpoint returns JSON:
import requests, json
query = json.dumps(
{"update_date": {"$gt": "2018-08-27 13:44"}}
)
resp = requests.get(
url="http://myDatabase.domain.io/api/v1/article/",
params={"q": query},
)
resp.raise_for_status()
data = resp.json()
print(data)

Retrieve date data from browser [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Closed 8 years ago.
This question appears to be off-topic because it lacks sufficient information to diagnose the problem. Describe your problem in more detail or include a minimal example in the question itself.
Questions concerning problems with code you've written must describe the specific problem — and include valid code to reproduce it — in the question itself. See SSCCE.org for guidance.
Improve this question
I am doing web application and I cannot retrieve a date value from the browser. I have selected the date 01/16/2014.
The query string looks like:
?dd=01%2F16%2F2014
and my code is:
from django.http import HttpResponse, HttpResponseNotFound
import cgi, cgitb
form = cgi.FieldStorage()
def pod_available(request):
dat = form.getvalue('dd')
print dat
return HttpResponse("<html><body>Hello<em>" + str(dat) + "</em></body></html>")
Output:
HelloNone

If you are building a Django application, you are not using CGI. The forms variable will only be filled once, and will forever be empty. You may as well remove the cgi and cgitb module imports, they are of no use to you here.
You need to re-read the Django tutorial; the request parameter to your view contains all your form variables:
def pod_available(request):
dat = request.GET['dd']
print dat
return HttpResponse("<html><body>Hello<em>" + str(dat) + "</em></body></html>")
The request.GET object is a dictionary holding the request parameters passed in via the URL.

If this is a Django application, then using CGI in any way makes absolutely no sense. I assume that pod_available is being called with a HttpRequest object passed to it, so if you want to read the request data, you have to access that object to get it.
You can do that by accessing they key—'dd' in this case—of the object:
def pod_available(request):
dat = request['dd']
return HttpResponse("<html><body>Hello<em>" + str(dat) + "</em></body></html>")
In any way, this is not a CGI application, so don’t use any CGI stuff in there.