We Keep Coding

Should I add third party data to web application database?

I have a Django application that users fill in some background information, such as "graduated university". These fields are optional, so for some users, they can be empty.
Given that, I have another service that crawls the web for the missing information. This service is completely decoupled from the Django application. It has its own schedule and saves the scraped data to S3 as JSON periodically.
The Django application has admin pages that summarize user information. For these pages, I need to use the real application data which is stored in the application database, as well as the scraped data that resides in S3.
Currently, I have a Django model named ScrapedUser that has a JSON field named data. To populate this model, I manually sync it with the data in S3. (Download files from S3 and create a ScrapedUser instance with it etc.)
My question is, to use those different data sources together, should I populate my real user data in the application database with the third party data that I scraped from the web?
In other words, I wonder if it would be better to map scraped information at ScrapedUser to the real User model.
To better illustrate it, here is a mocked version of the architecture:
I have a standard User model and a ScrapedUser.
# models.py
class User(AbstractUser):
...
university = CharField(...)
class ScrapedUser(Model):
user = ForeignKey(to="User", ...)
data = JSONField(...)
They look like this in the database
User
id
university
1
Harvard
2
NULL
ScrapedUser
user_id
data
2
{"university": "UC Berkeley", ...}
The final report I would like to see in the admin page
user_id
university
1
Harvard
2
UC Berkeley
At the end, should I keep these tables separate and use Django QuerySet features to merge them on view side? Or replace NULL fields in User.university with ScrapedUser.data['university']?

Yes, you will have to add the third party information to your database. If you don't, and the data remains resident in s3, which means every time you need to run a repost, the admin would have to fetch from s3 and process it.
In addition, instead of storing the information from s3 as a single field in ScrapedUser (you have stored it as 'data'), I would 'unpack' that information into fields in the ScrapedUser model.
class ScrapedUser(Model):
user = ForeignKey(to="User", related_name="scrapped_info")
university = CharField(...)
twitter = CharField(...)
facebook = CharField(...)
...
This would make the processing of the information much easier at a later date.
So given the related_name (as shown above), you can easily fetch the information like this:
user = User.objects.create()
user.scrapped_info.university
This displays the university information that was scraped.

How to detect mentions/hashtags when user is typing

I am trying to create a web app that is similar to Facebook and I was wondering how I can detect mentions/hashtags as the user is typing into an input field? For example, when they write "#jo", I want to create an autocomplete feature that will have the dropdown options of:
John Snow
Johnathan Smith
Joey Li
This is a web app running on Flask. My database is a MySQL database.
How do I accomplish this?

When the user is typing, you can use Javascript to detect the hashtag with a regEx and then request the database to know if the mention or/and hashtag exists.
RegEx : detecting hashtags and # in string
Use of database : Database design for apps using "hashtags"

Django Creating A Team Function

I want to design a simple app using Django. The design is as follows:
Each user has their own unique ID in the database called id which exists in the auth_user table already equipped with Django. Then I have a Team_ID which is another unique id that represents a team in table called team_profile. In this table I have the following columns: Member1, Member2, Member3. Currently, a user can create a team and this will set Member1 to the id if the creator.
Each user also has a profile page and in this profile page their is an invite button. This is where I am stuck. I am trying to write the invite function but I have absolutely no idea where to start. In the ideal world, I would like a notification to be sent to the invitee and the invitee can accept or decline the invitation. If the member accepted the invitation then Member2 will have this person's id. I am currently reading up on a lot of stuff but in the meantime if anyone of you guys have any suggestions that would be great.

There are multiple way to engineer your problem, I'm gonna suggest one, but eventually you may adapt it to your need ( i don't know your project in dept )
TeamMembership:
user1: user that send the request
user2: user that receive the request
status: here you can create a choice field where 1=pending, 2=accepted, 3=declined
The next thing you have to do is to manage the membership filtering based on the status.
This is a minimalistic approach that you can extend with for example the date of the invite etc.

Facebook graph api--get ids and names of all users who liked a public page

I am trying to get ids and names of all users who liked the public page called LeEcoGlobal. I tried on developers.facebook.com/tools/explorer by adding different field names and seemed like I can not get access to those information. Just wonder is there any way I can do that or Facebook just do not allow us to do things like that?
BTW, I can get all the user by searching "people who like LeEco" on Facebook, just do not know how to get all those data.
Thanks!

Adding to Google Group based on Google Spreadsheet data validation

So I'd like to preface this with the fact that I am NOT a programmer. I know a little Python, but that's about it.
So this is what I'm trying to do:
User fills out a Google Form and the responses are dumped into a spreadsheet. As a part of the form, their email address is captured in "'Form Responses'B:B"
Not sure if this is even possible, but I'd like to know:
for row X If column P = yes and column c = Accept and column F = Yes
Capture email from Column B and add them to GoogleGroup and send Welcome_Email
Please help! :)
Thanks in advance.

Google Apps Script is JavaScript.
When you build the form in Google Forms, set the Data Validation for the email field to require email address. That way you don't have to hassle with validating it yourself.
http://googledrive.blogspot.com/2013/09/four-new-ways-to-customize-your-google.html
Once the form is built, go to Tools, Script Editor, and make a new Blank Project.
Replace any code in there with this:
function addToGroup(e){
var itemResponses = e.response.getItemResponses();
var emailAddress = itemResponses[#].getResponse();
var group = GroupsManager.getGroup("group name").addMember(emailAddress);
GmailApp.sendEmail(emailaddress, "Welcome to Group Name!", "You have been successfully added to Group Name");
}
https://developers.google.com/apps-script/reference/gmail/gmail-app#sendEmail(String,String,String)
Replace # with the zero-indexed number of the column containing the email addresses, and group name is the mailbox name of the group on your Google Apps domain.
Then you just need to send the welcome email.
Save the project, and go to Resources> Current Script's Triggers. Set it run the addToGroup script on FormSubmit, Authorize it when prompted, and that should be it for you.
https://developers.google.com/apps-script/reference/domain/groups-manager
http://support.google.com/a/bin/answer.py?hl=en&answer=60757
This only works with Google Apps accounts, and only if you have the Provisioning API turned on in your domain admin console. If you're using a standard consumer Google account, they don't have anything available to do this.

Doing a few searches on this forum on words like email or groups you'll find quite a few scripts and if you spend a couple of hours looking at how they are built you'll be able to write something that does what you want...
If at that moment you meet some unexpected difficulties you'll be welcome to ask for help if you show what you've done.