Premise: I am a beginner in search for an easy way to send bug reports from users over sea.
I've made a script for some friends that are living on the other side of the sea (US - EUROPE)... I will like to gather automatic bug reports whenever they happen. So my first idea was to send myself an email with the smtplib module. It works fine when testing home, but as soon as the sender "sends", my email provider (gmail) blocks the connection because of course, its from an "unknown device". I've already enabled "Allow less secure apps" as someone suggested but with no avail.
What I am searching its a simple way of dealing with this.
Yes I could make the script to ignore the error if the email its not being sent, and then go into my google account and enable those devices so at least it will work from the second run..
But it doesn't seem what a programmer would do in this case. I am learning so a solution withing the language is what I am after.
A different provider that has no restriction its also a good start but I tried Yahoo, Live, Yandex but I couldn't make them work. Are there any?
So my question is: how others do? what is the best solution for some one like me?
I've read about sentry or other error/bug tracking but its obviously way too much for want I need
You should certainly not incorporate e.g. Gmail credentials in the code that is remotely executed on devices you do not control, given I understand correctly the Gmail less secure device issue happens as every "user" is running this code and using your credentials. This holds true for any other provider.
Now this won't exactly be simple but one way to go about it would be to create a server side API endpoint that can accept HTTP(s) or any other protocol requests that then will authenticate in a little more secure way on the server side with Gmail.
The concept for emails is:
Bug > Python Script > API call > Email
This could be implemented using Python on the API side (Flask e.g.) using an AWS Lambda Function with Amazon API Gateway, but again that is something to get through and understand by itself which will take a good chunk of time.
You need to touch a lot of concepts, like auth tokens to make this really secure.
Could you elaborate a little on where the code needs to run and if you are willing to try AWS or any other cloud provider, or would have access to an internet connected server ? This makes it easier to provide you with a full example on the solution in a hackish way while I would highlight the problems you could face on the security side.
I understand that this is not the way to go but as for my needs and my level of experience it works for me!
Yandex allows you to send email from different ip so Yandex is the way to go. What I was doing wrong in the first place was to use the wrong port (587 instead of 465)
I've tried to use smtpd but couldn't get it to work properly (Here's the link to the issue). I only need the server to receive emails and store in database. I've found few like MailGun, and although it looks great it is not free. Any suggestions?
There are so many options, it's an opinion question, and also related to the specific use case. So here are the main options you have.
It's mostly your decision and not "with django". The question is really choosing a mail server for a web application.
From the django point of view it's either local server(faster), or external API. As a developer, sending SMTP is probably easier than external API because django already includes the backend handling. But good email service providers usually have a good clients libraries, and it's not that complex to use their API as well.
Your own email server: Install and maintain your own mail server. Pros: cheaper, a cheap VPS will do, no limits on accounts (useful for testings) and mails. Cons: installing a mail server + auth backend etc is not easy if you didn't installed one before, and your emails can be easily marked as spam. If you select this option, Postfix is a safe choice.
External service: with API. Pros: easier to implement, and less chances to be marked as spammer. Cons: more expensive, possible specifc coding against the specific API
Your mail server + external API: the local server connection is much faster, so django sends the email quickly, and then the server handles the slower sending to an external service
Summary: If you just start something, and need a simple email facility, pick an external service that uses SMTP. You will have something up and running, using the very easy django SMTP utilities, without specific coding to the vendor API. Once you need to scale, do the research and pick a more advanced solution.
Note: usually sending email requires some async queue, that allows django to pass the email sending task and continue with the request, without waiting to the smtp connection. SMTP connections can be slow.
I have already asked a question about IP Authentication here: TastyPie Authentication from the same server
However, I need something more! An IP address could be very easily spoofed.
Scenario: My API (TastyPie) and Client App (in javascript) are on the same server/site/domain. My users don't login. I want to consume my API in my javascript client side.
Question: How can I make sure (authentication) that my AJAX requests are originating from the same server?
I'm using Tatypie. I need to authentication that the requests from the client are being made on the same server/domain etc. I cannot use 'logged in sessions' as my users don't login.
I have looked at private keys and generating a signature but they can viewed in the javascript making that method insecure. If I do it in a way to request a signature form the server (hiding the private key in some python code) anyone can make the same http request to get_signature that my javascript makes, thus defeating the point.
I also tried to have the Django view put the signature in the view eliminating the need to make the get_signature call. This is safe, but means that I have to now refresh the page every time to get a new signature. From a users point of view only the first call to the API would work, after which they need to refresh, again pointless.
I cannot believe I'm the only person with this requirement. This is a common scenario I'm sure. Please help :) An example using custom authentication in Tastypie would be welcome too.
Thanks
Added:
Depending on your infrastructure #dragonx's answer might interest you most.
my 2c
You want to make sure that only if a client visits your website can use the api? Hmm does the bot, robot, crawler fall in the same category with the client then? Or am I wrong? This can be easily exploited in case you really want to secure it really.
I cannot believe I'm the only person with this requirement.
Maybe not, but as you can see you are prone to several attacks to your API and that can be a reason for someone not sharing your design and making security stricter with auth.
EDIT
Since we are talking about AJAX requests what does the IP part has to do with this? The IP will always be the Client's IP! So probably, you want a public API...
I would Go with the tokens/session/cookie part.
I 'd go with a generated token that lasts a little while and a flow described below.
I'd go with a limiter per some time, like Github does. Eg 60 requests per hour per ip or more for registered users
To overcome the problem with the refreshing token I would just do this:
Client visits the site
-> server generates API TOKEN INIT
-> Client gets API TOKEN INIT which is valid only for starting 1 request.
Client makes AJAX Request to API
-> Client uses API TOKEN INIT
-> Server checks against API TOKEN INIT and limits
-> Server accepts request
-> Server passes back API TOKEN
-> Client consumes response data and stores API TOKEN for further usage (Will be stored in browser memory via JS)
Client Starts Comm with the API for a limited amount of time or requests. Notice that you know also the init token date so you can use it to check against the 1st visit on the page.
The 1st token is generated via the server when the client visits.
Then the client uses that token in order to obtain a real one, that lasts for some time or something else as of limitation.
This makes someone actually visit the webpage and then he can access the API for a limit amount of time, requests perhaps etc.
This way you don't need refreshing.
Of course the above scenario could be simplified with only one token and a time limit as mentioned above.
Of course the above scenario is prone to advanced crawlers, etc since you have no authentication.
Of course a clever attacker can grab tokens from server and repeat the steps but, then you already had that that problem from start.
Some extra points
As the comments provided please close writes to the API. You don't want to be a victim of DOS attacks with writes if you have doubts about your implementation(if not use auth) or for extra security
The token scenario as described above can also become more complicated eg by constantly exchanging tokens
Just for reference GAE Cloud storage uses signed_urls for kind of the same purpose.
Hope it helps.
PS. regarding IP spoofing and Defense against spoofing attacks wikipedia says so packet's won't be returned to the attacker:
Some upper layer protocols provide their own defense against IP
spoofing attacks. For example, Transmission Control Protocol (TCP)
uses sequence numbers negotiated with the remote machine to ensure
that arriving packets are part of an established connection. Since the
attacker normally can't see any reply packets, the sequence number
must be guessed in order to hijack the connection. The poor
implementation in many older operating systems and network devices,
however, means that TCP sequence numbers can be predicted.
If it's purely the same server, you can verify requests against 127.0.0.1 or localhost.
Otherwise the solution is probably at the network level, to have a separate private subnet that you can check against. It should be difficult for an attacker to spoof your subnet without being on your subnet.
I guess you're a bit confused (or I am, please correct me). That your JS code is published on the same server as your API does not mean AJAX requests will come from your server. The clients download the JS from your server and execute it, which results in requests to your API sent from the clients, not from the same server.
Now if the above scenario correctly describes your case, what you are probably trying to do is to protect your API from bot scraping. The easiest protection is CAPTCHA, and you can find some more ideas on the Wiki page.
If you are concerned that other sites may make AJAX calls to your API to copy your site functionality, you shouldn't be--AJAX requests can only be sent to the same server as the page the JS is running on, unless it is JSONP.
Short answer: It is not possible to prevent a dedicated attacker.
You have no method of identifying a client other than with the information that they give you. For instance, username/password authentication works under the assumption that only a valid client would be able to provide valid credentials. When someone logs in, all you know is that some person provided those credentials -- you assume that this means that this means that they are a legitimate user.
Let's take a look at your scenario here, as I understand it. The only method you have of authenticating a client is IP Address, a very weak form of authentication. As you stated, this can be easily spoofed, and in with some effort your server's response can be received back to the attacker's original IP address. If this happens, you can't do anything about it. The fact is, if you assume someone from a valid IP address is a valid user, then spoofers and legitimate users are indistinguishable. This is just like if someone steals your password and tries to log in to StackOverflow. To StackOverflow, the attacker and you are indistinguishable, since all they have to go on is the username and password.
You can do fancy things with the client as mentioned in other answers, such as tokens, time limits, etc., but an dedicated attacker would be able to mimic the actions of a legitimate client, and you wouldn't be able to tell them apart because they would both appear to be from valid IP addresses. For instance, in your last example, if I was an attacker looking to make API calls, I would spoof a legitimate IP address, get the signature, and use it to make an API call, just as a legitimate client would.
If your application is critical enough to deem this level of thought into security, you should at least think of implementing something like API tokens, public key encryption, or other authentication methods that are more secure than IP addresses to tell your clients apart from any attackers. Authentication by IP address (or other easily forged tokens like hostname or headers) simply won't cut it.
may be you could achieve this by using Same-origin policy
refer http://en.wikipedia.org/wiki/Same_origin_policy
As suggested by Venkatesh Bachu, Same Origin Policy and http://en.wikipedia.org/wiki/Cross-Origin_Resource_Sharing (CORS) could be used as a solution.
In your API, you can check Origin header and respond accordingly.
Need to check if Origin header can be modified by using extensions like tamper data.
A determined hacker can still snoop by pointing browser to a local proxy server.
If this app server is running on an ordinary web server that has configurable listening IP address, set it to 127.0.0.1. With the TCPServer module, it's like
SocketServer.TCPServer(("127.0.0.1", 12345), TheHandlerClass)
Use netstat command to verify the listening address is correct as "127.0.0.1"
tcp4 0 0 127.0.0.1.12345 *.* LISTEN
This will effectively making any connection originated outside the same host impossible on the TCP level.
There are two general solution types: in-band solutions using normal web server/client mechanisms, that are easy to implement but have limitations; and out-of-band solutions that rely on you to configure something externally, that take a little more work but don't have the same limitations as in-band.
If you prefer an in-band solution, then the typical approach used to prevent cross-site request forgery (XSRF) would work well. Server issues a token with a limited life span; client uses the token in requests; privacy of token is (sort of) assured by using an HTTPS connection. This approach is used widely, and works well unless you are worried about man-in-the-middle attacks that could intercept the token, or buggy browsers that could leak data to other client-side code that's being naughty.
You can eliminate those limitations, if you're motivated, by introducing client certificates. These are kind of the flip side to the SSL certificates we all use on web servers -- they operate the same way, but are used to identify the client rather than the server. Because the certificate itself never goes over the wire (you install it locally in the browser or other client), you don't have the same threats from man-in-the-middle and browser leakage. This solution isn't used much in the wild because it's confusing to set up (very confusing for the typical user), but if you have a limited number of clients and they are under your control, then it could be feasible to deploy and manage this limited number of client certificates. The certificate operations are handled by the browser, not in client code (i.e. not in JavaScript) so your concern about key data being visible in JavaScript would not apply in this scenario.
Lastly, if you want to skip over the client configuration nonsense, use the ultimate out-of-band solution -- iptables or a similar tool to create an application-level firewall that only allows sessions that originate from network interfaces (like local loopback) that you know for certain can't be accessed off the box.
I am trying to fetch mails from another mailbox (xxx#domail.com or xxx#gmail.com) in google-app-engine.
I don't want to read mails from appspotmail box as it is being used for different purpose.
Is there any efficient way in which i can make this happen.
Two options:
You could read an inbox via POP/IMAP, but this requires a bit of coding. You also need to have Outgoing Sockets API enabled, which requires you to have a paid app. This approach is async, which means you will constantly need to poll for new messages.
Forward emails to a new appspotmail address (you can have many). This is pretty easy, especially since you already process incoming emails. Since you can have multiple accounts, e.g. xyz#yourappid.appspotmail.com, you can distinguish between them in code.
You can use imap+oauth to read email from a google address. If you google it the very first result is what you need. https://developers.google.com/gmail/oauth_overview
Example scenario: Web based multi-user chat application through websocket connection. How can I ensure (or guarantee) that each connection in this application belongs to certain authenticated user and "can't be" exploited by false user impersonation or intervene during the connection.
by the way I am using tornado websocket on server side to implement basic chat room and already doing authentication for the non-websocket part of my app.
ps : suppose authenticated user posts what he wants and when other user open the listing page of item and automatically other user is added to list of websocket listeners what I want each user able to chat with buyer of the item individually not in a chatroom way but with one to one chat
First and foremost, there are two things you should remember about WebSockets: (a) it's an evolving standard and (b) it is designed with the intention of working with untrusted clients.
The biggest and most important thing you should always do with WebSockets is check their origin. If the origin is mismatched, obviously you don't want to deal with that client, so ignore their requests. Additionally, make sure you're using the "wss" secured WebSocket protocol rather than the "ws" unsecured protocol. This will ensure that your messages are encrypted.
The problem with just doing this, is that this information can be spoofed. See this blog post for a quick demonstration of this.
Additional Security:
Try sending a salted token, having it salted/hashed and sent back and validated in the handshake phase.
Limit requests that happen too frequently (just like the IRC protocol). If the user has submitted 10 lines or more within the span of a second, ignore that user.
Do a quick spam-check (there are lots of algorithms for this) -- stick to light heuristics, otherwise it will burden your server. Things like the presence of the words "free" or "viagra". Give the user a score that represents the likelihood that they are spamming or are a bot. When that is breached, boot them from the servers.
Hope that helps! Sorry if it doesn't. This is my frist answer on StackOverflow. :P
I'm pretty sure the websocket connection sends up any cookies that have been established in the non-websocket connection to your app. You should be able to query Django's session store for the connection's cookie and determine the user that socket belongs to.
Check out: https://docs.djangoproject.com/en/1.3/topics/http/sessions/#configuring-the-session-engine
This is the whole purpose of websocket connection. You authenticate them by handshaking protpcol. For further info see here:
https://datatracker.ietf.org/doc/html/draft-ietf-hybi-thewebsocketprotocol-08