I'm working on a project that allows a user to redirect his browsing through a proxy. The system works like this - a user runs this proxy on a remote PC and then also runs the proxy on his laptop. The user then changes his browser settings on the laptop to use localhost:8080 to make use of that local proxy, which in turn forwards all browser traffic to the proxy running on the remote PC.
This is where I ran into HTTPS. I was able to get normal HTTP requests working fine and dandy, but as soon as I clicked on google.com, Firefox skipped my proxy and connected to https://google.com directly.
My idea was to watch for browser requests the say CONNECT host:443 and then use the python ssl module to wrap that socket. This would give me a secure connection between the outer proxy and the target server. However, when I run wireshark to see how a browser request looks like before ssl kicks in, it's already there, meaning it looks like the browser connects to port 443 directly, which explains why it omitted my local proxy.
I would like to be able to handle to HTTPS as that would make for a complete browsing experience.
I'd really appreciate any tips that could push in the right direction.
Well, after doing a fair amount of reading on proxies, I found out that my understanding of the problem was insufficient.
For anyone else that might end up in the same spot as me, know that there's a pretty big difference between HTTP, HTTPS, and SOCKS proxies.
HTTP proxies usually take a quick look into the HTTP headers to determine where to forward the whole packet. These are quite easy to code on your own with some basic knowledge of sockets.
HTTPS proxies, on the other hand, have to work differently. They should either be able to do the whole SSL magic for the client or they could try to pass the traffic without changes, however if the latter solution is chosen, the users IP will be known. This is a wee bit more demanding when it comes to coding.
SOCKS proxies are a whole different, albeit really cool, beast. They work on the 5th layer of the OSI model and honestly, I have no clue as to where I would even begin creating one. They achieve both security and anonymity. However, I do know that a person may be able to use SSH to start a SOCKS proxy on their machine, just read this http://www.revsys.com/writings/quicktips/ssh-tunnel.html . That link also gave an idea that it should be possible to use SSH from a Python script to make it much more convenient.
Hope this helps anyone with the same question as I had. Good luck!
Related
I have a web page (front-end) created with ASP.NET (visual studio) on one laptop and a server (back-end) for my web page on another laptop. In the first place, I have a login page and I want to take the username and the password from textBox-es (front-end) and somehow transfer the data to the other laptop to verify if it's correct and then receive if the data it's correct or not. The front-end is on Windows and the server is on Linux. My teammate is using python on back-end.
I'm working with HTML, CSS and a little bit of C# on the front-end and I don't know how to transfer that username and password to my teammate and how to receive an answer from him.
He has the database on his laptop and he's the one who will verify if the pair is correct or not and send me an answer, also he wants to code in python.
We think that we can do some magic connection between ASP.NET and Node.js.
How do I do this data transfer ?
It's my first time doing this so please be gentle because I really don't know what to do.
Your question is quite broad and while I am not that fluent in web applications I will share what advice I can. Because the question is broad, you cannot get an exact, short and concise, answer.
Some clarifications and feedback regarding your question.
I have a web page (front-end) created with ASP.NET ...
You actually have a web application here (that, in your case, includes one, or more, web pages). Give more attention to such little details, programming in itself is a form of grammar natzy-ism but it’s not grammar and you are not being corrected by a person, but by machines and life. A nice definition of web application: https://blog.stackpath.com/web-application/
We think that we can do some magic connection between ASP.NET and Node.js.
I am not specialized in web application but it appears to me that this line comes completely out of the blue, you have not indicated that you are using or intend to use Node.js in your back-end; you indicated that you were using Python, so I will ignore this statement. Oh and there is no magic, there are only race conditions.
Structurally speaking, I would split this question into several smaller ones. Some suggestions follow.
Because you are attempting a somewhat more complicated task you cannot escape de idea of a design pattern (especially because of using ASP.NET which is designed to be patterning friendly).
So your first question should be similar to
how to communicate with a web server asp.net
Two-way communication between ASP.NET Web App and C# Application
back-end is not python but read the answer, it gives you a valid indicator, that request-response pattern could help you
search for questions regarding how to implement that pattern in C#/ASP.NET
Another possible way of how to-ing would be
how to transmit data to server C#
a bit old but reading might give you an understanding of how the ecosystem works
C# server and client communication - send/receive data
maybe helps?
https://codingvision.net/networking/c-sending-data-using-get-or-post
https://www.csharp-examples.net/socket-send-receive/
From you explication I presume that you already have implemented the UI forms that will retrieve the input data so I will skip UI related questions.
Since you wish to send user SENSITIVE data over the internet then you MUST think about security. Further questions
how to store password in database securely
this question will actually lead you to examples of transmitting the password and maybe even receiving the content
https://www.meziantou.net/how-to-store-a-password-in-a-web-application.htm
How to store passwords in database securely?
There are 2 more issues I wish to discuss.
While searching the internet for tutorials you will probably see a
lot working with client and server applications on the same machine.
Those are still valid tutorials for you! Read them! I will explain
further down why it's (almost) the same for you even if your case is
a remote server.
You need to somehow be able to test your application.
Addressing #1
Why working with localhost (127.0.0.1) is for you the same as working with a real server (in your case).
Let's say you finish the application, the back-end is also made and you wish to communicate with one another. You read on the internet that you need to known the IP of the server so you ask your partner to for him to give his IP. You search the wild wild web and reach the conclusion that to find your IP you would try:
executing ipconfig.exe (or what tool you use depending on your platform) and getting the IPv4 Address. . . . . . . . . . . : 192.168.1.12 value.
you would presume that that is the IP you need to put in your application to connect to. This is FALSE.
reaching the conclusion that that does not work, you find that you can get your IP with online sites such as https://whatismyipaddress.com/
and use that IP for your server address. This (in 99% cases) is FALSE.
A bit of poetry
Consider IP as you address in the world and your local (private) IP as the exact place in your house. The IP address you see there is a local IP and it would indicate in which chair you are sitting in, let’s say the 12 from the already provided example. Anyone in the world (who has 12 chairs anyway) can sit in the 12th chair in their house. You cannot use this information to identify the address of someone.
How would you know that that IP is a local IP?
One way would be to look at the the full output of ipconfig.exe (or whatever tool you are using) where you see a Wireless LAN adapter Wi-Fi: above from where you got your IP, the LAN there (hxxps://en.wikipedia.org/wiki/Local_area_network) is an indicator of that.
The other (more straightforward) is by recognizing the IP range (or visual pattern, if IP range is not a clear concept for you just yet). If it starts with 198.168. then it's a local/private IP address. Please read here more: https://en.wikipedia.org/wiki/Private_network
IPs are a valuable commodity (I joke you not). As the world has evolved you cannot now simply plug your computer with an internet cable, and voila you have your own IP. This was (relatively) true when the internet first started decades ago but now the IPs are managed by big (huge) companies that they allocate them as they consider.
The IP address space is managed globally by the Internet Assigned Numbers Authority (IANA), and by five regional Internet registries (RIRs) responsible in their designated territories for assignment to local Internet registries, such as Internet service providers, and other end users.
Wiki quote: https://en.wikipedia.org/wiki/IP_address
I also highly (please do) recommend you read the following answer
hxxps://superuser.com/questions/323801/how-can-i-own-an-ip-address
If you ask your (real world) neighbors to look up https://whatismyipaddress.com/ (and they have the same ISP and monthly plan as you) there is a strong possibility that they will see the same IP as you see. To put it simply, the IP you see there, when accessing sites such as whatismyipaddress, is the IP that that your ISP has assigned to the network node you are in. It would be a waste of IPs to assign each user an IP (not to mention there actually aren’t mathematically that many IPs to do so).
Regarding your current predicament. I can see 3 options.
Option 1 - buy hosting
If your response is "I really really wanna have a separate IP" (or domain) then you will need to buy from a hosting service an IP (the good kind) that will point to a device the hosting company has. You usually buy domains that point to IPs, but let's not got there for now. You will connect to that machine/devices and install your code on the server there. Then, and only then, you could get the IP/domain of that machine and use it in your code and it will work (this is the 1% cases where sites like whatismyipaddress will actually help you).
Recommended searches: server hosting cheap :D (or something free for students if you are students)
Option 2 - work locally
If your project is used for academia and there is no actual need for a separate server then you and your partner will run the client/server applications while connected to the same wireless modem. Ask him to get his IP in this way (here you actually need the LAN IP), you hardcoded it in your code (preferably a separate config file, this way no extra compilation is required) and voila, it works.
Option 3 - LAN over the internet
Technically this would be hard to describe but the idea is you use programs that communicate over the internet, you each install it on your computer, somehow connect with each other (usually a name/password pair) and that software make is seem for the programs on your computer that you are actually in a LAN setup (Option 2). In my days I used hamachi so that we could play CS online :D.
hxxps://www.howtogeek.com/172762/how-to-play-old-lan-games-over-the-internet/
Regarding #2
Until your partner provides a server you will need to test your code. So you will need to somehow fake a server response so that you can actually continue development.
Quite simply, the operation of faking something, a return/a class, in your code, so that you can correctly develop and test your code is known as mocking
how to mock server response asp.net
hxxps://gingter.org/2018/07/26/how-to-mock-httpclient-in-your-net-c-unit-tests/
For beginning programmers I think this is one of the most complicated parts. The concept of mocks and tests, while they do appear simple, are usually not engraved into the mind of young, innocent, previously untainted, future programmers. They are left stuck with printfs until graduations (even a dose of hate towards tests is sprinkled in the mix, what a pity).
As a courtesy, I wish to further extend that madness. Please, for all that is good and sacred in this world, do not use this code in production, burn it after you used it to understand some of the basics, BURN IT!
I modified the code from here Python 3.x BaseHTTPServer or http.server so that you could run a local HTTP server on your machine and test your application. You simply run it (python3) and test the sending of user name and password to the server using the following pagan syntax:
127.0.0.1:4242/?user=aba&password=Aba1 which would return I'al cohol you later
#!/usr/bin/python
from http.server import BaseHTTPRequestHandler,HTTPServer
from urllib.parse import urlparse, parse_qs
HOST = '127.0.0.1' # Standard loopback interface address (localhost)
PORT_NUMBER = 4242 # Port to listen on (non-privileged ports are > 1023)
# !!!!!!!!!!!!!!! THIS IS ONLY FOR DIDACTICAL PURPOSE< NEVER EVER DO SOMETHING LIKE THIS IN PRODUCTION!!!!!!!
# NEVER EVER STORE PASSWORDS LIKE THIS !!!!!!!
# LITERRLAY NEVER!!!!!!
data_base = [
{
'user': 'ionzapada',
'password': 'neaomat',
'data': b"I like pie!"
},
{
'user': 'aba',
'password': 'Aba1',
'data': b"I'al cohol you later"
}
]
class myHandler(BaseHTTPRequestHandler):
#Handler for the GET requests
def do_GET(self):
self.send_response(200)
self.send_header('Content-type','text/html')
self.end_headers()
# Send the html message
query_components = parse_qs(urlparse(self.path).query)
# DO NOT USE THIS LOGIC ON ANY PRODUCTION PLATFORM!!!!
# NEVER SEND PASSWORDS THIS WAY!!!
# NEVER SEND PASSWORDS!!!!
# NEVER SEND HASHES WITHOUT SALT!!!!
# NEVER IMPLEMENT YOUR OWN TRICKY ALGORITHM!!!!
user_name = query_components.get("user",[None])[0]
password = query_components.get("password",[None])[0]
if user_name and password:
for data_base_entry in data_base:
if data_base_entry['user'] == user_name:
print('Found searched user: {}'.format(user_name))
if data_base_entry['password'] == password:
print('Valid password for user {} given; returning db data'.format(user_name))
self.wfile.write(data_base_entry['data'])
else:
self.wfile.write(b"Not supported!")
else:
self.wfile.write(b"Not supported!")
return
try:
server = HTTPServer((HOST, PORT_NUMBER), myHandler)
print('Started httpserver on port', PORT_NUMBER)
server.serve_forever()
except KeyboardInterrupt:
print('CTRL+C received, shutting down the web server')
server.socket.close()
I burned my finger writing that, do not ever send user names and passwords in this way!!! DO NOT EVEN SEND PASSWORDS FOR THE LOVE OF GOD!!!!!!!! search for standard and secure ways of doing this!!!!
Please also visit the links indicated by #amc they are an also good start.
P.S. I am not allowed to have more then 8 link in one post, on some links that I have posted change the hxxp to http for them to work.
I am using django to build my web server, other people connect to me as clients. Now I need to know the clients' port number to distinguish them. If their browser opens two 'Tabs' of the same link, i.e. two pages but the same link, I also have to distinguish them.
Although I know I can use request.META['REMOTE_ADDR'] to get the client's IP in my django view function, but this realy is not enough for me.
Then I studied some TCP/IP basics and then I know that in TCP/IP layer, every IP packet has an IP header which contains the client's port number. But how can I access it in django?
Additional info:
I'm using python 2.6 and django 1.4
I know every TAB of a browser will be allocated a random unique port to access my django web page port. -- see this link 'The web server opens port 80, but the browser has a different, randomly-assigned port.' I really need to distinguish them. So my intuitive thoughts is to use the port number in the IP packet. If you have any other suggestion, also welcome.
I have found the similar question here, but I am not using Apache now. And this may be hard for me to config so maybe causing other more complex questions. This might make this simple question complex.
while I debug the django , I find this
request.environ["wsgi.input"].raw._sock.getpeername()
maybe it can work
Yes, after days of struggling, I answer it, with a working, but ugly solution on 'how to get client port in Django'.
in your python26/Lib/SocketServer.py, find def process_request_thread,add
global gClientPort; gClientPort = client_address
use this global value in yout project. Its format is ('12.34.56.78',55437) for example. 55437 is the port number.
Your assumption about 'every user connection opens connection at unique port' is wrong. All users are using the same port to connect.
To distinguish users Django (and almost every other frameworks) is using sessions. Every user gets a cookie with his unique session ID, and this cookie is passed to a server on every connection, so the application can distinguish users.
Here is documentation on sessions:
https://docs.djangoproject.com/en/1.8/topics/http/sessions/
I'm building a turn-based game and I'm hoping to implement client-server style networking. I really just need to send the position of a couple of objects and some other easily encodable data. I'm pretty new to networking, although I've coded some basic stuff in socket and twisted. Now, though, I need to be able to send the data to a computer that isn't on my local network, and I can't do port forwarding since I don't have admin access to the router and I'm also not totally sure that would do the trick anyways since I've never done it. So, I was thinking of running some Flask or Bottle or Django, etc. code off PythonAnywhere. The clients would then send data to the server code on PythonAnywhere, and when the turn passed, the other client would just go look up the information it needed on the server. I guess then the server would act as just a data bank with some simple getter and setter methods. My question is how can this be implemented? Can my Socket code on my client program talk to my Flask code on PythonAnywhere?
Yes, client code can talk to your project at PythonAnywhere, as you will be given a unique project url like http://yourblogname.pythonanywhere.com/. Your server will listen the 80 port at that url.
It depends what sort of connection your clients need to make to the server. PythonAnywhere supports WSGI, which means "normal" HTTP request/response interactions -- GET, POST, etc. That works well for "traditional" web pages or web apps.
If your client side needs dynamic, two-way connections using non-HTTP protocols, using raw sockets, or even websockets, PythonAnyhwere doesn't support that at present.
I'm attempting to use a proxy, via python, in order to log into a site from a different, specific IP address. It seems that certain websites, however, can detect the original (host) IP address. I've investigated the issue a bit and here's what I found.
There are four proxy methods I've tried:
Firefox with a proxy setting.
Python with mechanize.set_proxies.
Firefox in a virtual machine using an internal network, along with another virtual machine acting as a router (having two adapters: a NAT, and that internal network), set up such that the internal network traffic is routed through a proxy.
TorBrowser (which uses Firefox as the actual browser).
For the first three I used the same proxy. The Tor option was just for additional testing, not via my own proxy. The following things are behaviors I've noticed that are expected:
With all of these, if I go to http://www.whatismyip.com/, it gives the correct IP address (the IP address of the proxy, not the host computer).
whatismyip.com says "No Proxy Detected" for all of these.
Indeed, it seems like the websites I visit do think my IP is that of the proxy. However, there have been a few weird cases which makes me think that some sites can somehow detect my original IP address.
In one situation, visiting a non-US site via Firefox with a non-US proxy, the site literally was able to print my originating IP address (from the US) and deny me access. Shouldn't this be impossible? Visiting the site via the virtual machine with that same non-US proxy, or the TorBrowser with a non-US exit node, though, the site was unable to do so.
In a similar situation, I was visiting another non-US site from a non-US proxy. If I logged into the site from Firefox within the virtual machine, or from the TorBrowser with a non-US exit node, the site would work properly. However, if I attempted to log in via Firefox with a proxy (the same proxy the virtual machine uses), or with mechanize, it would fail to log in with an unrelated error message.
In a third situation, using the mechanize.set_proxies option, I overloaded a site with too many requests so it decided to block access (it would purposefully time out whenever I logged in). I thought it might have blocked the proxy's IP address. However, when I ran the code from a different host machine, but with the same proxy, it worked again, for a short while, until they blocked it again. (No worries, I won't be harassing the site any further - I re-ran the program as I thought it might have been a glitch on my end, not a block from their end.) Visiting that site with the Firefox+proxy solution from one of the blocked hosts also resulted in the purposeful timeout.
It seems to me that all of these sites, in the Firefox + proxy and mechanize cases, were able to find out something about the host machine's IP address, whereas in the TorBrowser and virtual machine cases, they weren't.
How are the sites able to gather this information? What is different about the TorBrowser and virtual machine cases that prevents the sites from gathering this information? And, how would I implement my python script so that the sites I'm visiting via the proxy can't detect the host/host's IP address?
It's possible that the proxy is reporting your real IP address in the X-Forwarded-For HTTP header, although if so, I'm surprised that the WhatIsMyIP site didn't tell you about it.
If you first visited the non-US site directly, and then later again using the proxy, it's also possible that the site might have set cookies in your browser on your first visit that let the site identify you even after your IP address changes. This could account for the differences you've observed between browser instances.
(I've noticed that academic journal sites like to do that. If I try to access a paywalled article from home and get blocked because I wasn't using my university's proxy server, I'll typically have to clear cookies after enabling the proxy to be allowed access.)
I am little stumped: I have a simple messenger client program (pure python, sockets), and I wanted to add proxy support (http/s, socks), however I am a little confused on how to go about it. I am assuming that the connection on the socket level will be done to the proxy server, at which point the headers should contain a CONNECT + destination IP (of the chat server) and authentication, (if proxy requires so), however the rest is a little beyond me. How is the subsequent connection handled, specifically the reading/writing, etc...
Are there any guides on proxy support implementation for socket based (tcp) programming in Python?
Thank you
Maybe use something like SocksiPy which does all the protocol details for you and would let you connect through a SOCKS proxy as you would without it?
It is pretty simple - after you send the HTTP request: CONNECT example.com:1234 HTTP/1.0\r\nHost: example.com:1234\r\n<additional headers incl. authentication>\r\n\r\n, the server responds with HTTP/1.0 200 Connection established\r\n\r\n and then (after the double line ends) you can communicate just as you would communicate with example.com port 1234 without the proxy (as I understand you already have the client-server communication part done).
Have a look at stunnel.
Stunnel can allow you to secure
non-SSL aware daemons and protocols
(like POP, IMAP, LDAP, etc) by having
Stunnel provide the encryption,
requiring no changes to the daemon's
code