i have a VPS which has 3 different IP addresses.
and i have a python script on it which crawl an specific website every 1 hour
for example this is my python request:
my_request = requests.get('https://example.com/timeline.json')
the only thing that i want is every time my python traffic goes out from the server it uses that IP addresses randomly
So after long deliberation in the comments, I can point you to some resources that hopefully will lead you to an answer. It's difficult to give a definitive answer to this question, because I'm unaware what kind of virtual infrastructure you are using or what the network looks like.
Here is a thread which had a similar desire, but was ultimately left unanswered. However, it is clear that if you are using AWS you will need to use VPC. And you would have to configure your ENI's in a special way to intermittently use your public IP addresses, to learn more about ENI's in VPC see here
This article details exactly what you are trying to accomplish, but using a Vyatta router with a special NAT configuration. There is a also mention of being able to accomplish this using advanced iptables rules, which might be worth looking into.
Regardless, you cannot accomplish delivering traffic using various public IP addresses from inside the script that is doing the crawling. To have this effect you will need to somehow modify the host network configuration in a special way (advanced iptable rules or periodically changing default routes), change the configuration of your virtual router (special NAT/routing rules), or using methods specific to your virtual hosting platform (Amazon VPC).
Related
I want to intercept all destinations, so I can reroute them, kind of like a virtual lan. How would I intercept and find the hostname of a destination packet?
I've searched the web but I haven't found anything. I would like it to be like a device driver, it starts and waits for web browsers to request a specific IP or domain name, and reroute it to a different IP or domain name.
You do that using a (local) "proxy" process. There are several solutions to set up such a "web proxy". You can even write one using a few lines of python capturing HTTP-traffic.
However, since most HTTPS web traffic is nowadays protected by SSL/TLS, you probably can't inspect the plain text details of the internet traffic without resorting to specific techniques.
What I want
I want to get client's IP in my TCP server and use it to whitelist/bind actions, detect regions (for language and currency), etc.
How I approached it
I'm using TornadoWeb framework for python to set up my tcp server.
It contains TCPServer.handler_stream(stream: IOStream, address: tuple[str, int])
from where i can get a hand on client's IP address. It's all good when running with host network (i.e. exposing my ports directly)
Problem
If launched multiple tasks (containers) in docker swarm, and therefore use docker's load balancer, client's IP address gets replaced with docker's inner one.
Question
How do i configure docker swarm or another load balancer to somehow send real client's ip to my server?
I'm not particularly bound to one piece of software or another, nor do i know which is better, the only thing I wouldn't consider - is using Kubernetes (not now at least), so any suggestion on software rather than configuration are also welcome!
Client and server can be altered, so other techniques I could use regardless of protocol itself will also be useful. However, something like using HTTPs requests to gather IP address via X-Forwarded-For or X-Real-IP headers is possible, but i'd like to refrain myself from it.
How do i configure docker swarm or another load balancer to somehow send real client's ip to my server?
Docker has already an open issue for that, see https://github.com/docker/roadmap/issues/157 . It is currently not possible to do any configuration to do that.
As you already mentioned earlier, you will have to use some custom way of handling that until the issue is closed.
The easiest solution would be replacing the load balancer with another one like nginx, here's a nifty guide for swarm
Where in the nginx configuration file you just need to keep the original request headers to keep tabs on them, you can see how to that here but the gist is to add an nginx directive with the X-Forwarded-For variable to it for when you proxy the request to one of your load balanced instances
Depending on how many instances you've got using docker-compose maybe easier than hassling with swarm which is a bit less bare bone
I am currently working on a developmental website and I need to be able to move it to a public domain, so I went and made the purchase of a domain and now i want to make the django app i have written publically available without purchasing a web hosting service quite yet or I might plan on investing in an actually windows server, if anyone knows how to go about doing this purely for developmental use and proff of concept practice, it would be great to see a video or a step by step guide.
This is a really broad question, so I will break it down for you in bits.
Your domain name has 'DNS records'. There are several types but the one you are looking for is the an A record. This type of DNS record points the 'domain.com' to an IP adress. The IP adress needs to be your external IP (can be found by going to api.ipify.org for example). This will make sure that when people on the internet will go to 'domain.com', the traffic is routed to you.
Then, on your own router, you will probably need to configure a NAT rule (e.g. all incoming traffic on port 80 needs to be routed to local IP:80 <ip_adress_of_your_computer>). Often this is called 'port forwarding' on your router, but that depends on the model you have.
After that, your web app is available through your domain name. Note that this is only very basic and does not take into account SSL (https).
I have a web page (front-end) created with ASP.NET (visual studio) on one laptop and a server (back-end) for my web page on another laptop. In the first place, I have a login page and I want to take the username and the password from textBox-es (front-end) and somehow transfer the data to the other laptop to verify if it's correct and then receive if the data it's correct or not. The front-end is on Windows and the server is on Linux. My teammate is using python on back-end.
I'm working with HTML, CSS and a little bit of C# on the front-end and I don't know how to transfer that username and password to my teammate and how to receive an answer from him.
He has the database on his laptop and he's the one who will verify if the pair is correct or not and send me an answer, also he wants to code in python.
We think that we can do some magic connection between ASP.NET and Node.js.
How do I do this data transfer ?
It's my first time doing this so please be gentle because I really don't know what to do.
Your question is quite broad and while I am not that fluent in web applications I will share what advice I can. Because the question is broad, you cannot get an exact, short and concise, answer.
Some clarifications and feedback regarding your question.
I have a web page (front-end) created with ASP.NET ...
You actually have a web application here (that, in your case, includes one, or more, web pages). Give more attention to such little details, programming in itself is a form of grammar natzy-ism but it’s not grammar and you are not being corrected by a person, but by machines and life. A nice definition of web application: https://blog.stackpath.com/web-application/
We think that we can do some magic connection between ASP.NET and Node.js.
I am not specialized in web application but it appears to me that this line comes completely out of the blue, you have not indicated that you are using or intend to use Node.js in your back-end; you indicated that you were using Python, so I will ignore this statement. Oh and there is no magic, there are only race conditions.
Structurally speaking, I would split this question into several smaller ones. Some suggestions follow.
Because you are attempting a somewhat more complicated task you cannot escape de idea of a design pattern (especially because of using ASP.NET which is designed to be patterning friendly).
So your first question should be similar to
how to communicate with a web server asp.net
Two-way communication between ASP.NET Web App and C# Application
back-end is not python but read the answer, it gives you a valid indicator, that request-response pattern could help you
search for questions regarding how to implement that pattern in C#/ASP.NET
Another possible way of how to-ing would be
how to transmit data to server C#
a bit old but reading might give you an understanding of how the ecosystem works
C# server and client communication - send/receive data
maybe helps?
https://codingvision.net/networking/c-sending-data-using-get-or-post
https://www.csharp-examples.net/socket-send-receive/
From you explication I presume that you already have implemented the UI forms that will retrieve the input data so I will skip UI related questions.
Since you wish to send user SENSITIVE data over the internet then you MUST think about security. Further questions
how to store password in database securely
this question will actually lead you to examples of transmitting the password and maybe even receiving the content
https://www.meziantou.net/how-to-store-a-password-in-a-web-application.htm
How to store passwords in database securely?
There are 2 more issues I wish to discuss.
While searching the internet for tutorials you will probably see a
lot working with client and server applications on the same machine.
Those are still valid tutorials for you! Read them! I will explain
further down why it's (almost) the same for you even if your case is
a remote server.
You need to somehow be able to test your application.
Addressing #1
Why working with localhost (127.0.0.1) is for you the same as working with a real server (in your case).
Let's say you finish the application, the back-end is also made and you wish to communicate with one another. You read on the internet that you need to known the IP of the server so you ask your partner to for him to give his IP. You search the wild wild web and reach the conclusion that to find your IP you would try:
executing ipconfig.exe (or what tool you use depending on your platform) and getting the IPv4 Address. . . . . . . . . . . : 192.168.1.12 value.
you would presume that that is the IP you need to put in your application to connect to. This is FALSE.
reaching the conclusion that that does not work, you find that you can get your IP with online sites such as https://whatismyipaddress.com/
and use that IP for your server address. This (in 99% cases) is FALSE.
A bit of poetry
Consider IP as you address in the world and your local (private) IP as the exact place in your house. The IP address you see there is a local IP and it would indicate in which chair you are sitting in, let’s say the 12 from the already provided example. Anyone in the world (who has 12 chairs anyway) can sit in the 12th chair in their house. You cannot use this information to identify the address of someone.
How would you know that that IP is a local IP?
One way would be to look at the the full output of ipconfig.exe (or whatever tool you are using) where you see a Wireless LAN adapter Wi-Fi: above from where you got your IP, the LAN there (hxxps://en.wikipedia.org/wiki/Local_area_network) is an indicator of that.
The other (more straightforward) is by recognizing the IP range (or visual pattern, if IP range is not a clear concept for you just yet). If it starts with 198.168. then it's a local/private IP address. Please read here more: https://en.wikipedia.org/wiki/Private_network
IPs are a valuable commodity (I joke you not). As the world has evolved you cannot now simply plug your computer with an internet cable, and voila you have your own IP. This was (relatively) true when the internet first started decades ago but now the IPs are managed by big (huge) companies that they allocate them as they consider.
The IP address space is managed globally by the Internet Assigned Numbers Authority (IANA), and by five regional Internet registries (RIRs) responsible in their designated territories for assignment to local Internet registries, such as Internet service providers, and other end users.
Wiki quote: https://en.wikipedia.org/wiki/IP_address
I also highly (please do) recommend you read the following answer
hxxps://superuser.com/questions/323801/how-can-i-own-an-ip-address
If you ask your (real world) neighbors to look up https://whatismyipaddress.com/ (and they have the same ISP and monthly plan as you) there is a strong possibility that they will see the same IP as you see. To put it simply, the IP you see there, when accessing sites such as whatismyipaddress, is the IP that that your ISP has assigned to the network node you are in. It would be a waste of IPs to assign each user an IP (not to mention there actually aren’t mathematically that many IPs to do so).
Regarding your current predicament. I can see 3 options.
Option 1 - buy hosting
If your response is "I really really wanna have a separate IP" (or domain) then you will need to buy from a hosting service an IP (the good kind) that will point to a device the hosting company has. You usually buy domains that point to IPs, but let's not got there for now. You will connect to that machine/devices and install your code on the server there. Then, and only then, you could get the IP/domain of that machine and use it in your code and it will work (this is the 1% cases where sites like whatismyipaddress will actually help you).
Recommended searches: server hosting cheap :D (or something free for students if you are students)
Option 2 - work locally
If your project is used for academia and there is no actual need for a separate server then you and your partner will run the client/server applications while connected to the same wireless modem. Ask him to get his IP in this way (here you actually need the LAN IP), you hardcoded it in your code (preferably a separate config file, this way no extra compilation is required) and voila, it works.
Option 3 - LAN over the internet
Technically this would be hard to describe but the idea is you use programs that communicate over the internet, you each install it on your computer, somehow connect with each other (usually a name/password pair) and that software make is seem for the programs on your computer that you are actually in a LAN setup (Option 2). In my days I used hamachi so that we could play CS online :D.
hxxps://www.howtogeek.com/172762/how-to-play-old-lan-games-over-the-internet/
Regarding #2
Until your partner provides a server you will need to test your code. So you will need to somehow fake a server response so that you can actually continue development.
Quite simply, the operation of faking something, a return/a class, in your code, so that you can correctly develop and test your code is known as mocking
how to mock server response asp.net
hxxps://gingter.org/2018/07/26/how-to-mock-httpclient-in-your-net-c-unit-tests/
For beginning programmers I think this is one of the most complicated parts. The concept of mocks and tests, while they do appear simple, are usually not engraved into the mind of young, innocent, previously untainted, future programmers. They are left stuck with printfs until graduations (even a dose of hate towards tests is sprinkled in the mix, what a pity).
As a courtesy, I wish to further extend that madness. Please, for all that is good and sacred in this world, do not use this code in production, burn it after you used it to understand some of the basics, BURN IT!
I modified the code from here Python 3.x BaseHTTPServer or http.server so that you could run a local HTTP server on your machine and test your application. You simply run it (python3) and test the sending of user name and password to the server using the following pagan syntax:
127.0.0.1:4242/?user=aba&password=Aba1 which would return I'al cohol you later
#!/usr/bin/python
from http.server import BaseHTTPRequestHandler,HTTPServer
from urllib.parse import urlparse, parse_qs
HOST = '127.0.0.1' # Standard loopback interface address (localhost)
PORT_NUMBER = 4242 # Port to listen on (non-privileged ports are > 1023)
# !!!!!!!!!!!!!!! THIS IS ONLY FOR DIDACTICAL PURPOSE< NEVER EVER DO SOMETHING LIKE THIS IN PRODUCTION!!!!!!!
# NEVER EVER STORE PASSWORDS LIKE THIS !!!!!!!
# LITERRLAY NEVER!!!!!!
data_base = [
{
'user': 'ionzapada',
'password': 'neaomat',
'data': b"I like pie!"
},
{
'user': 'aba',
'password': 'Aba1',
'data': b"I'al cohol you later"
}
]
class myHandler(BaseHTTPRequestHandler):
#Handler for the GET requests
def do_GET(self):
self.send_response(200)
self.send_header('Content-type','text/html')
self.end_headers()
# Send the html message
query_components = parse_qs(urlparse(self.path).query)
# DO NOT USE THIS LOGIC ON ANY PRODUCTION PLATFORM!!!!
# NEVER SEND PASSWORDS THIS WAY!!!
# NEVER SEND PASSWORDS!!!!
# NEVER SEND HASHES WITHOUT SALT!!!!
# NEVER IMPLEMENT YOUR OWN TRICKY ALGORITHM!!!!
user_name = query_components.get("user",[None])[0]
password = query_components.get("password",[None])[0]
if user_name and password:
for data_base_entry in data_base:
if data_base_entry['user'] == user_name:
print('Found searched user: {}'.format(user_name))
if data_base_entry['password'] == password:
print('Valid password for user {} given; returning db data'.format(user_name))
self.wfile.write(data_base_entry['data'])
else:
self.wfile.write(b"Not supported!")
else:
self.wfile.write(b"Not supported!")
return
try:
server = HTTPServer((HOST, PORT_NUMBER), myHandler)
print('Started httpserver on port', PORT_NUMBER)
server.serve_forever()
except KeyboardInterrupt:
print('CTRL+C received, shutting down the web server')
server.socket.close()
I burned my finger writing that, do not ever send user names and passwords in this way!!! DO NOT EVEN SEND PASSWORDS FOR THE LOVE OF GOD!!!!!!!! search for standard and secure ways of doing this!!!!
Please also visit the links indicated by #amc they are an also good start.
P.S. I am not allowed to have more then 8 link in one post, on some links that I have posted change the hxxp to http for them to work.
Hey I was wondering if anyone knew how to connect to a bitcoin wallet located on another server with bitcoinrpc
I am running a web program made in django and using a python library called bitcoinrpc to make connections.
When testing locally, I can use bitcoinrpc.connect_to_local), or even bitcoinrpc.connect_to_remote('account','password') and this works as well as long as the account and password match the values specified in my 'bitcoin.conf' file. I can then use the connection object to get values and do some tasks in my django site.
The third parameter in connect_to_local is default localhost. I was wondering:
A) What to specify for this third parameter in order to connect from my webserver to the wallet stored on my home comp (is it my IP address?)
B) Because the wallet is on my PC and not some dedicated server, does that mean that my IP will change and I won't be able to access the wallet?
C) The connection string is in the django app - which is hosted on heroku. Heroku apps are launched by pushing with git but I believe it is to a private repository. Still, if anyone could see the first few lines of my 'view' they would have all they need to take my BTC (or, more accurately, mBTC). Anyone know how bad this is - or any ways to go about doing btc payments/movements in a more secure way.
Thanks a lot.
I'm currently doing something very similar (heroku using express/nodejs instead of django/python tho) so I will try to share my thoughts.
In spite of using other library and other language, all the wallet remote libraries should be primarily a wrapper around JSON RPC (remote procedure call) API, which is actually the same for most of the coins out there (i would say all, but that would be a wild guess).
Specifically to your questions:
A)
To access the wallet from outside, use your external ip (fastest way to find it is to query google for it). Depending on your ISP you hopefully have static external address. You must provide this address to bitcoin.conf file under rpcallowip= option to allow incomming connections.
Moreover you should forward the used port in your home router (usually under NAT settings) to your local machine so the incoming connection from the server is allowed and redirected to your wallet computer.
There is one important thing to consider (https://en.bitcoin.it/wiki/Running_Bitcoin):
By default, only RPC connections from localhost are allowed. Specify
as many rpcallowip= settings as you like to allow connections from
other hosts (and you may use * as a wildcard character).
NOTE: opening up the RPC port to hosts outside your local
trusted network is NOT RECOMMENDED, because the rpcpassword
is transmitted over the network unencrypted.
I am yet to look into it further, from this comment alone it seems totally unusable for monetary transactions.
B)
As I said before, it depends on your home ISP, type of connection and the service provided to you.
C)
If I understand correctly from a django point of view, as long as the login parameters (username/password) are inside a view (views.py of your app) and the debug mode is turned off, source code of the server should not be publicly accessible. But the security concern from A still applies.
You can use SSL with RPC to hide the password.
rpcssl=1