I am running a Dask script on a EC2 instance from AWS. I would like to connect and see the dashboard provided by Dask, but I can't figure out how.
I am creating a Local Cluster on my EC2 instance, the scripts runs fine and I am connecting to my AWS instance via Putty. However, I would like to see the available dashboard: on my PC it is enough to connect to the provided IP and port, but I am not able to do that on the AWS machine.
Once the script is running, this is my output for the "parameters" of the local cluster:
<Client: 'inproc://172.31.29.4/7475/1' processes=1 threads=8, memory=27.94 GiB>
LocalCluster(b8be08dd, 'inproc://172.31.29.4/7475/1', workers=1, threads=8, memory=27.94 GiB)
dashboard address: {'dashboard': 8787}
For example, I tried to write 172.32.29.4:8787/status on my browser, but I wasn't able to connect to the dashboard.
I already checked this question: How to view Dask dashboard when running on a virtual machine? However I am using a Local Cluster and I would like to connect to its dashboard from remote. Is it possible? If so, how?
The answer is in the comments, but I will type it out here, so that the original question looks "answered".
You need two things to connect to a port on an EC2 machine: the external IP, and access. The former is most easily found from the AWS console. For the latter, you typically need to edit the security group to add an inbound TCP rule for the port (either open to the world, or just your IP). There are other ways to do this part, depending on whether your machine is inside a VPC, has any custom gateways or routers... but if you don't know what that means, find the security group first. Both the public IP and the security group will be linked from the machine's row in the EC2 "running instances" list.
I've setup dask-labextension visualizations to provide this type of UI.
Create the client object:
from dask.distributed import Client
client = Client()
Then click the magnifying glass provided by the extension to automatically connect with the cluster.
The detailed instructions are in this post.
Related
I have a question concerning the access of a ray multi node cluster:
Considering a deployed cluster on AWS or kubernetes, how can a
seperate python process from outside the cluster run tasks on the
cluster?
e.g. A Web client tries to invoke python tasks,that should run within the Remote cluster.
EDIT: It comes down to the question:
Is there a Ray native API to connect to a remote cluster from a computer/server outside of the actual cluster?
https://ray.readthedocs.io/en/latest/deploy-on-kubernetes.html#running-ray-programs says you can get a shell on the ray pods and run the tasks or create a new Kubernetes job.
So the question boils down to how do I connect to the Kubernetes cluster from a web application.
You can connect to the Kubernetes cluster from a web application using one of the clients in https://github.com/kubernetes-client. Here is the Python one https://github.com/kubernetes-client/python
You can connect to remote Ray servers (e.g. running in a different container or on a different machine within the same cluster [1]) on the client-server port (10001 by default) rather than the Ray Core (GCS) server (6379 by default), using the standard ray.init(), as long as you prefix the URL or IP with ray://:
For example:
import ray
ray_cluster_ip_or_url = "<remote url redacted>"
# ray_api_server_port = 6379 # Ray API (GCS) server port
ray_client_server_port = 10001 # Ray client-server port
# caution: it is necessary to prefix the URL or IP with "ray://"
# to be able to connect to the client-server port
ray.init(address=f"ray://{ray_cluster_ip_or_url}:{ray_client_server_port}",
ignore_reinit_error=True)
More info
This alternative method does not require the ray:// prefix, but is deprecated:
[..]
ray.util.connect(conn_str=f"{ray_cluster_ip_or_url}:{ray_client_server_port}", secure=False)
[1] The restriction of client-server connections to the same k8s cluster is due to the binary protocols being used by Ray, which cannot be passed though an HTTP proxy outside of the cluster. This is why you need to connect to service ports, not standard HTTP(S) ports.
Hey I was wondering if anyone knew how to connect to a bitcoin wallet located on another server with bitcoinrpc
I am running a web program made in django and using a python library called bitcoinrpc to make connections.
When testing locally, I can use bitcoinrpc.connect_to_local), or even bitcoinrpc.connect_to_remote('account','password') and this works as well as long as the account and password match the values specified in my 'bitcoin.conf' file. I can then use the connection object to get values and do some tasks in my django site.
The third parameter in connect_to_local is default localhost. I was wondering:
A) What to specify for this third parameter in order to connect from my webserver to the wallet stored on my home comp (is it my IP address?)
B) Because the wallet is on my PC and not some dedicated server, does that mean that my IP will change and I won't be able to access the wallet?
C) The connection string is in the django app - which is hosted on heroku. Heroku apps are launched by pushing with git but I believe it is to a private repository. Still, if anyone could see the first few lines of my 'view' they would have all they need to take my BTC (or, more accurately, mBTC). Anyone know how bad this is - or any ways to go about doing btc payments/movements in a more secure way.
Thanks a lot.
I'm currently doing something very similar (heroku using express/nodejs instead of django/python tho) so I will try to share my thoughts.
In spite of using other library and other language, all the wallet remote libraries should be primarily a wrapper around JSON RPC (remote procedure call) API, which is actually the same for most of the coins out there (i would say all, but that would be a wild guess).
Specifically to your questions:
A)
To access the wallet from outside, use your external ip (fastest way to find it is to query google for it). Depending on your ISP you hopefully have static external address. You must provide this address to bitcoin.conf file under rpcallowip= option to allow incomming connections.
Moreover you should forward the used port in your home router (usually under NAT settings) to your local machine so the incoming connection from the server is allowed and redirected to your wallet computer.
There is one important thing to consider (https://en.bitcoin.it/wiki/Running_Bitcoin):
By default, only RPC connections from localhost are allowed. Specify
as many rpcallowip= settings as you like to allow connections from
other hosts (and you may use * as a wildcard character).
NOTE: opening up the RPC port to hosts outside your local
trusted network is NOT RECOMMENDED, because the rpcpassword
is transmitted over the network unencrypted.
I am yet to look into it further, from this comment alone it seems totally unusable for monetary transactions.
B)
As I said before, it depends on your home ISP, type of connection and the service provided to you.
C)
If I understand correctly from a django point of view, as long as the login parameters (username/password) are inside a view (views.py of your app) and the debug mode is turned off, source code of the server should not be publicly accessible. But the security concern from A still applies.
You can use SSL with RPC to hide the password.
rpcssl=1
I'm trying to run a simple FTPServer on EC2. My client connects to it fine, and I can make and remove directories. However, I get the following error whenever I try to store or retrieve a file: Failure: twisted.protocols.ftp.CommandFailed: ["425 Can't open data connection."]
I've tested the server on my localhost, and everything behaves as normal. I was paranoid and decided to turn on all ports in my security groups, but it still doesn't work. Am I missing something when setting up my EC2 machine?
Your FTP client appears to be in active mode, and is (like most client computers these days) behind a NAT or firewall which does not permit inbound connections.
There's another question on Stack Overflow about this topic that quite clearly explains the differences between active and passive mode in FTP; you should read it.
Configure your client to use passive mode, or "PASV", and it ought to work.
I have built a messaging/chat application for my local network (all WINDOWS) using pyzmq and pyqt for UI, it is based on the majordomo pattern. It's setup this way:
each machine on the network has a client/worker pair
they connect to a 'server' broker via pyzmq and register sessions
sessions are broadcasted by 'server' broker to clients
when 'sender' client sends a message to a specific session, broker routes the message to the corresponding worker destination, a reply is generated by worker, and it gets routed by the broker back to the 'sender' client (ending loop, confirming delivery)
Everything is working well, text messages are formed in 'client' pyqt UI and received by 'worker'pyqt UI.
Now I'm looking to build upon this skeleton to add video chat to my application... I have been looking into webRTC and would like to find a way to implement it.
This is how webRTC works From what I gather (could be severely wrong here, please correct me):
Machine A's Chrome browser opens local video/audio stream from webcam/mic via javascript function
webkitGetUserMedia, then creates a (Machine A) URL for the stream via javascript function webkitURL
Sends (Machine A) URL to Machine B's Chrome browser via signaling server
Machine B's Chrome browser accepts and loads (Machine A) URL, sets up it's own local video/audio stream from webcam.mic via previously mentioned javascript functions and replies with a (Machine B) URL back to Machine A via signaling server
Machine A's Chrome browser is displaying (Machine B) video/audio | Machine B's Chrome browser is displaying (Machine A) video/audio
Is that the process? or is this a totally wring assumption of how peers connect to each other?
If Correct , I would like to adapt my current pyzmq application to act as a signaling server for creating connections between machines, Since IP addresses of my machines are known to me and I can configure my firewall to give access to needed ports I'm trying to eliminate any extra STUN/TURN servers for this setup, I am not planning to go outside of my LAN and access remote machines. And I would like to handle everything(as much as possible) with Python and included batteries(Avoiding Node.js).
So the main question is how should I go about integrating webRTC to my setup? Does webRTC need specific prerequisite libraries or API to be built and running on the signaling server or peer machines? any code examples/advice/links would be appreciated.
I have ubuntu instance on Ec2 cloud server and on same instance I have created xmlrpc server using simpleXMLRP. I'd like to access server methods from my local ubuntu machine.but when I tried to do so, it raised "Protocol Error" as below,
"XMLRPC Error : xmlrpclib.ProtocolError: ProtocolError for ec2-70-41-59-2.amazonaws.com:8000/Common: -1 >"
As per the link http://docs.python.org/library/xmlrpclib.html. protocol error will occur, if the server named by the URI does not exist. but server is running on the cloud.
What is this error and how do I fix it ? does any changes required on Amazon cloud, for giving access to particular host and port? if so, what changes should be apply?
This answer may help someone to solve the same problem,
1) Select your (or default) Security Group in Ec2 Section of cloud Server.
2) Select "Inbound" Tab and create new Rule for "All TCP" and give access to your required port.
as per my knowledge, second step will inform cloud server to open selected port for Inbound access from end users.