I got a existing software project, that has a relatively primitive plugin system and wanted to expand it by providing a web interface.
Since my application processes realtime data, websockets are the only option besides web rtc.
My previous attempt used zeromq domain sockets on the python side and a server in Node js that connected to the domain socket.
This solution works great and has some benefits over the plugin server, but I want to offer a simpler option for folks that don't need the benefits and don't want the extra complexity.
How would you go about implementing this and is it even possible to do so?
Otherwise, I'll still do a separate process, but use fastapi to build the stuff around the socket endpoint and start it up using subprocess to spawn a second process that also connects to the domain socket.
Hope my question is not stupid, or a rtfm case.
I've build before server-client programs (both sides where build in python by far).
Recently I started building app using swift and my goal is to add a backend to my apps using python (My app is a chat app)
I searched in the Internet a tutorials to do so, and I only saw two options to communicate between server side and mobile application, the first one is to create an API (REST) (request - response) - I can't use this solution because I want a real-time chat.
And the second option was web-sockets (socket.IO).
SO, my question is why not use the simple socket technology (like I used to use when it was only python server side to python client side -> import sockets) - no sockets over web
following Features You will get if you are using Socket.io or socketcluster.io (which is developed on the top of Socket IO)
scalability :- It will scale horizontally adding more nodes (scale-out) & Linearly(scale-up)
Reduces Payload size as message payload is compressed
Authorisation via middle ware functions
Reconnects Automatically if Connection drops
If You want to use your own implementation then you have to take care of the above features/Solutions to problems which arises when User-base is increases.
My understanding is Socket.IO isn't necessary anymore because all browsers worthwhile constantly keep each other in check. Socket.IO was for when browsers and servers didn't support the same technology. These days, everything is pretty much supported and Socket is perfectly safe to stick to without the use of Socket.IO. More of a breakdown here - https://codeburst.io/why-you-don-t-need-socket-io-6848f1c871cd
Which way is better?
Creating a while loop and then using the select module OR using ThreadedTCPServer with a custom class.
Im having problems with the Threaded TCP Server, although it could just be my coding.
My personal recommendation is to use Twisted. It's a Python-based framework intended primarily for writing event-driven network software. The documentation has a lot of great examples of how to create various types of servers and clients, as well.
I am sure there is no such a thing like the "correct" way.
If you want not, must not or cannot use any of the existing server implementations the general idea is (in pseudo code):
ss = serversocket()
ss.bind ()
while (True):
cs = ss.accept ()
spawnCommThread (cs)
In the CommThread for each client you take care of reading from the socket returned by accept, communicate with your client and die, when the client closes the connection or another criterion is given.
I'm working on writing a Python client for Direct Connect P2P networks. Essentially, it works by connecting to a central server, and responding to other users who are searching for files.
Occasionally, another client will ask us to connect to them, and they might begin downloading a file from us. This is a direct connection to the other client, and doesn't go through the central server.
What is the best way to handle these connections to other clients? I'm currently using one Twisted reactor to connect to the server, but is it better have multiple reactors, one per client, with each one running in a different thread? Or would it be better to have a completely separate Python script that performs the connection to the client?
If there's some other solution that I don't know about, I'd love to hear it. I'm new to programming with Twisted, so I'm open to suggestions and other resources.
Thanks!
Without knowing all the details of the protocol, I would still recommend using a single reactor -- a reactor scales quite well (especially advanced ones such as PollReactor) and this way you will avoid the overhead connected with threads (that's how Twisted and other async systems get their fundamental performance boost, after all -- by avoiding such overhead). In practice, threads in Twisted are useful mainly when you need to interface to a library whose functions could block on you.
The question: How do I create a python application that can connect and send packets over the internet to another computer running the same application? Is there any existing code/library I could use?
The background: I am pretty new to programming (HS senior). I've created a lot of simple things in python but I've recently decided to start on a bigger project. I'm considering creating a Magic: the Gathering booster draft simulator, but I'm not sure if it is feasible given my skill set so I'm asking around before I get started. The application would need to send data between computers about which cards are being picked/passed.
Thanks!
Twisted is a python event-driven networking engine licensed under MIT. Means that a single machine can communicate with one or more other machines, while doing other things between data being received and sent, all asynchronously, and running a in a single thread/process.
It supports many protocols out of the box, so you can just as well using an existing one. That's better because you get support for the protocol from 3rd party software (i.e. using HTTP for communication means middleware software that uses HTTP will be compatible: proxies etc.)
It also makes easy to create your own communication protocol, if that's what you want.
The documentation is filled with examples.
The standard library includes SocketServer (documented here), which might do what you want.
However I wonder if a better solution might be to use a message queue. Lots of good implementations already exist, including Python interfaces. I've used RabbitMQ before. The idea is that the computers both subscribe to the queue, and can either post or listen for events.
A great place to start looking is the Python Standard Library. In particular there are a few sections that will be relevant:
18. Interprocess Communication and Networking
19. Internet Data Handling
21. Internet Protocols and Support
Since you mentioned that you have no experience with this, I'd suggest starting with a HTTP based implementation. The HTTP protocol is fairly simple and easy to work with. In addition, there are nice frameworks to support this operation, such as webpy for the server and HTTPLib from the standard library for the client.
If you really want to dive into networking, then a socket based implementation might be educational. This will teach you the underlying concepts that are used in lots of networking code while resulting in an interface that is similar to a file stream.
Also, check out Pyro (Python remoting objects). Se this answer for more details.
Pyro basically allows you to publish Python object instances as services that can be called remotely. Pyro is probably the easiest way to implement Python-to-python process communication.
It's also worth looking at Kamaelia for this sort of thing - it's original usecase was network systems, and makes building such things relatively intuitive.
Some links: Overview of basic TCP system, Simple Chat server, Building a layered protocol, walk-through of how to evolve new components. Other extreme: P2P radio system: source, peer.
If it makes any difference, we've tested whether the system is accessible to novices through involvement in google summer of code 3 years in a row, actively taking on both experienced and inexperienced developers. All of them managed to build useful systems.
Essentially, if you've ever played with unix pipelines the ideas should be familiar.
Caveat: I wrote major chunks of Kamaelia :-)
If you want to learn how to do these things though, playing with a few different approaches makes sense, and you should definitely check out Twisted (the standard answer to this question), Pyro & the standard library tools. Each has a different approach, and learning them will definitely benefit you!
However, like nosklo, I would recommend against using the socket library directly and use a library instead - simply because it is much much harder to get sockets programming correct than people tend to realise.
Communication will take place with sockets, one way or another. Just a question of whether you use existing higher-level libraries, or roll your own.
If you're doing this as a learning experience, probably want to start as low-level as you can, to see the real nuts and bolts. Which means you probably want to start with a SocketServer, using a TCP connection (TCP is basically guaranteed delivery of data; UDP is not).
Google for some simple example code. Setting one up is very easy. But you will have to define all the details of your communications protocol: which end sends when and what, which end listens and when, what exactly the listener will expect, does it reply to confirm receipt, etc.