I have been looking into using ZODB as a persistence layer for a multiplayer video game. I quite like how seamlessly it integrates with arbitrary object-oriented data structures. However, I am stumbling over one issue, where I can't figure out, whether ZODB can resolve this for me.
Apparently, one can use the ClientStorage from ZEO to access a remote data storage used for persistence. While this is great in a trusted local network, one can't do this without proper authorization and authentication in an open network.
So I was wondering, if there is any chance to realize the following concept with ZODB:
On the server-side I would like to have a ZEO server running plus a simulation of the game world that might operate as a fully authorized client on the ZEO server (or use the same file storage as the ZEO server).
On the client side I'd need very restricted read/write access to the ZEO server, so that a client can only view the information its user is supposed to know about (e.g. the surrounding area of their character) and can only modify information related to the actions that their character can perform.
These restrictions would have to be imposed by the server using some sort of fine-grained authorisation scheme. So I would need to be able to tell the server whether user A has permissions to read/write object B.
Now is there way to do this in ZODB or third-party solutions for this kind of problem? Or is there a way to extend ZEO in this way?
No, ZEO was never designed for such use.
It is designed for scaling ZODB access across multiple processes instead, with authentication and authorisation left to the application on top of the data.
I would not use ZEO for anything beyond a local network anyway. Use a different protocol to handle communication between game clients and game server instead, keeping the ZODB server side only.
Related
I am trying to develop a python PyQt program that allow user to enter data about personal particulars and review them at a later time for processing purpose.
The program will be used by less than 5 persons at the same time. So, i am thinking to use Sqlite3 database as i believe it should be able to cope for that amount of traffic.
The frame work i have in mind is that, the clients will have their own copy of my python pyqt program on each machine. Whenever they perform any operations that required data read/write, it will connect to the server thorough internet and read/write from the sqlite.db on the server.
Basically, the server will be nothing but a remote data storage.
Currently, i am able to create the required GUI for data inputs by using various widgets like QlineEdit, QCombobox, QTextEdit and so on.
But i have never done network programming before, thus i have no idea how to implement a server that store the sqlite data file for my software. So my questions are
(1) if i have a PC that has 24/7 internet connection, how do i set it up so that it can act as a server that store the data file for my software?
(2)In what way can/should my program communicate to that server through internet.
Even if you can't give me exact answer, i would appreciate if you can provide me some information of so that i look up and study about it.
Any constructive advice will be appreciated.
FYI: all the PCs will be running windows XP SP3 32 bits.
There are different ways for a client to communicate with a server.
You can use
XMLRPC to create an object with methods that are called on the server side
You can use HTTP and REST for the server with the library requests or urllib for the client
For the latter you can use flask, bottle, django or other frameworks to create a website that serves the content
(tutorials)
You can use Pyro to remotely access the objects on the server. Useful if the clients should also communicate with eachother.
You can create your own protocol. You will learn a lot and value the other options.
The list is not complete
I suggest that you have a look at XMLRPC if that fits. For number 2 I can say that many APIs use such a HTTP-interface (twitter, github, facebok, google). It is easy to use also for other people.
Security is important. I am not an expert. If you send username and password in plain text then use SSL to encrypt the connection. If you can not get ssl to work with python you can use stunnel.
I spent quite some time now with researching Server Backends/API/Frameworks. I need a solution where I can store user content (JSON & Binary data).
The obvious choice would be a REST API. The only missing element is a push feature when data on server changed and clients should be notified instantly. With more research in this matter I discovered classic approaches (Comet, Push, Server sent events, Bayeux, BOSH, …) as well as the „new“ league, Websockets. I would definitely prefer the method with Websockets or using directly TCP Sockets. But this post is not about pros/cons of these two technologies so please restrain yourself from getting side tracked in comments.
At moment exists following projects which are very similar to my needs:
- Simperium (simperium.com), this looks very promising, but core/server is sadly not open source and god knows when, if ever, this step happens
- Realtime.co (framework.realtime.co/storage), hosted service, but same principle
- Some Frameworks for building servers such as Atmosphere (java, no WAMP), Cometd (java, project page looks like stuck in the 90’s), Autobahn (python, WAMP)
My actual favorite is the Autobahn framework (autobahn.ws). Especially using the WAMP protocol (subset of Websocket) as it offers exactly what I need. So the idea would be to build a python backend/server with Autobahn Python (based on Twisted framework) which manages all socket (WAMP) connections and include a Postgresql database for data storing. For all desired clients exists already WAMP libraries. The server would need to be able to do the typical REST API features:
- Send, update, delete requested data (JSON/Binary) from/to server/clients
- Synchronize & automatic conflict management
- Offline handling when connection breaks, automatic restart when connection available again
So finally the questions:
- Have I missed an open source project which covers exactly my needs?
- If I would like to develop my own server with autobahn and a database, could you point me to right direction? Have lot of concerns and not enough depth understanding.. I know Autobahn gives you already a server, but this one is not very close to my final needs.. how to build a server efficient so that he can handle all connected sockets? How handle when a client needs server push? Are there schemas, models or concept how such a server should look like?
- Twisted is a very powerful python framework but not regarded as the most convenient for writing apps.. But I guess a Socket based storage server with db access should be possible? When I run twisted as a web ressource and develop server components with other python framework, would this compromise the latency/performance much?
- Is such a desired server backend with lot of data storage (JSON fields and also binary data such as documents, images) reasonable to build with Sockets by a single devoloper/small team or is this smth. which only bigger companies like Dropbox can do at the moment?
Thank you very much for your help & time!
So finally the questions:
Have I missed an open source project which covers exactly my needs?
No you've covered the open source projects. Open source only gets you about halfway there though. To implement a Global Realtime Network requires equal parts implementation and equal parts operations. You have to think about dropped messages, retries, what happens if a particular geography gets hot how do you scale your servers ...etc. I would argue that an open source solution won't achieve what you want unless you're willing to invest significant resources into operations. I would recommend a service like PubNub: http://pubnub.com
If I would like to develop my own server with autobahn and a database, could you point me to right direction? Have lot of concerns and not enough depth understanding.. I know Autobahn gives you already a server, but this one is not very close to my final needs.. how to build a server efficient so that he can handle all connected sockets? How handle when a client needs server push? Are there schemas, models or concept how such a server should look like?
A good database to back a realtime framework would be Cassandra because it supports high write volumes and handles time series data well: http://cassandra.apache.org/.
Twisted is a very powerful python framework but not regarded as the most convenient for writing apps.. But I guess a Socket based storage server with db access should be possible? When I run twisted as a web ressource and develop server components with other python framework, would this compromise the latency/performance much?
I would not use Twisted. I would use Gevent:http://www.gevent.org/. Its coroutine based so you don't get into callback hell. To support more connections you just increase your greenlet pool to listen on the socket.
Is such a desired server backend with lot of data storage (JSON fields and also binary data such as documents, images) reasonable to build with Sockets by a single devoloper/small team or is this smth. which only bigger companies like Dropbox can do at the moment?
Again I would not build this on your own. A service like PubNub: http://pubnub.com which takes care of all the operational issues for you and has a clean API would service your needs with minimal cost. PubNub takes care of the protocol for you so if your on a mobile device that doesn't support WebSockets it will use TCP, HTTP or whatever the best transport is for the device.
I will keep it short.
Can someone please point me in the right direction in:
How to authenticate users in native applications written in Python?
I know in web there are sessions, but I can't think of a way to implement authentication, that will 'live' for some time and on expiry I can logout the user?
EDIT:
I am referring to desktop type of apps, I am fairly happy with the implementation for Web based development in Twisted
EDIT 2
The application I am thinking about will not authenticate against a server, but a self-contained application, an example the idea is a Cash Register/Point of Sale (my idea is kinda different, but parts of the functionality is the same), in which I need to authenticate the cashier, so I can log the transactions processed by him/her, print name on receipt and etc. All will be based in one single machine, no server communication or anything
It’s not entirely clear what kind of security you are expecting.
In general, if the end user has physical access to the machine and a screwdriver, you’re pretty much screwed—they can do whatever they want on that machine.
If you take hardware security as a given, but want to ensure software security, then you’re going to have to do server communication within the machine’s boundaries. You have to separate the server and the client, and run the server in a security context that is inaccessible to the user. The server will then do both the authentication and whatever operations need authentication (printing out receipts etc.). For example, under a Unix-like OS, you would run a daemon under a dedicated system user or under root; on Windows, you would have a system service running as LOCAL SERVICE or whatever that’s called. In this way, the operating system’s built-in security features will ensure (given proper maintenance, like timely application of security hotfixes) that the user cannot influence the behavior of the software that does the sensitive operations. The protocol between the client and the server can be anything, and you can do authentication in much the same way as in HTTP—indeed, you may even use HTTP itself.
Finally, if you’re certain that your users will not be tampering with your system at all—e.g. because they lack the technical skills, or are being watched by CCTV cameras—you can forget all that stuff and go with Puciek’s answer.
You seem to be very confused and fixated on "sessions" for some reasons, maybe because your background is in the web apps?
Any-who you don't need "sessions" because with desktop application you have no trouble telling who is using the software without needing some elaborate tools. You don't need server, you don't need authentication tools, you don't need anything - just store that user within your single application. That is all really - a variable within your application called "user" and maybe some interface at the boot to pick one from available users.
And if you need it to last between boots, just save it in a file and read from it.
If you're using Unix, rely on the fact that it's a multi user system. That is, the user has already logged in using his own credentials, so you don't need to do anything, just use its home directory to store the data, taking care to block other users from accessing it by using permissions. You can improve this to provide encryption too. For global application data, you can specify a "manager" user or group, with its own directory, where the application can write.
All this might be possible on Windows systems too.
I am having a little trouble with a server / client design and
wonder if anyone had any advice.
I have a Thrift server that abstracts a data store. The idea is that
there will be a number of clients that are essentially
out of process plugins that use the interface provided by the server
to receive, manipulate the underlying data store and also provide
their own data.
There will be a number of other clients which simply access the data
provided by the server and its "plugins".
The problem case is when one of these "plugins" wishes to provide its
own data and provide an interface to that data.
The server should have no knowledge of the plugins data or interface.
I would ideally like all clients to access functionality through the
main thrift server so it acts as a facade for the plugins. If a client
requested some data provided by a plugin the main server could
delegate to the plugin to provide that data. I guess this would mean
have each plugin being a thrift client and server. I have written the
server in python so could probably handle thrift calls that are not
yet defined but would it be possible to forward these calls another
thrift server IE act as a proxy ?
An alternative is maybe have the plugins be clients only and push data
to the server. But the format of these messages
would unknown to the server and would have to be generic enough to
accommodate different types of data. I not sure how I would provide a
useful interface to this data to other clients.
As far as I can see only the plugins knows how to store and manipulate
the data it owns so this idea probably would not work.
Thanks for any advice. Any suggestions welcomed.
Sounds like you need some sort of a mechanism to correlate requests to the different plugins available. Ideally, there should be a different URL path per set of operations published for each plugin.
I would consider implementing a sort of map/dictionary of URL paths to plugins. Then for each request received, do a lookup in the map and get the associated plugin and send it the request accordingly. If there is no entry in the map, then a redirect/proxy could be sent. For example if URL = http://yourThriftServer/path/operation, the operation or the path and operation would map to a plugin.
An extra step would be to implement a sort of meta request, whereby a client could query what URL paths/operations are available in the server.
Here is where I am at presently. I am designing a card game with the aim of utilizing major components for future work. The part that is hanging me up is creating a layer of abstraction between the server and the client(s). A server is started, and then one or more clients can connect (locally or remotely). I am designing a thick client but my friend is looking at doing a web-based client. I would like to design the server in a manner that allows a variety of different clients to call a common set of server commands.
So, for a start, I would like to create a 'server' which manages the game rules and player interactions, and a 'client' on the local CLI (I'm running Ubuntu Linux for convenience). I'm attempting to flesh out how the two pieces are supposed to interact, without mandating that future clients be CLI-based or on the local machine.
I've found the following two questions which are beneficial, but don't quite answer the above.
Client Server programming in python?
Evaluate my Python server structure
I don't require anything full-featured right away; I just want to establish the basic mechanisms for abstraction so that the resulting mock-up code reflects the relationship appropriately: there are different assumptions at play with a client/server relationship than with an all-in-one application.
Where do I start? What resources do you recommend?
Disclaimers:
I am familiar with code in a variety of languages and general programming/logic concepts, but have little real experience writing substantial amounts of code. This pet project is an attempt at rectifying this.
Also, I know the information is out there already, but I have the strong impression that I am missing the forest for the trees.
Read up on RESTful architectures.
Your fat client can use REST. It will use urllib2 to make RESTful requests of a server. It can exchange data in JSON notation.
A web client can use REST. It can make simple browser HTTP requests or a Javascript component can make more sophisticated REST requests using JSON.
Your server can be built as a simple WSGI application using any simple WSGI components. You have nice ones in the standard library, or you can use Werkzeug. Your server simply accepts REST requests and makes REST responses. Your server can work in HTML (for a browser) or JSON (for a fat client or Javascript client.)
I would consider basing all server / client interactions on HTTP -- probably with JSON payloads. This doesn't directly allow server-initiated interactions ("server push"), but the (newish but already traditional;-) workaround for that is AJAX-y (even though the X makes little sense as I suggest JSON payloads, not XML ones;-) -- the client initiates an async request (via a separate thread or otherwise) to a special URL on the server, and the server responds to those requests to (in practice) do "pushes". From what you say it looks like the limitations of this approach might not be a problem.
The key advantage of specifying the interactions in such terms is that they're entirely independent from the programming language -- so the web-based client in Javascript will be just as doable as your CLI one in Python, etc etc. Of course, the server can live on localhost as a special case, but there is no constraint for that as the HTTP URLs can specify whatever host is running the server; etc, etc.
First of all, regardless of the locality or type of the client, you will be communicating through an established message-based interface. All clients will be operating based on a common set of requests and responses, and the server will handle and reject these based on their validity according to game state. Whether you are dealing with local clients on the same machine or remote clients via HTTP does not matter whatsoever from an abstraction standpoint, as they will all be communicating through the same set of requests/responses.
What this comes down to is your protocol. Your protocol should be a well-defined and technically sound language between client and server that will allow clients to a) participate effectively, and b) participate fairly. This protocol should define what messages ('moves') a client can do, and when, and how the server will react.
Your protocol should be fully fleshed out and documented before you even start on game logic - the two are intrinsically connected and you will save a lot of wasted time and effort by competely defining your protocol first.
You protocol is the abstraction between client and server and it will also serve as the design document and programming guide for both.
Protocol design is all about state, state transitions, and validation. Game servers usually have a set of fairly common, generic states for each game instance e.g. initialization, lobby, gameplay, pause, recap, close game, etc...
Each one of these states has important state data related with it. For example, a 'lobby' state on the server-side might contain the known state of each player...how long since the last message or ping, what the player is doing (selecting an avatar, switching settings, going to the fridge, etc.). Organizing and managing state and substate data in code is important.
Managing these states, and the associated data requirements for each is a process that should be exquisitely planned out as they are directly related to volume of work and project complexity - this is very important and also great practice if you are using this project to step up into larger things.
Also, you must keep in mind that if you have a game, and you let people play, people will cheat. It's a fact of life. In order to minimize this, you must carefully design your protocol and state management to only ever allow valid state transitions. Never trust a single client packet.
For every permutation of client/server state, you must enforce a limited set of valid game messages, and you must be very careful in what you allow players to do, and when you allow them to do it.
Project complexity is generally exponential and not linear - client/server game programming is usually a good/painful way to learn this. Great question. Hope this helps, and good luck!