Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have to call external REST APIs from Django. The external data source schemas resemble my Django models. I'm supposed to keep the remote data and local ones in sync (maybe not relevant for the question)
Questions:
What is the most logical place from where to call external web services: from a model method or from a view?
Should I put the code that call the remote API in external modules that will be then called by the views?
Is it possible to conditionally select the data source? Meaning presenting the data from the REST API or the local models depending on their "freshness"?
Thanks
EDIT: for the people willing to close this question: I've broken down the question in three simple questions from the beginning and I've received good answers so far, thanks.
I think it is an opinion where to call web services. I would say don't pollute your models because it means you probably need instances of those models to call these web services. That might not make any sense. Your other choice there is to make things #classmethod on the models, which is not very clean design I would argue.
Calling from the view is probably more natural if accessing the view itself is what triggers the web service call. Is it? You said that you need to keep things in sync, which points to a possible need for background processing. At that point, you can still use views if your background processes issue http requests, but that's often not the best design. If anything, you would probably want your own REST API for this, which necessitates separating the code from your average web site view.
My opinion is these calls should be placed in modules and classes specifically encapsulated for your remote calls and processing. This makes things flexible (background jobs, signals, etc.) and it is also easier to unit test. You can trigger calling this code in the views or elsewhere, but the logic itself should be separate from both the views and the models to decouple things nicely.
You should imagine that this logic should exist on its own if there was no Django around it, then build other pieces that connect that logic to Django (ex: syncing the models). In other words, keep things atomic.
Yes, same reasons as above, especially flexibility. Is there any reason not to?
Yes, simply create the equivalent of an interface. Have each class map to the interface. If the fields are the same and you are lazy, in python you can just dump the fields you need as dicts to the constructor (using **kwargs) and be done with it, or rename the keys using some convetion you can process. I usually build some sort of simple data mapper class for this and process the django or rest models in a list comprehension, but no need if things match up as I mentioned.
Another related option to the above is you can dump things into a common structure in a cache such as Redis or Memcache. It might be wise to atomically update this info if you are concerned with "freshness." But in general you should have a single source of authority that can tell you what is actually fresh. In sync situations, I think it's better to pick one or the other to keep things predictable and clear though.
One last thing that might influence your design is that by definition, keeping things in sync is a difficult process. Syncs tend to be very prone to failure, so you should have some sort of durable mechanism such as a task queue or job system for retries. Always assume when calling a remote REST API that calls can fail for crazy reasons such as network hicups. Also keep in mind transactions and transactional behavior when syncing. Since these are important, it points again to the fact that if you put all this logic in a view directly, you will probably run into trouble reusing it in the background without abstracting things a bit anyway.
What is the most logical place from where to call external web services: from a model method or from a view?
Ideally your models should only talk to database and have no clue what's happening with your business logic.
Should I put the code that call the remote API in external modules that will be then called by the views?
If you need to access them from multiple modules, then yes, placing them in a module makes sense. That way you can reuse them efficiently.
Is it possible to conditionally select the data source? Meaning presenting the data from the REST API or the local models depending on their "freshness"?
Of course it's possible. You can just implement how you fetch your data on request. But the more efficient way might just be avoiding that logic and just sync your local data with remote data and show the local data on the views.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
This is less of a question, and more of a "am I on the right track with my train of thought?" topic.
I have an idea for a fairly large scale application written using Pyramid, using multiple databases, and wanted to post some of my musings to see if what I have in mind would work. I'm in the process of designing the application, but haven't started writing any code yet, to test out if it would work or not.
I'm considering it large scale because I anticipate the database growing significantly.
Main points:
The main route would be in the form of www.domain.com/[name]/whatever, where [name] is the key parameter that will decide what data to present to the client.
There will be multiple databases: 1 database, let's say site.db that contains site-wide information and configs, and the rest of the databases will be [name].db, containing each individuals user-specific data. This is where I expect the bulk of the scaling to happen, as I want to design the application to accept 100s of databases, one per user, to compartmentalize the data and prevent data cross-contamination.
Options:
2 databases. site.db and userdata.db. I would use the same database models, but the table names would be determined dynamically using the [name] parameter as such: [name]_table1, [name]_table2, etc, where table1/2/n would be a lot more descriptive than that. This may be simpler in the sense that I'd only ever have 2 database sessions to work with, and as long as I keep the sessions separate, I should be able to query each one individually. The downside is that the userdata.db can grow large with 100s of users. In theory, shouldn't be a problem if I can query the right tables using the [name] parameter.
1 + n databases. Managing database sessions could be a pain here, but one possible solution might be to scan a folder for all .db files, and create a session for each database, and build out a dictionary of sessions, where the dictionary key would be the file name. Something like sessions['site'] points to the DB session that handles site-wide data, while session['name'] points to a session that manipulates name.db - if that makes sense. This way, I can use the [name] parameter in the route to use the appropriate database session.
Aside from the above, there will be a sub-application as well, in the form of an IRC bot, that will monitor for chat events and commands to manipulate the same databases. I'm thinking one way to do this would be to run instances of the IRC bot, passing in [name] as a parameter, so that each instance of the IRC bot is only accessing one database. There's still the possibility that the IRC bot and Pyramid application may be trying to manipulate the same database, which could be problematic. Maybe one way around this would be for the IRC bot to import the database sessions (as mentioned above in point #2), and use the same sessions within the IRC bot application (would that take advantage of Pyramid's transaction manager? Presumably yes).
That's all I have right now. I hope I'm thinking about this correctly. Is there anything that I'm grossly mistaken on?
There's a lot to unpack in your question and it's probably too much for SO but I have a few comments.
I would strongly urge avoiding a per-tenant schema where you're doing dynamic naming of tables. So I'd avoid option 1. Session management is always difficult, and sort of depends on your choice of database (for example, pgbouncer can greatly help by keeping persistent connections open to a lot of databases without complicating your code as much).
Manipulating the same database isn't the end of the world - any decent database has concurrency support and can handle multiple connections so long as you handle your locking correctly.
Consider whether it's really worth separating the databases as it definitely adds a lot of complexity up front. It depends on the problem domain and use cases whether it's worth it.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
In order to make an web application with RESTful capabilities I have read and watched tons of articles and videos and I still do not get a complete picture of how it works. And which I should chose. Every other answer is the not helpful ”it depends”. I have boiled it down to a first choice between Django and Node. But nowhere I find the whole ”picture” of how the pieces works together and which modules are needed. Therefore I have tried to put all into a rough illustration. Note that I am a complete newbie on this.
I develop an ERP application with accounting modules. Basically it is mainly about CRUD besides viewing diagrams, printing and storing documents. So this is the ”it depends”
The only thing I have managed to make decisions about is to use nginx, Postgresql and Debian 8 as tools/os. These are the fixed stars.
My questions are not really the common Django vs Node.js and it is not just an opinion I want:
Is the picture below correct? Any comments?
Is there any further components that will be needed? To get started?
You have a lot of questions - and on StackOverflow there should be one question that can be answered without generating a lot of debate or have opinions rather than facts.
As such, I think your question might be closed as "too broad"; however I think it deserves an answer.
I am not going to say "it depends", although that's really all it boils down to - but here is my attempt to explain it.
nodejs is a runtime. It is an environment which allows you to develop code on the server using javascript. In order to do anything useful with nodejs, beyond "hello world"; you'll need to use a framework, and there are tons of those around and various stacks have been developed by the community to tie in all the components together. An example of such a stack is MEAN, which is MongoDB for the database, Express for the framework, Angular to assist with the front-end, and Node to run it all.
django is a framework - it is not a runtime. This means that it is one step removed from the node world. The runtime for django is Python. django also is not a "stack" like MEAN, you can develop your own stack on top of it - but since django is a "batteries included" framework, you only really need to add a database to it - it includes everything else you need.
REST is just a way of designing web-services. Its not a language, or a platform or a library. Its a set of rules that describe a way to design APIs such that they take advantage of the semantic verbs of HTTP.
You can use any library and programming language to develop a RESTful service. All you really need is two things [a] a library to communicate over HTTP [b] a way to serialize data, preferably in JSON (but even that's not a requirement).
nginx is just a very fast webserver and a reverse proxy. The reason it is mentioned often - is because it is very expensive for a framework to serve static media. All requests to a framework (either in django world, or in nodejs world) have to go through a large chain of components that help decode the HTTP request and create a data structure that is easy for developers to use. This chain of components is often called middleware. Since each and every request has to go through this middleware, it is better for performance reasons that requests that don't need the "power" of the application to execute (like a request for an image, a stylesheet, a video file) be handled by something else. This is what nginx is used for, since its a very fast webserver.
Now that those are explained, you need to see what stack works best for your application. To do that, you need to know a bit about the philosophy/justification or problem that each stack is trying to solve.
For django - this is easy. Django was created by a team working on multiple newspapers to help them manage content that was published on different sites. As such, it is designed so that the management of content is of primary concern. That is why it has a very robust administration console as a standard component; and a built-in quite robust ORM and its own templating engine. Django leaves it up to you to figure out how best to actually run and deploy it; although they do provide a lot of suggestions and examples - but in the end, its upto you to decide which database to use, which web server to use, and how to deploy the application.
In the nodejs world - the primary focus is nonblocking I/O and speed of response. Nodejs excels at being able to serve a lot of simultaneous requests on limited resources. Therefore, it provides you a very powerful foundation to develop applications that need to quickly respond to requests ... and that's it. When you program in node or any other specialized lower-level library, you need to make sure your code is taking complete advantage of the library. So, if you start writing blocking code in node, you'll find that the performance that you expect hasn't been achieved.
nodejs doesn't care what the application actually does. Think of it like a very fast, very powerful tool. You can build anything with it, but you need to know what the tool is designed to do best in order to know when to use it.
nodejs has you working at a lower level - which is why there are a lot of packages that help you do all sorts of things with node; and multiple ways you can take components and create your own stack - depending on what you are building on top of node. Think of it like Lego building blocks.
nodejs and django are not mutually exclusive. You can utilize both in your application and exploit their strengths and take advantage of what each does best.
As far as your specific questions:
Did I get the picture? Any comments?
I don't know. Did you?
Is there any further components that will be needed? To get started?
The answer to this is yes, because you don't want to build everything from scratch. Each stack has its own libraries components for developing services. For django, there is django rest framework (DRF).
Which framework are best for CRUD?
Which framework are best for RESTful? Any other module needed?
Best report generator for printing?
Best diagram tools?
There is nothing that is "best" for anything. This question is just asking for opinions. Its like asking, what is the best fruit juice?
Which framework are fastest and most reliable for CRUD using Postgresql
People have developed many robust applications on top of postgresql; however as nodejs is bound to javascript - there is still a lot of work being done in this area.
Can I lock the library (trade secrets) in both environments?
Yes.
Is there better tools for creating ERP/Accounting?
ERP and accounting are two very different things. There are tons of accounting packages/applications written in Python. There are very few ERP systems written in Python.
You cannot combine the two and lump it together.
What is the benefits using Angular on top of Node.js?
The same benefit of using Angular on top of _____ (insert your favorite backend). Angular is just a front end library.
An anecdotal benefit is that both Angular and Node use the same programming language.
I am building my first Django web application and I need a bit of advice on code layout.
I need to integrate it with several other applications that are exposed through RESTful APIs and additionally Django's internal data. I need to develop the component that will pull data from various sources, django's DB, format it consistently for output and return it for rendering by the template.
I am thinking of the best way to write this component. There are a couple of ways to proceed and I wanted to solicit some feedback from more experienced web developers on things I may have missed.
Way #1
Develop a completely standalone objects for interacting with other applications via their APIs, etc... This would not have anything related with django and test independently. Then in django, import this module in the views that need it, run object methods to get required data, etc...
If I need to access any of this functionality via a GET request (like through JavaScript), I can have a dedicated view that imports the module and returns json.
Way #2
Develop this completely as django view(s) expose as a series of GET/POST calls that would all call each other to get the work done. This would be directly tied in the application.
Way #3
Start with Way #1, but instead of creating a view, package it as a django app. Develop unit tests on the app as well as the individual objects.
I think that way 1 or 3 would be very much encapsulated and controlled.
Way 2 would be more complicated, but facilitate higher component re-use.
What is better from a performance standpoint? If I roll with Way #1 or 3, would an instance of the object be instantiated for each request?
If so this approach may be a bit too heavy for this. If I proceed with this, can they be singletons?
Anyway, I hope this makes sense.
thanks in advance.
Definitely go for "way #1". Keeping an independent layer for your service(s) API will help a lot later when you have to enhance that layer for reliability or extending the API.
Stay away from singletons, they're just global variables with a new name. Use an appropriate life cycle for your interface objects. The obvious "instantiate for each request" is not the worst idea, and it's easier to optimize that if needed (by caching and/or memoization) than it's to unroll global vars everywhere.
Keep in mind that web applications are supposed to be several processes on a "shared nothing" design; the only shared resources must be external to the app: database, queue managers, cache storage.
Finally, try to avoid using this API directly from the view functions/CBV. Either use them from your models, or write a layer conceptually similar to the way models are used from the views. No need of an ORM-like api, but keep any 'business process' away from the views, which should be concerned only with the presentation and high level operations. Think "thick model, thin views" with your APIs as a new kind of models.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
First I want to clearify that I mean by reverse engineering something like "decompiling" and getting back the original source code or something similiar.
Yesterday I read a question about someone who wanted to protect his python code from "getting stolen" in other words: he didn't like that someone can read his python code.
The interesting thing I read was that someone said that the only reliable way to "protect" his code from getting reverse engineered is by using a Webservice.
So I could actually only write some GUIs in Python, PHP, whatever and do the "very secret code" I want to protect via a Webservice. (Basically sending variables to the host and getting results back).
Is it really impossible to reverse engineer a Webservice (via code and without hacking into the Server)? Will this be the future of modern commercial applications? The cloud-hype is already here. So I wouldn't wonder.
I'm very sorry if this topic was already discussed, but I couldn't find any resources about this.
EDIT: The whole idea reminds me of AJAX. The code is executed on the server and the content is sent to the client and "prettified". The client himself doesnt see what php-code or other technology is behind.
Wow, this is awesome! I've never thought it this way, but you could create a program that crawls an api, and returns as an output a django/tastypie software that mimics everything the api does.
By calling the service, and reading what it says, you can parse it, and begin to see the relationships between objects inside the api. Having this, you can create the models, and tastypie takes it from this point.
The awesome thing about this, is that normal people (or at least not backend developers) could create an api just by describing what they want to be as an output. I've seen many android/iphone developers creating a bunch of static xml or json, so they can call their service, and start the frontend development. Well what if that was enough? Take some xml/json files as input, get a backend as an output.
Yes,
All they could do is treat your web service as a black box, query the WSDL for all the parameters it accepts and the data that it returns.
They could then submit different variables and see what different results are. The "code" could not be seen or stolen (with proper security) but the inputs and outputs could be duplicated.
If you want to secure your "very secret code" a web service is a great way to protect the actual code.
-sb
It depends on what you mean by reverse engineering: by repeatedly sending input and analyzing the output the behaviour of your code can still be seen. I wouldn't have your code but I can still see what the system does. This means I could build a similar system that does the same thing, given the same input.
It would be hard to catch exceptional cases (such as output that is different on one day of the year only) but the common behaviour can certainly be copied. It is similar to analyzing the protocol of an instant messaging client: you may not have the original code but you can still build a copy.
A few weeks ago I asked the question "Is a PHP, Python, PostgreSQL design suitable for a non-web business application?" Is a PHP, Python, PostgreSQL design suitable for a business application?
A lot of the answers recommended skipping the PHP piece and using Django to build the application. As I've explored Django, I've started to question one specific aspect of my goals and how Django comes into play for a non-web business application.
Based on my understanding, Django would manage both the view and controller pieces and PostgreSQL or MySQL would handle the data. But my goal was to clearly separate the layers so that the database, domain logic, and presentation could each be changed without significantly affecting the others. It seems like I'm only separating the M from the VC layers with the Django solution.
So, is it counterproductive for me to build the domain layer in Python with an SQL Alchemy/Elixir ORM tool, PostgreSQL for the database layer, and then still use Django or PHP for the presentation layer? Is this possible or pure insanity?
Basically, I'd be looking at an architecture of Django/PHP > Python/SQLAlchemy > PostgreSQL/MySQL.
Edit: Before the fanboys get mad at me for asking a question about Django, just realize: It's a question, not an accusation. If I knew the answer or had my own opinion, I wouldn't have asked!
You seem to be saying that choosing Django would prevent you from using a more heterogenous solution later. This isn't the case. Django provides a number of interesting connections between the layers, and using Django for all the layers lets you take advantage of those connections. For example, using the Django ORM means that you get the great Django admin app almost for free.
You can choose to use a different ORM within Django, you just won't get the admin app (or generic views, for example) along with it. So a different ORM takes you a step backward from full Django top-to-bottom, but it isn't a step backward from other heterogenous solutions, because those solutions didn't give you intra-layer goodness the admin app in the first place.
Django shouldn't be criticized for not providing a flexible architecture: it's as flexible as any other solution, you just forgo some of the Django benefits if you choose to swap out a layer.
If you choose to start with Django, you can use the Django ORM now, and then later, if you need to switch, you can change over to SQLalchemy. That will be no more difficult than starting with SQLalchemy now and later moving to some other ORM solution.
You haven't said why you anticipate needing to swap out layers. It will be a painful process no matter what, because there is necessarily much code that relies on the behavior of whichever toolset and library you're currently using.
Django will happily let you use whatever libraries you want for whatever you want to use them for -- you want a different ORM, use it, you want a different template engine, use it, and so on -- but is designed to provide a common default stack used by many interoperable applications. In other words, if you swap out an ORM or a template system, you'll lose compatibility with a lot of applications, but the ability to take advantage of a large base of applications typically outweighs this.
In broader terms, however, I'd advise you to spend a bit more time reading up on architectural patterns for web applications, since you seem to have some major conceptual confusion going on. One might just as easily say that, for example, Rails doesn't have a "view" layer since you could use different file systems as the storage location for the view code (in other words: being able to change where and how the data is stored by your model layer doesn't mean you don't have a model layer).
(and it goes without saying that it's also important to know why "strict" or "pure" MVC is an absolutely horrid fit for web applications; MVC in its pure form is useful for applications with many independent ways to initiate interaction, like a word processor with lots of toolbars and input panes, but its benefits quickly start to disappear when you move to the web and have only one way -- an HTTP request -- to interact with the application. This is why there are no "true" MVC web frameworks; they all borrow certain ideas about separation of concerns, but none of them implement the pattern strictly)
You seem to be confusing "separate layers" with "separate languages/technologies." There is no reason you can't separate your concerns appropriately within a single programming language, or within an appropriately modular framework (such as Django). Needlessly multiplying programming languages / frameworks is just needlessly multiplying complexity, which is likely to slow down your initial efforts so much that your project will never reach the point where it needs a technology switch.
You've effectively got a 3 layer architecture whether you use Django's ORM or SQLAlchemy, though your forgo some of the Django's benefits if you choose the latter.
Based on my understanding, Django would manage both the view and controller pieces and PostgreSQL or MySQL would handle the data.
Not really, Django has its own ORM, so it does separate data from view/controller.
here's an entry from the official FAQ about MVC:
Where does the “controller” fit in, then? In Django’s case, it’s probably the framework itself: the machinery that sends a request to the appropriate view, according to the Django URL configuration.
If you’re hungry for acronyms, you might say that Django is a “MTV” framework – that is, “model”, “template”, and “view.” That breakdown makes much more sense.
At the end of the day, of course, it comes down to getting stuff done. And, regardless of how things are named, Django gets stuff done in a way that’s most logical to us.
There's change and there's change. Django utterly seperates domain model, business rules and presentation. You can change (almost) anything with near zero breakage. And by change I am talking about meaningful end-user focused business change.
The technology mix-n-match in this (and the previous) question isn't really very helpful.
Who -- specifically -- benefits from replacing Django templates with PHP. It's not more productive. It's more complex for no benefit. Who benefits from changing ORM's. The users won't and can't tell.