I have the problem that for a project I need to work with a framework (Python), that has a poor documentation. I know what it does since it is the back end of a running application. I also know that no framework is good if the documentation is bad and that I should prob. code it myself. But, I have a time constraint. Therefore my question is: Is there a cooking recipe on how to understand a poorly documented framework?
What I tried until now is checking some functions and identify the organizational units in the framework but I am lacking a system to do it more effectively.
If I were you, with time constaraints, and bound to use a specific framework. I'll go in the following manner:
List down the use cases I desire to implement using the framework
Identify the APIs provided by the framework that helps me implement the use cases
Prototype the usecases based on the available documentation and reading
The prototyping is not implementing the entire use case, but to identify the building blocks around the case and implementing them. e.g., If my usecase is to fetch the Students, along with their courses, and if I were using Hibernate to implement, I would prototype the database accesss, validating how easily am I able to access the database using Hibernate, or how easily I am able to get the relational data by means of joining/aggregation etc.
The prototyping will help me figure out the possible limitations/bugs in the framework. If the limitations are more of show-stoppers, I will implement the supporting APIs myself; or I can take a call to scrap out the entire framework and write one for myself; whichever makes more sense.
You may also use python debugging library: pdb. After importing it with import pdb you may set traces in the body of functions and classes pdb.set_trace(). Then it will stop the execution of the program in the line and you may look at existing variables and processes.
Related
We have a back end that exposes 50-60 Rest APIs. These will largely be consumed by standalone applications like a Python script or a Java program.
One issue we have is the APIs are at a very granular level, they do not match the business use case. For example to perform a business use case end user might have to call 4 to 5 APIs.
I want to develop a DSL or some solution that will help provide a high level abstraction that will enable end users to implement business use cases with ease. This can either be a standalone abstraction or a "library" for Python or or some much high level programming language.
For the specific purpose of combining multiple Rest API calls to create a business use case transaction, what are the approaches available.
Thanks
I think this is a nice idea. To determine what kind of solution you could build you should consider different aspects:
Who would write these API combinations?
What kind of tool support would be appropriate? I mean validation, syntax highlighting, autocompletion, typesystem checks, etc
How much time would make sense to invest on it?
Depending on these answers you could consider different options. The simplest one is to build a DSL using ANTLR. You get a parser, then you build some program to process the AST and generate the code to call the APIs. Your user will just have to edit these programs in a text editor with not support. The benefit of this is that the cost of implementing this is reduced and your user could write these programs using a simple text editor.
Alternatively you could use a Language Workbench like Xtext or Jetbrains MPS to build some specific editors for your language and provide a better editing experience to your users.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
In order to make an web application with RESTful capabilities I have read and watched tons of articles and videos and I still do not get a complete picture of how it works. And which I should chose. Every other answer is the not helpful ”it depends”. I have boiled it down to a first choice between Django and Node. But nowhere I find the whole ”picture” of how the pieces works together and which modules are needed. Therefore I have tried to put all into a rough illustration. Note that I am a complete newbie on this.
I develop an ERP application with accounting modules. Basically it is mainly about CRUD besides viewing diagrams, printing and storing documents. So this is the ”it depends”
The only thing I have managed to make decisions about is to use nginx, Postgresql and Debian 8 as tools/os. These are the fixed stars.
My questions are not really the common Django vs Node.js and it is not just an opinion I want:
Is the picture below correct? Any comments?
Is there any further components that will be needed? To get started?
You have a lot of questions - and on StackOverflow there should be one question that can be answered without generating a lot of debate or have opinions rather than facts.
As such, I think your question might be closed as "too broad"; however I think it deserves an answer.
I am not going to say "it depends", although that's really all it boils down to - but here is my attempt to explain it.
nodejs is a runtime. It is an environment which allows you to develop code on the server using javascript. In order to do anything useful with nodejs, beyond "hello world"; you'll need to use a framework, and there are tons of those around and various stacks have been developed by the community to tie in all the components together. An example of such a stack is MEAN, which is MongoDB for the database, Express for the framework, Angular to assist with the front-end, and Node to run it all.
django is a framework - it is not a runtime. This means that it is one step removed from the node world. The runtime for django is Python. django also is not a "stack" like MEAN, you can develop your own stack on top of it - but since django is a "batteries included" framework, you only really need to add a database to it - it includes everything else you need.
REST is just a way of designing web-services. Its not a language, or a platform or a library. Its a set of rules that describe a way to design APIs such that they take advantage of the semantic verbs of HTTP.
You can use any library and programming language to develop a RESTful service. All you really need is two things [a] a library to communicate over HTTP [b] a way to serialize data, preferably in JSON (but even that's not a requirement).
nginx is just a very fast webserver and a reverse proxy. The reason it is mentioned often - is because it is very expensive for a framework to serve static media. All requests to a framework (either in django world, or in nodejs world) have to go through a large chain of components that help decode the HTTP request and create a data structure that is easy for developers to use. This chain of components is often called middleware. Since each and every request has to go through this middleware, it is better for performance reasons that requests that don't need the "power" of the application to execute (like a request for an image, a stylesheet, a video file) be handled by something else. This is what nginx is used for, since its a very fast webserver.
Now that those are explained, you need to see what stack works best for your application. To do that, you need to know a bit about the philosophy/justification or problem that each stack is trying to solve.
For django - this is easy. Django was created by a team working on multiple newspapers to help them manage content that was published on different sites. As such, it is designed so that the management of content is of primary concern. That is why it has a very robust administration console as a standard component; and a built-in quite robust ORM and its own templating engine. Django leaves it up to you to figure out how best to actually run and deploy it; although they do provide a lot of suggestions and examples - but in the end, its upto you to decide which database to use, which web server to use, and how to deploy the application.
In the nodejs world - the primary focus is nonblocking I/O and speed of response. Nodejs excels at being able to serve a lot of simultaneous requests on limited resources. Therefore, it provides you a very powerful foundation to develop applications that need to quickly respond to requests ... and that's it. When you program in node or any other specialized lower-level library, you need to make sure your code is taking complete advantage of the library. So, if you start writing blocking code in node, you'll find that the performance that you expect hasn't been achieved.
nodejs doesn't care what the application actually does. Think of it like a very fast, very powerful tool. You can build anything with it, but you need to know what the tool is designed to do best in order to know when to use it.
nodejs has you working at a lower level - which is why there are a lot of packages that help you do all sorts of things with node; and multiple ways you can take components and create your own stack - depending on what you are building on top of node. Think of it like Lego building blocks.
nodejs and django are not mutually exclusive. You can utilize both in your application and exploit their strengths and take advantage of what each does best.
As far as your specific questions:
Did I get the picture? Any comments?
I don't know. Did you?
Is there any further components that will be needed? To get started?
The answer to this is yes, because you don't want to build everything from scratch. Each stack has its own libraries components for developing services. For django, there is django rest framework (DRF).
Which framework are best for CRUD?
Which framework are best for RESTful? Any other module needed?
Best report generator for printing?
Best diagram tools?
There is nothing that is "best" for anything. This question is just asking for opinions. Its like asking, what is the best fruit juice?
Which framework are fastest and most reliable for CRUD using Postgresql
People have developed many robust applications on top of postgresql; however as nodejs is bound to javascript - there is still a lot of work being done in this area.
Can I lock the library (trade secrets) in both environments?
Yes.
Is there better tools for creating ERP/Accounting?
ERP and accounting are two very different things. There are tons of accounting packages/applications written in Python. There are very few ERP systems written in Python.
You cannot combine the two and lump it together.
What is the benefits using Angular on top of Node.js?
The same benefit of using Angular on top of _____ (insert your favorite backend). Angular is just a front end library.
An anecdotal benefit is that both Angular and Node use the same programming language.
I am building my first Django web application and I need a bit of advice on code layout.
I need to integrate it with several other applications that are exposed through RESTful APIs and additionally Django's internal data. I need to develop the component that will pull data from various sources, django's DB, format it consistently for output and return it for rendering by the template.
I am thinking of the best way to write this component. There are a couple of ways to proceed and I wanted to solicit some feedback from more experienced web developers on things I may have missed.
Way #1
Develop a completely standalone objects for interacting with other applications via their APIs, etc... This would not have anything related with django and test independently. Then in django, import this module in the views that need it, run object methods to get required data, etc...
If I need to access any of this functionality via a GET request (like through JavaScript), I can have a dedicated view that imports the module and returns json.
Way #2
Develop this completely as django view(s) expose as a series of GET/POST calls that would all call each other to get the work done. This would be directly tied in the application.
Way #3
Start with Way #1, but instead of creating a view, package it as a django app. Develop unit tests on the app as well as the individual objects.
I think that way 1 or 3 would be very much encapsulated and controlled.
Way 2 would be more complicated, but facilitate higher component re-use.
What is better from a performance standpoint? If I roll with Way #1 or 3, would an instance of the object be instantiated for each request?
If so this approach may be a bit too heavy for this. If I proceed with this, can they be singletons?
Anyway, I hope this makes sense.
thanks in advance.
Definitely go for "way #1". Keeping an independent layer for your service(s) API will help a lot later when you have to enhance that layer for reliability or extending the API.
Stay away from singletons, they're just global variables with a new name. Use an appropriate life cycle for your interface objects. The obvious "instantiate for each request" is not the worst idea, and it's easier to optimize that if needed (by caching and/or memoization) than it's to unroll global vars everywhere.
Keep in mind that web applications are supposed to be several processes on a "shared nothing" design; the only shared resources must be external to the app: database, queue managers, cache storage.
Finally, try to avoid using this API directly from the view functions/CBV. Either use them from your models, or write a layer conceptually similar to the way models are used from the views. No need of an ORM-like api, but keep any 'business process' away from the views, which should be concerned only with the presentation and high level operations. Think "thick model, thin views" with your APIs as a new kind of models.
I'm trying to pull similar data in from several third party APIs, all of which have slightly varying schemas, and convert them all into a unified schema to store in a DB and expose through a unified API. This is actually a re-write of a system that already does this, minus storing in the DB, but which is hard to test and not very elegant. I figured I'd turn to the community for some wisdom.
Here are some thoughts/what I'd like to achieve.
An easy way to specify schema mappings from the external APIs schema to the internal schema. I realize that some nuances in the data might be lost by converting to a unified schema, but that's life. This schema mapping might not be easy to do and perhaps overkill from the academic papers I've found on the matter.
An alternative solution would be to allow third parties to develop the interfaces to the external APIs. The code quality of these third parties may or may not be known, but could be established via thorough tests.
Therefore the system should be easy to test, I'm thinking by mocking the external API calls to have reproducible data and ensure that parsing and conversion is being done correctly.
One of the external API interfaces crashing should not bring down the rest of them.
Some sort of schema validation/way to detect if the external API schemas have changed without warning
This will end up being integrated into a Django project, so it could be written as a Django app, which would likely make unit and integration testing easier. On the other hand, I would like to keep it as decoupled as possible from Django. Although the API interfaces would have to know what format to convert to, could this be specified at runtime?
Am I missing anything in the wishlist? Unrealistic? Headed down the wrong path? Would love to get some feedback.
I'm not sure if there are libraries/OS project which already do some of this. The less wheels I have to reinvent the better. Would any part of this be valuable as an OS project?
In the previous version I spawned a bunch of threads that would handle individual requests. Although I've never used it, I've been told I should look at gevent as a way to handle this.
For your second bullet point you should check out Temboo. Temboo normalizes access to over 100 APIs, meaning that you can talk to them all using a common syntax in the language of your choice. In this case you would use the Temboo Python SDK - available here.
(Full disclosure: I work at Temboo)
is there any good reason not to use XML-RPC for an object-broker server/client architecture? Maybe something like "no it's already outfashioned, there is X for that now".
To give you more details: I want to build a framework which allows for standardized interaction and the exchange of results between many little tools (e. g. command-line tools). In case someone wants to integrate another tool she writes a wrapper for that purpose. The wrapper could, e. g., convert the STDOUT of a tool into objects usable by the architecture.
Currently I'm thinking of writing the proof-of-concept server in Python. Later it could be rewritten in C/C++. Just to make sure clients can be written in as many languages as possible I thought of using XML-RPC. CORBA seems to be too bloated for that purpose, since the server shouldn't be too complex.
Thanks for your advice and opinions,
Rainer
XML-RPC has a lot going for it. It's simple to create and to consume, easy to understand and easy to code for.
I'd say avoid SOAP and CORBA like the plague. They are way too complex, and with SOAP you have endless problems because only implementations from single vendors tend to interact nicely - probably because the complexity of the standard leads to varying interpretations.
You may want to consider a RESTful architecture. REST and XML-RPC cannot be directly compared. XML-RPC is a specific implementation of RPC, and REST is an architectural style. REST does not mandate anything much - it's more a style of approach with a bunch of conventions and suggestions. REST can look a lot like XML-RPC, but it doesn't have to.
Have a look at http://en.wikipedia.org/wiki/Representational_State_Transfer and some of the externally linked articles.
One of the goals of REST is that by creating a stateless interface over HTTP, you allow the use of standard caching mechanisms and load balancing mechanisms without having to invent new ways of doing what has already been well solved by HTTP.
Having read about REST, which hopefully is an interesting read, you may decide that for your project XML-RPC is still the best solution, which would be a perfectly reasonable conclusion depending on what exactly you are trying to achieve.