I am designing/prototyping a Domain Specific Language... in Python, for now, at least. The design is straightforward - but requiring support to specify an arbitrary function (the domain of which is a map from labels to integers - the range is an integer.) In many cases, the function will merely select a label in the domain to yield a result... but I want to allow the specification of any function that could be easily (and efficiently) implemented in a general purpose programming language.
A caveat is that I want the function to be 'safe'... by this I mean:
A 'pure' function: deterministic with no side effects. (i.e. no external state; no interaction with files, I/O, devices - etc.)
Terminating - either successfully, or after specific (small-scale) allocated computational resources have expired.
I am keen that this function should be implemented efficiently - I expect definitions to be provided infrequently - and evaluated very frequently. I would also like the functions to be defined using a familiar syntax.
I've considered supporting the implementation of functions in python... I'm aware that I could impose restrictions using the eval() function, and I've found the AST module - suggesting an approach involving parsing to an AST, then interpreting (or verifying, prior to evaluation) the AST tree. I've also read about pyparse and consdered implementing a bespoke, interpreted, language.
I can't help think that trying to block undesirable behaviour from eval() is to be tackling the problem "backwards" (trying to block undesirable functionality ex-post) whereas implementing a bespoke language would involve re-inventing the wheel.
Does Python already have a safe, efficient, embeddable, expression interpreter?
PyPy has a sandbox.
If you're running this in the web browser (the usual place for untrusted code concerns) consider running it client-side with something like Brython. No-one cares if the user hacks his own machine.
If you do implement a bespoke interpreter, you don't have to re-implement all of the wheel. It's thought to be relatively safe to use compile() on untrusted code, but beware of large constants eating time and memory. Run the compiler in a separate process you can kill. Then you just need to write a Python bytecode interpreter that lacks access to anything important.
I was reading Dietel's C++ programming book. In this book they mention how a programmer should release only the interface part of his code and not the implementation.
So carrying this over to python:
I have 2 files:
1) the implementation file = accountClass.py and
2) the interface file = useAccountClass.py
I have compiled the implementation file and have obtained the .pyc file. So when I provide my code to someone else, I would provide him with the .pyc file and the interface file, right?
Also, if I provide someone else with ONLY the .pyc file, can I expect him to write the interface on his own? I'm going to say no. But there's this one nagging doubt that I have:
The creators of numpy and scipy did not share the implementation with us end users. And I don't think they shared any interfaces either. But we can still search for the different classes and their methods inside both numpy and scipy. So, using this example of numpy and scipy, I guess what I'm trying to ask is:
Is it possible for someone else to create an interface to my code if I provide him/ her with only the compiled implementation file (in this case accountClass.pyc)? How will that person know what classes and methods I have defined in my implementation? I mean, will they use the
if __name__ = "__main__" :
blah blah
or is there some other way??
You got that entirely wrong. Or perhaps it's a horrible book whose author got something seriously wrong. Code using other code should indeed, barring significant counterarguments, adhere to an interface and not care about the details of the implementation. However, even in the world of static compilation to machine code (e.g. C++), this does not mean you should lock away the source code of the implementation.
Whether someone has access to the implementation, and whether they make use of that knowledge while writing a specific piece of code, are completely different issues. Heck, even the author of the implementation can/should still program to an interface when working on other code (e.g. other modules). Likewise, even if you lock the implementation away from someone, they may very well rely on implementation quirks which are not part of the interface. If anyone in the world of static compilation to machine code provides only headers and object files, and not the source code, it's because the projects are closed source, not to encourage good programming practices among clients.
In Python, your question makes no sense - there are no "interface" and "implementation" files, there's just code which is run and defines functions, classes, and other values. There is no such thing as an interface file you'd provide. You provide an implementation - and (hopefully) documentation which details both interface and possibly implementation details. And once a module is imported, the class objects, function objects, and other objects, contain plenty of information (including, in many cases, the text from which large parts of the documentation was generated). This is also true for extension modules like numpy. And note that their implementation is accessible, it's just not included in all distributions because it's of little use. With Python code, you practically have to distribute the source code because anything else is platform-specific.
On a side note, .pyc files are pretty high level, and easily understood when disassembled (which is as easy as importing the module and running the stdlib module dis on any function inside). I consider this a minor technicality as it's already the wrong question to ask.
Deitel's advice to C++ programmers doesn't apply to Python, for a number of reasons:
Python isn't compiled to machine code, so no matter what form you provide the program in, it will be relatively easy for someone to read the code.
Python doesn't have .h and .c files, all you can provide is the .py or .pyc files.
Treating code as a secret is kind of silly anyway. What is in your code that you need to keep hidden from others?
Numpy and Scipy are largely implemented in C, which is why you don't have the source, for your own convenience. You can get the source if you like. The "interface" to that code is the module that you can import and then call.
You should not confuse "user interface" with "class interface". If you have a useAccountClass file, that file probably performs some task using the classes and methods defined in the accountClass file, if I understood right.
If you send the file to other person, they are not supposed to "guess" what your compiled class does. That's what DOCUMENTATION is for: a description of the functions contained in the module (compiled or not), which parameters they take, which values they return, and what they are expected to do, the "meaning" of the task they perform.
As an abstract example, let's suppose you have an image processing class. If that class has the function findCircles(image), the documentation should explain that it takes an image, possibly containing circles, and returns a list or array of coordinates of the centers of circles contained in the image. HOW the circles are detected is not important, you don't need to know that to use the function. Now if the function was called like findCircles(image, gaussian_threshold=10), the caller would have to know the function uses some "gaussian_threshold" parameter, that is, the caller would NEED to know about the function's entrails, and in OOP this is Not Good. If you decided to use another algorithm in the future, every code using that function would have to be rewritten, because the gaussian_threshold most probably wouldn't make sense anymore.
So, the interface, in OOP, is the abstraction used to communicate to the object only the canonical parameters or inputs it needs to know to perform a task in the language of the problem, not in the language of the implementation (that can change anytime).
The documentation, in this sense, is a contract that assures to the user (in this case, another developer) that the function will perform as expected if sane inputs are given to it.
Now the FINAL USER, a non-technical person wanting to use your program, would need the WHOLE working program (controls and views), not only the class definitions (the model).
Hope this helps, and I must recommend the books "Code Complete 2nd ed." and "Pragmatic Programmer - From Journeyman to Master" as VERY enlightening readings on the broad topic.
To ask my very specific question I find I need quite a long introduction to motivate and explain it -- I promise there's a proper question at the end!
While reading part of a large Python codebase, sometimes one comes across code where the interface required of an argument is not obvious from "nearby" code in the same module or package. As an example:
def make_factory(schema):
entity = schema.get_entity()
...
There might be many "schemas" and "factories" that the code deals with, and "def get_entity()" might be quite common too (or perhaps the function doesn't call any methods on schema, but just passes it to another function). So a quick grep isn't always helpful to find out more about what "schema" is (and the same goes for the return type). Though "duck typing" is a nice feature of Python, sometimes the uncertainty in a reader's mind about the interface of arguments passed in as the "schema" gets in the way of quickly understanding the code (and the same goes for uncertainty about typical concrete classes that implement the interface). Looking at the automated tests can help, but explicit documentation can be better because it's quicker to read. Any such documentation is best when it can itself be tested so that it doesn't get out of date.
Doctests are one possible approach to solving this problem, but that's not what this question is about.
Python 3 has a "parameter annotations" feature (part of the function annotations feature, defined in PEP 3107). The uses to which that feature might be put aren't defined by the language, but it can be used for this purpose. That might look like this:
def make_factory(schema: "xml_schema"):
...
Here, "xml_schema" identifies a Python interface that the argument passed to this function should support. Elsewhere there would be code that defines that interface in terms of attributes, methods & their argument signatures, etc. and code that allows introspection to verify whether particular objects provide an interface (perhaps implemented using something like zope.interface / zope.schema). Note that this doesn't necessarily mean that the interface gets checked every time an argument is passed, nor that static analysis is done. Rather, the motivation of defining the interface is to provide ways to write automated tests that verify that this documentation isn't out of date (they might be fairly generic tests so that you don't have to write a new test for each function that uses the parameters, or you might turn on run-time interface checking but only when you run your unit tests). You can go further and annotate the interface of the return value, which I won't illustrate.
So, the question:
I want to do exactly that, but using Python 2 instead of Python 3. Python 2 doesn't have the function annotations feature. What's the "closest thing" in Python 2? Clearly there is more than one way to do it, but I suspect there is one (relatively) obvious way to do it.
For extra points: name a library that implements the one obvious way.
Take a look at plac that uses annotations to define a command-line interface for a script. On Python 2.x it uses plac.annotations() decorator.
The closest thing is, I believe, an annotation library called PyAnno.
From the project webpage:
"The Pyanno annotations have two functions:
Provide a structured way to document Python code
Perform limited run-time checking "
In Java IoC / DI is a very common practice which is extensively used in web applications, nearly all available frameworks and Java EE. On the other hand, there are also lots of big Python web applications, but beside of Zope (which I've heard should be really horrible to code) IoC doesn't seem to be very common in the Python world. (Please name some examples if you think that I'm wrong).
There are of course several clones of popular Java IoC frameworks available for Python, springpython for example. But none of them seems to get used practically. At least, I've never stumpled upon a Django or sqlalchemy+<insert your favorite wsgi toolkit here> based web application which uses something like that.
In my opinion IoC has reasonable advantages and would make it easy to replace the django-default-user-model for example, but extensive usage of interface classes and IoC in Python looks a bit odd and not »pythonic«. But maybe someone has a better explanation, why IoC isn't widely used in Python.
I don't actually think that DI/IoC are that uncommon in Python. What is uncommon, however, are DI/IoC frameworks/containers.
Think about it: what does a DI container do? It allows you to
wire together independent components into a complete application ...
... at runtime.
We have names for "wiring together" and "at runtime":
scripting
dynamic
So, a DI container is nothing but an interpreter for a dynamic scripting language. Actually, let me rephrase that: a typical Java/.NET DI container is nothing but a crappy interpreter for a really bad dynamic scripting language with butt-ugly, sometimes XML-based, syntax.
When you program in Python, why would you want to use an ugly, bad scripting language when you have a beautiful, brilliant scripting language at your disposal? Actually, that's a more general question: when you program in pretty much any language, why would you want to use an ugly, bad scripting language when you have Jython and IronPython at your disposal?
So, to recap: the practice of DI/IoC is just as important in Python as it is in Java, for exactly the same reasons. The implementation of DI/IoC however, is built into the language and often so lightweight that it completely vanishes.
(Here's a brief aside for an analogy: in assembly, a subroutine call is a pretty major deal - you have to save your local variables and registers to memory, save your return address somewhere, change the instruction pointer to the subroutine you are calling, arrange for it to somehow jump back into your subroutine when it is finished, put the arguments somewhere where the callee can find them, and so on. IOW: in assembly, "subroutine call" is a Design Pattern, and before there were languages like Fortran which had subroutine calls built in, people were building their own "subroutine frameworks". Would you say that subroutine calls are "uncommon" in Python, just because you don't use subroutine frameworks?)
BTW: for an example of what it looks like to take DI to its logical conclusion, take a look at Gilad Bracha's Newspeak Programming Language and his writings on the subject:
Constructors Considered Harmful
Lethal Injection
A Ban on Imports (continued)
IoC and DI are super common in mature Python code. You just don't need a framework to implement DI thanks to duck typing.
The best example is how you set up a Django application using settings.py:
# settings.py
CACHES = {
'default': {
'BACKEND': 'django_redis.cache.RedisCache',
'LOCATION': REDIS_URL + '/1',
},
'local': {
'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
'LOCATION': 'snowflake',
}
}
Django Rest Framework utilizes DI heavily:
class FooView(APIView):
# The "injected" dependencies:
permission_classes = (IsAuthenticated, )
throttle_classes = (ScopedRateThrottle, )
parser_classes = (parsers.FormParser, parsers.JSONParser, parsers.MultiPartParser)
renderer_classes = (renderers.JSONRenderer,)
def get(self, request, *args, **kwargs):
pass
def post(self, request, *args, **kwargs):
pass
Let me remind (source):
"Dependency Injection" is a 25-dollar term for a 5-cent concept. [...] Dependency injection means giving an object its instance variables. [...].
Part of it is the way the module system works in Python. You can get a sort of "singleton" for free, just by importing it from a module. Define an actual instance of an object in a module, and then any client code can import it and actually get a working, fully constructed / populated object.
This is in contrast to Java, where you don't import actual instances of objects. This means you are always having to instantiate them yourself, (or use some sort of IoC/DI style approach). You can mitigate the hassle of having to instantiate everything yourself by having static factory methods (or actual factory classes), but then you still incur the resource overhead of actually creating new ones each time.
Django makes great use of inversion of control. For instance, the database server is selected by the configuration file, then the framework provides appropriate database wrapper instances to database clients.
The difference is that Python has first-class types. Data types, including classes, are themselves objects. If you want something to use a particular class, simply name the class. For example:
if config_dbms_name == 'postgresql':
import psycopg
self.database_interface = psycopg
elif config_dbms_name == 'mysql':
...
Later code can then create a database interface by writing:
my_db_connection = self.database_interface()
# Do stuff with database.
Instead of the boilerplate factory functions that Java and C++ need, Python does it with one or two lines of ordinary code. This is the strength of functional versus imperative programming.
It seems that people really don't get what Dependency injection and inversion of control mean anymore.
The practice of using inversion of control is to have classes or functions that depend on other classes or functions, but instead of creating the instances whithin the class or function code it is better to receive them as parameters, so loose coupling can be achieved. That has many benefits as more testability and to achieve the liskov substitution principle.
You see, by working with interfaces and injections, your code gets more maintainable, since you can change the behavior easily, because you won't have to rewrite a single line of code (maybe a line or two on the DI configuration) of your class to change its behavior, since the classes that implement the interface your class is waiting for can vary independently as long as they follow the interface. One of the best strategies to keep code decoupled and easy to maintain is to follow at least the single responsibility, substitution and dependency inversion principles.
What's a DI library good for if you can instantiate an object yourself inside a package and import it to inject it yourself? The chosen answer is right, since java has no procedural sections (code outside of classes), all that goes into boring configuration xml's, hence the need of a class to instantiate and inject dependencies on a lazy load fashion so you don't blow away your performance, while on python you just code the injections in the "procedural" (code outside classes) sections of your code.
Haven't used Python in several years, but I would say that it has more to do with it being a dynamically typed language than anything else. For a simple example, in Java, if I wanted to test that something wrote to standard out appropriately I could use DI and pass in any PrintStream to capture the text being written and verify it. When I'm working in Ruby, however, I can dynamically replace the 'puts' method on STDOUT to do the verify, leaving DI completely out of the picture. If the only reason I'm creating an abstraction is to test the class that's using it (think File system operations or the clock in Java) then DI/IoC creates unnecessary complexity in the solution.
Actually, it is quite easy to write sufficiently clean and compact code with DI (I wonder, will it be/stay pythonic then, but anyway :) ), for example I actually perefer this way of coding:
def polite(name_str):
return "dear " + name_str
def rude(name_str):
return name_str + ", you, moron"
def greet(name_str, call=polite):
print "Hello, " + call(name_str) + "!"
_
>>greet("Peter")
Hello, dear Peter!
>>greet("Jack", rude)
Hello, Jack, you, moron!
Yes, this can be viewed as just a simple form of parameterizing functions/classes, but it does its work. So, maybe Python's default-included batteries are enough here too.
P.S. I have also posted a larger example of this naive approach at Dynamically evaluating simple boolean logic in Python.
IoC/DI is a design concept, but unfortunately it's often taken as a concept that applies to certain languages (or typing systems). I'd love to see dependency injection containers become far more popular in Python. There's Spring, but that's a super-framework and seems to be a direct port of the Java concepts without much consideration for "The Python Way."
Given Annotations in Python 3, I decided to have a crack at a full featured, but simple, dependency injection container: https://github.com/zsims/dic . It's based on some concepts from a .NET dependency injection container (which IMO is fantastic if you're ever playing in that space), but mutated with Python concepts.
I think due to the dynamic nature of python people don't often see the need for another dynamic framework. When a class inherits from the new-style 'object' you can create a new variable dynamically (https://wiki.python.org/moin/NewClassVsClassicClass).
i.e.
In plain python:
#application.py
class Application(object):
def __init__(self):
pass
#main.py
Application.postgres_connection = PostgresConnection()
#other.py
postgres_connection = Application.postgres_connection
db_data = postgres_connection.fetchone()
However have a look at https://github.com/noodleflake/pyioc this might be what you are looking for.
i.e. In pyioc
from libs.service_locator import ServiceLocator
#main.py
ServiceLocator.register(PostgresConnection)
#other.py
postgres_connection = ServiceLocator.resolve(PostgresConnection)
db_data = postgres_connection.fetchone()
pytest fixtures all based on DI (source)
Check out FastAPI, it has dependency injection built-in. For example:
from fastapi import Depends, FastAPI
async def get_db():
db = DBSession()
try:
yield db
except Exception:
db.rollback()
raise
finally:
db.close()
app = FastAPI()
#app.get("/items")
def get_items(db=Depends(get_db)):
return db.get_items()
I back "Jörg W Mittag" answer: "The Python implementation of DI/IoC is so lightweight that it completely vanishes".
To back up this statement, take a look at the famous Martin Fowler's example ported from Java to Python: Python:Design_Patterns:Inversion_of_Control
As you can see from the above link, a "Container" in Python can be written in 8 lines of code:
class Container:
def __init__(self, system_data):
for component_name, component_class, component_args in system_data:
if type(component_class) == types.ClassType:
args = [self.__dict__[arg] for arg in component_args]
self.__dict__[component_name] = component_class(*args)
else:
self.__dict__[component_name] = component_class
My 2cents is that in most Python applications you don't need it and, even if you needed it, chances are that many Java haters (and incompetent fiddlers who believe to be developers) consider it as something bad, just because it's popular in Java.
An IoC system is actually useful when you have complex networks of objects, where each object may be a dependency for several others and, in turn, be itself a dependant on other objects. In such a case you'll want to define all these objects once and have a mechanism to put them together automatically, based on as many implicit rules as possible. If you also have configuration to be defined in a simple way by the application user/administrator, that's an additional reason to desire an IoC system that can read its components from something like a simple XML file (which would be the configuration).
The typical Python application is much simpler, just a bunch of scripts, without such a complex architecture. Personally I'm aware of what an IoC actually is (contrary to those who wrote certain answers here) and I've never felt the need for it in my limited Python experience (also I don't use Spring everywhere, not when the advantages it gives don't justify its development overhead).
That said, there are Python situations where the IoC approach is actually useful and, in fact, I read here that Django uses it.
The same reasoning above could be applied to Aspect Oriented Programming in the Java world, with the difference that the number of cases where AOP is really worthwhile is even more limited.
You can do dependency injection with Python manually, but manual approach has its downsides:
lots of boilerplate code to do the wiring. You can use dynamic features of Python to do the injection, but then you're loosing IDE support (e.g. Ctrl+Space in PyCharm), and you're making code harder to understand and debug
no standards: every programmer has its own way for solving same problems, this leads to reinventing the wheel, understanding each other's code can quickly become a pain. Dependency injection library provides easy framework to plug-in
To have it all we NEED a dependency injection framework, for example this one https://python-dependency-injector.ets-labs.org/index.html seems to be the most mature DI framework for Python.
For smaller apps DI container is not necessary, for anything that has few hundred lines of code or more, DI container is a must have to keep your code maintaineable.
I agree with #Jorg in the point that DI/IoC is possible, easier and even more beautiful in Python. What's missing is the frameworks supporting it, but there are a few exceptions. To point a couple of examples that come to my mind:
Django comments let you wire your own Comment class with your custom logic and forms. [More Info]
Django let you use a custom Profile object to attach to your User model. This is not completely IoC but is a good approach. Personally I'd like to replace the hole User model as the comments framework does. [More Info]
IoC containers are "mimicked" mostly using **kwargs
class A:
def __init__(self, **kwargs):
print(kwargs)
Class B:
pass
Class C:
pass
Ainstance = A(b=B, c=C)
In my opinion, things like dependency injection are symptoms of a rigid and over-complex framework. When the main body of code becomes much too weighty to change easily, you find yourself having to pick small parts of it, define interfaces for them, and then allowing people to change behaviour via the objects that plug into those interfaces. That's all well and good, but it's better to avoid that sort of complexity in the first place.
It's also the symptom of a statically-typed language. When the only tool you have to express abstraction is inheritance, then that's pretty much what you use everywhere. Having said that, C++ is pretty similar but never picked up the fascination with Builders and Interfaces everywhere that Java developers did. It is easy to get over-exuberant with the dream of being flexible and extensible at the cost of writing far too much generic code with little real benefit. I think it's a cultural thing.
Typically I think Python people are used to picking the right tool for the job, which is a coherent and simple whole, rather than the One True Tool (With A Thousand Possible Plugins) that can do anything but offers a bewildering array of possible configuration permutations. There are still interchangeable parts where necessary, but with no need for the big formalism of defining fixed interfaces, due to the flexibility of duck-typing and the relative simplicity of the language.
Unlike the strong typed nature in Java. Python's duck typing behavior makes it so easy to pass objects around.
Java developers are focusing on the constructing the class strcuture and relation between objects, while keeping things flexible. IoC is extremely important for achieving this.
Python developers are focusing on getting the work done. They just wire up classes when they need it. They don't even have to worry about the type of the class. As long as it can quack, it's a duck! This nature leaves no room for IoC.