Why has the syntax changed from flask.ext.* to flask_*? - python

It looks like there was a deprecation. How was that decided? Is there a difference between Python 3 and Python 2?

The old flask.ext was deprecated in issue #1135, which was created back in 2014. The actual deprecation notice was turned on in 2016. The reasoning behind the deprecation is:
Some introductory information for new contributors:
Flask used to have flaskext as a namespace for extensions, so they were importable as flaskext.foo. This didn't work well, so the new form flask_foo was introduced. flask.ext.foo is a compatibility layer that will try to import both variants. See http://flask.pocoo.org/docs/0.10/extensions/
flask.ext.foo is hard to maintain, and since now all extensions have switched to the new package naming scheme, it is no longer worth it. We want to deprecate it for 1.0, so we need some sort of tool which can help users to rewrite all their old imports in their apps.
One could write a Python script similar to this beast. This will get the job done, but as its docstring says, it's a terrible hack.
lib2to3 proved useful for writing larger migration tools, but it's nontrivial to use it.
https://github.com/mitsuhiko/python-modernize/ is one based on it, and it seems to me that's the easiest project one could rip off from.
I wasn't able to find complete tutorials that are useful for this. Most seem to be focused on porting to Python 3, which would imply running the default 2to3 fixers on the user's codebase (which we definetly don't want)
One will have to read the sourcecode of 2to3 and lib2to3 to understand, i think. This is doable by entering libraryname hg.python.org into Google, where the libraryname is either 2to3 or lib2to3.
The current state for doing sourcecode manipulation in Python sucks, and i'd like to see a library which wraps lib2to3 and provides a more concise API.
The old .ext was a compatibility layer to support the old flaskext module while waiting for flask_ to standardize.
This separates the flask. namespace from each module's namespace, as the module now lives completely in its own module (flask_module) instead of being loaded into a general namespace for all extensions in Flask. It's also clearer that the module is not bundled as a part of Flask.

Related

How to protect my Python code before distribution?

I have written a python code which takes an input data file, performs some processing on the data and writes another data file as output.
I should distribute my code now but the users should not see the source code but be able to just giving the input and getting the output!
I have never done this before.
I would appreciate any advice on how to achieve this in the easiest way.
Thanks a lot in advance
As Python is an interpreted language by design; and as it compiles code to a bytecode (- which doesn't help the fact you're trying to conceal it, as bytecodes are easier to reverse -) there's no real secure way to hide your source code whereby it is not recoverable, as is true for any programming language, really.
Initially, if you'd wanted to work with a language that can't be so easily reversed- you should've gone for a more native language which compiles directly to the underlying architecture's machine code which is significantly harder to reproduce in the original language let alone read due to neat compiler optimizations, the overhead given by CISC et cetera.
However, some libraries that do convert your source code into an executable format (by packing the Python interpreter and the bytecode alongside it) can be used such as:
cx_Freeze - for freezing any code >=Python 2.7 for any platform, allegedly.
PyInstaller - for freezing general purpose code, it does state additionally that it works with third-party libraries.
py2exe -for freezing code into Windows-only executable format.
Or you might consider a substitute for this, which is code obfuscation which still allows the user to read the source code however make it near-to-impossible to read.
However, an issue brought up with this is that, it'd be harder for code addition as bad code obfuscation techniques could make the code static. Also, on the latter case, the code could have overhead brought by redundant code meant to fool or trick the user into thinking the code is doing something which it is not.
Also in general it negates the standard practice of open-source which is what Python loves to do and support.
So to really conclude, if you don't want to read everything above; the first thing you did wrong was choose Python for this, a language that supports open source and is open source as well. Thus to mitigate the issue you should either reconsider the language, or follow the references above to links to modules which might help aide basic source code concealment.
Firstly, as Python is an interpreted language, I think you cannot completely protect your Python code, .pyc files can be uncompiled to get back .py files (using uncompyle6 for example).
So the only thing you can do is make it very hard to read.
I recommend to have a look at code obfuscation, which consists in making your code unreadable by changing variables/function names, removing comments and docstrings, removing useless spaces, etc. Pyminifier does that kind of things.
You can also write your own obfuscation script.
Then you can also turn your program into a single executable (using pyinstaller for example). I am pretty sure there is a way to get .py files back from the executable, but it just makes it harder. Also beware of cross-platform compatibility when making an executable.
Going through above responses, my understanding is that some of the strategies mentioned may not work if your client wants to execute your protected script along with other unprotected scripts.
One other option is to encrypt your script and then use an interpreter that can decrypt and execute it. It too has some limitations.
ipepycrypter is a suite that helps protect python scripts. This is accomplished by hiding script implementation through encryption. The encrypted script is executed by modifed python interpreter. ipepycrypter consists of encryption tool ipepycrypt and python interpreter ipepython.
More information is available at https://ipencrypter.com/user-guides/ipepycrypter/
One other option, of course, is to expose the functionality over the web, so that the user can interact through the browser without ever having access to the actual code.
There are several tools which compile Python code into either (a) compiled modules usable with CPython, or (b) a self-contained executable.
https://cython.org/ is the best known, and probably? oldest, and it only takes a very small amount of effort to prepare a traditional Python package so that it can be compiled with Cython.
http://numba.pydata.org/ and https://pythran.readthedocs.io/ can also be used in this way, to produce Python compiled modules such that the source doesnt need to be distributed, and it will be very difficult to decompile the distributable back into usable source code.
https://mypyc.readthedocs.io is newer player, an offshoot of the mypy toolkit.
Nuitka is the most advanced at creating a self-contained executable. https://github.com/Nuitka/Nuitka/issues/392#issuecomment-833396517 shows that it is very hard to de-compile code once it has passed through Nuitka.
https://github.com/indygreg/PyOxidizer is another tool worth considering, as it creates a self-contained executable of all the needed packages. By default, only basic IP protection is provided, in that the packages inside it are not trivial to inspect. However for someone with a bit of knowledge of the tool, it is trivial to see the packages enclosed within the binary. However it is possible to add custom module loaders, so that the "modules" in the binary can be stored in unintelligible formats.
Finally, there are many Python to C/go/rust/etc transpilers, however these will very likely not be usable except for small subsets of the language (e.g. will 3/0 throw the appropriate exception in the target language?), and likely will only support a very limited subset of the standard library, and are unlikely to support any imports of packages beyond the standard library. One example is https://github.com/py2many/py2many , but a search for "Python transpiler" will give you many to consider.

python 3 import from subdir

My project has to be extensible, i have a lot of scripts with the same interface that lookup things online. Before i was using __import__ but that does not let me put my 'plugins' on a dedicated directory:
root/
main.py
plugins/
[...]
So my question is: Is there a way to individually import modules from that subdirectory? I'm guessing importlib, but i'm so lost in how the python module loading process works... What i want to do is something like this:
for pluginname in plugins:
plugin = somekindofimport("plugins/{name}".format(name=pluginname))
plugin.unififedinterface()
Also, as a side question, the way am i trying to achieve extensibility is a good way?
I'm on python3.3
Stop thinking in terms of pathnames and start thinking in terms of packages. Read Packages in the tutorial, and if you want more detail see The import system.
But the basic idea is this:
Create a file name plugins/__init__.py. It can be empty; that's enough to turn plugins into a package. Which means you can import modules from that package with:
import plugins.plugin
So, how do you do this dynamically? That's what importlib is for. (You can also use __import__ here, but it's less flexible, and less readable in non-trivial cases, so unless you need pre-3.3 compatibility, don't.)
plugin = importlib.import_module('plugins.{name}'.format(name=pluginname))
It would probably be cleaner to import plugins to get the package, and then use relative imports from within that package, as shown in the examples in the import_module docs.
This also means Python takes care of the .pyc creation and caching, etc.
And it means that you can later expand plugins to be a "namespace package", which can be split across multiple directories like /usr/share/myapp/plugins for stock plugins, /etc/myapp/plugins for site plugins and ~/myapp/plugins for user-specific plugins.
If you really, really want to import from a directory that isn't a package, you can create a module loader and use it, but that's a whole lot of work for no actual benefit. (It's actually not that hard in 3.3 (SourceLoader and friends will do most of the work for you), but you will find almost no examples out there to guide you; instead, you'll find examples of the 2.6-3.2 way, or the 2.0-2.5 way, both of which are hard.) Plus, it means that if someone creates a plugin named, say, gzip, you can end up blocking the stdlib gzip module with the plugin. (That's especially fun if the gzip plugin tries to use the gzip stdlib module, as it likely will…) If the plugin ends up being named plugins.gzip, there's no problem.
Also, as a side question, the way am i trying to achieve extensibility is a good way?
As long as you only want to support 3.3+, yes, I think this is a great solution.
Before 3.3, using a package for plugins was a lot more problematic. People have come up with a variety of different plugin systems—in one case going so far as to dynamically create module objects and execfile into them. If you need to deal with that, I would suggest looking at existing Python apps with plugins (e.g., MusicBrainz Picard) to get different ideas.

A tool to validate the structure of a Python Package?

I started writing Python code not too long ago and everything just works, but I have been having problem writing a package. I was wondering if there is such a thing as a "package validation tool". I know, I could just start up a REPL and start importing the module but...is there a better way? Is there a tool that could tell me "you have these possible errors"?
Or maybe there is something in the middle: is there a way to test a Python's package structure?
As always, thanks in advance!
If you call a module using:
python -m module
Python will load/execute the module, so you should catch crude syntax errors. Also, if module has a block like:
if __name__ = "__main__":
do_something()
It will be called. For some small self-contained modules I often use this this block to run tests.
Given the very dynamic nature of Python, it is very hard to check for correctness if the module author is not using TTD. There is no silver bullet here. There are tools that will check for "code smells" and compliance with standards (dynamic languages tend to generate a profusion of linters).
pylint
PyChecker
PyFlakes
PEP8
A good IDE like PyCharm can help, if you like IDEs.
These tools can help, but are still far from the assurance of static languages where the compiler can catch many errors at compile time. For example, Go seems to be designed to have a very pedantic compiler. Haskell programs are said to be like mathematical proofs.
If you are coming from a Language with strong compile time checks, just relax. Python is kind of a "throw against the wall and see if it sticks", language. Some of the Python "macho" principles:
duck typing
EAFP
We are all consenting adults
There is no tool to test the package structure per se, and I'm unsure of what would be tested. Almost any structure is a valid structure...
But there are some tools to help you test your package data if you are distributing your module, they may be useful:
Pyroma will check the packages meta data.
check-manifest will check the MANIFEST.in file.
I have both of them installed and also uses zest.releaser which also has some basic sanity-checks. But none of these will check that the code is OK, so it won't look for the __init__ files, for example.

Can i take package of cpython?

I used cpython api to load py from C/C++.
But, if i want not setup cpython in client, can I take package dll of cpython in my program?
How to do that?
Installer-builders like PyInstaller (cross-platform) and py2exe (Windows only) basically do that job for you in a general way, except that the executable at the heart of the produced package is their own instead of yours.
But basically, you can imitate their behavior in terms of setting up a .zip file with all the Python library modules you need (or just zip up everything in the standard python library if you want to allow python code running form your app to import anything from there), and follow the simple advice in the Embedding Python in Another Application section of the Python docs.
Note that embedding Python equals extending Python plus a little bit of code to initialize and finalize the interpreter itself and a little bit of packaging as I just mentioned; if you've never writted Python extensions I would suggest practicing that first since it's the most substantial part of the task (not all that hard with helpers such as boost python, but more work if you choose to do it as the "bare C" level instead).
You don't need to setup Python to embed it in applications. The core of the Python interpreter is available as a shared library which you can dynamically load in your application and distribute with it.
Read on embedding Python in the official docs. Also, this article seems nice and comprehensive for Linux. For Windows, read the notes here.
Here's another SO question that discusses this issue.
The Python license is probably hard to understand for a non-lawyer, non-native English speaker. So yes, you can redistribute the unmodified DLL as it contains the copyright notice within it.
It would be polite to give credit like "This program contains the Python Language Interpreter version X.XX http://python.org for more information" or similar somewhere in the program or documentation.

Real-world Jython applications

I recently started learning Python. Not yet ventured into coding.
During one of my learning sessions, i came accross the term Jython.
I googled it & got some information.
I would like to know if anyone has implemented any real-world program using Jython.
Most of the time, Jython isn't used directly to write full read-world programs, but a lot of programs actually embed Jython to use it as a scripting language.
The official Jython website gives a list of projects, some written in Jython, others using Jython for scripting:
http://wiki.python.org/jython/JythonUsers
I am writing a full application in Jython at the moment, and would highly recommend it. Having all of the Java libraries at your disposal is very handy, and the Python syntax and language features actually make using some of them easier than it is in Java (I'm mostly talking about Swing here).
Check out the chapter on GUI Applications from the Jython book. It does a lot of comparisons like 'Look at all this Java code, and now look at it reduced to Python code of half the length!'.
The only caveats I've found are:
Jython development tends to run slightly behind Python, which can be annoying if you find a cool way of doing something in Python, only to discover it's not supported in the current Jython version.
Occasionally you might have hiccups with the interface between Python and Java (I have a couple of unsolved problems here and here, although there are always workarounds for this kind of thing).
Distribution is not as simple as it could be, although once you figure out how to do it, it's fairly painless. I recommend following the method here. It essentially consists of:
Exploding jython.jar and adding your own modules into it.
Writing and compiling a small Java class that creates a Python interpreter and loads up your Python modules.
Creating an executable .jar file consisting of the jython.jar modules, your own Python modules, and the Java class.
Jython really shines for dependency injection.
You know those pesky variables you have to give your program, like
file system paths
server names
ports
Jython provides a really nice way of injecting those variables by putting them in a script. It works equally well for injecting java dependencies, as well.
WebSphere and WebLogic use it as their default scripting engine for administrative purposes.
A lot of other Oracle products ship it as part of their "oracle_commons" module (Oracle Universal Installer, Oracle HTTP Server etc). It's mostly version 2.2 being deployed though, which is a bit old and clunky.
There is a list of application that uses jython at http://wiki.python.org/jython/JythonUsers

Categories