Can Pandas run on Google App Engine for Python? - python

Can the pandas data analysis module run on Google App Engine?
My first inclination is no: the web page states "critical code paths compiled to C". So since this is not a purely python package, you cannot simply copy a directory or ZIP file into your app engine project.
Is it possible to "disable" the C extensions and have the module run in pure python (albeit slower)?

As of today, Google App Engine Python 2.7 runtime doesn't currently allow running arbitrary C extensions.
But it does allow inclusion of third party library that have been hand selected by the engineering team based on their popularity and developer demand.
You can request for a specific library to be included by filling a feature request on the public issue tracker. If it gets enough ☆☆☆ it is likely to be considered for inclusion in an upcoming release.

Related

Best practices with google cloud app engine: Python2 or Python3?

I am new to Google Cloud Platform and in my whole I have been working on Python 3. I am trying to find out which version of Python is more complete for Google App Engine: Python 2.7 or Python 3.
As I'm starting to work with Google App Engine I have realised that continuing using Python 3 seems too painful as basic tools like dev_appserver.py are written for Python 2 only. Now I am hitting the opposite problem: cloudstorage module seems to exist only for python3. Again, when I install it, seems the only way I can test read/write to google bucket locally is by authenticating with google.appengine.ext, which in turn only works within dev_appserver.py or remotely. This leaves me confused which environment to chose.
What is a general agreement / what is the focus of Google App Engine: Python 2 or Python 3?
In App Engine, you have to options: the Standard environment and the Flexible environment.
Python 2.7 is available in both Standard and Flexible, while Python 3.6 is only available in Flexible.
Also, the choice between Standard and Flexible depends on what you want to do/what libraries you need:
There are some third-party libraries already built-in in the Standard Environment, and you can include other libraries, but, those libraries can't include C extensions, they must be written in pure Python. If you need libraries with C extensions, you will have to move to Flexible.
In Standard, you can use propietary libraries (like google.appengine.ext, as you mentioned) to do tasks like accessing databases, while in Flexible you can use other libraries (like the client you mentioned).
There are also another important differences, like pricing, scaling, etc. The choice will depend, as I said, in your needs for your application.
EDIT
dev_appserver.py is only used when developing in Standard. There is a tutorial in here, with Flask. If you are in Flexible, you can test the app locally as if you were running as usual a python file, like in this other example.
You can use buckets in both Standard and Flexible
The python3-only cloudstorage support assumption based on the SO post you referenced is not correct:
the import appears to be done in a regular python shell or as a standalone script, not from a standard environment GAE app - different things, see import cloudstorage, ImportError: No module named google.appengine.api.
it is not specified where that library comes from
GCS is definitely supported in the standard env GAE (i.e. on python 2), you just need to follow the steps from the official documentation: Setting Up Google Cloud Storage and Reading and Writing to Google Cloud Storage.
Both were good. But the question is what kind of environment do you want? Standard environment or Flexible environment.
Find your answer in this document: https://cloud.google.com/appengine/docs/python/
It kind of depends on what you're using it for. If you're doing data science, for example, I'm seeing a few notices of Python libraries that are (finally) dropping support for Python 2. numpy is one that is dropping support.
Generally speaking, I would recommend Python 3 over Python 2. Why spend time developing in an aging version when its replacement has matured nicely and is more consistent?

Shopify App Development that supports multiple platforms For eg. Shopify, Bigcommerce etc

If we are to build an app that supports multiple e-comm platforms like Shopify, Bigcommerce, Magento, Generic 3rd party API etc. , what should be our approach ? Should we build separate apps ( codebases ) for each of them or a common codebase with extensions ?
P.S : I have built an app on Shopify using Python, Django stack
My recommendation would be to focus on writing just the code unique to your app (completely separate and irrelevant to any specific platform) and then build your own public (or internal) REST/SOAP API around this code base.
This way, from there, you can then focus on building platform-specific applications without ever needing to mess with the core code, while establishing a common/set methodology of building unique platform versions (at least 50% of the code will be reusable amongst the individual platform extensions). Also though, having your own API made publicly available would even allow users on other platforms to use your API to build their own integrations into their own eCommerce platform - this is actually a critical business move of becoming first to market as your app is now actually made available for every platform, even if a platform specific version has not been internally developed yet.

Is it possible to use a C library with python AppEngine?

I am investigating into if I can use a library like GHMM with my python web service in which runs on AppEngine.
Short answer: no
https://developers.google.com/appengine/kb/commontasks
What third party libraries can I use in my application?
You can use any pure Python third party libraries in your Google App Engine application. In order to use a third party library, simply include the files in your application's directory, and they will be uploaded with your application when you deploy it to our system. You can import the files as you would any other Python files with your application.
As #gahooa has said, the generic answer is no.
For more popular libraries that have C dependencies your best option right now is to file a ticket[1], get other to upvote (star) your ticket and have the App Engine add it as a supported library.
[1] http://code.google.com/p/googleappengine/issues/entry?template=Feature%20request
In 2021 yes you can.
The flexible version of AppEngine allows you to do this, standard does not.
If, like me, you cannot justify the full time running costs of flexible, an alternative is to host the C library on Cloud Run and make API calls to it. Then you have costs of AppEngine standard and Cloud Run, but both are on-demand only.

Django: speech to text application

first do you know what language google used to create the speech to text search?
then can you create a speech to text django or google apps engine using python? i'm not a vb.net programmer and yes i know that such applications are usually created in java, couldn't they be created in jython? i have a django app i would like to introduce this feature to it.
i can use ispeech (not free!) or similar services, but before working with APIs (i don't like working with APIs to be honest unless it is facebook or google API), i'm looking to see if there's a python or jython library for that (i prefer python)
i found so far pyspeech it works on windows and my vps is linux so won't do me any good
Unlikely.
As you cannot install binaries in the Google App Engine environment, you would need build an entire speech recognition engine from scratch. This would be ..non-trival.
If you are not limited to Google App Engine however, there are Python bindings to existing packages, such as Sphinx

Breaking out of the Google App Engine Python lock-in? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
Are there any guidelines to writing Google App Engine Python code that would work without Google's infrastructure on other platforms?
Is there any known attempt to create an open source framework which can run applications designed for Google App Engine on other platforms?
Edit:
To clarify, the question really is:
If I develop an application on Google App Engine now, will I be able to migrate to another platform later, or is it a lock in?
There's a number of components required to make an app fully portable:
The runtime environment itself. This can be ported relatively simply, by setting up a CGI or FastCGI server that emulates the App Engine environment (which itself is basically slightly-enhanced CGI). Most of the code to do this is already in the SDK. Unfortunately, there's no easy pre-packaged toolkit for this yet.
The datastore. By far the most complicated API to port. There are a number of efforts underway: AppScale runs on EC2/Eucalyptus/Xen and uses a HyperTable or HBase backend; it's still very much beta quality and they don't distribute the data connector separately (it's the beginnings of a complete run-your-app-on-your-own-cloud solution). Jens is/was writing an SQLite backend, and there's my own project, BDBDatastore, which uses BDB-JE as the backend, and is fully functional (though beta quality). AppDrop, which others have mentioned, simply uses the development server as a backend, and hence isn't suitable for production use.
The users API needs replacing with something else, such as an OpenID based API. Again, fairly simple, but no premade solutions yet.
The Memcache API needs a backend that uses standard C memcache backends.
The other APIs have perfectly functional backends as part of the SDK, so don't really need porting.
Cron support would also need implementing, as would background processing, XMPP, etc, when they become available.
As you can see, there's a lot of work to be done, but no fundamental barriers to making your App Engine app run outside Google's environment. In fact, if you're interested, you're more than welcome to participate - I and others have plans to combine the solutions for the various pieces into a single 'OpenEngine' solution for hosting your own apps.
Use a high level framework that works on App-Engine. That way you can port your code to other servers when you want.
django has been patched and ported to work in the Appengine patch project and is the most used FW on appengine.
You may want to refer this step by step intro to running a django app on App engine
As far as the parallel infrastructure to run an app engine application is concerned, it is still way far. App Engine itself hasn't got as popular as people believed it to and google wanted it to be. Plus it is harder to develop on the builtin WebApp framework than on django.
Its quite unlikely to see a parallel infrastructure to run app engine application on, atleast for years to come. Rather it is likely to see django and other popular frameworks work out of the box on app engine and the work on this is currently underway in the referred project.
You can build AppEngine applications using the Django python framework (although the supported version is a bit behind the most recent Django release). Where you lose portability (at least right now) is when using GQL/BigTable for persistence. This is Google proprietary database platform. As Hank mentioned this is one of the biggest reasons to actually use AppEngine, but it is also the single largest locking point.
Here are a couple of links to Django support in AppEngine and GQL/BigTable:
http://code.google.com/appengine/articles/django.html
http://code.google.com/appengine/docs/python/datastore/gqlreference.html
So far, I found an experimental host called app-drop which is capable of hosting google app-engine projects. This should mean that it's atleast possible to run app engine projects outside of google's infrastructure.
This is however clearly not yet suitable for production.
The code should be mostly portable (they do a pretty good job of indicating which modules you can't use on AppEngine and which AppEngine-specific code corresponds to which forbidden modules), but the whole point of AppEngine is to get access to Google's infrastructure. There's not much point to writing your code to the AppEngine restrictions if you aren't going to be using their infrastructure.
AppDrop is a proof of concept port of AppEngine to Amazon Web Services / Elastic Computing completed in April of 2008. It uses a flat file instead of BigTable and runs in a single instance, so there are scaling issues; but it's developer says it took him only four days, and perhaps these limitations can be addressed by others.
I did the reverse migration from vanilla Unix to app engine recently very easily by using WHIFF resources. Basically configure anything platform dependant as a resource and then swap/replace the resources on different configurations.
http://piopio.appspot.com/W1000_1000.resources
also see
http://aaron.oirt.rutgers.edu/myapp/docs/W1100_1200.wwiki
for a detailed example of resource swapping/configuration.
(note: links may go away eventually, app is experimental.)
Check out typhoonae. it's in beta, but quite usable – we moved one of our apps to inhouse server running this stack.
AppScale is the most mature open source implementation of Google App Engine. It's been in development since 2008 and currently support all four languages: Python, Java, Go, and PHP. It has users running their application in production today.
The FAQ explains what APIs are supported and what is lacking:
https://github.com/AppScale/appscale/wiki/FAQs
(Disclaimer: I work on the project)

Categories