We Keep Coding

Delete local data generated by Google App Engine's dev_appserver.py

I run my application locally using dev_appserver.py
using the following command
dev_appserver.py mydir --port=xxxx
Is there a way to delete all the local data generated by app engine server.
Currently, I go to localhost:8000/datastore, and delete entity groups manually.
IS there a way to automate it
Where does dev_appserver.py write local data to? (File or directory? may be in can delete that)

Yes there is!
dev_appserver.py accepts some arguments that clears things before it starts up. Two arguments that I use consistently are:
dev_appserver.py --clear_datastore=true --clear_search_indexes=true ...
dev_appserver.py --help can probably give you more information about other things that you may want to clear -- But this always gets everything I need it too.

Access Google cloud endpoints on a non-existent version label

I have two apps, my_app and my_endpoint_app. I can access my_endpoint_app with any version label in the URL I want and it will automatically route to the default version if it does not match an existing version.
Example:
https://josh-dot-my_endpoint_app.appspot.com/ will respond with the default version since there is no josh version deployed.
If I try to do the same with a Google Cloud Endpoint service call, I get a Not Found error.
Example:
The unsuccessful https://josh-dot-my_endpoint_app.appspot.com/_ah/api/myendpoint vs the working https://my_endpoint_app.appspot.com/_ah/api/myendpoint
I have a couple of Google AppEngine applications that communicate with each other via Cloud Endpoints.
Under normal usage this is OK because I know the version beforehand and avoid these errors. In our development environment, this falls apart. In order to support feature branches and testing in isolation, we push our code up to appspot using the -V switch of appcfg.py.
Example:
appcfg.py -A my_app -V josh update .
Now I can access my feature branch at https://josh-dot-my_app.appspot.com. In order to support some version label hackery, I dynamically calculate the right endpoint app to call with something like s/my_app/my_endpont_app/g and then make my service calls there. This fails because of the dynamic version label not existing. If I push a version label with that name it completes as expected.
Is there any way to get Cloud Endpoints to answer on non-existent version label hostnames?
Scenarios that I want to support
https://my_endpoint_app.appspot.com/_ah/api/myendpoint
Main application URL, routes to default version
https://josh-dot-my_endpoint_app.appspot.com/_ah/api/myendpoint
Version does not exist, should route to default version
https://new-feature-dot-my_endpoint_app.appspot.com/_ah/api/myendpoint
Version new-feature exists, should route to new-feature version so that we can test new code in isolation before merging into the main code branch. This would be internal apis that the current endpoints might make use of without changing what the endpoint accomplishes. (performance improvements, etc)

You can reroute any Url to any module/version via the dispatch file.

App Engine Bulkloader

I am trying to use Bulkloader of google app engine but unfortunately could not understand what to do from documentation. It says add this part to app.yaml
builtins:
- remote_api: on
ok i have added. Then says that i have to execute this command
appcfg.py update
but i don't have any appcfg.py file. And also what is the command which executes this line?
Please somebody tell me what i am missing I use AppEngineLauncher to upload my project to server.. I have naver used a command to update or upload it.
Thanks in advance..

If you're using App Engine Launcher, simply click the Deploy button; it runs appcfg.py update. appcfg is installed as part of the SDK.

In order to call it globally you must set the python path to include google appengine library, or call it from its excact location
~/google_appengine/appcfg.py update ~/project/path_to_your_yaml_file/
where ~ stands for you home folder.

For those using the the Java GAE SDK, it comes wtih appcfg.sh, but doesn't support full features of apppcfg.py. You need to also install the Python GAELauncher. On Mac, it installs appcfg.py in /usr/local/bin.
Working only with my local datastore, I did not have an app.yaml, nor did I need to update it as per the uploading data GAE documentation. Just needed to update the web.xml with the RemoteApiServlet.
Main gotcha is the Authentication error while connecting to the local data store. It may be specific to Mac and the Charles proxy work around mentioned in that thread likewise worked for me. I did have to modify the default proxy port from 8888 to avoid conflict with my app. The workaround is tedious and the trial version of Charles comes with limitations such as shutdown after 30 minutes.

To use the main database for development, first download it from app engine:
bulkloader.py --dump --url http://<app name>.appspot.com/_ah/remote_api --filename dump.bin
It will ask for an email and password (use your google login details).
To upload it to the development environment, make sure remote_api is
enabled in app.yaml
https://developers.google.com/appengine/docs/python/tools/uploadingdata#Setting_Up_remote_api
Then run:
bulkloader.py --restore --url http://localhost:8080/_ah/remote_api --filename dump.bin --application dev~<app name>
This time for the email, just leave it blank and hit enter.

Where is my local App Engine datastore?

How can I find where my local development datastore is located? I am using the Python SDK and Linux.

I think it depends on if you got Java or Python SDK.
For Python, here's what the instructions say from Google:
"The web server prints the location of the datastore file it is using to the terminal when it starts up. You can make a copy of the file, then restore them later to reset the datastore to a known state. Be sure to restart the web server after replacing the datastore file.
To change the location used for the datastore file, use the --datastore_path option:
dev_appserver.py --datastore_path=/tmp/myapp_datastore myapp
more info here: http://code.google.com/appengine/docs/python/tools/devserver.html

I'm using Windows 7 with the Python SDK. My local datastore is located at
C:\Users\[username]\AppData\Local\Temp\dev_appserver.datastore

To find the file location for the local AppEngine datastore on MacOSX/Python, you can run the following command:
dev_appserver.py -help
Mine was at something like:
/var/folders/uP/uP1GHkGKGqO7QPq+eGMmb++++TI/-Tmp-/dev_appserver.datastore

I think a lot of the answers on this page are out of date. Under the current Python dev kit (1.8.6) on Windows 7 I eventually found the datastore at:
c:\Users\[username]\AppData\Local\Temp\appengine.[appname]\datastore.db
I couldn't find this info in anything dev_appserver.py printed out, either with normal startup options or with --help. On other OSes you might try searching for a file called datastore.db.

For Python u can make access to datastore admin interface path:' /_ah/'
or
add the app handler to app.yaml
- url: /admin/.*
script: $PYTHON_LIB/google/appengine/ext/admin
login: admin
and access it at /admin/

I use OS X Mavericks (10.9), Python 2.7.5, and Google App Engine SDK 1.9.3 (Python).
None of the above worked for me, however, referencing #alsmola's answer, I executed sudo find / | grep datastore.db and found the file in /private/var/folders/vw/7w1zhkls4gb1wd8r160c36300000gn/T/appengine.YYYY.XXXXX/datastore.db (YYYY is the project name, XXXXX is my username).

Since it's top question on Google search and I spent quite amount of time searching for an answer I'll say that on Windows/Java mix DB file called local_db.bin.

With Maven the files are sitting here:
target/{buildName}/WEB-INF/appengine-generated/

I'll restate a solution to getting permanent datastore as it worked for me (circa Feb 2017), running GoogleAppEngineLauncher on OS X v10.10.
Create the folder path for permanent datastore
In GAEL, click on the project in question e.g. PROJECTNAME
Click Edit-Application Settings
in Extra Flags field:
--datastore_path=/Users/foo/GAE_datastore/PROJECTNAME/datastore.db
Filename has to be included; in my config, datastore.db works.
Having searched all over for GAE datastore path, and head-bonked on dev_appserver.py --datastore_path command line, it was very helpful to find this.
Application Settings under the Edit menu is an odd choice, Google :-)

The default location of the datastore for the platform you're running the app engine on is provided in the README that comes with the platform (at least, in the one for Linux). The README is in google_appengine_x.x.xx/google_appengine/README. This is what is says in the Linux'es one:
--datastore_path=DS_FILE Path to file to use for storing Datastore file
stub data.
(Default /tmp/dev_appserver.datastore)

Is it possible to deploy one GAE application from another GAE application?

In order to redeploy a GAE application, I currently have to install the GAE deployment tools on the system that I am using for deployment. While this process is relatively straight forward, the deployment process is a manual process that does not work from behind a firewall and the deployment tools must be installed on every machine that will be used for updating GAE apps. A more ideal solution would be if I could update a GAE application from another GAE application that I have deployed previously. This would remove the need to have multiple systems configured to deploy apps.
Since the GAE deployment tools are written in Python and the GAE App Engine supports Python, is it possible to modify appcfg.py to work from within GAE? The use case would be to pull a project from GitHub or some other online repository and update one GAE application from another GAE app. If this is not possible, what is the limiting constraint?

Is it possible? Yes. The protocol appcfg uses to update apps is entirely HTTP-based, so there's absolutely no reason you couldn't write an app that's capable of deploying other apps (or redeploying itself - self-modifying code)! You may even be able to reuse large parts of appcfg.py to do it.
Is it easy? Probably not. It's quite likely you'll need to understand a decent chunk of appcfg's internals, and the RPCs it uses to upload new apps - not a trivial undertaking. You'll also need to store your credentials in the app, in all likelihood - though you can use a role account that is and admin only for the apps it's deploying to minimize risk there.

One limiting constraint could be the protocol that the python sdk uses to communicate with the GAE servers. If it only uses HTTP, you might be OK. but if it's anything else, you might be out of luck because you can't open a socket directly from within GAE.

What problem did you have by trying to update behind a firewall?
I've got some, but finally I manage to work around them.
About your question, the constraint is that you cannot write files into a GAE app, so even though you could possibly pull from the VCS you can't write those pulled files.
So you would have to update from outside the GAE in first place.
Anyway every machine that needs to update the GAE should have the SDK anyway just to see if they changes work.
So, If you really want to do this you have two alternatives:
Host your own "updater" site and istall the SDK there, then when you want to update log into your side ( or run a script ) and do the remote update.
Although I don't know Amazon EC2 well, I think you can do pretty much the same thing as op 1 from there.
Finally I think the password to update has to be typed always. ( you could have the SDK of the App engine and modify that, because it is open source )