I have some CSV files for cities,state and countries with their ids, names etc. I want to put all this data into Google app engine datastore.
Can someone please suggest an efficient way of doing this on development server as well as on the production server?
Thanks in advance.
You're in luck. The functionality you described is baked into appcfg.py:
http://code.google.com/appengine/docs/python/tools/uploadingdata.html
Related
I want to upload and retrieve file (that includes image and csv) for my django application.
My project is currently hosted on Google App Engine Flexible. From my understanding, I need to use Google Cloud Buckets for the process. But I could not find relevant material online for the process.
GoogleAppEngineCloudStorageClient API also provides the feature of writing a file and reading it:
cloudstorage.open(filename, mode='r', content_type=None, options=None, read_buffer_size=storage_api.ReadBuffer.DEFAULT_BUFFER_SIZE, retry_params=None)
Please point me to the relevant resources for the same. Since I am new to Django and Google Cloud, I would really appreciate if you could share code snippets with me.
Thanks in advance.
Check the Google storage section of Django Storages. It may help to read generally about the storage system used in Django before going to Google storage in detail.
As per the official documentation of Google App Engine Flexible, you can configure upload and retrieve files in different runtimes environments. I went through the documentation and I found that you need to create a Cloud Storage bucket in order to stock your files.Keep in mind that it's only for static content. I recommend to take a look at this documentation where you may find valuable information.
Please let me know how it goes.
Is there any way to alter the Property names in the Google application engine for a Kind, or in other words is there a way to alter the column names of a table in Google application Engine (though it follows a different way to handle the data)? I am using python.
Please suggest.
Thanks in advance.
Refactoring on Google AppEngine involves you having to either modify all of the records in your datastore as you make the change, or write the code so that it will still read the old value if the new value doesn't exist.
Removing a column from the datastore is possible but not easy. More information can be found here.
I'm from Brazil and study at FATEC (college located in Brazil).
I'm trying to learn about AppEngine.
Now, I'm trying to load a large database from MySQL to AppEngine to perform some queries, but I don't know how i can do it. I did some testing with CSV files,but is there any way to perform the direct import from MySQL?
This database is from Pentaho BI Server (www.pentaho.com).
Thank you for your attention.
Regards,
Daniel Naito
It isn't clear from your tags, but the documented bulkloader is preferable to trying to hoist your csv files directly to the app-server.
Advanced Bulk Loading by Nick Johnson is what you are looking for.
If you need live synchronization between App Engine and MySQL, you should look into AppRocket. AppRocket seems to require that you have your data in App Engine before the first synchronization. It will also require some minor changes to your model.
If you're using Pentaho BI Server as your data source, why don't you consider using Pentaho Data Integration (ETL tool) to move the data over? At the very least PDI automate any movement of data between your data source and any AppEngine bulk loader tool (it can easily trigger any app with a shell step).
Hi I want some help in building a Phone book application on python and put it on google app engine. I am running a huge db of 2 million user lists and their contacts in phonebook. I want to upload all that data from my servers directly onto the google servers and then use a UI to retrieve the phone book contacts of each user based on his name.
I am using MS SQL sever 2005 as my DB.
Please help in putting together this application.
Your inputs are much appreciated.
For building your UI, AppEngine has it's own web framework called webapp that is pretty easy to get working. I've also had a good experience using the Jinja2 templating engine, which you can include in your source, or package as a zip file (example shows Django, you can do the same type of thing for Jinja).
As for loading all of your data into the DataStore, you should take a look at the bulk uploader documentation.
I think you're going to need to be more specific as to what problem you're having. As far as bulk loading goes, there's lots of bulkloader documentation around; or are you asking about model design? If so, we need to know more about how you plan to search for users. Do you need partial string matches? Sorting? Fuzzy matching?
Does anyone have any good information aside from the Google App Engine docs provided by Google that gives a good overview for people with MS SQL background to porting their knowledge and using Google App Engine Data Store API effectively.
For Example, if you have a self created Users Table and a Message Table
Where there is a relationship between Users and Message (connected by the UserID), how would this structure be represented in Google App Engine?
SELECT * FROM Users INNER JOIN Message ON Users.ID = Message.UserID
Here is a good link: One to Many Join using Google App Engine.
http://blog.arbingersys.com/2008/04/google-app-engine-one-to-many-join.html
Here is another good link: Many to Many Join using Google App Engine:
http://blog.arbingersys.com/2008/04/google-app-engine-many-to-many-join.html
Here is a good discussion regarding the above two links:
http://groups.google.com/group/google-appengine/browse_thread/thread/e9464ceb131c726f/6aeae1e390038592?pli=1
Personally I find this comment in the discussion very informative about the Google App Engine Data Store:
http://groups.google.com/group/google-appengine/msg/ee3bd373bd31e2c7
At scale you wind up doing a bunch of
things that seem wrong, but that are
required by the numbers we are
running. Go watch the EBay talks. Or
read the posts about how many database
instances FaceBook is running.
The simple truth is, what we learned
about in uni was great for the
business automation apps of small to
medium enterprise applications, where
the load was predictable, and there
was money enough to buy the server
required to handle the load of 50
people doing data entry into an
accounts or business planning and
control app....
Searched around a bit more and came across this Google Doc Article:
http://code.google.com/appengine/articles/modeling.html
App Engine allows the creation of easy
to use relationships between datastore
entities which can represent
real-world things and ideas. Use
ReferenceProperty when you need to
associate an arbitrary number of
repeated types of information with a
single entity. Use key-lists when you
need to allow lots of different
objects to share other instances
between each other. You will find that
these two approaches will provide you
with most of what you need to create
the model behind great applications.
Can I supplement the excellent answer further above with a link to a video:
http://sites.google.com/site/io/building-scalable-web-applications-with-google-app-engine
It's a great talk by Google's Brett Slatkin who talks for an hour about the special way you need to think about your application before you can expect it to scale well. There are some genuine WTFs (such as no count() in db queries) that will cause you to struggle if you are coming from a relational background.
I think this is the basics : Keys and Entity Groups
look for it in appengine docs. (I'm new here so can't post a link)
I have worked on it but not a expert though Google app engine is very good thing and it is the future as it implements Platform as a Service and Software as a Service. Google app engine provides a non- relational database. So you cantreally write relationships here.
Regards,
Gaurav J
These links are great, but are predominantly python biased, I am using GWT, and therefore have to use the java flavour of GAE, does anyone have any examples of how to achieve these "join" equivalencies in the java version of GAE?
Cheers,
John
The standalone GAE SDK is pretty difficult to use for putting data into and retrieving data from the Google App Engine data store.
"Objectify" is a GAE extension that makes these operations much easier. The Objectify wiki and source code can be found here. I strongly recommend using Objectify in your GAE project.
http://code.google.com/p/objectify-appengine/
Here are a couple of tutorials on using Objectify with the app engine. Follow these tutorials and you will be storing and retrieving data in no time.
http://www.fishbonecloud.com/2010/11/use-objectify-to-store-data-in-google.html