Jump to content

Check out our Community Blogs


Member Since 05 Nov 2010
Offline Last Active Feb 16 2011 12:02 PM

Topics I've Started

Python on Google App Engine: Creating blog engine. Part 3

05 December 2010 - 12:27 PM

In this part I am going to discuss some limitations I met at GAE. To be honest there are a lot of them, but here I listed only issues for which nice workaround was found.

1. pre- and post- hooks. Flask signals.
There is an excelent article by Nick Johnson: Pre- and post- put hooks for Datastore models - Nick's Blog . I fixed and enhanced code provided by Nick to support Flask signals and created BaseModel that extends db.Model:

from blinker import Namespace

model_signals = Namespace()
pre_put = model_signals.signal('pre-put')
post_put = model_signals.signal('post-put')

class BaseModel(db.Model):
    def pre_put(self):

    def post_put(self):

    def put(self, **kwargs):
        if self.pre_put() == False:
            return None # do not save model if pre_put() returned False
        for (receiver, ret) in signals.pre_put.send(self.__class__, model=self): # send signal to subscribers
            if ret == False:
                return None # do not save model if handler returned False

        key = super(BaseModel, self).put(**kwargs) # save model

        # here goes post hooks
        signals.post_put.send(self.__class__, model=self)

        return key

old_put = db.put

def put(models, **kwargs):
    # we can fallback to BaseModel.put() if user want to save only one model
    if not isinstance(models, (list, tuple)):
        return models.put()

    # call hooks and send signals to all models
    for model in models:
        if isinstance(model, BaseModel):
            if model.pre_put() == False:
            for (_, ret) in signals.pre_put.send(model.__class__, model):
                if ret == False:

    # check it there are models on the list
    if models:
        # batch save
        keys = old_put(models, **kwargs)
        keys = None

    # call hooks and send signals to all models
    for model in models:
        if isinstance(model, BaseModel):
            signals.post_put.send(model.__class__, model)

    return keys

# monkey path GAE ORM
db.put = put

This code is used in my app like this:

class Post(BaseModel):
    def pre_put(self):
        # autogenerate html from markdown markup
        self.content_html = markdown(self.content)

# here we listen for post_put signal on Post model to icrement post_count for post category
def increment_post_count_on_category(sender, model=None):
    category = model.category
    category.post_count += 1

2. Datastore transactions

Quote from docs:

All datastore operations in a transaction must operate on entities in the same entity group. This includes querying for entities by ancestor, retrieving entities by key, updating entities, and deleting entities. Notice that each root entity belongs to a separate entity group, so a single transaction cannot create or operate on more than one root entity.

In my case I want to check if category exists in database before saving it. To be able to do this I have to put all categories under the same root entity, but there is no suitable entity for this. The good news is that GAE does not really care if you provide real (saved) entity or just dummy key with random id. So I use dummy key as entity root to be able to use transaction:

Category(parent=db.Key.from_path(Category.kind(), sys.maxint))

Code that saves category looks like this:

        def create(cls, name):
            def txn():
                entity = Category.all().ancestor(our_dummy_key).filter('name =', name).get()
                if entity:
                    return (entity, False)

                entity = Category(name=name, parent=our_dummy_key)

                return (entity, True)

            return db.run_in_transaction(txn)

category, created = Category.create('name')

3. Denormalization and model instance method

GAE does not support joins. To be able to show user category name of the post I have to duplicate category name in the Post model:

class Post(BaseModel):
    # ...

    category_name = db.CategoryProperty(required=False)
    category = db.ReferenceProperty(Category, required=True)

Template looks like this:

<a href="{{ url_for('blog.category_view', slug=slugify(post.category_name)) }}">{{ post.category_name }}</a>

Not bad, but ususally above code looks like this:

<a href="{{ post.category.get_absolute_url() }}">{{ post.category }}</a>

To accomplish this I add such code to the post:

class Post(BaseModel):
    def category_c(self):
        """Creates and caches dummy category"""
        return Category(name=self.category_name)

The only change you need in template is replacing "category" with "category_c". Category must implement __unicode__() and get_absolute_url() methods:

class Category(BaseModel):
    # ...

    def get_absolute_url(self):
        return url_for('blog.category_view', slug=self.slug)

    def __unicode__(self):
        return unicode(self.name) # don't forget to force unicode conversion on property

4. key_id and key_name

It is not clear when I have to use key_id or key_name on the model and how to migrate from one to another. What do you think?

5.Model validation and required property

If True, the property cannot have a value of None. A model instance must initialize all required properties in its constructor so that the instance is not created with missing values. An attempt to create an instance without initializing a required property, or an attempt to assign None to a required property, raises a BadValueError.

I am not happy with this, because I definitely want most properties to present in datastore, but I also want to be able to autogenerate missing properties (and key_name) in pre_put hook. What do you think?

Python on Google App Engine: Creating blog engine. Part 2

28 November 2010 - 02:44 PM

In last part we configured very basic Flask application for Google App Engine. In this part we will add some models and forms to our application. Let's start with model Post which represents blog post. Create file blog/models.py:

from google.appengine.ext import db

class Post(db.Model):
    title = db.StringProperty(required=True)

    content = db.TextProperty(required=True)
    content_html = db.TextProperty(required=True)

    # we cache category_name within Post to avoid JOIN query
    category_name = db.CategoryProperty(required=True)
    category = db.ReferenceProperty(Category, required=True)

    # we also cache tags for same reason
    tags = db.StringListProperty(required=True)

    # these 2 fields will be autopopulated by GAE, see docs for more info
    created_at = db.DateTimeProperty(required=True, auto_now_add=True)
    updated_at = db.DateTimeProperty(required=True, auto_now=True)

The code is pretty straightforward. Now we have to create form that will represent our Post model. Flask has very nice integration with WTForms and we are going to use it, because WTForms officially supports standard App Engine models. That means that WTForms is able to generate form from supplied model. You have to download WTForms and Flask-WTF extension. To generate form from existing model we have to use method model_form(). Create file blog/forms.py:

from flaskext import wtf
from wtforms.ext.appengine.db import model_form

# model_form() will collect properties from models.Port and convert them to appropriate fields
PostForm = model_form(models.Post, base_class=wtf.Form)

Now we have model that can store our post and form that can display it. We can implement view that will create posts using model and form. Create file blog/views.py:

# this view is accessible via URL http://localhost:8080/create-post using GET and POST methods
@module.route('/create-post', methods=['GET', 'POST'])
def create_post():
    form = forms.PostForm()
    if form.validate_on_submit(): # form was submitted and validated
        flash(u'Post successfully create') # notify user
        # create post 
        post = models.Post(title=form.title.data, content=form.content.data) using data from form
        # save post in datastore
        # redirect user to the post
        return redirect(post.get_absolute_url())
    # if form was not submitted or user provided invalid data just render template with our form
    return render_template('blog/create_post.html', form=form)

The only thing that I still did not explained is rendering form. We are using Jinja2 as template engine and we will use some reusable functions (macros in Jinja2 terms) that will help us render our form. You can see these macros at github. Template that renders form is located at blog/post/_form.html:

<!-- Import form macro I mentioned above -->
{% from '_form_macros.html' import form_field_td %}

<form method="post" action="" enctype="multipart/form-data">
    <!-- Here we manually render each field -->
    <!-- It is boring but with this approach we can easily move and group fields -->
    <tr>{{ form_field_td(form.title) }}</tr>
    <tr>{{ form_field_td(form.category_name) }}</tr>
    <tr>{{ form_field_td(form.content) }}</tr>
  <fieldset class="submit"> 
    <input type="submit" name="submit" value="Submit"> 

The last thing that we will do today is displaying posts. To do this we have to create new view:

# this view will be accessible via URL like http://localhost/view/flask-on-gae
@module.route('/view/<slug>', methods=['GET'])
def view_post(slug):
    # slug is used as key for model
    # get post by key
    post = models.Post.get_by_key_name(slug)
    return render_template('blog/post/post.html', post=post)

Template that renders post is very simple:

{% extends 'base.html' %}

{% block body %}
<h1>{{ post.title }}</h1>

<p>Category: {{ post.category_name }} @ {{ post.created_at }}</p>

{{ post.content_html|safe }}
{% endblock body %}

I think it is enough for today. I don't explain a lot of things that are covered by official docs. If you don't understand anything please ask questions. Demo is here: Flask on GAE

Python on Google App Engine: Creating blog engine

21 November 2010 - 07:00 AM

I am going to write series of articles describing process of creating a blog engine for Google App Engine. I suppose that you have some experience with Python and web frameworks around it. I will try to cover non-trivial things that you will meet on GAE (emulating joins on Datastore, using blobstore, queues, map reduce etc), but I will also describe some basic things that are specific to GAE.

I use this environment in my daily development:
1. Windows 7
2. Eclipse + PyDev
3. GAE latest Python SDK: Downloads - Google App Engine - Google Code

You can setup your own environment, but this tutorial will cover mine :)

There are a couple of frameworks you can use on GAE:
1. Flask (get started) - extremely simple micro framework built on top of Werkzeug and Jinja2. You can directly use GAE Datastore, but your views can be easily ported to another platform.
2. Tipfy (get started) has a lot of extensions that are specific to GAE (like auth, sharded counters, appstats/blobstore utilities), but you can not use this framework outside GAE.
3. Django-nonrel (get started) - you will get most of the power of Django, but you will miss GAE specific features. You should use it only if you want to port existing Django app to GAE.
4. webapp, webapp2, kay, web2py and many others - less popular and less documented.

I will use Flask because I want directly use GAE APIs to show more features, but tipfy is probably better choice if you are not going to move from GAE. First, download and copy flask, werzeug and jinja2 to your project root. Then you have to create app.yaml file:

application: flaskapp # application name, http://flaskapp.appspot.com/
version: part1 # this version will be available at http://part1.latest.flaskapp.appspot.com/
runtime: python
api_version: 1

default_expiration: '365d' # expiration for static files

builtins: # enables various  builtin GAE apps
- admin_redirect: on
- datastore_admin: on
- appstats: on
- remote_api: on

handlers: # handlers for urls
- url: /static # we can access our statics at this url, for example http://flaskapp.appspot.com/static/img/logo.png
  static_dir: static # path to folder that conatins static files, static/img/logo.png

- url: /.* # all other requests will be served by main.py script
  script: main.py

You may notice that app.yaml contains reference to file "main.py" and dir "static". Create them. We will also need a separate dir to store templates and file to store our config. You project structure should look like this:


We created basic structure of our project. Now we can start programming. Very basic main.py file can look like this:

from google.appengine.ext.webapp.util import run_wsgi_app

from flask import *

app = Flask(__name__)
app.config.from_pyfile('config.py') # config.py is empty now, but we will add some lines later

def index():
    return render_template('index.html') # index.html will be searched in templates/ dir

def main():

if __name__ == '__main__':

Code is very short and self-explaining. index.html is very simple:


<title>Flask on GAE</title>


<h1>Flask on GAE</h1>



To run app execute following command in Command Prompt:

dev_appserver.py .

Site should be accessible at http://localhost:8080/ . There is nothing specific to GAE in the code. If you don't understand anything, please refer to the official Flask docs: Welcome to Flask — Flask v0.6.1dev documentation . To deploy app to GAE use this command:

appcfg.py update .

Site should be accessible at Flask on GAE . Source code is available at bitbucket: vladimir_webdev / flaskapp / source . In next part I am going to cover models (Post, Tag, Category) and views (creating new posts, categories, tagging).

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download