App Engine Datastore Overview

by

José M Vidal

Navigation

  • [arrow keys] to go next and previous

  • [mouse scroll wheel] to go next and previous

  • [Ctrl or Command] + [+/-] to zoom in and out

  • [touch gestures] for mobile devices

Options

Definitions

  • An item in the datastore is called an entitty
  • An entity has properties of some data type.
  • An entity has as kind: name of python class that its an instance of. Entities of the same kind can have different properties (run-time addition of properties). That is, entities are schemaless.

Example

  • First example: See how multiple instances are created.
  • Second example: See how we create instances with a name from the URL.

Keys

  • Each entity has a key (name or ID) which is either assigned automatically or given using the key_name argument.
  • Keys cannot be changed later on.
  • Fetching by key is superfast (according to the video presentation).
  • Example: creating entity with key from URL path, and fetching by key.
  • Batch operations:
          db.put([k1, k2, k3]])
          

Queries

  • Example: /list entries with a given key_name
  • .order('age'): sorts. '-age'
  • .fetch(5): gets 5 results
  • .with_cursor: for paging
  • All queried properties must have index. These are created automatically if you excersize all your code in dev_appserver.py. Stored in index.yaml
  • Multiple inequality .filter()s are allowed for one property only.
  • Properties in inequality filter must be sorted before other properties:
    SELECT * FROM Person WHERE birth_year >= 1970 ORDER BY last_name # ERROR
    	    
    SELECT * FROM Person WHERE birth_year >= 1970 ORDER BY last_name, birth_year  # ERROR
    
    SELECT * FROM Person WHERE birth_year >= 1970 ORDER BY birth_year, last_name #OK
    

Transactions

  • Transactions give use atomic operations: all or nothing.
  • Example: transactions
  • All datastore operations in a transaction must operate on entities in the same entity group. To put entities on an entity group use parent=x attribute on creation of the entity.
    employee = Employee() #a db.Model
    employee.put()
    
    # Create an Address in the same entity group as the Employee by setting
    # the Employee as the Address's parent.
    address = Address(parent=employee)
    
    # Another way...using the db.Key of the Employee.
    e_key = employee.key()
    address = Address(parent=e_key)
    
  • See Sharding Counters for better implementation of counters (less contention).
  • Transactions also useful for generating a consistent view of many tables.

Expando Models

  • They let you add more properties later on...
class Person(db.Expando):
    first_name = db.StringProperty()
    last_name = db.StringProperty()
    hobbies = db.StringListProperty()

p = Person(first_name="Albert", last_name="Johnson")
p.hobbies = ["chess", "travel"]

p.chess_elo_rating = 1350

p.travel_countries_visited = ["Spain", "Italy", "USA", "Brazil"]
p.travel_trip_count = 13     
    

PolyModel Model

  • Lets you create hierarchies of Models.
from google.appengine.ext import db
from google.appengine.ext.db import polymodel

class Contact(polymodel.PolyModel):
    phone_number = db.PhoneNumberProperty()
    address = db.PostalAddressProperty()

class Person(Contact):
    first_name = db.StringProperty()
    last_name = db.StringProperty()
    mobile_number = db.PhoneNumberProperty()

class Company(Contact):
    name = db.StringProperty()
    fax_number = db.PhoneNumberProperty()
   

Choosing a Datastore

Master/Slave High Replication
Cost
    Storage 1x 3x
    Put/Delete CPU 1x 3x
    Get CPU 1x 1x
    Query CPU 1x 1x
Performance
    Put/Delete Latency 1x 1x–2x
    Get Latency 1x 1x
    Query Latency 1x 1x
Consistency
    Get/Put/Delete Strong Strong
    Most Queries Strong Eventual
    Ancestor Queries Strong Strong
Occasional Planned Read-Only Period Yes No
Unplanned Downtime Rare. Possible to lose a small % of writes that occurred near the downtime (recoverable after event). Extremely rare. No data loss.

Metadata Queries

  • You can ask the datastor to tell you all the tables (Entity) and their properties.
Model Kind
google.appengine.ext.db.metadata.Namespace __namespace__
google.appengine.ext.db.metadata.Kind __kind__
google.appengine.ext.db.metadata.Property __property__
import google.appengine.ext.db
from google.appengine.ext.db.metadata import Kind

q = Kind.all()
q.filter('__key__ >=', Kind.key_for_kind('a'))
q.filter('__key__ <', Kind.key_for_kind(chr(ord('z') + 1)))
for kind in q.fetch(100):
  print kind.kind_name