Getting started with the Google Cloud Datastore API

This page provides a short exercise in building a simple command-line TaskList application with the Google Cloud Datastore API. The TaskList application stores, lists, updates, and removes tasks.

Prerequisites

  1. Ability to write and run a command line application in the programming languages used in this topic
    In addition to a basic understanding of how to develop applications, you should be able to download and install additional libraries before attempting this tutorial.
  2. A Google Cloud Platform Console project with the Cloud Datastore API enabled
    Applications that use Cloud Datastore must be associated with a Google Cloud Platform Console project with the Cloud Datastore API enabled. This project provides authentication credentials you use in your application to identify it to Google and authorize its use of the Cloud Datastore API.
    Follow these instructions to create a project, enable the Cloud Datastore API for it, and set up your local development environment with authentication credentials using the gcloud auth login command. Make a note of the project's ID, which you'll use later on.
  3. An active App Engine Application
    Projects that use the Cloud Datastore API require an active App Engine application. Open the App Engine dashboard and confirm your project has an active App Engine app.
    Create an App Engine app if needed. The app must not be disabled.

Installation and setup

Install client libraries and configure any additional settings for your development environment.

C#

  1. Ensure you have Visual Studio (version 2013 or later) installed.
  2. Download the TaskList sample application from here.
  3. Extract the files from the zip into a directory in your Documents folder.
  4. In Visual Studio, open the file dotnet-docs-samples-master\datastore\api\Datastore.sln.
  5. In Visual Studio's Solution Explorer window, right click the TaskList project and choose Set as StartUp Project.
  6. Right click the TaskList project again and choose Properties.
  7. In the Properties window, click Debug and type the ID of your Google Cloud Platform project into the Command line arguments: box.

    Visual Studio Debug Window

  8. Click File and then click Save to save your changes.

  9. Run the application! Press F5 on your keyboard.

Go

  1. Clone the TaskList sample application.

    go get github.com/GoogleCloudPlatform/golang-samples/datastore/tasks
    
  2. Change directories to where you cloned the sample:

    cd $GOPATH/src/github.com/GoogleCloudPlatform/golang-samples/datastore/tasks
    
  3. At a command prompt, run the following, where <project-id> is the ID of your Google Cloud Platform project.

    export DATASTORE_PROJECT_ID=<project-id>
    

    (Windows users: use set instead of export.)

  4. Run the application!

    go run tasks.go
    

Java

  1. Ensure you have Maven and Java (version 7 or later) installed.

  2. Download the TaskList sample application from here.

  3. At a command prompt, unzip the download:

    unzip java-docs-samples-master.zip
    
  4. Change directories to the TaskList application:

    cd java-docs-samples-master/datastore
    
  5. Run the following, where <project-id> is the ID of your Google Cloud Platform project.

    gcloud config set project <project-id>
    
  6. Compile and run the application!

    mvn clean compile
    mvn exec:java
    

Node.js

  1. Ensure you have Node.js and npm installed.

  2. Download the TaskList sample application from here.

  3. Unzip the download:

    unzip nodejs-docs-samples-master.zip
    
  4. Change directories to the TaskList application:

    cd nodejs-docs-samples-master/datastore
    
  5. Install the dependencies and link the application:

    npm install
    
  6. At a command prompt, run the following, where <project-id> is the ID of your Google Cloud Platform project.

    export GCLOUD_PROJECT=<project-id>
    

    (Windows users: use set instead of export.)

  7. Run the application!

    node tasks.js
    

PHP

  1. Ensure you have PHP (version 5.6 or later) and Composer installed.
  2. Download the TaskList sample application from here.
  3. Unzip the download:

    unzip php-docs-samples-master.zip
    
  4. Change directories to the TaskList application:

    cd php-docs-samples-master/datastore/tutorial
    
  5. Install dependencies.

    If Composer is installed locally:

    php composer.phar install
    

    If Composer is installed globally:

    composer install
    
  6. Run the application!

    php tasks.php
    

Python

  1. Ensure you have Python (version 2.7.9 or later), pip, and virtualenv installed.
  2. Activate a virtualenv session.

    virtualenv venv
    source venv/bin/activate
    
  3. Download the TaskList sample application from here.

  4. Unzip the download:

    unzip python-docs-samples-master.zip
    
  5. Change directories to the TaskList application:

    cd python-docs-samples-master/datastore/cloud-client
    
  6. Install dependencies:

    pip install -r requirements.txt
    
  7. Run the application! Use the ID of your Google Cloud Platform project for <project-id>.

    python tasks.py --project-id <project-id>
    

Ruby

  1. Ensure you have Ruby and Bundler installed.

  2. Download the TaskList sample application from here.

  3. Unzip the download:

    unzip ruby-docs-samples-master.zip
    
  4. Change directories to the TaskList application:

    cd ruby-docs-samples-master/datastore
    
  5. Install the dependencies:

    bundle install
    
  6. At a command prompt, run the following, where <project-id> is the ID of your Google Cloud Platform project.

    export GOOGLE_CLOUD_PROJECT=<project-id>
    

    (Windows users: use set instead of export.)

  7. Run the application!

    bundle exec ruby tasks.rb
    

Creating an Authorized Service Object

In order to make authenticated requests to Google Cloud APIs using the Google APIs Client libraries, you must:

  • Fetch the credential to use for requests.
  • Create a service object that uses that credential.

You can then make API calls by calling methods on the Cloud Datastore service object.

For this example, you'll fetch Application Default Credentials from the environment, and pass it as an argument to create the service object.

Here's the call to create the authorized Cloud Datastore service object:

C#

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

// Create an authorized Datastore service using Application Default Credentials.
_db = DatastoreDb.Create(projectId);
// Create a Key factory to construct keys associated with this project.
_keyFactory = _db.CreateKeyFactory("Task");

Go

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

ctx := context.Background()
client, err := datastore.NewClient(ctx, projID)

Java

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

// Create an authorized Datastore service using Application Default Credentials.
private final Datastore datastore = DatastoreOptions.getDefaultInstance().getService();

// Create a Key factory to construct keys associated with this project.
private final KeyFactory keyFactory = datastore.newKeyFactory().setKind("Task");

Node.js

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

// By default, the client will authenticate using the service account file
// specified by the GOOGLE_APPLICATION_CREDENTIALS environment variable and use
// the project specified by the GCLOUD_PROJECT environment variable. See
// https://googlecloudplatform.github.io/google-cloud-node/#/docs/datastore/latest/guides/authentication
const Datastore = require('@google-cloud/datastore');

// Instantiates a client
const datastore = Datastore();

PHP

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

use Google\Cloud\Datastore\DatastoreClient;

/**
 * Create a Cloud Datastore client.
 *
 * @param string $projectId
 * @return DatastoreClient
 */
function build_datastore_service($projectId)
{
    $datastore = new DatastoreClient(['projectId' => $projectId]);
    return $datastore;
}

Python

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

from google.cloud import datastore


def create_client(project_id):
    return datastore.Client(project_id)

Ruby

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

require "google/cloud/datastore"

@datastore = Google::Cloud::Datastore.new project: project_id

Storing data

Objects in Cloud Datastore are known as entities, and each entity is of a particular kind. The TaskList application will store entities of kind Task, with the following properties:

  • description: a string specified by the user as the task description
  • created: a date that shows when the task was initially created
  • done: a boolean that indicates whether the task has been completed

When the user adds a new task, the TaskList application will create a Task entity with values for the properties listed above:

C#

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/// <summary>
///  Adds a task entity to the Datastore
/// </summary>
/// <param name="description">The task description.</param>
/// <returns>The key of the entity.</returns>
Key AddTask(string description)
{
    Entity task = new Entity()
    {
        Key = _keyFactory.CreateIncompleteKey(),
        ["description"] = new Value()
        {
            StringValue = description,
            ExcludeFromIndexes = true
        },
        ["created"] = DateTime.UtcNow,
        ["done"] = false
    };
    return _db.Insert(task);
}

Go

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

// Task is the model used to store tasks in the datastore.
type Task struct {
	Desc    string    `datastore:"description"`
	Created time.Time `datastore:"created"`
	Done    bool      `datastore:"done"`
	id      int64     // The integer ID used in the datastore.
}

// AddTask adds a task with the given description to the datastore,
// returning the key of the newly created entity.
func AddTask(ctx context.Context, client *datastore.Client, desc string) (*datastore.Key, error) {
	task := &Task{
		Desc:    desc,
		Created: time.Now(),
	}
	key := datastore.IncompleteKey("Task", nil)
	return client.Put(ctx, key, task)
}

Java

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/**
 * Adds a task entity to the Datastore.
 *
 * @param description The task description
 * @return The {@link Key} of the entity
 * @throws DatastoreException if the ID allocation or put fails
 */
Key addTask(String description) {
  Key key = datastore.allocateId(keyFactory.newKey());
  Entity task = Entity.newBuilder(key)
      .set("description", StringValue.newBuilder(description).setExcludeFromIndexes(true).build())
      .set("created", Timestamp.now())
      .set("done", false)
      .build();
  datastore.put(task);
  return key;
}

Node.js

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

function addTask (description) {
  const taskKey = datastore.key('Task');
  const entity = {
    key: taskKey,
    data: [
      {
        name: 'created',
        value: new Date().toJSON()
      },
      {
        name: 'description',
        value: description,
        excludeFromIndexes: true
      },
      {
        name: 'done',
        value: false
      }
    ]
  };

  datastore.save(entity)
    .then(() => {
      console.log(`Task ${taskKey.id} created successfully.`);
    })
    .catch((err) => {
      console.error('ERROR:', err);
    });
}

PHP

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/**
 * Create a new task with a given description.
 *
 * @param DatastoreClient $datastore
 * @param $description
 * @return Google\Cloud\Datastore\Entity
 */
function add_task(DatastoreClient $datastore, $description)
{
    $taskKey = $datastore->key('Task');
    $task = $datastore->entity(
        $taskKey,
        [
            'created' => new DateTime(),
            'description' => $description,
            'done' => false
        ],
        ['excludeFromIndexes' => ['description']]
    );
    $datastore->insert($task);
    return $task;
}

Python

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

def add_task(client, description):
    key = client.key('Task')

    task = datastore.Entity(
        key, exclude_from_indexes=['description'])

    task.update({
        'created': datetime.datetime.utcnow(),
        'description': description,
        'done': False
    })

    client.put(task)

    return task.key

Ruby

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

def add_task description
  task = @datastore.entity "Task" do |t|
    t["description"] = description
    t["created"]     = Time.now
    t["done"]        = false
    t.exclude_from_indexes! "description", true
  end

  @datastore.save task

  puts task.key.id

  task.key.id
end

For this application, we'll also provide a method to update the done property, to indicate the task is complete:

C#

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/// <summary>
/// Marks a task entity as done.
/// </summary>
/// <param name="id">The ID of the task entity as given by Key.</param>
/// <returns>true if the task was found.</returns>
bool MarkDone(long id)
{
    using (var transaction = _db.BeginTransaction())
    {
        Entity task = transaction.Lookup(_keyFactory.CreateKey(id));
        if (task != null)
        {
            task["done"] = true;
            transaction.Update(task);
        }
        transaction.Commit();
        return task != null;
    }
}

Go

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

// MarkDone marks the task done with the given ID.
func MarkDone(ctx context.Context, client *datastore.Client, taskID int64) error {
	// Create a key using the given integer ID.
	key := datastore.IDKey("Task", taskID, nil)

	// In a transaction load each task, set done to true and store.
	_, err := client.RunInTransaction(ctx, func(tx *datastore.Transaction) error {
		var task Task
		if err := tx.Get(key, &task); err != nil {
			return err
		}
		task.Done = true
		_, err := tx.Put(key, &task)
		return err
	})
	return err
}

Java

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/**
 * Marks a task entity as done.
 *
 * @param id The ID of the task entity as given by {@link Key#id()}
 * @return true if the task was found, false if not
 * @throws DatastoreException if the transaction fails
 */
boolean markDone(long id) {
  Transaction transaction = datastore.newTransaction();
  try {
    Entity task = transaction.get(keyFactory.newKey(id));
    if (task != null) {
      transaction.put(Entity.newBuilder(task).set("done", true).build());
    }
    transaction.commit();
    return task != null;
  } finally {
    if (transaction.isActive()) {
      transaction.rollback();
    }
  }
}

Node.js

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

function markDone (taskId) {
  const transaction = datastore.transaction();
  const taskKey = datastore.key([
    'Task',
    taskId
  ]);

  transaction.run()
    .then(() => transaction.get(taskKey))
    .then((results) => {
      const task = results[0];
      task.done = true;
      transaction.save({
        key: taskKey,
        data: task
      });
      return transaction.commit();
    })
    .then(() => {
      // The transaction completed successfully.
      console.log(`Task ${taskId} updated successfully.`);
    })
    .catch(() => transaction.rollback());
}

PHP

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/**
 * Mark a task with a given id as done.
 *
 * @param DatastoreClient $datastore
 * @param int $taskId
 */
function mark_done(DatastoreClient $datastore, $taskId)
{
    $taskKey = $datastore->key('Task', $taskId);
    $transaction = $datastore->transaction();
    $task = $transaction->lookup($taskKey);
    $task['done'] = true;
    $transaction->upsert($task);
    $transaction->commit();
}

Python

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

def mark_done(client, task_id):
    with client.transaction():
        key = client.key('Task', task_id)
        task = client.get(key)

        if not task:
            raise ValueError(
                'Task {} does not exist.'.format(task_id))

        task['done'] = True

        client.put(task)

Ruby

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

def mark_done task_id
  task = @datastore.find "Task", task_id

  task["done"] = true

  @datastore.save task
end

Here's the method to delete a Task entity, using the Task entity's key:

C#

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/// <summary>
/// Deletes a task entity.
/// </summary>
/// <param name="id">The ID of the task entity as given by Key.</param>
void DeleteTask(long id)
{
    _db.Delete(_keyFactory.CreateKey(id));
}

Go

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

// DeleteTask deletes the task with the given ID.
func DeleteTask(ctx context.Context, client *datastore.Client, taskID int64) error {
	return client.Delete(ctx, datastore.IDKey("Task", taskID, nil))
}

Java

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/**
 * Deletes a task entity.
 *
 * @param id The ID of the task entity as given by {@link Key#id()}
 * @throws DatastoreException if the delete fails
 */
void deleteTask(long id) {
  datastore.delete(keyFactory.newKey(id));
}

Node.js

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

function deleteTask (taskId) {
  const taskKey = datastore.key([
    'Task',
    taskId
  ]);

  datastore.delete(taskKey)
    .then(() => {
      console.log(`Task ${taskId} deleted successfully.`);
    })
    .catch((err) => {
      console.error('ERROR:', err);
    });
}

PHP

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/**
 * Delete a task with a given id.
 *
 * @param DatastoreClient $datastore
 * @param $taskId
 */
function delete_task(DatastoreClient $datastore, $taskId)
{
    $taskKey = $datastore->key('Task', $taskId);
    $datastore->delete($taskKey);
}

Python

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

def delete_task(client, task_id):
    key = client.key('Task', task_id)
    client.delete(key)

Ruby

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

def delete_task task_id
  task = @datastore.find "Task", task_id

  @datastore.delete task
end

Running a query

In addition to retrieving entities from Cloud Datastore directly by their keys, an application can perform a query to retrieve them by the values of their properties. A typical query includes the following:

  • An entity kind to which the query applies
  • Zero or more filters, for example to select kinds whose properties match a value
  • Zero or more sort orders, to sequence the results

For this application, we'll query Cloud Datastore for Task entities sorted by creation time:

C#

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/// <summary>
/// Returns a list of all task entities in ascending order of creation time.
/// </summary>
IEnumerable<Entity> ListTasks()
{
    Query query = new Query("Task")
    {
        Order = { { "created", PropertyOrder.Types.Direction.Descending } }
    };
    return _db.RunQuery(query).Entities;
}

Go

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

// ListTasks returns all the tasks in ascending order of creation time.
func ListTasks(ctx context.Context, client *datastore.Client) ([]*Task, error) {
	var tasks []*Task

	// Create a query to fetch all Task entities, ordered by "created".
	query := datastore.NewQuery("Task").Order("created")
	keys, err := client.GetAll(ctx, query, &tasks)
	if err != nil {
		return nil, err
	}

	// Set the id field on each Task from the corresponding key.
	for i, key := range keys {
		tasks[i].id = key.ID
	}

	return tasks, nil
}

Java

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/**
 * Returns a list of all task entities in ascending order of creation time.
 *
 * @throws DatastoreException if the query fails
 */
Iterator<Entity> listTasks() {
  Query<Entity> query =
      Query.newEntityQueryBuilder().setKind("Task").setOrderBy(OrderBy.asc("created")).build();
  return datastore.run(query);
}

Node.js

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

function listTasks () {
  const query = datastore.createQuery('Task')
    .order('created');

  datastore.runQuery(query)
    .then((results) => {
      const tasks = results[0];

      console.log('Tasks:');
      tasks.forEach((task) => {
        const taskKey = task[datastore.KEY];
        console.log(taskKey.id, task);
      });
    })
    .catch((err) => {
      console.error('ERROR:', err);
    });
}

PHP

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

/**
 * Return an iterator for all the tasks in ascending order of creation time.
 *
 * @param DatastoreClient $datastore
 * @return EntityIterator<Google\Cloud\Datastore\Entity>
 */
function list_tasks(DatastoreClient $datastore)
{
    $query = $datastore->query()
        ->kind('Task')
        ->order('created');
    return $datastore->runQuery($query);
}

Python

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

def list_tasks(client):
    query = client.query(kind='Task')
    query.order = ['created']

    return list(query.fetch())

Ruby

For more on installing and creating a Cloud Datastore client, refer to Cloud Datastore Client Libraries.

def list_tasks
  query = @datastore.query("Task").order("created")
  tasks = @datastore.run query

  tasks.each do |t|
    puts t["description"]
    puts t["done"] ? "  Done" : "  Not Done"
    puts "  ID: #{t.key.id}"
  end
end

Next Steps

This tutorial covers only the most basic steps necessary to make calls to the Cloud Datastore API from a command-line application. Cloud Datastore supports fast and highly scalable ACID transactions, SQL-like queries, indexes and more.

Send feedback about...

Cloud Datastore Documentation