Adding commands to your mobile app using a Dialogflow agent

This tutorial shows how to make your iOS app respond to voice and text commands that are processed by a Dialogflow agent. The client app in this tutorial captures the command, sends it to the agent, and receives a response. The agent processes the command and matches it to a preconfigured intent to return an appropriate response. This tutorial also shows how to retrieve short-lived credentials that client apps can use to authenticate calls to Google Cloud APIs.

Solution overview

The solution includes the following components:

Client app
The client component is an iOS app that receives voice and text commands and sends them to the stopwatch agent. The app sends the command using an authenticated request that includes credentials obtained from the token service.
Token service
The token service is implemented using Cloud Functions for Firebase. The service provides short-lived credentials that client apps can use to send authenticated requests to the stopwatch agent. The token service retrieves the credentials from the Cloud IAM API, stores them in a Cloud Firestore database, and validates that credentials haven't expired.
Stopwatch agent
Dialogflow hosts the stopwatch agent, which provides the natural language processing capabilities to translate the commands to instructions to control the stopwatch.

The following diagram shows the interaction between the components:

Solution high-level architecture

Costs

Consider the following potential costs associated with running the samples in this tutorial:

  • Firebase has a free level of usage. If your usage of these services is less than the limits specified in the Spark plan, there is no charge for using Firebase. For more information, check Firebase Pricing plans.
  • Firebase defines quotas for Cloud Functions usage that specify resource, time, and rate limits. For more information, check Quotas and limits in the Firebase documentation.
  • Dialogflow is priced monthly based on the edition, plan, and number and duration of requests. You can use a limited number of requests without charge using the Standard edition. For more information, check Dialogflow pricing.

You can also estimate the costs using the Google Cloud Pricing Calculator.

Before you begin

To complete this tutorial, you need the following software:

The sample app was tested with Xcode version 10.2.1, CocoaPods version 1.5.2, and Node.js version 8.16.1.

Cloning the sample code

Clone the client app code:

git clone https://github.com/GoogleCloudPlatform/ios-docs-samples.git

Clone the token service code:

git clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples.git

Create a Firebase project

  1. Create a Firebase account or log into an existing account.

  2. Click Add project.

  3. In Project name, enter: stopwatch. Make a note of the Project ID assigned to your project, it's used in multiple steps in this tutorial.

  4. Follow the remaining setup steps and click Create project.

  5. After the wizard provisions your project, click Continue.

  6. In the Overview page of your project, click the Settings gear and then click Project settings.

  7. Click Add Firebase to your iOS app.

  8. In iOS bundle ID, enter: com.sample.dialogflow.

  9. Click Register app.

  10. Follow the steps in the Download config file section to add the GoogleService-Info.plist file to the ios-docs-samples/dialogflow/stopwatch/ folder in your project.

  11. Click Next in the Download config file section.

  12. Make a note of the instructions to use CocoaPods to install and manage project dependencies. The CocoaPods dependency manager is already configured in the sample code.

  13. Run the following command to install the dependencies. The command might take a long time to complete.

    cd ios-docs-samples/dialogflow/stopwatch/
    ./INSTALL-COCOAPODS
    

    You might need to run pod repo update if your CocoaPods installation cannot find the Firebase dependency.

    After this step, use the newly created .xcworkspace file instead of the .xcodeproj file for all future development on the iOS app.

  14. Click Next in the Add Firebase SDK section.

  15. Make a note of the code required to initialize Firebase in your project.

  16. Click Next in the Add initialization code section.

  17. Click Skip this step in the Run your app to verify installation section.

  18. Replace the your-project-identifier placeholder in the ApplicationConstants.swift file with the Firebase project ID.

Creating a service account

The token service uses the Cloud IAM API to request short-lived credentials that provide client access to the Dialogflow API. For more information, check Creating short-lived service account credentials.

To create a service account with the required permissions:

  1. From the left menu of the Firebase console, next to the stopwatch project home, select the Settings gear and then Project settings.

  2. Select Service accounts and then Manage service account permissions.

  3. Click Create Service Account.

  4. Configure the following settings:

    1. In Service account name, enter dialogflow-client.
    2. In Role, select Project > Dialogflow API Client.
  5. Click Create.

Adding token creator permissions

The Cloud Function for Firebase runs under the App Engine default service account, which requires permissions to create short-lived credentials. To add the required permissions, add the Service Account Token Creator role to the App Engine default service account:

  1. Open the IAM page in the Cloud Console.

    Open the IAM page

  2. Find the entry that displays App Engine default service account in the Name column and click the Edit member button at the end of the row.

  3. Click Add Another Role.

  4. From the Select a role menu, select Service Accounts > Service Account Token Creator.

  5. Click Save.

Enabling billing and APIs for the Google Cloud project

This tutorial requires the Dialogflow API. To enable the API:

  1. In the Google Cloud Console, select the stopwatch project.

    Go to the Projects page

  2. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

  3. Enable the Dialogflow API.

    Enable the API

Building and deploying the token service

To build and deploy the token service, you must install the Firebase CLI, create the project structure for the Cloud Function for Firebase, and replace the default function with token service function.

  1. Install the Firebase CLI.

    npm install -g firebase-tools
    
  2. Sign in to Firebase.

    firebase login
    
  3. Create the project structure for the function.

    firebase init functions
    
  4. Replace the default index.js file in the project structure with the index.js file in the functions/tokenservice/functions folder of the nodejs-docs-samples repository.

  5. Replace the SERVICE-ACCOUNT-NAME and YOUR_PROJECT_ID placeholders in the generateAccessToken() function with dialogflow-client and your Firebase project ID, respectively. The placeholders are part of the service account name, which is in the form:

    SERVICE-ACCOUNT-NAME@YOUR_PROJECT_ID.iam.gserviceaccount.com
    
  6. Deploy the token service function.

    firebase deploy --only functions
    

Deploying the stopwatch agent

To deploy the stopwatch agent, create a new agent in the Dialogflow console:

  1. If you don't have any agents in your Dialogflow console, click Create new agent in the left menu. Otherwise, click the current agent name in the left menu and choose Create new agent.

  2. Enter stopwatch as the name of your agent.

  3. Choose a default language.

  4. Choose a default time zone.

  5. Choose stopwatch from the Google Project menu.

  6. Click Create.

The client app project includes a StopwatchAgent.zip file that you can use to import the agent:

  1. Click the gear icon settings next to the agent name in the left menu.
  2. Click the Export and Import tab.
  3. Click Import From Zip.
  4. Select the StopwatchAgent.zip file.
  5. Type IMPORT in the text field to confirm the operation.
  6. Click Import.

Exploring the code

Client apps can get short-lived credentials by calling the getOAuthToken() function of the token service. The getOAuthToken() method performs the following tasks:

  1. Checks that the current request is authenticated. The function throws an error if the request isn't authenticated.
  2. Tries to retrieve a token from the database and performs one of the following actions:
    1. If there's a token in the database and the token doesn't expire within five minutes, then the function returns the token to the client.
    2. If the database doesn't have a token or the token expires within five minutes, then the function retrieves a token from the Cloud IAM API and passes it to the client. The function saves the token in the database to serve future client requests.

The following code example shows the getOAuthToken() function:

exports.getOAuthToken = functions.https.onCall(async (data, context) => {
  // Checking that the user is authenticated.
  if (!context.auth) {
    // Throwing an HttpsError so that the client gets the error details.
    throw new functions.https.HttpsError(
      'failed-precondition',
      'The function must be called ' + 'while authenticated.'
    );
  }
  // Retrieve the token from the database
  const docRef = db.collection('ShortLivedAuthTokens').doc('OauthToken');

  const doc = await docRef.get();

  try {
    if (doc.exists && isValid(doc.data().expireTime)) {
      //push notification
      pushNotification(
        data['deviceID'],
        doc.data().accessToken,
        doc.data().expireTime
      );
      return doc.data();
    } else {
      const result = await retrieveCredentials(context);
      console.log('Print result from retrieveCredentials functions');
      console.log(result);
      pushNotification(
        data['deviceID'],
        result['accessToken'],
        result['expireTime']
      );
      return result;
    }
  } catch (err) {
    console.log('Error retrieving token', err);
    pushNotification(data['deviceID'], 'Error retrieving token', 'Error');
    // return 'Error retrieving token';
    return 'Error retrieving token';
  }
});

To retrieve credentials, the token service makes a request to the /computeMetadata/v1/instance/service-accounts/default/token endpoint of the Cloud IAM API, as shown in the following code:

const retrieveCredentials = (context) => {
  return new Promise((resolve) => {
    // To create a new access token, we first have to retrieve the credentials
    // of the service account that will make the generateTokenRequest().
    // To do that, we will use the App Engine Default Service Account.
    const options = {
      host: 'metadata.google.internal',
      path: '/computeMetadata/v1/instance/service-accounts/default/token',
      method: 'GET',
      headers: {'Metadata-Flavor': 'Google'},
    };

    const get_req = http.get(options, (res) => {
      let body = '';

      res.on('data', (chunk) => {
        body += chunk;
      });

      res.on('end', async () => {
        const response = JSON.parse(body);
        const result = await generateAccessToken(
          context,
          response.access_token,
          response.token_type
        );
        return resolve(result);
      });
    });
    get_req.on('error', (e) => {
      //console.log('Error retrieving credentials', e.message);
      return `Error retrieving token${e.message}`;
    });
    get_req.end();
  });
};
exports.retrieveCredentials = retrieveCredentials;

For more details about the token service, check the index.js file that defines the Cloud Function for Firebase.

Cleaning up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, delete the Google Cloud and Firebase projects. Although you created the project in the Firebase console, you can also delete it in the Cloud Console.

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project that you want to delete and then click Delete .
  3. In the dialog, type the project ID and then click Shut down to delete the project.