Google Cloud to Neo4j template

The Google Cloud to Neo4j template lets you import a dataset into a Neo4j database through a Dataflow job, sourcing data from CSV files hosted in Cloud Storage buckets. It also lets you manipulate and transform the data at various steps of the import. You can use the template for both first-time imports and incremental imports.

Pipeline requirements

  • A running Neo4j instance
  • A Cloud Storage bucket
  • A dataset to import, in the form of CSV files
  • A job specification file to use

Create a job specification file

The job specification file consists of a JSON object with the following sections:

  • config: global flags affecting how the import is performed.
  • sources: data source definitions (relational).
  • targets: data target definitions (graph: nodes/relationships).
  • actions: pre/post-load actions.

For more information, see Create a job specification file in the Neo4j documentation.

Template parameters

Parameter Description
jobSpecUri The path to the job specification file, which contains the configuration for source and target metadata.
neo4jConnectionUri The path to the Neo4j connection metadata JSON file.
optionsJson Optional: Options JSON. Use runtime tokens. For example: {token1:value1,token2:value2}. Defaults to empty.
readQuery Optional: Override SQL query. Defaults to empty.
inputFilePattern Optional: Override text file pattern. For example: gs://your-bucket/path/*.json. Defaults to empty.
disabledAlgorithms Optional: Comma-separated algorithms to disable. If this value is set to none then no algorithm is disabled. Use with care, because the algorithms that are disabled by default are known to have either vulnerabilities or performance issues. For example: SSLv3, RC4.
extraFilesToStage Optional: Comma-separated Cloud Storage paths or Secret Manager secrets for files to stage in the worker. These files are saved under the /extra_files directory in each worker. For example: gs://your-bucket/file.txt,projects/project-id/secrets/secret-id/versions/version-id.

Run the template

Console

  1. Go to the Dataflow Create job from template page.
    2. Go to Create job from template
  2. In the Job name field, enter a unique job name.
  3. Optional: For Regional endpoint, select a value from the drop-down menu. The default region is us-central1.

    For a list of regions where you can run a Dataflow job, see Dataflow locations.

  4. From the Dataflow template drop-down menu, select the Google Cloud to Neo4j template.
  5. In the provided parameter fields, enter your parameter values.
  6. Click Run job.

gcloud

In your shell or terminal, run the template:

gcloud dataflow flex-template run JOB_NAME \
    --template-file-gcs-location=gs://dataflow-templates-REGION_NAME/VERSION/flex/Google_Cloud_to_Neo4j \
    --project=PROJECT_ID \
    --region=REGION_NAME \
    --parameters \
       jobSpecUri=JOB_SPEC_URI,\
       neo4jConnectionUri=NEO4J_CONNECTION_URI,\

Replace the following:

  • JOB_NAME: a unique job name of your choice
  • VERSION: the version of the template that you want to use

    You can use the following values:

  • REGION_NAME: the region where you want to deploy your Dataflow job—for example, us-central1
  • JOB_SPEC_URI: the path to the job specification file
  • NEO4J_CONNECTION_URI: the path to the Neo4j connection metadata

API

To run the template using the REST API, send an HTTP POST request. For more information on the API and its authorization scopes, see projects.templates.launch.

POST https://dataflow.googleapis.com/v1b3/projects/PROJECT_ID/locations/LOCATION/flexTemplates:launch
{
   "launchParameter": {
     "jobName": "JOB_NAME",
     "parameters": {
       "jobSpecUri": "JOB_SPEC_URI",
       "neo4jConnectionUri": "NEO4J_CONNECTION_URI",
     },
     "containerSpecGcsPath": "gs://dataflow-templates-LOCATION/VERSION/flex/Google_Cloud_to_Neo4j",
     "environment": { "maxWorkers": "10" }
  }
}

Replace the following:

  • PROJECT_ID: the Google Cloud project ID where you want to run the Dataflow job
  • JOB_NAME: a unique job name of your choice
  • VERSION: the version of the template that you want to use

    You can use the following values:

  • LOCATION: the region where you want to deploy your Dataflow job—for example, us-central1
  • JOB_SPEC_URI: the path to the job specification file
  • NEO4J_CONNECTION_URI: the path to the Neo4j connection metadata

What's next