Bulk importing and exporting existing Google Cloud resources

This page describes the config-connector bulk-export command and how to use it to export Google Cloud resources into Config Connector YAML files which you can subsequently import into Config Connector.

config-connector bulk-export uses Cloud Asset Inventory's Export functionality to discover existing Google Cloud resources. You can provide a Cloud Asset Inventory export or config-connector can perform the export on your behalf.

Cloud Asset Inventory exports JSON structures. Each structure has the name of the resource, its Asset Inventory type, and its ancestor resources: projects, folders, and organization. To discover the types supported by asset inventory, see Supported asset types.

Before you begin

Install the config-connector tool
If you want to use the config-connector tool to export directly from Cloud Asset Inventory, enable the Cloud Asset Inventory API on your Google Cloud Identity's project with gcloud.
```
gcloud services enable cloudasset.googleapis.com
```

Bulk export example

In this example, you create a PubSubTopic with the Google Cloud CLI and then import it into Config Connector.

Create a topic named sample-topic with the Google Cloud CLI:
```
gcloud pubsub topics create sample-topic
```
You receive confirmation that the topic has been created.
```
Created topic [projects/PROJECT_ID/topics/sample-topic].
```
In the output, PROJECT_ID is replaced with your Google Cloud project.
Get the topic's Google Cloud resource name and save it into an environment variable with the following command:
```
TOPIC_RESOURCE_NAME=$(gcloud pubsub topics describe sample-topic --format "value(name)")
```
To identify objects, the config-connector tool uses Cloud Asset Inventory JSON structures. Save the topic asset's JSON structure into an environment variable:
```
TOPIC_ASSET='{"name":"//pubsub.googleapis.com/'"${TOPIC_RESOURCE_NAME}"'","asset_type":"pubsub.googleapis.com/Topic"}'
```

Pass the asset into config-connector bulk-export by running the following command:

echo ${TOPIC_ASSET} | config-connector bulk-export

The output is a Config Connector resource in YAML format.

---
apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubTopic
metadata:
  annotations:
    cnrm.cloud.google.com/project-id: PROJECT_ID
  name: sample-topic
...

In the output, PROJECT_ID is replaced with your Google Cloud project.

You can pass this resource into Config Connector with kubectl apply -f -. To pass the resource directly, run the following command:
```
echo ${TOPIC_ASSET} | config-connector bulk-export | kubectl apply -f -  --namespace CC_NAMESPACE
```
Replace CC_NAMESPACE with the namespace that Config Connector manages resources from.

Config Connector acquires the resource.
Confirm that Config Connector is managing the resource with kubectl describe:
```
kubectl describe pubsubtopic sample-topic --namespace CC_NAMESPACE
```
Replace CC_NAMESPACE with the namespace that Config Connector manages resources from.

Cleaning up

You can delete your PubSubTopic with config-connector bulk-export and kubectl delete.

echo ${TOPIC_ASSET} | config-connector bulk-export | kubectl delete -f - --namespace CC_NAMESPACE

Replace CC_NAMESPACE with the namespace that Config Connector manages resources from.

Discovering resources to import

When importing resources, you can perform a Cloud Asset Inventory export and supply the results to config-connector bulk-export or have config-connector bulk-export perform one on your behalf.

Importing from an Cloud Asset Inventory export

You can provide an asset inventory export by providing a path to a local file containing the export or piping the results of an export to config-connector on STDIN.

Importing from a local file

You can provide an asset inventory export to config-connector bulk-export by using a local file with the --input parameter.

config-connector bulk-export --input ASSET_INVENTORY_EXPORT

Replace ASSET_INVENTORY_EXPORT with the filename of your Cloud Asset Inventory export.

Importing from STDIN

To provide an asset inventory export on STDIN, pipe the results of an export to config-connector bulk-export. For example, if your export is in a local file named export.json, pipe the contents of the file to config-connector bulk-export without supplying any of the export parameters.

cat export.json | config-connector bulk-export

Filtering an Asset Inventory export on STDIN

To filter an asset inventory export, you can use the jq tool and pipe to input the results to config-connector bulk-export. For example, if you want to only import PubSubTopic assets from the file EXPORT_FILE, run the following command:

cat EXPORT_FILE | jq '. | select( .asset_type == "pubsub.googleapis.com/Topic" )' | config-connector bulk-export

Exporting an inventory with config-connector

The config-connector bulk-export tool can export resources from a Google Cloud Resource hierarchy.

Exporting your project

To export all resources from your project, use the --project parameter.

config-connector bulk-export --project PROJECT_ID

Replace PROJECT_ID with your Google Cloud project.

Exporting your folder

To export all resources from a folder, use the --folder parameter.

config-connector bulk-export --folder FOLDER_NUMBER

Replace FOLDER_NUMBER with your Google Cloud folder number.

Exporting your organization

To export all resources from your organization, use the --organization parameter.

config-connector bulk-export --organization ORGANIZATION_ID

Replace ORGANIZATION_ID with your Google Cloud organization ID.

Cloud Storage location

Asset inventory export's output location is a Cloud Storage URI. When config-connector bulk-export performs an export, it uses a Cloud Storage bucket. By default, config-connector bulk-export creates a temporary bucket. You can also specify the bucket name.

Temporary Cloud Storage bucket

If you do not provide the --storage-key parameter, config-connector bulk-export creates a temporary Cloud Storage bucket on your behalf. The bucket is created in the default location for storage buckets; the US multi-region. The bucket is deleted when the export is complete.

Specifying a temporary bucket

To specify a bucket, use a Cloud Storage URI with the storage-key parameter. If the URI is only the name of the bucket, then a name is generated for the export storage object. If the URI is a full path to a storage object, then the full path is used.

config-connector bulk-export --storage-key gs://BUCKET_NAME

Output

The output from the config-connector bulk-export command is Config Connector resources in YAML format. The YAML file is written to STDOUT by default. You can direct the output of resources to files with the output option.

Output to a single file

When you set the --output parameter, config-connector bulk-export writes its results to a single file if one of the following is true:

The file specified by output exists and is a regular file.
The file specified by output does not exist and the parent directory represented by output does exist.

Output to a directory

config-connector writes its results to multiple files when the --output parameter is a directory that ends in a /. config-connector bulk-export creates one file per resource, and file names match their resource names.

config-connector bulk-export --project PROJECT_ID --on-error continue --output OUTPUT_DIRECTORY/

Replace PROJECT_ID with your Google Cloud project.

For example, to output assets from the project my-project to the sample directory, run the following command:

config-connector bulk-export --project my-project --on-error continue --output sample/

Supported resources

You can print the list of resources for the config-connector tool along with resource name formats by running the following command:

config-connector print-resources

Command-line options

The config-connector bulk-export command has the following options:

config-connector bulk-export
    --input FILENAME \
    --output FILENAME \
    --storage-key gs://BUCKET_NAME \
    --project PROJECT_ID \
    --folder FOLDER_NUMBER \
    --organization ORGANIZATION_ID \
    --oauth2-token TOKEN \
    --on-error [halt | continue | ignore] \
    --iam-format [policy | policymember | none] \
    --filter-deleted-iam-members [true | false] \
    --verbose

--input: Cloud Asset Inventory input file.
--output: An optional output file path that disables standard output. When a file, the result contains all the command output; when a directory, the directory will contain a new file for each resource in the output.
--storage-key: Target temporary Cloud Storage bucket for export.
--project: Google Cloud project ID to export
--folder: Google Cloud folder ID to export
--organization: Google Cloud organization ID to export.
--oauth2-token: An OAUTH2 token as the Google Cloud identity. By default, config-connector uses the Google Cloud CLI default credentials.
--on-error: Control the behavior when a recoverable error occurs. Options are 'continue', 'halt', or 'ignore'.
- halt: stop execution on any error (default)
- continue: continue processing resources, print the error to STDERR
- ignore: continue processing resources and don't print the error
--iam-format: Specifies the kind of IAM resources output with your export. Options are policy (default), policymember, or none.
--filter-deleted-iam-members: Specifies whether to filter out deleted IAM principals. Options are true or false. The default value is false.
--verbose: Enables verbose logging.

What's next

Read about how Config Connector acquires existing Google Cloud resources
Learn about Cloud Asset Inventory and Exporting assets to Cloud Storage.
Learn about the resources that Config Connector supports.