Detect crop hints

Crop Hints suggests vertices for a crop region on an image.

Image before crop
Image credit: Yasmin Dangor, Unsplash (image below cropped).

Crop hint applied (2:1 ratio):

Image after crop

Crop hint detection requests

Set up your GCP project and authentication

Detect crop hints on a local image

The Vision API can perform feature detection on a local image file by sending the contents of the image file as a base64 encoded string in the body of your request.

Command-line

To make a crop hint detection request using curl from the Linux or MacOS command line, make a POST request to the https://vision.googleapis.com/v1/images:annotate endpoint and specify CROP_HINTS as the value of features.type, as shown in the following example:

curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
https://vision.googleapis.com/v1/images:annotate -d "{
  'requests': [
    {
      "image": {
        "content": "/9j/7QBEUGhvdG9zaG9...base64-encoded-image-content...fXNWzvDEeYxxxzj/Coa6Bax//Z"
      },
      'features': [
        {
          'type': 'CROP_HINTS'
        }
      ],
      'imageContext': {
        'cropHintsParams': {
          'aspectRatios': [
             2.0
          ]
        }
      }
    }
  ]
}"

Where:

  • cropHintsParams.aspectRatios - A float that corresponds to your specified ratio(s) for your images (width:height). You can supply up to 16 crop ratios.

See the AnnotateImageRequest reference documentation for more information on configuring the request body.

If the request is successful, the server returns a 200 OK HTTP status code and the response in JSON format.

Response:

{
  "responses": [
    {
      "cropHintsAnnotation": {
        "cropHints": [
          {
            "boundingPoly": {
              "vertices": [
                {
                  "y": 520
                },
                {
                  "x": 2369,
                  "y": 520
                },
                {
                  "x": 2369,
                  "y": 1729
                },
                {
                  "y": 1729
                }
              ]
            },
            "confidence": 0.79999995,
            "importanceFraction": 0.66999996
          }
        ]
      }
    }
  ]
}

C#

Before trying this sample, follow the C# setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API C# API reference documentation .

// Load an image from a local file.
var image = Image.FromFile(filePath);
var client = ImageAnnotatorClient.Create();
CropHintsAnnotation annotation = client.DetectCropHints(image);
foreach (CropHint hint in annotation.CropHints)
{
    Console.WriteLine("Confidence: {0}", hint.Confidence);
    Console.WriteLine("ImportanceFraction: {0}", hint.ImportanceFraction);
    Console.WriteLine("Bounding Polygon:");
    foreach (Vertex vertex in hint.BoundingPoly.Vertices)
    {
        Console.WriteLine("\tX:\t{0}\tY:\t{1}", vertex.X, vertex.Y);
    }
}

Go

Before trying this sample, follow the Go setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Go API reference documentation .

// detectCropHints gets suggested croppings the Vision API for an image at the given file path.
func detectCropHints(w io.Writer, file string) error {
	ctx := context.Background()

	client, err := vision.NewImageAnnotatorClient(ctx)
	if err != nil {
		return err
	}

	f, err := os.Open(file)
	if err != nil {
		return err
	}
	defer f.Close()

	image, err := vision.NewImageFromReader(f)
	if err != nil {
		return err
	}
	res, err := client.CropHints(ctx, image, nil)
	if err != nil {
		return err
	}

	fmt.Fprintln(w, "Crop hints:")
	for _, hint := range res.CropHints {
		for _, v := range hint.BoundingPoly.Vertices {
			fmt.Fprintf(w, "(%d,%d)\n", v.X, v.Y)
		}
	}

	return nil
}

Java

Before trying this sample, follow the Java setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Java API reference documentation .

public static void detectCropHints(String filePath, PrintStream out) throws Exception,
    IOException {
  List<AnnotateImageRequest> requests = new ArrayList<>();

  ByteString imgBytes = ByteString.readFrom(new FileInputStream(filePath));

  Image img = Image.newBuilder().setContent(imgBytes).build();
  Feature feat = Feature.newBuilder().setType(Type.CROP_HINTS).build();
  AnnotateImageRequest request =
      AnnotateImageRequest.newBuilder().addFeatures(feat).setImage(img).build();
  requests.add(request);

  try (ImageAnnotatorClient client = ImageAnnotatorClient.create()) {
    BatchAnnotateImagesResponse response = client.batchAnnotateImages(requests);
    List<AnnotateImageResponse> responses = response.getResponsesList();

    for (AnnotateImageResponse res : responses) {
      if (res.hasError()) {
        out.printf("Error: %s\n", res.getError().getMessage());
        return;
      }

      // For full list of available annotations, see http://g.co/cloud/vision/docs
      CropHintsAnnotation annotation = res.getCropHintsAnnotation();
      for (CropHint hint : annotation.getCropHintsList()) {
        out.println(hint.getBoundingPoly());
      }
    }
  }
}

Node.js

Before trying this sample, follow the Node.js setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Node.js API reference documentation .

// Imports the Google Cloud client library
const vision = require('@google-cloud/vision');

// Creates a client
const client = new vision.ImageAnnotatorClient();

/**
 * TODO(developer): Uncomment the following line before running the sample.
 */
// const fileName = 'Local image file, e.g. /path/to/image.png';

// Find crop hints for the local file
const [result] = await client.cropHints(fileName);
const cropHints = result.cropHintsAnnotation;
cropHints.cropHints.forEach((hintBounds, hintIdx) => {
  console.log(`Crop Hint ${hintIdx}:`);
  hintBounds.boundingPoly.vertices.forEach((bound, boundIdx) => {
    console.log(`  Bound ${boundIdx}: (${bound.x}, ${bound.y})`);
  });
});

PHP

Before trying this sample, follow the PHP setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API PHP API reference documentation .

namespace Google\Cloud\Samples\Vision;

use Google\Cloud\Vision\V1\ImageAnnotatorClient;

// $path = 'path/to/your/image.jpg'

function detect_crop_hints($path)
{
    $imageAnnotator = new ImageAnnotatorClient();

    # annotate the image
    $image = file_get_contents($path);
    $response = $imageAnnotator->cropHintsDetection($image);
    $annotations = $response->getCropHintsAnnotation();

    # print the crop hints from the annotation
    if ($annotations) {
        print("Crop hints:" . PHP_EOL);
        foreach ($annotations->getCropHints() as $hint) {
            # get bounds
            $vertices = $hint->getBoundingPoly()->getVertices();
            $bounds = [];
            foreach ($vertices as $vertex) {
                $bounds[] = sprintf('(%d,%d)', $vertex->getX(),
                    $vertex->getY());
            }
            print('Bounds: ' . join(', ',$bounds) . PHP_EOL);
        }
    } else {
        print('No crop hints' . PHP_EOL);
    }

    $imageAnnotator->close();
}

Python

Before trying this sample, follow the Python setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Python API reference documentation .

def detect_crop_hints(path):
    """Detects crop hints in an image."""
    from google.cloud import vision
    client = vision.ImageAnnotatorClient()

    with io.open(path, 'rb') as image_file:
        content = image_file.read()
    image = vision.types.Image(content=content)

    crop_hints_params = vision.types.CropHintsParams(aspect_ratios=[1.77])
    image_context = vision.types.ImageContext(
        crop_hints_params=crop_hints_params)

    response = client.crop_hints(image=image, image_context=image_context)
    hints = response.crop_hints_annotation.crop_hints

    for n, hint in enumerate(hints):
        print('\nCrop Hint: {}'.format(n))

        vertices = (['({},{})'.format(vertex.x, vertex.y)
                    for vertex in hint.bounding_poly.vertices])

        print('bounds: {}'.format(','.join(vertices)))

Ruby

Before trying this sample, follow the Ruby setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Ruby API reference documentation .

# image_path = "Path to local image file, eg. './image.png'"

require "google/cloud/vision"

image_annotator = Google::Cloud::Vision::ImageAnnotator.new

response = image_annotator.crop_hints_detection image: image_path

response.responses.each do |res|
  puts "Crop hint bounds:"
  res.crop_hints_annotation.crop_hints.each do |crop_hint|
    crop_hint.bounding_poly.vertices.each do |bound|
      puts "#{bound.x}, #{bound.y}"
    end
  end
end

Detect crop hints on a remote image

For your convenience, the Vision API can perform feature detection directly on an image file located in Google Cloud Storage or on the Web without the need to send the contents of the image file in the body of your request.

Command-line

To make a crop hints detection request using curl from the Linux or MacOS command line, make a POST request to the https://vision.googleapis.com/v1/images:annotate endpoint and specify CROP_HINTS as the value of features.type, as shown in the following example:

curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
https://vision.googleapis.com/v1/images:annotate -d "{
  'requests': [
    {
      'image': {
        'source': {
          'gcsImageUri': 'gs://cloud-samples-data/vision/crop_hints/bubble.jpeg'
        }
      },
      'features': [
        {
          'type': 'CROP_HINTS'
        }
      ],
      'imageContext': {
        'cropHintsParams': {
          'aspectRatios': [
             2.0
          ]
        }
      }
    }
  ]
}"

Where:

  • cropHintsParams.aspectRatios - A float that corresponds to your specified ratio(s) for your images (width:height). You can supply up to 16 crop ratios.

See the AnnotateImageRequest reference documentation for more information on configuring the request body.

If the request is successful, the server returns a 200 OK HTTP status code and the response in JSON format.

Response:

{
  "responses": [
    {
      "cropHintsAnnotation": {
        "cropHints": [
          {
            "boundingPoly": {
              "vertices": [
                {
                  "y": 520
                },
                {
                  "x": 2369,
                  "y": 520
                },
                {
                  "x": 2369,
                  "y": 1729
                },
                {
                  "y": 1729
                }
              ]
            },
            "confidence": 0.79999995,
            "importanceFraction": 0.66999996
          }
        ]
      }
    }
  ]
}

C#

Before trying this sample, follow the C# setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API C# API reference documentation .

// Specify a Google Cloud Storage uri for the image
// or a publicly accessible HTTP or HTTPS uri.
var image = Image.FromUri(uri);
var client = ImageAnnotatorClient.Create();
CropHintsAnnotation annotation = client.DetectCropHints(image);
foreach (CropHint hint in annotation.CropHints)
{
    Console.WriteLine("Confidence: {0}", hint.Confidence);
    Console.WriteLine("ImportanceFraction: {0}", hint.ImportanceFraction);
    Console.WriteLine("Bounding Polygon:");
    foreach (Vertex vertex in hint.BoundingPoly.Vertices)
    {
        Console.WriteLine("\tX:\t{0}\tY:\t{1}", vertex.X, vertex.Y);
    }
}

Java

Before trying this sample, follow the Java setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Java API reference documentation .

public static void detectCropHintsGcs(String gcsPath, PrintStream out) throws Exception,
    IOException {
  List<AnnotateImageRequest> requests = new ArrayList<>();

  ImageSource imgSource = ImageSource.newBuilder().setGcsImageUri(gcsPath).build();
  Image img = Image.newBuilder().setSource(imgSource).build();
  Feature feat = Feature.newBuilder().setType(Type.CROP_HINTS).build();
  AnnotateImageRequest request =
      AnnotateImageRequest.newBuilder().addFeatures(feat).setImage(img).build();
  requests.add(request);

  try (ImageAnnotatorClient client = ImageAnnotatorClient.create()) {
    BatchAnnotateImagesResponse response = client.batchAnnotateImages(requests);
    List<AnnotateImageResponse> responses = response.getResponsesList();

    for (AnnotateImageResponse res : responses) {
      if (res.hasError()) {
        out.printf("Error: %s\n", res.getError().getMessage());
        return;
      }

      // For full list of available annotations, see http://g.co/cloud/vision/docs
      CropHintsAnnotation annotation = res.getCropHintsAnnotation();
      for (CropHint hint : annotation.getCropHintsList()) {
        out.println(hint.getBoundingPoly());
      }
    }
  }
}

Go

Before trying this sample, follow the Go setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Go API reference documentation .

// detectCropHints gets suggested croppings the Vision API for an image at the given file path.
func detectCropHintsURI(w io.Writer, file string) error {
	ctx := context.Background()

	client, err := vision.NewImageAnnotatorClient(ctx)
	if err != nil {
		return err
	}

	image := vision.NewImageFromURI(file)
	res, err := client.CropHints(ctx, image, nil)
	if err != nil {
		return err
	}

	fmt.Fprintln(w, "Crop hints:")
	for _, hint := range res.CropHints {
		for _, v := range hint.BoundingPoly.Vertices {
			fmt.Fprintf(w, "(%d,%d)\n", v.X, v.Y)
		}
	}

	return nil
}

Node.js

Before trying this sample, follow the Node.js setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Node.js API reference documentation .

// Imports the Google Cloud client libraries
const vision = require('@google-cloud/vision');

// Creates a client
const client = new vision.ImageAnnotatorClient();

/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// const bucketName = 'Bucket where the file resides, e.g. my-bucket';
// const fileName = 'Path to file within bucket, e.g. path/to/image.png';

// Find crop hints for the remote file
const [result] = await client.cropHints(`gs://${bucketName}/${fileName}`);
const cropHints = result.cropHintsAnnotation;
cropHints.cropHints.forEach((hintBounds, hintIdx) => {
  console.log(`Crop Hint ${hintIdx}:`);
  hintBounds.boundingPoly.vertices.forEach((bound, boundIdx) => {
    console.log(`  Bound ${boundIdx}: (${bound.x}, ${bound.y})`);
  });
});

PHP

Before trying this sample, follow the PHP setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API PHP API reference documentation .

namespace Google\Cloud\Samples\Vision;

use Google\Cloud\Vision\V1\ImageAnnotatorClient;

// $path = 'gs://path/to/your/image.jpg'

function detect_crop_hints_gcs($path)
{
    $imageAnnotator = new ImageAnnotatorClient();

    # annotate the image
    $response = $imageAnnotator->cropHintsDetection($path);
    $annotations = $response->getCropHintsAnnotation();

    # print the crop hints from the annotation
    if ($annotations) {
        print("Crop hints:" . PHP_EOL);
        foreach ($annotations->getCropHints() as $hint) {
            # get bounds
            $vertices = $hint->getBoundingPoly()->getVertices();
            $bounds = [];
            foreach ($vertices as $vertex) {
                $bounds[] = sprintf('(%d,%d)', $vertex->getX(),
                    $vertex->getY());
            }
            print('Bounds: ' . join(', ',$bounds) . PHP_EOL);
        }
    } else {
        print('No crop hints' . PHP_EOL);
    }

    $imageAnnotator->close();
}

Python

Before trying this sample, follow the Python setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Python API reference documentation .

def detect_crop_hints_uri(uri):
    """Detects crop hints in the file located in Google Cloud Storage."""
    from google.cloud import vision
    client = vision.ImageAnnotatorClient()
    image = vision.types.Image()
    image.source.image_uri = uri

    crop_hints_params = vision.types.CropHintsParams(aspect_ratios=[1.77])
    image_context = vision.types.ImageContext(
        crop_hints_params=crop_hints_params)

    response = client.crop_hints(image=image, image_context=image_context)
    hints = response.crop_hints_annotation.crop_hints

    for n, hint in enumerate(hints):
        print('\nCrop Hint: {}'.format(n))

        vertices = (['({},{})'.format(vertex.x, vertex.y)
                    for vertex in hint.bounding_poly.vertices])

        print('bounds: {}'.format(','.join(vertices)))

Ruby

Before trying this sample, follow the Ruby setup instructions in the Vision API Quickstart Using Client Libraries . For more information, see the Vision API Ruby API reference documentation .

# image_path = "Google Cloud Storage URI, eg. 'gs://my-bucket/image.png'"

require "google/cloud/vision"

image_annotator = Google::Cloud::Vision::ImageAnnotator.new

response = image_annotator.crop_hints_detection image: image_path

response.responses.each do |res|
  puts "Crop hint bounds:"
  res.crop_hints_annotation.crop_hints.each do |crop_hint|
    crop_hint.bounding_poly.vertices.each do |bound|
      puts "#{bound.x}, #{bound.y}"
    end
  end
end

PowerShell

To make a crop hint detection request using Windows PowerShell, make a POST request to the https://vision.googleapis.com/v1/images:annotate endpoint and specify CROP_HINTS as the value of features.type, as shown in the following example:

$cred = gcloud auth application-default print-access-token
$headers = @{ Authorization = "Bearer $cred" }

Invoke-WebRequest `
  -Method Post `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -Body "{
      'requests': [
        {
          'image': {
            'source': {
              'imageUri': 'gs://cloud-samples-data/vision/crop_hints/bubble.jpeg'
            }
          },
          'features': [
            {
              'type': 'CROP_HINTS'
            }
          ],
          'imageContext': {
            'cropHintsParams': {
              'aspectRatios': [
                 2.0
              ]
            }
          }
        }
      ]
    }" `
  -Uri "https://vision.googleapis.com/v1/images:annotate" | Select-Object -Expand Content

See the AnnotateImageRequest reference documentation for more information on configuring the request body.

GCLOUD COMMAND

To perform text detection, use the gcloud ml vision suggest-crop command as shown in the following example:

gcloud ml vision suggest-crop gs://cloud-samples-data/vision/crop_hints/bubble.jpeg

Try it

Try crop hint detection below. You can use the image specified already (gs://cloud-samples-data/vision/crop_hints/bubble.jpeg) or specify your own image in its place. Send the request by selecting Execute.

Image before crop
Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Vision API Documentation
Need help? Visit our support page.