RANGE Function

Computes an array of integers, from a beginning integer to an end (stop) integer, stepping by a third parameter.

NOTE: If the function generates more than 100,000 values for a cell, the output is a null value.

Basic Usage

Numeric literal example:

derive type:single value: RANGE(0,3,1 ) as: 'threeSteps'

Output: Generates a column containing the following array:

[0,1,2]

Column reference example:

derive type:single value: RANGE(0,MaxValue,stepValue) as: 'mySteps'

Output: Generates the mySteps column containing arrays of values from zero to the value in the MaxValue column stepping by the stepValue column value.

Syntax

derive type:single value: RANGE(column_integer_start, column _integer_end, column_integer_step)

ArgumentRequired?Data TypeDescription
column_integer_startYstring or integerName of column or Integer literal that represents the start of the range
column_integer_endYstring or integerName of column or Integer literal that represents the end of the range
column_integer_stepYstring or integerName of column or Integer literal that represents the steps in integers between values in the range

For more information on syntax standards, see Language Documentation Syntax Notes.

column_integer_start

Name of the column or value of the starting integer used to compute the range.

NOTE: This value is always included in the range, unless it is equal to the value for col-integer-stop, which results in a blank array.

  • Missing input values generate missing results.
  • Multiple columns and wildcards are not supported.

Usage Notes:

Required?Data TypeExample Value
YesInteger0

column_integer_end

Name of the column or value of the end integer used to compute the range.

NOTE: This value is not included in the output.

  • Missing input values generate missing results.
  • Multiple columns and wildcards are not supported.

Usage Notes:

Required?Data TypeExample Value
YesInteger20

column_integer_step

Name of the column or value of the integer used to compute the integer interval (step) between each value in the range.

NOTE: This value must be a positive integer. If col-integer-start is greater than col-integer-stop, steps are negative values of this parameter.

  • Missing input values generate missing results.
  • Multiple columns and wildcards are not supported.

Usage Notes:

Required?Data TypeExample Value
YesInteger2

Examples

Example - Breaking out log messages

Source:

Your dataset contains log data that is gathered each minute, yet each entry can contain multiple error messages in an array. The key fields might look like the following:

TimestampErrors
02/16/16 15:31["Unable to connect","File not found","Proxy down","conn. timeout"]
02/16/16 15:30[]
02/16/16 15:29["Access forbidden","Invalid password"]

Transform:

You can use the following steps to break out the array values into separate rows. The following transform generates a column containing the number of elements in each row's Errors array.

derive type:single value:ARRAYLEN(Errors) as:'arraylength_Errors'

This transform deletes rows that contain no errors:

delete row:(arraylength_Errors == 0)

For the remaining rows, you can generate a column containing an array of numbers to match the count of error messages:

derive type:single value:RANGE(0,arraylength_Errors,1) as:'range_Errors'

You can then use the ARRAYZIP function to zip together the two arrays into a single one:

derive type:single value:ARRAYZIP([range_Errors,Errors]) as:'zipped_Errors'

The unnest transform uses the values in an array column as key values to break out rows in your dataset:

unnest col:zipped_Errors

You might rename the above as individual_Errors. To clean up your dataset, you can now drop the following columns:

  • arraylength_Errors
  • range_Errors
  • zipped_Errors

Results:

TimestampErrorsindividual_Errors
02/16/16 15:31["Unable to connect","File not found","Proxy down","conn. timeout"][0, "Unable to connect"]
02/16/16 15:31["Unable to connect","File not found","Proxy down","conn. timeout"][1, "File not found"]
02/16/16 15:31["Unable to connect","File not found","Proxy down","conn. timeout"][2, "Proxy down"]
02/16/16 15:31["Unable to connect","File not found","Proxy down","conn. timeout"][3, "conn. timeout"]
02/16/16 15:29["Access forbidden","Invalid password"][0, "Access forbidden"]
02/16/16 15:29["Access forbidden","Invalid password"][1, "Invalid password"]

Example - unnest test scores

The following example includes a range example to define a new index array.

Source:

You have the following data on student test scores. Scores on individual scores are stored in the Scores array, and you need to be able to track each test on a uniquely identifiable row. This example has two goals:

  1. One row for each student test
  2. Unique identifier for each student-score combination
LastNameFirstNameScores
AdamsAllen[81,87,83,79]
BurnsBonnie[98,94,92,85]
CannonCharles[88,81,85,78]

Transform:

When the data is imported from CSV format, you must add a header transform and remove the quotes from the Scores column:

header

replace col:Scores with:'' on:`"` global:true

Validate test date: To begin, you might want to check to see if you have the proper number of test scores for each student. You can use the following transform to calculate the difference between the expected number of elements in the Scores array (4) and the actual number:

derive type:single value: (4 - ARRAYLEN(Scores)) as: 'numMissingTests'

When the transform is previewed, you can see in the sample dataset that all tests are included. You might or might not want to include this column in the final dataset, as you might identify missing tests when the recipe is run at scale.

Unique row identifier: The Scores array must be broken out into individual rows for each test. However, there is no unique identifier for the row to track individual tests. In theory, you could use the combination of LastName-FirstName-Scores values to do so, but if a student recorded the same score twice, your dataset has duplicate rows. In the following transform, you create a parallel array called Tests, which contains an index array for the number of values in the Scores column. Index values start at 0:

derive type:single value:RANGE(0,ARRAYLEN(Scores)) as:'Tests'

Also, we will want to create an identifier for the source row using the SOURCEROWNUMBER function:

derive type:single value:SOURCEROWNUMBER() as:'orderIndex'

One row for each student test: Your data should look like the following:

LastNameFirstNameScoresTestsorderIndex
AdamsAllen[81,87,83,79][0,1,2,3]2
BurnsBonnie[98,94,92,85][0,1,2,3]3
CannonCharles[88,81,85,78][0,1,2,3]4

Now, you want to bring together the Tests and Scores arrays into a single nested array using the ARRAYZIP function:

derive type:single value:ARRAYZIP([Tests,Scores])

Your dataset has been changed:

LastNameFirstNameScoresTestsorderIndexcolumn1
AdamsAllen[81,87,83,79][0,1,2,3]2[[0,81],[1,87],[2,83],[3,79]]
AdamsBonnie[98,94,92,85][0,1,2,3]3[[0,98],[1,94],[2,92],[3,85]]
CannonCharles[88,81,85,78][0,1,2,3]4[[0,88],[1,81],[2,85],[3,78]]

With the flatten transform, you can unpack the nested array:

flatten col: column1

Each test-score combination is now broken out into a separate row. The nested Test-Score combinations must be broken out into separate columns using unnest:

unnest col:column1 keys:'[0]','[1]'

After you drop column1, which is no longer needed you should rename the two generated columns:

rename mapping:[column_0,'TestNum']

rename mapping:[column_1,'TestScore']

Unique row identifier: You can do one more step to create unique test identifiers, which identify the specific test for each student. The following uses the original row identifier OrderIndex as an identifier for the student and the TestNumber value to create the TestId column value:

derive type:single value: (orderIndex * 10) + TestNum as: 'TestId'

The above are integer values. To make your identifiers look prettier, you might add the following:

merge col:'TestId00','TestId'


Extending:
You might want to generate some summary statistical information on this dataset. For example, you might be interested in calculating each student's average test score. This step requires figuring out how to properly group the test values. In this case, you cannot group by the LastName value, and when executed at scale, there might be collisions between first names when this recipe is run at scale. So, you might need to create a kind of primary key using the following:

merge col:'LastName','FirstName' with:'-' as:'studentId'

You can now use this as a grouping parameter for your calculation:

derive type:single value:AVERAGE(TestScore) group:studentId as:'avg_TestScore'

Results:

After you drop unnecessary columns and move your columns around, the dataset should look like the following:

TestIdLastNameFirstNameTestNumTestScorestudentIdavg_TestScore
TestId0021AdamsAllen081Adams-Allen82.5
TestId0022AdamsAllen187Adams-Allen82.5
TestId0023AdamsAllen283Adams-Allen82.5
TestId0024AdamsAllen379Adams-Allen82.5
TestId0031AdamsBonnie098Adams-Bonnie92.25
TestId0032AdamsBonnie194Adams-Bonnie92.25
TestId0033AdamsBonnie292Adams-Bonnie92.25
TestId0034AdamsBonnie385Adams-Bonnie92.25
TestId0041CannonChris088Cannon-Chris83
TestId0042CannonChris181Cannon-Chris83
TestId0043CannonChris285Cannon-Chris83
TestId0044CannonChris378Cannon-Chris83

Was this page helpful? Let us know how we did:

Send feedback about...

Google Cloud Dataprep Documentation
Need help? Visit our support page.