EXAMPLE - Statistical Functions

This example illustrates how you can apply statistical functions to your dataset. Calculations include average (mean), max, min, standard deviation, and variance.

Source:

Students took a test and recorded the following scores. You want to perform some statistical analysis on them:

StudentScore
Anna84
Ben71
Caleb76
Danielle87
Evan85
Faith92
Gabe85
Hannah99
Ian73
Jane68

Transform:

You can use the following transforms to calculate the average (mean), minimum, and maximum scores:

derive type:single value:AVERAGE(Score) as:'avgScore'

derive type:single value:MIN(Score) as:'minScore'

derive type:single value:MAX(Score) as:'maxScore'

To apply statistical functions to your data, you can use the VAR and STDEV functions, which can be used as the basis for other statistical calculations.

derive type:single value:VAR(Score)

derive type:single value:STDEV(Score)

For each score, you can now calculate the variation of each one from the average, using the following:

derive type:single value:((Score - avg_Score) / stdev_Score) as:'stDevs'

Now, you want to apply grades based on a formula:

Gradestandard deviations from avg (stDevs)
AstDevs > 1
BstDevs > 0.5
C-1 <= stDevs <= 0.5
DstDevs < -1
FstDevs < -2

You can build the following transform using the IF function to calculate grades.

derive type:single value:IF((stDevs > 1),'A',IF((stDevs < -2),'F',IF((stDevs < -1),'D',IF((stDevs > 0.5),'B','C'))))

For more information, see IF Function.

To clean up the content, you might want to apply some formatting to the score columns. The following reformats the stdev_Score and stDevs columns to display two decimal places:

set col:stdev_Score value:NUMFORMAT(stdev_Score, '##.00')

set col:stDevs value:NUMFORMAT(stDevs, '##.00')

derive type:single value:MODE(Score) as:'modeScore'

Results:

StudentScoremodeScoreavgScoreminScoremaxScorevar_Scorestdev_ScorestDevsGrade
Anna8485826899

87.00000000000001

9.330.21C
Ben718582689987.000000000000019.33-1.18D
Caleb768582689987.000000000000019.33-0.64C
Danielle878582689987.000000000000019.330.54B
Evan858582689987.000000000000019.330.32C
Faith928582689987.000000000000019.331.07A
Gabe858582689987.000000000000019.330.32C
Hannah998582689987.000000000000019.331.82A
Ian738582689987.000000000000019.33-0.96C
Jane688582689987.000000000000019.33-1.50D
Was this page helpful? Let us know how we did:

Send feedback about...

Google Cloud Dataprep Documentation