Send feedback
Class PslSparkUtils (1.0.0)
Stay organized with collections
Save and categorize content based on your preferences.
Version latest keyboard_arrow_down
public class PslSparkUtils
Inheritance
java.lang.Object >
PslSparkUtils
Static Methods
convertAttributesToSparkMap(ListMultimap<String,ByteString> attributeMap)
public static ArrayBasedMapData convertAttributesToSparkMap(ListMultimap<String,ByteString> attributeMap)
Parameter Name Description attributeMap
com.google.common.collect.ListMultimap <String ,ByteString >
Returns Type Description org.apache.spark.sql.catalyst.util.ArrayBasedMapData
getSparkEndOffset(SparkSourceOffset headOffset, SparkSourceOffset startOffset, long maxMessagesPerBatch, long topicPartitionCount)
public static SparkSourceOffset getSparkEndOffset(SparkSourceOffset headOffset, SparkSourceOffset startOffset, long maxMessagesPerBatch, long topicPartitionCount)
toInternalRow(SequencedMessage msg, SubscriptionPath subscription, Partition partition)
public static InternalRow toInternalRow(SequencedMessage msg, SubscriptionPath subscription, Partition partition)
Parameters Name Description msg
com.google.cloud.pubsublite.SequencedMessage
subscription
com.google.cloud.pubsublite.SubscriptionPath
partition
com.google.cloud.pubsublite.Partition
Returns Type Description org.apache.spark.sql.catalyst.InternalRow
toPslPartitionOffset(SparkPartitionOffset sparkPartitionOffset)
public static PslPartitionOffset toPslPartitionOffset(SparkPartitionOffset sparkPartitionOffset)
Parameter Name Description sparkPartitionOffset
com.google.cloud.pubsublite.spark.SparkPartitionOffset
toPslSourceOffset(SparkSourceOffset sparkSourceOffset)
public static PslSourceOffset toPslSourceOffset(SparkSourceOffset sparkSourceOffset)
toPubSubMessage(StructType inputSchema, InternalRow row)
public static Message toPubSubMessage(StructType inputSchema, InternalRow row)
Parameters Name Description inputSchema
org.apache.spark.sql.types.StructType
row
org.apache.spark.sql.catalyst.InternalRow
Returns Type Description com.google.cloud.pubsublite.Message
toSparkSourceOffset(PslSourceOffset pslSourceOffset)
public static SparkSourceOffset toSparkSourceOffset(PslSourceOffset pslSourceOffset)
public static void verifyWriteInputSchema(StructType inputSchema)
Make sure data fields for publish have expected Spark DataType if they exist.
Parameter Name Description inputSchema
org.apache.spark.sql.types.StructType
input table schema to write to Pub/Sub Lite.
Constructors
PslSparkUtils()
Send feedback
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License , and code samples are licensed under the Apache 2.0 License . For details, see the Google Developers Site Policies . Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-03-22 UTC.
[{
"type": "thumb-down",
"id": "hardToUnderstand",
"label":"Hard to understand"
},{
"type": "thumb-down",
"id": "incorrectInformationOrSampleCode",
"label":"Incorrect information or sample code"
},{
"type": "thumb-down",
"id": "missingTheInformationSamplesINeed",
"label":"Missing the information/samples I need"
},{
"type": "thumb-down",
"id": "otherDown",
"label":"Other"
}]
[{
"type": "thumb-up",
"id": "easyToUnderstand",
"label":"Easy to understand"
},{
"type": "thumb-up",
"id": "solvedMyProblem",
"label":"Solved my problem"
},{
"type": "thumb-up",
"id": "otherUp",
"label":"Other"
}]
Need to tell us more?