Send feedback
Class PslSparkUtils (0.3.4)
Stay organized with collections
Save and categorize content based on your preferences.
Version 0.3.4 keyboard_arrow_down
public class PslSparkUtils
Inheritance
java.lang.Object >
PslSparkUtils
Constructors
PslSparkUtils()
Methods
convertAttributesToSparkMap(ListMultimap<String,ByteString> attributeMap)
public static ArrayBasedMapData convertAttributesToSparkMap(ListMultimap<String,ByteString> attributeMap)
Parameter
Name Description attributeMap com.google.common.collect.ListMultimap <String ,ByteString >
Returns
Type Description org.apache.spark.sql.catalyst.util.ArrayBasedMapData
getSparkEndOffset(SparkSourceOffset headOffset, SparkSourceOffset startOffset, long maxMessagesPerBatch, long topicPartitionCount)
public static SparkSourceOffset getSparkEndOffset(SparkSourceOffset headOffset, SparkSourceOffset startOffset, long maxMessagesPerBatch, long topicPartitionCount)
Parameters
Returns
getSparkStartOffset(CursorClient cursorClient, SubscriptionPath subscriptionPath, long topicPartitionCount)
public static SparkSourceOffset getSparkStartOffset(CursorClient cursorClient, SubscriptionPath subscriptionPath, long topicPartitionCount)
Parameters
Name Description cursorClient com.google.cloud.pubsublite.internal.CursorClient
subscriptionPath com.google.cloud.pubsublite.SubscriptionPath
topicPartitionCount long
Returns
toInternalRow(SequencedMessage msg, SubscriptionPath subscription, Partition partition)
public static InternalRow toInternalRow(SequencedMessage msg, SubscriptionPath subscription, Partition partition)
Parameters
Name Description msg com.google.cloud.pubsublite.SequencedMessage
subscription com.google.cloud.pubsublite.SubscriptionPath
partition com.google.cloud.pubsublite.Partition
Returns
Type Description org.apache.spark.sql.catalyst.InternalRow
toPslPartitionOffset(SparkPartitionOffset sparkPartitionOffset)
public static PslPartitionOffset toPslPartitionOffset(SparkPartitionOffset sparkPartitionOffset)
Parameter
Name Description sparkPartitionOffset com.google.cloud.pubsublite.spark.SparkPartitionOffset
Returns
toPslSourceOffset(SparkSourceOffset sparkSourceOffset)
public static PslSourceOffset toPslSourceOffset(SparkSourceOffset sparkSourceOffset)
Parameter
Returns
toPubSubMessage(StructType inputSchema, InternalRow row)
public static Message toPubSubMessage(StructType inputSchema, InternalRow row)
Parameters
Name Description inputSchema org.apache.spark.sql.types.StructType
row org.apache.spark.sql.catalyst.InternalRow
Returns
Type Description com.google.cloud.pubsublite.Message
toSparkSourceOffset(PslSourceOffset pslSourceOffset)
public static SparkSourceOffset toSparkSourceOffset(PslSourceOffset pslSourceOffset)
Parameter
Returns
public static void verifyWriteInputSchema(StructType inputSchema)
Make sure data fields for publish have expected Spark DataType if they exist.
Parameter
Name Description inputSchema org.apache.spark.sql.types.StructType
input table schema to write to Pub/Sub Lite.
Send feedback
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License , and code samples are licensed under the Apache 2.0 License . For details, see the Google Developers Site Policies . Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-03-22 UTC.
[{
"type": "thumb-down",
"id": "hardToUnderstand",
"label":"Hard to understand"
},{
"type": "thumb-down",
"id": "incorrectInformationOrSampleCode",
"label":"Incorrect information or sample code"
},{
"type": "thumb-down",
"id": "missingTheInformationSamplesINeed",
"label":"Missing the information/samples I need"
},{
"type": "thumb-down",
"id": "otherDown",
"label":"Other"
}]
[{
"type": "thumb-up",
"id": "easyToUnderstand",
"label":"Easy to understand"
},{
"type": "thumb-up",
"id": "solvedMyProblem",
"label":"Solved my problem"
},{
"type": "thumb-up",
"id": "otherUp",
"label":"Other"
}]
Need to tell us more?