Compteurs distribués

De nombreuses applications en temps réel contiennent des documents qui agissent comme compteurs. Par exemple, vous pouvez comptabiliser les "J'aime" pour un post ou les "Favoris" pour un élément spécifique.

Dans Firestore, vous ne pouvez mettre à jour qu'un seul document environ une fois par seconde, ce qui peut être trop faible pour certaines applications supposant un trafic élevé.

Solution : compteurs distribués

Pour permettre des mises à jour plus fréquentes du compteur, créez un compteur distribué. Chaque compteur est un document avec une sous-collection de "segments", et la valeur du compteur est la somme de la valeur des segments.

Le débit d'écriture augmente linéairement avec le nombre de segments. Par conséquent, un compteur distribué avec 10 segments peut gérer 10 fois plus d'écritures qu'un compteur traditionnel.

Web

// counters/${ID}
{
  "num_shards": NUM_SHARDS,
  "shards": [subcollection]
}

// counters/${ID}/shards/${NUM}
{
  "count": 123
}

Swift

// counters/${ID}
struct Counter {
    let numShards: Int

    init(numShards: Int) {
        self.numShards = numShards
    }
}

// counters/${ID}/shards/${NUM}
struct Shard {
    let count: Int

    init(count: Int) {
        self.count = count
    }
}

Objective-C

// counters/${ID}
@interface FIRCounter : NSObject
@property (nonatomic, readonly) NSInteger shardCount;
@end

@implementation FIRCounter
- (instancetype)initWithShardCount:(NSInteger)shardCount {
  self = [super init];
  if (self != nil) {
    _shardCount = shardCount;
  }
  return self;
}
@end

// counters/${ID}/shards/${NUM}
@interface FIRShard : NSObject
@property (nonatomic, readonly) NSInteger count;
@end

@implementation FIRShard
- (instancetype)initWithCount:(NSInteger)count {
  self = [super init];
  if (self != nil) {
    _count = count;
  }
  return self;
}
@end

Java
Android

// counters/${ID}
public class Counter {
    int numShards;

    public Counter(int numShards) {
        this.numShards = numShards;
    }
}

// counters/${ID}/shards/${NUM}
public class Shard {
    int count;

    public Shard(int count) {
        this.count = count;
    }
}

Kotlin + KTX
Android

// counters/${ID}
data class Counter(var numShards: Int)

// counters/${ID}/shards/${NUM}
data class Shard(var count: Int)

Python

import random

from google.cloud import firestore

class Shard(object):
    """
    A shard is a distributed counter. Each shard can support being incremented
    once per second. Multiple shards are needed within a Counter to allow
    more frequent incrementing.
    """

    def __init__(self):
        self._count = 0

    def to_dict(self):
        return {"count": self._count}

class Counter(object):
    """
    A counter stores a collection of shards which are
    summed to return a total count. This allows for more
    frequent incrementing than a single document.
    """

    def __init__(self, num_shards):
        self._num_shards = num_shards

Node.js

Non applicable, voir l'extrait d'incrémentation du compteur ci-dessous.

Go

import (
	"context"
	"fmt"
	"math/rand"
	"strconv"

	"cloud.google.com/go/firestore"
	"google.golang.org/api/iterator"
)

// Counter is a collection of documents (shards)
// to realize counter with high frequency.
type Counter struct {
	numShards int
}

// Shard is a single counter, which is used in a group
// of other shards within Counter.
type Shard struct {
	Count int
}

PHP

Non applicable, voir l'extrait d'initialisation du compteur ci-dessous.

C#

/// <summary>
/// Shard is a document that contains the count.
/// </summary>
[FirestoreData]
public class Shard
{
    [FirestoreProperty(name: "count")]
    public int Count { get; set; }
}

Ruby

import random

from google.cloud import firestore

class Shard(object):
    """
    A shard is a distributed counter. Each shard can support being incremented
    once per second. Multiple shards are needed within a Counter to allow
    more frequent incrementing.
    """

    def __init__(self):
        self._count = 0

    def to_dict(self):
        return {"count": self._count}

class Counter(object):
    """
    A counter stores a collection of shards which are
    summed to return a total count. This allows for more
    frequent incrementing than a single document.
    """

    def __init__(self, num_shards):
        self._num_shards = num_shards

Le code suivant initialise un compteur distribué :

Web

function createCounter(ref, num_shards) {
    var batch = db.batch();

    // Initialize the counter document
    batch.set(ref, { num_shards: num_shards });

    // Initialize each shard with count=0
    for (let i = 0; i < num_shards; i++) {
        let shardRef = ref.collection('shards').doc(i.toString());
        batch.set(shardRef, { count: 0 });
    }

    // Commit the write batch
    return batch.commit();
}

Swift

func createCounter(ref: DocumentReference, numShards: Int) {
    ref.setData(["numShards": numShards]){ (err) in
        for i in 0...numShards {
            ref.collection("shards").document(String(i)).setData(["count": 0])
        }
    }
}

Objective-C

- (void)createCounterAtReference:(FIRDocumentReference *)reference
                      shardCount:(NSInteger)shardCount {
  [reference setData:@{ @"numShards": @(shardCount) } completion:^(NSError * _Nullable error) {
    for (NSInteger i = 0; i < shardCount; i++) {
      NSString *shardName = [NSString stringWithFormat:@"%ld", (long)shardCount];
      [[[reference collectionWithPath:@"shards"] documentWithPath:shardName]
          setData:@{ @"count": @(0) }];
    }
  }];
}

Java
Android

public Task<Void> createCounter(final DocumentReference ref, final int numShards) {
    // Initialize the counter document, then initialize each shard.
    return ref.set(new Counter(numShards))
            .continueWithTask(new Continuation<Void, Task<Void>>() {
                @Override
                public Task<Void> then(@NonNull Task<Void> task) throws Exception {
                    if (!task.isSuccessful()) {
                        throw task.getException();
                    }

                    List<Task<Void>> tasks = new ArrayList<>();

                    // Initialize each shard with count=0
                    for (int i = 0; i < numShards; i++) {
                        Task<Void> makeShard = ref.collection("shards")
                                .document(String.valueOf(i))
                                .set(new Shard(0));

                        tasks.add(makeShard);
                    }

                    return Tasks.whenAll(tasks);
                }
            });
}

Kotlin + KTX
Android

fun createCounter(ref: DocumentReference, numShards: Int): Task<Void> {
    // Initialize the counter document, then initialize each shard.
    return ref.set(Counter(numShards))
            .continueWithTask { task ->
                if (!task.isSuccessful) {
                    throw task.exception!!
                }

                val tasks = arrayListOf<Task<Void>>()

                // Initialize each shard with count=0
                for (i in 0 until numShards) {
                    val makeShard = ref.collection("shards")
                            .document(i.toString())
                            .set(Shard(0))

                    tasks.add(makeShard)
                }

                Tasks.whenAll(tasks)
            }
}

Python

def init_counter(self, doc_ref):
    """
    Create a given number of shards as
    subcollection of specified document.
    """
    col_ref = doc_ref.collection("shards")

    # Initialize each shard with count=0
    for num in range(self._num_shards):
        shard = Shard()
        col_ref.document(str(num)).set(shard.to_dict())

Node.js

Non applicable, voir l'extrait d'incrémentation du compteur ci-dessous.

Go

// initCounter creates a given number of shards as
// subcollection of specified document.
func (c *Counter) initCounter(ctx context.Context, docRef *firestore.DocumentRef) error {
	colRef := docRef.Collection("shards")

	// Initialize each shard with count=0
	for num := 0; num < c.numShards; num++ {
		shard := Shard{0}

		if _, err := colRef.Doc(strconv.Itoa(num)).Set(ctx, shard); err != nil {
			return fmt.Errorf("Set: %v", err)
		}
	}
	return nil
}

PHP

$numShards = 10;
$colRef = $ref->collection('SHARDS');
for ($i = 0; $i < $numShards; $i++) {
    $doc = $colRef->document($i);
    $doc->set(['Cnt' => 0]);
}

C#

/// <summary>
/// Create a given number of shards as a
/// subcollection of specified document.
/// </summary>
/// <param name="docRef">The document reference <see cref="DocumentReference"/></param>
private static async Task CreateCounterAsync(DocumentReference docRef, int numOfShards)
{
    CollectionReference colRef = docRef.Collection("shards");
    var tasks = new List<Task>();
    // Initialize each shard with Count=0
    for (var i = 0; i < numOfShards; i++)
    {
        tasks.Add(colRef.Document(i.ToString()).SetAsync(new Shard() { Count = 0 }));
    }
    await Task.WhenAll(tasks);
}

Ruby

# project_id = "Your Google Cloud Project ID"
# num_shards = "Number of shards for distributed counter"

require "google/cloud/firestore"

firestore = Google::Cloud::Firestore.new project_id: project_id

shards_ref = firestore.col "shards"

# Initialize each shard with count=0
num_shards.times do |i|
  shards_ref.doc(i).set(count: 0)
end

puts "Distributed counter shards collection created."

Pour incrémenter le compteur, choisissez un segment aléatoire et incrémentez le compteur d'éléments :

Web

function incrementCounter(db, ref, num_shards) {
    // Select a shard of the counter at random
    const shard_id = Math.floor(Math.random() * num_shards).toString();
    const shard_ref = ref.collection('shards').doc(shard_id);

    // Update count
    return shard_ref.update("count", firebase.firestore.FieldValue.increment(1));
}

Swift

func incrementCounter(ref: DocumentReference, numShards: Int) {
    // Select a shard of the counter at random
    let shardId = Int(arc4random_uniform(UInt32(numShards)))
    let shardRef = ref.collection("shards").document(String(shardId))

    shardRef.updateData([
        "count": FieldValue.increment(Int64(1))
    ])
}

Objective-C

- (void)incrementCounterAtReference:(FIRDocumentReference *)reference
                         shardCount:(NSInteger)shardCount {
  // Select a shard of the counter at random
  NSInteger shardID = (NSInteger)arc4random_uniform((uint32_t)shardCount);
  NSString *shardName = [NSString stringWithFormat:@"%ld", (long)shardID];
  FIRDocumentReference *shardReference =
      [[reference collectionWithPath:@"shards"] documentWithPath:shardName];

  [shardReference updateData:@{
    @"count": [FIRFieldValue fieldValueForIntegerIncrement:1]
  }];
}

Java
Android

public Task<Void> incrementCounter(final DocumentReference ref, final int numShards) {
    int shardId = (int) Math.floor(Math.random() * numShards);
    DocumentReference shardRef = ref.collection("shards").document(String.valueOf(shardId));

    return shardRef.update("count", FieldValue.increment(1));
}

Kotlin + KTX
Android

fun incrementCounter(ref: DocumentReference, numShards: Int): Task<Void> {
    val shardId = Math.floor(Math.random() * numShards).toInt()
    val shardRef = ref.collection("shards").document(shardId.toString())

    return shardRef.update("count", FieldValue.increment(1))
}

Python

def increment_counter(self, doc_ref):
    """Increment a randomly picked shard."""
    doc_id = random.randint(0, self._num_shards - 1)

    shard_ref = doc_ref.collection("shards").document(str(doc_id))
    return shard_ref.update({"count": firestore.Increment(1)})

Node.js

function incrementCounter(docRef, numShards) {
  const shardId = Math.floor(Math.random() * numShards);
  const shardRef = docRef.collection('shards').doc(shardId.toString());
  return shardRef.set({count: FieldValue.increment(1)}, {merge: true});
}

Go

// incrementCounter increments a randomly picked shard.
func (c *Counter) incrementCounter(ctx context.Context, docRef *firestore.DocumentRef) (*firestore.WriteResult, error) {
	docID := strconv.Itoa(rand.Intn(c.numShards))

	shardRef := docRef.Collection("shards").Doc(docID)
	return shardRef.Update(ctx, []firestore.Update{
		{Path: "Count", Value: firestore.Increment(1)},
	})
}

PHP

$colRef = $ref->collection('SHARDS');
$numShards = 0;
$docCollection = $colRef->documents();
foreach ($docCollection as $doc) {
    $numShards++;
}
$shardIdx = random_int(0, $numShards-1);
$doc = $colRef->document($shardIdx);
$doc->update([
    ['path' => 'Cnt', 'value' => FieldValue::increment(1)]
]);

C#

/// <summary>
/// Increment a randomly picked shard by 1.
/// </summary>
/// <param name="docRef">The document reference <see cref="DocumentReference"/></param>
/// <returns>The <see cref="Task"/></returns>
private static async Task IncrementCounterAsync(DocumentReference docRef, int numOfShards)
{
    int documentId;
    lock (s_randLock)
    {
        documentId = s_rand.Next(numOfShards);
    }
    var shardRef = docRef.Collection("shards").Document(documentId.ToString());
    await shardRef.UpdateAsync("count", FieldValue.Increment(1));
}

Ruby

# project_id = "Your Google Cloud Project ID"
# num_shards = "Number of shards for distributed counter"

require "google/cloud/firestore"

firestore = Google::Cloud::Firestore.new project_id: project_id

# Select a shard of the counter at random
shard_id = rand 0..num_shards
shard_ref = firestore.doc "shards/#{shard_id}"

# increment counter
shard_ref.update count: firestore.field_increment(1)

puts "Counter incremented."

Pour obtenir le nombre total, effectuez une requête sur tous les segments et additionnez leurs champs count :

Web

function getCount(ref) {
    // Sum the count of each shard in the subcollection
    return ref.collection('shards').get().then(snapshot => {
        let total_count = 0;
        snapshot.forEach(doc => {
            total_count += doc.data().count;
        });

        return total_count;
    });
}

Swift

func getCount(ref: DocumentReference) {
    ref.collection("shards").getDocuments() { (querySnapshot, err) in
        var totalCount = 0
        if err != nil {
            // Error getting shards
            // ...
        } else {
            for document in querySnapshot!.documents {
                let count = document.data()["count"] as! Int
                totalCount += count
            }
        }

        print("Total count is \(totalCount)")
    }
}

Objective-C

- (void)getCountWithReference:(FIRDocumentReference *)reference {
  [[reference collectionWithPath:@"shards"]
      getDocumentsWithCompletion:^(FIRQuerySnapshot *snapshot,
                                   NSError *error) {
        NSInteger totalCount = 0;
        if (error != nil) {
          // Error getting shards
          // ...
        } else {
          for (FIRDocumentSnapshot *document in snapshot.documents) {
            NSInteger count = [document[@"count"] integerValue];
            totalCount += count;
          }

          NSLog(@"Total count is %ld", (long)totalCount);
        }
  }];
}

Java
Android

public Task<Integer> getCount(final DocumentReference ref) {
    // Sum the count of each shard in the subcollection
    return ref.collection("shards").get()
            .continueWith(new Continuation<QuerySnapshot, Integer>() {
                @Override
                public Integer then(@NonNull Task<QuerySnapshot> task) throws Exception {
                    int count = 0;
                    for (DocumentSnapshot snap : task.getResult()) {
                        Shard shard = snap.toObject(Shard.class);
                        count += shard.count;
                    }
                    return count;
                }
            });
}

Kotlin + KTX
Android

fun getCount(ref: DocumentReference): Task<Int> {
    // Sum the count of each shard in the subcollection
    return ref.collection("shards").get()
            .continueWith { task ->
                var count = 0
                for (snap in task.result!!) {
                    val shard = snap.toObject<Shard>()
                    count += shard.count
                }
                count
            }
}

Python

def get_count(self, doc_ref):
    """Return a total count across all shards."""
    total = 0
    shards = doc_ref.collection("shards").list_documents()
    for shard in shards:
        total += shard.get().to_dict().get("count", 0)
    return total

Node.js

async function getCount(docRef) {
  const querySnapshot = await docRef.collection('shards').get();
  const documents = querySnapshot.docs;

  let count = 0;
  for (const doc of documents) {
    count += doc.get('count');
  }
  return count;
}

Go

// getCount returns a total count across all shards.
func (c *Counter) getCount(ctx context.Context, docRef *firestore.DocumentRef) (int64, error) {
	var total int64
	shards := docRef.Collection("shards").Documents(ctx)
	for {
		doc, err := shards.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			return 0, fmt.Errorf("Next: %v", err)
		}

		vTotal := doc.Data()["Count"]
		shardCount, ok := vTotal.(int64)
		if !ok {
			return 0, fmt.Errorf("firestore: invalid dataType %T, want int64", vTotal)
		}
		total += shardCount
	}
	return total, nil
}

PHP

$result = 0;
$docCollection = $ref->collection('SHARDS')->documents();
foreach ($docCollection as $doc) {
    $result += $doc->data()['Cnt'];
}

C#

/// <summary>
/// Get total count across all shards.
/// </summary>
/// <param name="docRef">The document reference <see cref="DocumentReference"/></param>
/// <returns>The <see cref="int"/></returns>
private static async Task<int> GetCountAsync(DocumentReference docRef)
{
    var snapshotList = await docRef.Collection("shards").GetSnapshotAsync();
    return snapshotList.Sum(shard => shard.GetValue<int>("count"));
}

Ruby

# project_id = "Your Google Cloud Project ID"

require "google/cloud/firestore"

firestore = Google::Cloud::Firestore.new project_id: project_id

shards_ref = firestore.col_group "shards"

count = 0
shards_ref.get do |doc_ref|
  count += doc_ref[:count]
end

puts "Count value is #{count}."

Limites

La solution présentée ci-dessus est un moyen évolutif de créer des compteurs partagés dans Firestore, mais vous devez tenir compte des limites suivantes :

  • Nombre de segments : le nombre de segments influe directement sur les performances du compteur distribué. Si le nombre de segments est trop faible, certaines transactions devront être réessayées avant de réussir, ce qui ralentira les écritures. Avec un trop grand nombre de segments, les lectures deviennent plus lentes et plus onéreuses. Pour limiter les frais de lecture, vous pouvez conserver le total du compteur dans un document cumulé distinct mis à jour à une cadence plus lente (une fois par seconde, par exemple) et demander aux clients de consulter ce document pour obtenir le total. Le compromis est que les clients devront attendre la mise à jour du document cumulé au lieu de calculer le total en lisant toutes les partitions immédiatement après la mise à jour.
  • Coût : le coût de lecture d'une valeur de compteur augmente linéairement avec le nombre de segments, car l'intégralité de la sous-collection de segments doit être chargée.