Cloud Firestore collection count

Question

Is it possible to count how many items a collection has using the new Firebase database  Cloud Firestore   If so  how do I do that

User · Answer

Be careful counting number of documents for large collections. It is a little bit complex with firestore database if you want to have a precalculated counter for every collection.

Code like this doesn't work in this case:

export const customerCounterListener = 
    functions.firestore.document('customers/{customerId}')
    .onWrite((change, context) => {

    // on create
    if (!change.before.exists && change.after.exists) {
        return firestore
                 .collection('metadatas')
                 .doc('customers')
                 .get()
                 .then(docSnap =>
                     docSnap.ref.set({
                         count: docSnap.data().count + 1
                     }))
    // on delete
    } else if (change.before.exists && !change.after.exists) {
        return firestore
                 .collection('metadatas')
                 .doc('customers')
                 .get()
                 .then(docSnap =>
                     docSnap.ref.set({
                         count: docSnap.data().count - 1
                     }))
    }

    return null;
});

The reason is because every cloud firestore trigger has to be idempotent, as firestore documentation say: https://firebase.google.com/docs/functions/firestore-events#limitations_and_guarantees

Solution

So, in order to prevent multiple executions of your code, you need to manage with events and transactions. This is my particular way to handle large collection counters:

const executeOnce = (change, context, task) => {
    const eventRef = firestore.collection('events').doc(context.eventId);

    return firestore.runTransaction(t =>
        t
         .get(eventRef)
         .then(docSnap => (docSnap.exists ? null : task(t)))
         .then(() => t.set(eventRef, { processed: true }))
    );
};

const documentCounter = collectionName => (change, context) =>
    executeOnce(change, context, t => {
        // on create
        if (!change.before.exists && change.after.exists) {
            return t
                    .get(firestore.collection('metadatas')
                    .doc(collectionName))
                    .then(docSnap =>
                        t.set(docSnap.ref, {
                            count: ((docSnap.data() && docSnap.data().count) || 0) + 1
                        }));
        // on delete
        } else if (change.before.exists && !change.after.exists) {
            return t
                     .get(firestore.collection('metadatas')
                     .doc(collectionName))
                     .then(docSnap =>
                        t.set(docSnap.ref, {
                            count: docSnap.data().count - 1
                        }));
        }

        return null;
    });

Use cases here:

/**
 * Count documents in articles collection.
 */
exports.articlesCounter = functions.firestore
    .document('articles/{id}')
    .onWrite(documentCounter('articles'));

/**
 * Count documents in customers collection.
 */
exports.customersCounter = functions.firestore
    .document('customers/{id}')
    .onWrite(documentCounter('customers'));

As you can see, the key to prevent multiple execution is the property called eventId in the context object. If the function has been handled many times for the same event, the event id will be the same in all cases. Unfortunately, you must have "events" collection in your database.

User · Answer

UPDATE 11 20 I created an npm package for easy access to a counter function  https   fireblog io blog post firestore-counters  I created a universal function using all these ideas to handle all counter situations  except queries    The only exception would be when doing so many writes a second  it slows you down  An example would be likes on a trending post  It is overkill on a blog post  for example  and will cost you more  I suggest creating a separate function in that case using shards  https   firebase google com docs firestore solutions counters     trigger collections exports myFunction   functions firestore      document   colId   docId         onWrite async  change  any  context  any    gt            return runCounter change  context               trigger sub-collections exports mySubFunction   functions firestore      document   colId   docId   subColId   subDocId         onWrite async  change  any  context  any    gt            return runCounter change  context               add change the count const runCounter   async function  change  any  context  any         const col   context params colId       const eventsDoc     events       const countersDoc     counters           ignore helper collections     if  col startsWith                 return null               simplify event types     const createDoc   change after exists  amp  amp   change before exists      const updateDoc   change before exists  amp  amp  change after exists       if  updateDoc            return null               check for sub collection     const isSubCol   context params subDocId       const parentDoc      countersDoc    context params colId        const countDoc   isSubCol              parentDoc    context params docId    context params subColId                parentDoc            collection references     const countRef   db doc countDoc       const countSnap   await countRef get            increment size if doc exists     if  countSnap exists               createDoc or deleteDoc         const n   createDoc   1   -1          const i   admin firestore FieldValue increment n               create event for accurate increment         const eventRef   db doc    eventsDoc    context eventId              return db runTransaction async  t  any   Promise lt any gt    gt                const eventSnap   await t get eventRef                  do nothing if event exists             if  eventSnap exists                    return null                               add event and update size             await t update countRef    count  i                 return t set eventRef                    completed  admin firestore FieldValue serverTimestamp                              catch  e  any    gt                console log e                          otherwise count all docs in the collection and add size       else           const colRef   db collection change after ref parent path           return db runTransaction async  t  any   Promise lt any gt    gt                   update size             const colSnap   await t get colRef               return t set countRef    count  colSnap size                catch  e  any    gt                console log e                         This handles events  increments  and transactions   The beauty in this  is that if you are not sure about the accuracy of a document  probably while still in beta   you can delete the counter to have it automatically add them up on the next trigger  Yes  this costs  so don t delete it otherwise  Same kind of thing to get the count  const collectionPath    buildings 138faicnjasjoa89 buildingContacts   const colSnap   await db doc   counters     collectionPath  get    const count   colSnap get  count     Also  you may want to create a cron job  scheduled function  to remove old events to save money on database storage  You need at least a blaze plan  and there may be some more configuration  You could run it every sunday at 11pm  for example  https   firebase google com docs functions schedule-functions This is untested  but should work with a few tweaks  exports scheduledFunctionCrontab   functions pubsub schedule  5 11              timeZone  America New York        onRun async  context    gt                get yesterday         const yesterday   new Date            yesterday setDate yesterday getDate   - 1            const eventFilter   db collection   events   where  completed     lt     yesterday           const eventFilterSnap   await eventFilter get            eventFilterSnap forEach async  doc  any    gt                await doc ref delete                        return null           And last  don t forget to protect the collections in firestore rules  match   counters  document      allow read    allow write  if false    match   events  document      allow read  write  if false     Update  Queries Adding to my other answer if you want to automate query counts as well  you can use this modified code in your cloud function      if  col      posts                 counter reference - user doc ref         const userRef   after   after userDoc   before userDoc             query reference         const postsQuery   db collection  posts   where  userDoc    quot    quot   userRef              add the count - postsCount on userDoc         await addCount change  context  postsQuery  userRef   postsCount               return delEvents      Which will automatically update the postsCount in the userDocument  You could easily add other one to many counts this way  This just gives you ideas of how you can automate things   I also gave you another way to delete the events  You have to read each date to delete it  so it won t really save you to delete them later  just makes the function slower         Adds a counter to a doc     param change - change ref     param context - context ref     param queryRef - the query ref to count     param countRef - the counter document ref     param countName - the name of the counter on the counter document     const addCount   async function  change  any  context  any     queryRef  any  countRef  any  countName  string            events collection     const eventsDoc     events           simplify event type     const createDoc   change after exists  amp  amp   change before exists          doc references     const countSnap   await countRef get            increment size if field exists     if  countSnap get countName                createDoc or deleteDoc         const n   createDoc   1   -1          const i   admin firestore FieldValue increment n               create event for accurate increment         const eventRef   db doc    eventsDoc    context eventId              return db runTransaction async  t  any   Promise lt any gt    gt                const eventSnap   await t get eventRef                  do nothing if event exists             if  eventSnap exists                    return null                               add event and update size             await t set countRef     countName   i      merge  true                 return t set eventRef                    completed  admin firestore FieldValue serverTimestamp                              catch  e  any    gt                console log e                          otherwise count all docs in the collection and add size       else           return db runTransaction async  t  any   Promise lt any gt    gt                   update size             const colSnap   await t get queryRef               return t set countRef     countName   colSnap size      merge  true                catch  e  any    gt                console log e                               Deletes events over a day old     const delEvents   async function              get yesterday     const yesterday   new Date        yesterday setDate yesterday getDate   - 1        const eventFilter   db collection   events   where  completed     lt     yesterday       const eventFilterSnap   await eventFilter get        eventFilterSnap forEach async  doc  any    gt            await doc ref delete                return null      I should also warn you that universal functions will run on every onWrite call period  It may be cheaper to only run the function on onCreate and on onDelete instances of your specific collections  Like the noSQL database we are using  repeated code and data can save you money

User · Answer

There is no direct option available  You cant t do db collection  CollectionName   count    Below are the two ways by which you can find the count of number of documents within a collection   1  - Get all the documents in the collection and then get it s size  Not the best Solution   db collection  CollectionName   get   subscribe doc  gt   console log doc size       By using above code your document reads will be equal to the size of documents within a collection and that is the reason why one must avoid using above solution   2 - Create a separate document with in your collection which will store the count of number of documents in the collection  Best Solution   db collection  CollectionName   doc  counts  get   subscribe doc  gt   console log doc count       Above we created a document with name counts to store all the count information You can update the count document in the following way -   Create a firestore triggers  on the document counts Increment the count property of counts document when a new document is created  Decrement the count property of counts document when a document is deleted    w r t price  Document Read   1  and fast data retrieval the above solution is good

User · Answer

I agree with  Matthew  it will cost a lot if you perform such query      ADVICE FOR DEVELOPERS BEFORE STARTING THEIR PROJECTS   Since we have foreseen this situation at the beginning  we can actually make a collection namely counters with a document to store all the counters in a field with type number    For example   For each CRUD operation on the collection  update the counter document    When you create a new collection subcollection    1 in the counter   1 write operation  When you delete a collection subcollection   -1 in the counter   1 write operation  When you update an existing collection subcollection  do nothing on the counter document   0   When you read an existing collection subcollection  do nothing on the counter document   0     Next time  when you want to get the number of collection  you just need to query point to the document field   1 read operation   In addition  you can store the collections name in an array  but this will be tricky  the condition of array in firebase is shown as below      we send this   a    b    c    d    e      Firebase stores this  0   a   1   b   2   c   3   d   4   e       since the keys are numeric and sequential     if we query the data  we get this   a    b    c    d    e       however  if we then delete a  b  and d     they are no longer mostly sequential  so    we do not get back an array  2   c   4   e     So  if you are not going to delete the collection   you can actually use array to store list of collections name instead of querying all the collection every time   Hope it helps

User · Answer

firebaseFirestore collection        addSnapshotListener new EventListener lt QuerySnapshot gt               Override         public void onEvent QuerySnapshot documentSnapshots  FirebaseFirestoreException e                 int Counter   documentSnapshots size

User · Answer

A workaround is to   write a counter in a firebase doc  which you increment within a transaction everytime you create a new entry  You store the count in a field of your new entry  i e  position  4    Then you create an index on that field  position DESC    You can do a skip limit with a query Where  position     lt   x  OrderBy  position   DESC   Hope this helps

User · Answer

This uses counting to create numeric unique ID   In my use  I will not be decrementing ever  even when the document that the ID is needed for is deleted  Upon a collection creation that needs unique numeric value  Designate a collection appData with one document  set with  doc id only Set uniqueNumericIDAmount to 0 in the firebase firestore console Use doc data   uniqueNumericIDAmount   1 as the unique numeric id Update appData collection uniqueNumericIDAmount with firebase firestore FieldValue increment 1   firebase      firestore        collection  quot appData quot        doc  quot only quot        get        then doc   gt            var foo   doc data            foo id   doc id              your collection that needs a unique ID         firebase              firestore                collection  quot uniqueNumericIDs quot                doc user uid    user id in my case              set     I use this in login  so this document doesn t                      exist yet  otherwise use update instead of set                 phone  this state phone    whatever else you need                 uniqueNumericID  foo uniqueNumericIDAmount   1                             then      gt                        upon success of new ID  increment uniqueNumericIDAmount                 firebase                      firestore                        collection  quot appData quot                        doc  quot only quot                        update                           uniqueNumericIDAmount  firebase firestore FieldValue increment                              1                                                                       catch err   gt                            console log err                                                       catch err   gt                    console log err

User · Answer

I have try a lot with different approaches   And finally  I improve one of the methods   First you need to create a separate collection and save there all events  Second you need to create a new lambda to be triggered by time  This lambda will Count events in event collection and clear event documents  Code details in article  https   medium com  ihor malaniuk how-to-count-documents-in-google-cloud-firestore-b0e65863aeca

User · Answer

No  there is no built-in support for aggregation queries right now   However there are a few things you could do   The first is documented here  You can use transactions or cloud functions to maintain aggregate information   This example shows how to use a function to keep track of the number of ratings in a subcollection  as well as the average rating   exports aggregateRatings   firestore    document  restaurants  restId  ratings  ratingId       onWrite event   gt           Get value of the newly added rating     var ratingVal   event data get  rating            Get a reference to the restaurant     var restRef   db collection  restaurants   document event params restId           Update aggregations in a transaction     return db transaction transaction   gt          return transaction get restRef  then restDoc   gt               Compute new number of ratings         var newNumRatings   restDoc data  numRatings     1              Compute new average rating         var oldRatingTotal   restDoc data  avgRating     restDoc data  numRatings            var newAvgRating    oldRatingTotal   ratingVal    newNumRatings              Update restaurant info         return transaction update restRef              avgRating  newAvgRating            numRatings  newNumRatings                                     The solution that jbb mentioned is also useful if you only want to count documents infrequently   Make sure to use the select   statement to avoid downloading all of each document  that s a lot of bandwidth when you only need a count    select   is only available in the server SDKs for now so that solution won t work in a mobile app

User · Answer

In 2020 this is still not available in the Firebase SDK however it is available in Firebase Extensions  Beta  however it s pretty complex to setup and use    A reasonable approach Helpers     create delete seems redundant but is cheaper than onUpdate  export const onCreateCounter        gt  async     change    context     gt      const collectionPath   change ref parent path    const statsDoc   db doc  quot counters  quot    collectionPath     const countDoc         countDoc  quot count quot     admin firestore FieldValue increment 1     await statsDoc set countDoc    merge  true         export const onDeleteCounter        gt  async     change    context     gt      const collectionPath   change ref parent path    const statsDoc   db doc  quot counters  quot    collectionPath     const countDoc         countDoc  quot count quot     admin firestore FieldValue increment -1     await statsDoc set countDoc    merge  true         export interface CounterPath     watch  string    name  string      Exported Firestore hooks  export const Counters  CounterPath               name   quot count buildings quot       watch   quot buildings  id2  quot               name   quot count buildings subcollections quot       watch   quot buildings  id2   id3   id4  quot           Counters forEach item   gt      exports item name     create     functions firestore      document item watch       onCreate onCreateCounter        exports item name     delete     functions firestore      document item watch       onDelete onDeleteCounter           In action The building root collection and all sub collections will be tracked   Here under the  counters  root path  Now collection counts will update automatically and eventually  If you need a count  just use the collection path and prefix it with counters  const collectionPath    buildings 138faicnjasjoa89 buildingContacts   const collectionCount   await db    doc  counters     collectionPath     get      then snap   gt  snap get  count      Limitations As this approach uses a single database and document  it is limited to the Firestore constraint of 1 Update per Second for each counter  It will be eventually consistent  but in cases where large amounts of documents are added removed the counter will lag behind the actual collection count

User · Answer

Increment a counter using admin firestore FieldValue increment   exports onInstanceCreate   functions firestore document  projects  projectId  instances  instanceId       onCreate  snap  context    gt      db collection  projects   doc context params projectId  update         instanceCount  admin firestore FieldValue increment 1                exports onInstanceDelete   functions firestore document  projects  projectId  instances  instanceId       onDelete  snap  context    gt      db collection  projects   doc context params projectId  update         instanceCount  admin firestore FieldValue increment -1                 In this example we increment an instanceCount field in the project each time a document is added to the instances sub collection  If the field doesn t exist yet it will be created and incremented to 1   The incrementation is transactional internally but you should use a distributed counter if you need to increment more frequently than every 1 second   It s often preferable to implement onCreate and onDelete rather than onWrite as you will call onWrite for updates which means you are spending more money on unnecessary function invocations  if you update the docs in your collection

User · Answer

var variable 0 variable variable querySnapshot count  then if you are to use it on a String variable then let stringVariable  String variable

User · Answer

Solution using pagination with offset  amp  limit  public int collectionCount String collection            Integer page   0          List lt QueryDocumentSnapshot gt  snaps   new ArrayList lt  gt             findDocsByPage collection  page  snaps           return snaps size           public void findDocsByPage String collection  Integer page                              List lt QueryDocumentSnapshot gt  snaps        try           Integer limit   26000          FieldPath   selectedFields   new FieldPath     FieldPath of  quot id quot              List lt QueryDocumentSnapshot gt  snapshotPage          snapshotPage   fireStore                            collection collection                           select selectedFields                           offset page   limit                           limit limit                           get   get   getDocuments                if  snapshotPage size    gt  0                snaps addAll snapshotPage               page                findDocsByPage collection  page  snaps                   catch  InterruptedException   ExecutionException e            e printStackTrace              findDocsPage it s a recursive method to find all pages of collection  selectedFields for otimize query and get only id field instead full body of document  limit max size of each query page  page define inicial page for pagination    From the tests I did it worked well for collections with up to approximately 120k records

User · Answer

Simplest way to do so is to read the size of a  querySnapshot    db collection  cities   get   then function querySnapshot              console log querySnapshot size          You can also read the length of the docs array inside  querySnapshot    querySnapshot docs length    Or if a  querySnapshot  is empty by reading the empty value  which will return a boolean value   querySnapshot empty

User · Answer

As far as I know there is no build-in solution for this and it is only possible in the node sdk right now  If you have a  db collection  someCollection     you can use   select  fields     to define which field you want to select  If you do an empty select   you will just get an array of document references   example   db collection  someCollection   select   get   then     snapshot    gt  console log snapshot docs length      This solution is only a optimization for the worst case of downloading all documents and does not scale on large collections   Also have a look at this  How to get a count of number of documents in a collection with Cloud Firestore

User · Answer

Took me a while to get this working based on some of the answers above  so I thought I d share it for others to use  I hope it s useful    use strict    const functions   require  firebase-functions    const admin   require  firebase-admin    admin initializeApp    const db   admin firestore     exports countDocumentsChange   functions firestore document  library  categoryId  documents  documentId    onWrite  change  context    gt         const categoryId   context params categoryId      const categoryRef   db collection  library   doc categoryId      let FieldValue   require  firebase-admin   firestore FieldValue       if   change before exists                new document created   add one to count         categoryRef update  numberOfDocs  FieldValue increment 1             console log   s numberOfDocs incremented by 1   categoryId          else if  change before exists  amp  amp  change after exists                updating existing document   Do nothing        else if   change after exists                deleting document   subtract one from count         categoryRef update  numberOfDocs  FieldValue increment -1             console log   s numberOfDocs decremented by 1   categoryId               return 0

User · Answer

As with many questions  the answer is - It depends  You should be very careful when handling large amounts of data on the front end  On top of making your front end feel sluggish  Firestore also charges you  0 60 per million reads you make   Small collection  less than 100 documents  Use with care - Frontend user experience may take a hit Handling this on the front end should be fine as long as you are not doing too much logic with this returned array  db collection        get   then snap   gt       size   snap size    will return the collection size       Medium collection  100 to 1000 documents  Use with care - Firestore read invocations may cost a lot Handling this on the front end is not feasible as it has too much potential to slow down the users system  We should handle this logic server side and only return the size  The drawback to this method is you are still invoking firestore reads  equal to the size of your collection   which in the long run may end up costing you more than expected  Cloud Function      db collection        get   then snap   gt        res status 200  send  length  snap size         Front End  yourHttpClient post yourCloudFunctionUrl  toPromise   then snap   gt         size   snap length    will return the collection size      Large collection  1000  documents  Most scalable solution   FieldValue increment    As of April 2019 Firestore now allows incrementing counters  completely atomically  and without reading the data prior  This ensures we have correct counter values even when updating from multiple sources simultaneously  previously solved using transactions   while also reducing the number of database reads we perform   By listening to any document deletes or creates we can add to or remove from a count field that is sitting in the database  See the firestore docs - Distributed Counters Or have a look at Data Aggregation by Jeff Delaney  His guides are truly fantastic for anyone using AngularFire but his lessons should carry over to other frameworks as well  Cloud Function  export const documentWriteListener        functions firestore document  collection  documentUid         onWrite  change  context    gt         if   change before exists               New document Created   add one to count          db doc docRef  update  numberOfDocs  FieldValue increment 1                else if  change before exists  amp  amp  change after exists               Updating existing document   Do nothing        else if   change after exists               Deleting document   subtract one from count          db doc docRef  update  numberOfDocs  FieldValue increment -1             return       Now on the frontend you can just query this numberOfDocs field to get the size of the collection

[firebase] Cloud Firestore collection count

Solution

Examples related to firebase

Examples related to google-cloud-firestore