How to listen for changes to a MongoDB collection

Question

I m creating a sort of background job queue system with MongoDB as the data store  How can I  listen  for inserts to a MongoDB collection before spawning workers to process the job  Do I need to poll every few seconds to see if there are any changes from last time  or is there a way my script can wait for inserts to occur  This is a PHP project that I am working on  but feel free to answer in Ruby or language agnostic

User · Answer

Check out this  Change Streams  January 10  2018  - Release 3 6   EDIT  I wrote an article about how to do this https   medium com riow mongodb-data-collection-change-85b63d96ff76  https   docs mongodb com v3 6 changeStreams     It s new in mongodb 3 6 https   docs mongodb com manual release-notes 3 6  2018 01 10        mongod --version db version v3 6 2     In order to use changeStreams the database must be a Replication Set        More about Replication Sets     https   docs mongodb com manual replication       Your Database will be a  Standalone  by default         How to Convert a Standalone to a Replica Set  https   docs mongodb com manual tutorial convert-standalone-to-replica-set        The following example is a practical application for how you might use this    Specifically for Node        file js     use strict    module exports   function       app      io      User    Collection Name            SET WATCH ON COLLECTION      const changeStream   User watch              Socket Connection       io on  connection   function  socket            console log  Connection                 USERS - Change         changeStream on  change   function change                console log  COLLECTION CHANGED                 User find      err  data    gt                    if  err  throw err                   if  data                           RESEND ALL USERS                     socket emit  users   data                                                               END - file js        Useful links  https   docs mongodb com manual tutorial convert-standalone-to-replica-set https   docs mongodb com manual tutorial change-streams-example    https   docs mongodb com v3 6 tutorial change-streams-example http   plusnconsulting com post MongoDB-Change-Streams

User · Answer

Since MongoDB 3 6 there will be a new notifications API called Change Streams which you can use for this  See this blog post for an example  Example from it   cursor   client my db my collection changes          match              operationType      in     insert    replace                   match              newDocument n      gte   1               Loops forever  for change in cursor      print change  newDocument

User · Answer

There is an working java example which can be found here    MongoClient mongoClient   new MongoClient        DBCollection coll   mongoClient getDatabase  local   getCollection  oplog rs         DBCursor cur   coll find   sort BasicDBObjectBuilder start   natural   1  get                 addOption Bytes QUERYOPTION TAILABLE   Bytes QUERYOPTION AWAITDATA        System out println     open cursor            Runnable task      - gt            System out println   tWaiting for events            while  cur hasNext                  DBObject obj   cur next                System out println  obj                          new Thread task  start      The key is QUERY OPTIONS given here   Also you can change find query  if you don t need to load all the data every time   BasicDBObject query  new BasicDBObject    query put  ts   new BasicDBObject   gt   new BsonTimestamp 1471952088  1        timestamp is within some range query put  op    i      Only insert operation  DBCursor cur   coll find query  sort BasicDBObjectBuilder start   natural   1  get     addOption Bytes QUERYOPTION TAILABLE   Bytes QUERYOPTION AWAITDATA

User · Answer

MongoDB has what is called capped collections and tailable cursors that allows MongoDB to push data to the listeners   A capped collection is essentially a collection that is a fixed size and only allows insertions  Here s what it would look like to create one   db createCollection  messages     capped  true  size  100000000      MongoDB Tailable cursors  original post by Jonathan H  Wage   Ruby  coll   db collection  my collection   cursor   Mongo  Cursor new coll   tailable   gt  true  loop do   if doc   cursor next document     puts doc   else     sleep 1   end end   PHP   mongo   new Mongo     db    mongo- gt selectDB  my db    coll    db- gt selectCollection  my collection     cursor    coll- gt find  - gt tailable true   while  true        if   cursor- gt hasNext               doc    cursor- gt getNext            print r  doc         else           sleep 1             Python  by Robert Stewart   from pymongo import Connection import time  db   Connection   my db coll   db my collection cursor   coll find tailable True  while cursor alive      try          doc   cursor next           print doc     except StopIteration          time sleep 1    Perl  by Max   use 5 010   use strict  use warnings  use MongoDB   my  db   MongoDB  Connection- gt new  my  coll    db- gt my db- gt my collection  my  cursor    coll- gt find- gt tailable 1   for            if  defined my  doc    cursor- gt next                 say  doc            else               sleep 1            Additional Resources   Ruby Node js Tutorial which walks you through creating an application that listens to inserts in a MongoDB capped collection   An article talking about tailable cursors in more detail   PHP  Ruby  Python  and Perl examples of using tailable cursors

User · Answer

Actually  instead of watching output  why you dont get notice when something new is inserted by using middle-ware that was provided by mongoose schema   You can catch the event of insert a new document and do something after this insertion done

User · Answer

After 3 6 one is allowed to use database the following database triggers types    event-driven triggers - useful to update related documents automatically  notify downstream services  propagate data to support mixed workloads  data integrity  amp  auditing scheduled triggers - useful for scheduled data retrieval  propagation  archival and analytics workloads   Log into your Atlas account and select Triggers interface and add new trigger     Expand each section for more settings or details

User · Answer

What you are thinking of sounds a lot like triggers  MongoDB does not have any support for triggers  however some people have  rolled their own  using some tricks  The key here is the oplog   When you run MongoDB in a Replica Set  all of the MongoDB actions are logged to an operations log  known as the oplog   The oplog is basically just a running list of the modifications made to the data  Replicas Sets function by listening to changes on this oplog and then applying the changes locally    Does this sound familiar   I cannot detail the whole process here  it is several pages of documentation  but the tools you need are available   First some write-ups on the oplog  - Brief description  - Layout of the local collection  which contains the oplog   You will also want to leverage tailable cursors  These will provide you with a way to listen for changes instead of polling for them  Note that replication uses tailable cursors  so this is a supported feature

User · Answer

Alternatively  you could use the standard Mongo FindAndUpdate method  and within the callback  fire an EventEmitter event  in Node  when the callback is run   Any other parts of the application or architecture listening to this event will be notified of the update  and any relevant data sent there also  This is a really simple way to achieve notifications from Mongo

User · Answer

MongoDB version 3 6 now includes change streams which is essentially an API on top of the OpLog allowing for trigger notification-like use cases     Here is a link to a Java example  http   mongodb github io mongo-java-driver 3 6 driver tutorials change-streams   A NodeJS example might look something like    var MongoClient   require  mongodb   MongoClient      MongoClient connect  mongodb   localhost 22000 MyStore readConcern majority         then function client          let db   client db  MyStore           let change streams   db collection  products   watch             change streams on  change   function change               console log JSON stringify change

User · Answer

There is an awesome set of services available called MongoDB Stitch  Look into stitch functions triggers  Note this is a cloud-based paid service  AWS   In your case  on an insert  you could call a custom function written in javascript

User · Answer

Many of these answers will only give you new records and not updates and or are extremely ineffecient  The only reliable  performant way to do this is to create a tailable cursor on local db  oplog rs collection to get ALL changes to MongoDB and do with it what you will   MongoDB even does this internally more or less to support replication    Explanation of what the oplog contains  https   www compose com articles the-mongodb-oplog-and-node-js   Example of a Node js library that provides an API around what is available to be done with the oplog  https   github com cayasso mongo-oplog

[mongodb] How to listen for changes to a MongoDB collection?

Check out this: Change Streams

January 10, 2018 - Release 3.6

https://docs.mongodb.com/v3.6/changeStreams/

Examples related to mongodb