MongoDB logging all queries

Question

The question is as basic as it is simple    How do you log all queries in a  tail able log file in mongodb   I have tried    setting the profiling level  setting the slow ms parameter starting mongod with the -vv option   The  var log mongodb mongodb log keeps showing just the current number of active connections

User · Answer

Setting profilinglevel to 2 is another option to log all queries

User · Answer

db adminCommand    getLog            Then  db adminCommand    getLog    global

User · Answer

I ended up solving this by starting mongod like this  hammered and ugly  yeah    but works for development environment    mongod --profile 1 --slowms 1  amp    This enables profiling and sets the threshold for  slow queries  as 1ms  causing all queries to be logged as  slow queries  to the file    var log mongodb mongodb log   Now I get continuous log outputs using the command   tail -f  var log mongodb mongodb log   An example log   Mon Mar  4 15 02 55  conn1  query dendro quads query    graph   u http   example org people    ntoreturn 0 ntoskip 0 nscanned 6 keyUpdates 0 locks micros  r 73163 nreturned 6 reslen 9884 88ms

User · Answer

I made a command line tool to activate the profiler activity and see the logs in a  tail able way   mongotail    But the more interesting feature  also like tail  is to see the changes in  real time  with the -f option  and occasionally filter the result with grep to find a particular operation   See documentation and installation instructions in  https   github com mrsarm mongotail

User · Answer

Try out this package to tail all the queries  without oplog operations   https   www npmjs com package mongo-tail-queries   Disclaimer  I wrote this package exactly for this need

User · Answer

Because its google first answer      For version 3    mongo MongoDB shell version  3 0 2 connecting to  test  gt  use myDb switched to db  gt  db setLogLevel 1    http   docs mongodb org manual reference method db setLogLevel

User · Answer

The profiler data is written to a collection in your DB  not to file  See http   docs mongodb org manual tutorial manage-the-database-profiler   I would recommend using 10gen s MMS service  and feed development profiler data there  where you can filter and sort it in the UI

User · Answer

if you want the queries to be logged to mongodb log file  you have to set both  the log level and the profiling  like for example    db setLogLevel 1  db setProfilingLevel 2     see https   docs mongodb com manual reference method db setLogLevel   Setting only the profiling would not have the queries logged to file  so you can only get it from   db system profile find   pretty

User · Answer

MongoDB has a sophisticated feature of profiling  The logging happens in system profile collection  The logs can be seen from    db system profile find     There are 3 logging levels  source     Level 0 - the profiler is off  does not collect any data  mongod always writes operations longer than the slowOpThresholdMs threshold to its log  This is the default profiler level  Level 1 - collects profiling data for slow operations only  By default slow operations are those slower than 100 milliseconds  You can modify the threshold for    slow    operations with the slowOpThresholdMs runtime option or the setParameter command  See the Specify the Threshold for Slow Operations section for more information  Level 2 - collects profiling data for all database operations    To see what profiling level the database is running in  use  db getProfilingLevel     and to see the status   db getProfilingStatus     To change the profiling status  use the command   db setProfilingLevel level  milliseconds    Where level refers to the profiling level and milliseconds is the ms of which duration the queries needs to be logged  To turn off the logging  use   db setProfilingLevel 0    The query to look in the system profile collection for all queries that took longer than one second  ordered by timestamp descending will be   db system profile find    millis      gt 1000       sort    ts   -1

User · Answer

I recommend checking out mongosniff  This can tool can do everything you want and more  Especially it can help diagnose issues with larger scale mongo systems and how queries are being routed and where they are coming from since it works by listening to your network interface for all mongo related communications   http   docs mongodb org v2 2 reference mongosniff

User · Answer

Once profiling level is set using db setProfilingLevel 2     The below command will print the last executed query   You may change the limit 5  as well to see less more queries    nin - will filter out profile and indexes queries  Also  use the query projection    query  1  for only viewing query field  db system profile find         ns              nin     meteor system profile   meteor system indexes              limit 5  sort    ts   -1     pretty     Logs with only query projection  db system profile find         ns              nin     meteor system profile   meteor system indexes              query  1    limit 5  sort    ts   -1     pretty

User · Answer

I wrote a script that will print out the system profile log in real time as queries come in   You need to enable logging first as stated in other answers   I needed this because I m using Windows Subsystem for Linux  for which tail still doesn t work   https   github com dtruel mongo-live-logger

User · Answer

You can log all queries     mongo MongoDB shell version  2 4 9 connecting to  test  gt  use myDb switched to db myDb  gt  db getProfilingLevel   0  gt  db setProfilingLevel 2     was    0   slowms    1   ok    1    gt  db getProfilingLevel   2  gt  db system profile find   pretty     Source  http   docs mongodb org manual reference method db setProfilingLevel   db setProfilingLevel 2  means  log all operations

User · Answer

I think that while not elegant  the oplog could be partially used for this purpose  it logs all the writes - but not the reads      You have to enable replicatoon  if I m right  The information is from this answer from this question  How to listen for changes to a MongoDB collection

User · Answer

This was asked a long time ago but this may still help someone   MongoDB profiler logs all the queries in the capped collection system profile  See this  database profiler   Start mongod instance with --profile 2 option that enables logging all queries OR if mongod instances is already running  from mongoshell  run db setProfilingLevel 2  after selecting database    it can be verified by  db getProfilingLevel    which should return 2  After this  I have created a script which utilises mongodb s tailable cursor to tail this system profile collection and write the entries in a file  To view the logs I just need to tail it tail -f    logs mongologs txt  This script can be started in background and it will log all the operation on the db in the file    My code for tailable cursor for the system profile collection is in nodejs  it logs all the operations along with queries happening in every collection of MyDb   const MongoClient   require  mongodb   MongoClient  const assert   require  assert    const fs   require  fs    const file       logs mongologs     Connection URL const url    mongodb   localhost 27017       Database Name const dbName    MyDb     Mongodb connection  MongoClient connect url  function  err  client       assert equal null  err      const db   client db dbName      listen db           function listen db  conditions    var filter     ns     ne   MyDb system profile         filter for query   e g  if we need to log only insert queries  use  op  insert     e g  if we need to log operation on only  MyCollection  collection  use  ns   MyDb MyCollection     we can give a lot of filters  print and check the  document  variable below     set MongoDB cursor options var cursorOptions         tailable  true      awaitdata  true      numberOfRetries  -1        create stream and listen var stream   db collection  system profile   find filter  cursorOptions  stream        call the callback stream on  data   function  document          this will run on every operation query done on our database       print  document  to check the keys based on which we can filter       delete data which we dont need in our log file      delete document execStats      delete document keysExamined        -----       -----        append the log generated in our log file which can be tailed from command line     fs appendFile file  JSON stringify document      n   function  err            if  err   console log  err                     For tailable cursor in python using pymongo  refer the following code which filters for MyCollection and only insert operation   import pymongo import time client   pymongo MongoClient   oplog   client MyDb system profile first   oplog find   sort   natural   pymongo ASCENDING  limit -1  next    ts   first  ts   while True      cursor   oplog find   ts      gt   ts    ns    MyDb MyCollection    op    insert                            cursor type pymongo CursorType TAILABLE AWAIT      while cursor alive          for doc in cursor              ts   doc  ts               print doc              print   n           time sleep 1    Note  Tailable cursor only works with capped collections  It cannot be used to log operations on a collection directly  instead use filter   ns    MyDb MyCollection    Note  I understand that the above nodejs and python code may not be of much help for some  I have just provided the codes for reference   Use this link to find documentation for tailable cursor in your languarge driver choice Mongodb Drivers  Another feature that i have added after this logrotate

User · Answer

db setProfilingLevel 2 -1   This worked  it logged all query info in mongod log file

[mongodb] MongoDB logging all queries

Examples related to mongodb

Examples related to logging