How to retrieve unique count of a field using Kibana Elastic Search

Question

Is it possible to query for a distinct unique count of a field using Kibana  I am using elastic search as my backend to Kibana    If so  what is the syntax of the query  Heres a link to the Kibana interface I would like to make my query  http   demo kibana org   dashboard  I am parsing nginx access logs with logstash and storing the data into elastic search   Then  I use Kibana to run queries and visualize my data in charts   Specifically  I want to know the count of unique IP addresses for a specific time frame using Kibana

User · Answer

Be aware with Unique count you are using 'cardinality' metric, which does not always guarantee exact unique count. :-)

the cardinality metric is an approximate algorithm. It is based on the HyperLogLog++ (HLL) algorithm. HLL works by hashing your input and using the bits from the hash to make probabilistic estimations on the cardinality.

Depending on amount of data I can get differences of 700+ entries missing in a 300k dataset via Unique Count in Elastic which are otherwise really unique.

Read more here: https://www.elastic.co/guide/en/elasticsearch/guide/current/cardinality.html

User · Answer

Now Kibana 4 allows you to use aggregations  Apart from building a panel like the one that was explained in this answer for Kibana 3  now we can see the number of unique IPs in different periods  that was  IMO  what the OP wanted at the first place    To build a dashboard like this you should go to Visualize -  Select your Index -  Select a Vertical Bar chart and then in the visualize panel    In the Y axis we want the unique count of IPs  select the field where you stored the IP  and in the X axis we want a date histogram with our timefield       After pressing the Apply button  we should have a graph that shows the unique count of IP distributed on time  We can change the time interval on the X axis to see the unique IPs hourly daily        Just take into account that the unique counts are approximate  For more information check also this answer

User · Answer

For Kibana 4 go to this answer  This is easy to do with a terms panel     If you want to select the count of distinct IP that are in your logs  you should specify in the field clientip  you should put a big enough number in length  otherwise  it will join different IP under the same group  and specify in the style table  After adding the panel  you will have a table with IP  and the count of that IP

User · Answer

Unique counts of field values are achieved by using facets  See ES documentation for the full story  but the gist is that you will create a query and then ask ES to prepare facets on the results for counting values found in fields  It s up to you to customize the fields used and even describe how you want the values returned  The most basic of facet types is just to group by terms  which would be like an IP address above  You can get pretty complex with these  even requiring a query within your facet          query              match all                  facets              terms                  field    ip address

User · Answer

Create  topN  query on  clientip  and then histogram with count on  clientip  and set  topN  query as source  Then you will see count of different ips per time

User · Answer

Using Aggs u can easily do that  Writing down query for now   GET index  search      size  0     aggs          source            terms              field    field            size   100000                        This would return the different values of field with there doc counts

[elasticsearch] How to retrieve unique count of a field using Kibana + Elastic Search

Examples related to elasticsearch

Examples related to logstash

Examples related to kibana