Remove Duplicate objects from JSON Array

Question

I have an array that looks like this   var standardsList           Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Geometry          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Orders of Operation          Grade    Math 2    Domain    Geometry          Grade    Math 2    Domain    Geometry        And I need to remove the duplicates so that something like this remains   var standardsList           Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Geometry          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Orders of Operation          Grade    Math 2    Domain    Geometry        I ve tried installing underscore js and using   uniq but that only seems to work when a single key value pair appears in the object  I can t seem to get it to work across multiple keys    When I try something like   var uniqueStandards     uniq standardsList  function item  key  Domain       return item Domain        I only get the first three unique values  one per grade   But I need all the unique values across both grade and domain  Is there a simple way to feed both keys to the   uniq function   Ultimately  I need a list with the each unique grade as the header and the unique domains as the list items to pass into an HTML page  I may be going about this wrong  so if there is an easier way to accomplish that end goal  I am open to ideas    Thanks in advance   Edit  Getting some good responses and wanted to clarify what my end goal was  I m trying to create a series of lists in HTML of the form    lt div gt       lt h3 gt Math K lt  h3 gt       lt li gt Counting  amp  Cardinality lt  li gt       lt li gt Geometry lt  li gt   lt  div gt   lt div gt       lt h3 gt Math 1 lt  h3 gt       lt li gt Counting  amp  Cardinality lt  li gt       lt li gt Orders of Operation lt  li gt   lt  div gt   lt div gt       lt h3 gt Math 2 lt  h3 gt       lt li gt Geometry lt  li gt   lt  div gt    My original though was to create an array and push that into the  lt div gt  element on the page with     divid   append array

User · Answer

The following works for me     uniq standardsList  JSON stringify    This would probably be slow for very long lists  however

User · Answer

function arrUnique arr        var cleaned           arr forEach function itm            var unique   true          cleaned forEach function itm2                if    isEqual itm  itm2   unique   false                      if  unique   cleaned push itm               return cleaned     var standardsList   arrUnique standardsList     FIDDLE  This will return  var standardsList           Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Geometry          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Orders of Operation          Grade    Math 2    Domain    Geometry        Which is exactly what you asked for

User · Answer

I needed to do some de-duping of JSON objects so I stumbled across this page  However  I went with the short ES6 solution  no need for external libs   running this in Chrome Dev Tools Snippets   const data        any list of objects       const set   new Set data map item   gt  JSON stringify item     const dedup       set  map item   gt  JSON parse item    console log  Removed   data length - dedup length  elements    console log dedup

User · Answer

Ultimately  I need a list with the each unique grade as the header and the unique domains as the list items to pass into an HTML page  I may be going about this wrong  so if there is an easier way to accomplish that end goal  I am open to ideas    So you don t actually need that output array in the format you asked about   That being the case  I would cut directly to the chase with a very simple and efficient solution   var grades       standardsList forEach  function  item         var grade   grades item Grade    grades item Grade             grade item Domain    true       console log  JSON stringify  grades  null  4        The resulting grades object is          Math K              Counting  amp  Cardinality   true           Geometry   true             Math 1              Counting  amp  Cardinality   true           Orders of Operation   true             Math 2              Geometry   true           One interesting thing about this approach is that it is very fast  Note that it makes only a single pass through the input array  unlike other solutions that require multiple passes  whether you write them yourself or whether   uniq   does it for you   For a small number of items this won t matter  but it s good to keep in mind for larger lists   And with this object you now have everything you need to run any code or generate any other format you want  For example  if you do need the exact array output format you mentioned  you can use   var outputList       for  var grade in grades         for  var domain in grades grade              outputList push   Grade  grade  Domain  domain              JSON stringify  outputList  null  4      This will log                    Grade    Math K            Domain    Counting  amp  Cardinality                        Grade    Math K            Domain    Geometry                        Grade    Math 1            Domain    Counting  amp  Cardinality                        Grade    Math 1            Domain    Orders of Operation                        Grade    Math 2            Domain    Geometry            Rai asks in a comment how this line of code works   var grade   grades item Grade    grades item Grade           This is a common idiom for fetching an object property or providing a default value if the property is missing  Note that the   assignments are done in right-to-left order  So we could translate it literally to use an if statement and a temp variable      Fetch grades item Grade  and save it in temp var temp   grades item Grade   if    temp            It was missing  so use an empty object as the default value     temp            Now save the result in grades item Grade   in case it was missing     and in grade grades item Grade    temp  var grade   temp    You may notice that in the case where grades item Grade  already exists  we take the value we just fetched and store it back into the same property  This is unnecessary  of course  and  you probably wouldn t do it if you were writing the code out like this  Instead  you would simplify it   var grade   grades item Grade   if    grade         grade   grades item Grade            That would be a perfectly reasonable way to write the same code  and it s more efficient too  It also gives you a way to do a more specific test than the  truthiness  that the    idiom relies on  For example instead of if    grade   you might want to use if  grade     undefined

User · Answer

Javascript solution for your case   console log unique standardsList     function unique obj       var uniques         var stringify         for var i 0 i lt obj length i            var keys Object keys obj i           keys sort function a b   return a-b           var str             for var j 0 j lt keys length j                str   JSON stringify keys j               str   JSON stringify obj i  keys j                       if  stringify hasOwnProperty str                uniques push obj i                stringify str  true                      return uniques

User · Answer

x000D   x000D  var standardsList     x000D        Grade    Math K    Domain    Counting  amp  Cardinality    x000D        Grade    Math K    Domain    Counting  amp  Cardinality    x000D        Grade    Math K    Domain    Counting  amp  Cardinality    x000D        Grade    Math K    Domain    Counting  amp  Cardinality    x000D        Grade    Math K    Domain    Geometry    x000D        Grade    Math 1    Domain    Counting  amp  Cardinality    x000D        Grade    Math 1    Domain    Counting  amp  Cardinality    x000D        Grade    Math 1    Domain    Orders of Operation    x000D        Grade    Math 2    Domain    Geometry    x000D        Grade    Math 2    Domain    Geometry   x000D     x000D   x000D  standardsList   standardsList filter  li  idx  self    gt  self map itm   gt  itm Grade itm Domain  indexOf li Grade li Domain      idx  x000D   x000D  document write JSON stringify standardsList   x000D   x000D   x000D    here is a functional way of doing it that is much easier  standardsList   standardsList filter  li  idx  self    gt  self map itm   gt  iem Grade itm domain  indexOf li Grade li domain      idx

User · Answer

The following method does the way you want  It filters the array based on all properties values          var standardsList          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Geometry          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Orders of Operation          Grade    Math 2    Domain    Geometry          Grade    Math 2    Domain    Geometry        const removeDupliactes    values    gt      let concatArray   values map eachValue   gt        return Object values eachValue  join            let filterValues   values filter  value  index    gt        return concatArray indexOf concatArray index       index         return filterValues   removeDupliactes standardsList     Results this    Grade   Math K   Domain   Counting  amp  Cardinality     Grade   Math K   Domain   Geometry     Grade   Math 1   Domain   Counting  amp  Cardinality     Grade   Math 1   Domain   Orders of Operation     Grade   Math 2   Domain   Geometry

User · Answer

Use this Pseudocode  var standardsList           Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Geometry          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Orders of Operation          Grade    Math 2    Domain    Geometry          Grade    Math 2    Domain    Geometry      var newArr     for var i in standardsList       newArr push JSON stringify standardsList i      var obj       newArr  newArr filter  item   gt       return obj hasOwnProperty item    false    obj item    true      standardsList length   0 for var i in newArr       standardsList push JSON parse newArr i      console log standardsList    I have choose a sample array similar to yours  Its easier to compare objects once you stringfy them  Then you just have to compare strings

User · Answer

You can use lodash  download here   4 17 15  Example code  var object       x   1   y   2       x   2   y   1       x   1   y   2        uniqWith object    isEqual         gt      x   1   y   2       x   2   y   1

User · Answer

Here s a short one-liner with es6   const nums        AC8818E1      AC8818E1      AC8818E1      AC8818E1      AC8818E1      AC9233F2015      AC9233F2015      AC9233F2015      AC8818E1      AC8818E1      AC8818E1      AC8818E1      AC8818E1      AC8818E2      AC8818E2      AC8818E2      AC8818E2      AC9233F2015      AC9233F2015      AC9233F2015      AC9233F2015      AC8818E1      AC8818E1      AC8818E1      AC8818E2      AC8818E2      AC9233F2015      AC9233F2015      AC8818E1      AC8818E1      AC8818E1      AC8818E2      AC8818E2      AC8818E2      AC8818E2      ACB098F25      ACB098F25      ACB098F25      ACB098F25      AC8818E2      AC8818E2      AC8818E1      AC8818E1      AC8818E1         Set is a new data object introduced in ES6    Because Set only lets you store unique values  When you pass in an array  it will remove any duplicate values    export const  uniquenums       new Set nums   sort

User · Answer

I know there are many answers already but the best one that worked for me for a complex json structure is    x000D   x000D  var arr       State    RWCW    desc    WEST    code    RWCW    level   0   save    RWCW   WEST    attribute1        attribute2            State    RSCW    desc    SOUTHEAST    code    RSCW    level   0   save    RSCW   SOUTHEAST    attribute1        attribute2            State    RCSW    desc    SOUTHWEST    code    RCSW    level   0   save    RCSW   SOUTHWEST    attribute1        attribute2            State    RECW    desc    NORTHEAST    code    RECW    level   0   save    RECW   NORTHEAST    attribute1        attribute2            State    RWCW    desc    WEST    code    RWCW    level   0   save    RWCW   WEST    attribute1        attribute2            State    RWCW    desc    WEST    code    RWCW    level   0   save    RWCW   WEST    attribute1        attribute2            State    RSCW    desc    SOUTHEAST    code    RSCW    level   0   save    RSCW   SOUTHEAST    attribute1        attribute2            State    RCSW    desc    SOUTHWEST    code    RCSW    level   0   save    RCSW   SOUTHWEST    attribute1        attribute2            State    RECW    desc    NORTHEAST    code    RECW    level   0   save    RECW   NORTHEAST    attribute1        attribute2            State    RWCW    desc    WEST    code    RWCW    level   0   save    RWCW   WEST    attribute1        attribute2            State    RSCW    desc    SOUTHEAST    code    RSCW    level   0   save    RSCW   SOUTHEAST    attribute1        attribute2            State    RECW    desc    NORTHEAST    code    RECW    level   0   save    RECW   NORTHEAST    attribute1        attribute2            State    RWCW    desc    WEST    code    RWCW    level   0   save    RWCW   WEST    attribute1        attribute2            State    RCSW    desc    SOUTHWEST    code    RCSW    level   0   save    RCSW   SOUTHWEST    attribute1        attribute2            State    RWCW    desc    WEST    code    RWCW    level   0   save    RWCW   WEST    attribute1        attribute2            State    RCNW    desc    MIDWEST    code    RCNW    level   0   save    RCNW   MIDWEST    attribute1        attribute2            State    RCSW    desc    SOUTHWEST    code    RCSW    level   0   save    RCSW   SOUTHWEST    attribute1        attribute2            State    RECW    desc    NORTHEAST    code    RECW    level   0   save    RECW   NORTHEAST    attribute1        attribute2            State    RCNW    desc    MIDWEST    code    RCNW    level   0   save    RCNW   MIDWEST    attribute1        attribute2            State    RSCW    desc    SOUTHEAST    code    RSCW    level   0   save    RSCW   SOUTHEAST    attribute1        attribute2            State    RECW    desc    NORTHEAST    code    RECW    level   0   save    RECW   NORTHEAST    attribute1        attribute2            State    RCNW    desc    MIDWEST    code    RCNW    level   0   save    RCNW   MIDWEST    attribute1        attribute2            State    RSCW    desc    SOUTHEAST    code    RSCW    level   0   save    RSCW   SOUTHEAST    attribute1        attribute2            State    RSCW    desc    SOUTHEAST    code    RSCW    level   0   save    RSCW   SOUTHEAST    attribute1        attribute2            State    RSCW    desc    SOUTHEAST    code    RSCW    level   0   save    RSCW   SOUTHEAST    attribute1        attribute2            State    RCNW    desc    MIDWEST    code    RCNW    level   0   save    RCNW   MIDWEST    attribute1        attribute2            State    RSCW    desc    SOUTHEAST    code    RSCW    level   0   save    RSCW   SOUTHEAST    attribute1        attribute2            State    RECW    desc    NORTHEAST    code    RECW    level   0   save    RECW   NORTHEAST    attribute1        attribute2          x000D   x000D  var clean   arr filter  arr  index  self    gt  x000D      index     self findIndex  t    gt   t save     arr save  amp  amp  t State     arr State    x000D   x000D  console log clean   x000D   x000D   x000D    You can try this directly to chrome browser console and edit according to your need   I hope this helps someone

User · Answer

var standardsList           Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Geometry          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Orders of Operation          Grade    Math 2    Domain    Geometry          Grade    Math 2    Domain    Geometry                  function uniqurArray array                            var a   array concat                            for var i 0  i lt a length  i                                  for var j i 1  j lt a length  j                                      if a i  Grade     a j  Grade                                       a splice j--  1                                                                                                                      return a                             uniqurArray standardsList     put this js in console and you get uniq object in array

User · Answer

Use Map to remove the duplicates   For new readers   x000D   x000D  var standardsList           Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Counting  amp  Cardinality          Grade    Math K    Domain    Geometry          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Counting  amp  Cardinality          Grade    Math 1    Domain    Orders of Operation          Grade    Math 2    Domain    Geometry          Grade    Math 2    Domain    Geometry       var grades   new Map    standardsList forEach  function  item         grades set JSON stringify item   item        console log      grades values                Grade   Math K   Domain   Counting  amp  Cardinality         Grade   Math K   Domain   Geometry         Grade   Math 1   Domain   Counting  amp  Cardinality         Grade   Math 1   Domain   Orders of Operation         Grade   Math 2   Domain   Geometry         x000D   x000D   x000D

User · Answer

Reviving an old question  but I wanted to post an iteration on  adeneo s answer  That answer is completely general  but for this use case it could be more efficient  it s slow on my machine with an array of a few thousand objects   If you know the specific properties of the objects you need to compare  just compare them directly   var sl   standardsList  var out        for  var i   0  l   sl length  i  lt  l  i          var unique   true      for  var j   0  k   out length  j  lt  k  j              if   sl i  Grade     out j  Grade   amp  amp   sl i  Domain     out j  Domain                 unique   false                      if  unique            out push sl i             console log sl length      10 console log out length      5

[javascript] Remove Duplicate objects from JSON Array

Examples related to javascript

Examples related to jquery

Examples related to underscore.js