Code works (mostly). But failure to remove duplicates

Question

Summary:

Hey I've got a filter for some data. It's filtering fine but it's not removing duplicate objects. Arguments taken in order:

1. datasource
2. filters (remove all matching)
3. unique: (whether to remove duplicate items)
4. prop ( object properties - used for Listing a an attribute of unfiltered items - not in use))

Data:

var moduleData = [
     {manufacturer: "SunPower Corp.", productNo: "SPR-M475-COM-MLSD", watts: 475, cells: 72, warranty: 25, degradation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc001", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc001", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc002", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc002.5", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCoDuplicate", productNo: "TE-STc002.5", watts: 430, cells: 71, warranty: 23, degredation: 2, volume: 1, smartModule: true, adder: 5, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc003", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
    ]

Function:

const filters = {}

const getFilteredArray = ((data, filters, isItUnique, prop  ) => {
//Arrays

    // filtered Array =======================================
    if ((prop.length > 0 || prop != undefined || prop != null) && (prop.length == 0 || prop == undefined || prop == null)){ 
        let nonUniqueArray = data.filter(p => 
            filters.every(f=> Object.keys(f).every(k => p[k] === f[k]))
        )
        console.log('filtered Array - (unique, no prop) ')
        
        var unique = []
        //XXXXXXXXXXXXX make array unique XXXXXXXXXXXXXXXXX
        
            const uniqueArray = nonUniqueArray.filter(element => {
                const isDuplicate = unique.includes(element);

                if (!isDuplicate) {
                    unique.push(element)
            
                    return true
                }
            })

        return uniqueArray
    }
})


filters.available = true
filters.manufacturer = "TestCo"


//console.log(filters)
console.log(getFilteredArray(moduleData, [filters], "unique", "" ))

So I'm looking for products with a manufacturer of TestCo, that's available and looking to remove duplicates.

ITS RETURNING:

Console:[Object, Object, Object, Object, Object] (5)<br>
0 {manufacturer: "TestCo", productNo: "TE-STc001", watts: 475, cells: 72, warranty: 25, …}<br>
1 {manufacturer: "TestCo", productNo: "TE-STc001", watts: 475, cells: 72, warranty: 25, …}<br>
2 {manufacturer: "TestCo", productNo: "TE-STc002", watts: 475, cells: 72, warranty: 25, …}<br>
3 {manufacturer: "TestCo", productNo: "TE-STc002.5", watts: 475, cells: 72, warranty: 25, …}<br>
4 {manufacturer: "TestCo", productNo: "TE-STc003", watts: 475, cells: 72, warranty: 25, …}<br>

Its clear you can see prod No TE-STc001 twice. (not removed). Being a complete copy, I would think one of them wouldn't be added to the uniqueArray. Why is that and how can I fix my code? Thanks in advance!

Answer 1

Depending on your use case you can remove duplicate objects from your array in several ways.

1.) Filter duplicates by one object property

const uniqueData = moduleData.filter((v,i,a) => a.findIndex(v2 => (v2.productNo === v.productNo)) === i)

2.) Filter by several properties

const uniqueData = moduleData.filter((v,i,a) => a.findIndex(v2 => ['productNo','manufacturer'].every(k => v2[k] === v[k])) === i)

3.) Check all properties and filter duplicates (if you know the quality of your data this is likely overkill)

const uniqueData = moduleData.filter((v,i,a) => a.findIndex(v2 => [...Object.keys(moduleData[0])].every(k => v2[k] === v[k])) === i)

In the second half of your solution instead of looking again for duplicates within the list after you create an array of object properties for your dropdown values the way you are, you can easily filter out duplicates by simply using the Set constructor and the spread syntax

doubleUniquePropList = [...new Set(uniquePropList)]

Answer 2

So, I found the solution:

It seemed to make the difference to stringify the object and then compare. And I took the advice of jFriend00 and removed duplicates first before filtering.

The solution:

// filtered Array =======================================
    if ((prop.length > 0 || prop != undefined || prop != null) && (prop.length == 0 || prop == undefined || prop == null)){ 
        console.log('filtered Array - (unique, no prop) ')
        
        //Make data unique
        const uniqueArray = data.filter((object,index) => index === data.findIndex(obj => JSON.stringify(obj) === JSON.stringify(object)));
            
        return uniqueArray
    }

... and that made it not all that difficult then to provide filtered Lists as well. (for dropdowns.)

// filtered List =======================================
    if ((prop.length > 0 || prop != undefined || prop != null) && (prop.length !== 0 || prop.length !== undefined || prop.length !== null)){ 
        console.log('filtered List - (unique, prop) ')

        //Make data unique
        const uniqueArray = data.filter((object,index) => index === data.findIndex(obj => JSON.stringify(obj) === JSON.stringify(object)));

        //filter the list for matches
        let UniqueArray = uniqueArray.filter(p => 
            filters.every(f=> Object.keys(f).every(k => p[k] === f[k]))
        )

        //map Array into the desired list
        let uniquePropList = UniqueArray.map(item => item[prop])

        //look again for duplicates within the list
        var unique = []
        let doubleUniquePropList = uniquePropList.filter(element => {
            const isDuplicate = unique.includes(element);

            if (!isDuplicate) {
                unique.push(element)
        
                return true
            }
        })

        return doubleUniquePropList
    }

Now I can use this function for many different use cases! Thank you guys for your help! and I hope this can help someone else!