[arrays] Removing duplicate values from a PowerShell array

How can I remove duplicates from a PowerShell array?

$a = @(1,2,3,4,5,5,6,7,8,9,0,0)

With my method you can completely remove duplicate values, leaving you with values from the array that only had a count of 1. It was not clear if this is what the OP actually wanted however I was unable to find an example of this solution online so here it is.

$array=@'
Bananna
Apple
Carrot
Pear
Apricot
Pear
Bananna
'@ -split '\r\n'

($array | Group-Object -NoElement | ?{$_.count -eq 1}).Name

This is how you get unique from an array with two or more properties. The sort is vital and the key to getting it to work correctly. Otherwise you just get one item returned.

PowerShell Script:

$objects = @(
    [PSCustomObject] @{ Message = "1"; MachineName = "1" }
    [PSCustomObject] @{ Message = "2"; MachineName = "1" }
    [PSCustomObject] @{ Message = "3"; MachineName = "1" }
    [PSCustomObject] @{ Message = "4"; MachineName = "1" }
    [PSCustomObject] @{ Message = "5"; MachineName = "1" }
    [PSCustomObject] @{ Message = "1"; MachineName = "2" }
    [PSCustomObject] @{ Message = "2"; MachineName = "2" }
    [PSCustomObject] @{ Message = "3"; MachineName = "2" }
    [PSCustomObject] @{ Message = "4"; MachineName = "2" }
    [PSCustomObject] @{ Message = "5"; MachineName = "2" }
    [PSCustomObject] @{ Message = "1"; MachineName = "1" }
    [PSCustomObject] @{ Message = "2"; MachineName = "1" }
    [PSCustomObject] @{ Message = "3"; MachineName = "1" }
    [PSCustomObject] @{ Message = "4"; MachineName = "1" }
    [PSCustomObject] @{ Message = "5"; MachineName = "1" }
    [PSCustomObject] @{ Message = "1"; MachineName = "2" }
    [PSCustomObject] @{ Message = "2"; MachineName = "2" }
    [PSCustomObject] @{ Message = "3"; MachineName = "2" }
    [PSCustomObject] @{ Message = "4"; MachineName = "2" }
    [PSCustomObject] @{ Message = "5"; MachineName = "2" }
)

Write-Host "Sorted on both properties with -Unique" -ForegroundColor Yellow
$objects | Sort-Object -Property Message,MachineName -Unique | Out-Host

Write-Host "Sorted on just Message with -Unique" -ForegroundColor Yellow
$objects | Sort-Object -Property Message -Unique | Out-Host

Write-Host "Sorted on just MachineName with -Unique" -ForegroundColor Yellow
$objects | Sort-Object -Property MachineName -Unique | Out-Host

Output:

Sorted on both properties with -Unique

Message MachineName
------- -----------
1       1          
1       2          
2       1          
2       2          
3       1          
3       2          
4       1          
4       2          
5       1          
5       2          


Sorted on just Message with -Unique

Message MachineName
------- -----------
1       1          
2       1          
3       1          
4       1          
5       2          


Sorted on just MachineName with -Unique

Message MachineName
------- -----------
1       1          
3       2  

Source: https://powershell.org/forums/topic/need-to-unique-based-on-multiple-properties/

Whether you're using SORT -UNIQUE, SELECT -UNIQUE or GET-UNIQUE from Powershell 2.0 to 5.1, all the examples given are on single Column arrays. I have yet to get this to function across Arrays with multiple Columns to REMOVE Duplicate Rows to leave single occurrences of a Row across said Columns, or develop an alternative script solution. Instead these cmdlets have only returned Rows in an Array that occurred ONCE with singular occurrence and dumped everything that had a duplicate. Typically I have to Remove Duplicates manually from the final CSV output in Excel to finish the report, but sometimes I would like to continue working with said data within Powershell after removing the duplicates.

In case you want to be fully bomb prove, this is what I would advice:

@('Apples', 'Apples ', 'APPLES', 'Banana') | 
    Sort-Object -Property @{Expression={$_.Trim()}} -Unique

Output:

Apples
Banana

This uses the Property parameter to first Trim() the strings, so extra spaces are removed and then selects only the -Unique values.

More info on Sort-Object:

Get-Help Sort-Object -ShowWindow

To get unique items from an array and preserve their order, you can use .NET HashSet:

$Array = @(1, 3, 1, 2)
$Set = New-Object -TypeName 'System.Collections.Generic.HashSet[int]' -ArgumentList (,[int[]]$Array)

# PS>$Set
# 1
# 3
# 2

Works best with string arrays that contain both uppercase and lowercase items where you need to preserve first occurrence of each item in case-insensitive manner:

$Array = @("B", "b", "a", "A")
$Set = New-Object -TypeName 'System.Collections.Generic.HashSet[string]' -ArgumentList ([string[]]$Array, [StringComparer]::OrdinalIgnoreCase)

# PS>$Set
# B
# a

Works as expected with other types.

Shortened syntax, compatible with PowerShell 5.1 and newer:

$Array = @("B", "b", "a", "A")
$Set = [Collections.Generic.HashSet[string]]::new([string[]]$Array, [StringComparer]::OrdinalIgnoreCase)

$Array = @(1, 3, 1, 2)
$Set = [Collections.Generic.HashSet[int]]::new([int[]]$Array)

If the list is sorted, you can use the Get-Unique cmdlet:

 $a | Get-Unique

Another option is to use Sort-Object (whose alias is sort, but only on Windows) with the
-Unique switch, which combines sorting with removal of duplicates:

$a | sort -unique

$a | sort -unique

This works with case-insensitive, therefore removing duplicates strings with differing cases. Solved my problem.

$ServerList = @(
    "FS3",
    "HQ2",
    "hq2"
) | sort -Unique

$ServerList

The above outputs:

FS3
HQ2