[sql] Removing duplicates from a SQL query (not just "use distinct")

It's probably simple, here is my query:

SELECT DISTINCT U.NAME, P.PIC_ID
FROM USERS U, PICTURES P, POSTINGS P1
WHERE U.EMAIL_ID = P1.EMAIL_ID AND P1.PIC_ID = P.PIC_ID AND P.CAPTION LIKE '%car%';

but this will only remove duplicates where a row has both the same u.name and p.pic_id. I want it so if there is any duplicates of the names, it just leaves out the other rows. It's a weird query, but in general, how can I apply the distinct to a single column of the SELECT clause?

This question is related to sql duplicates distinct

The answer is


You need to tell the query what value to pick for the other columns, MIN or MAX seem like suitable choices.

 SELECT
   U.NAME, MIN(P.PIC_ID)
 FROM
   USERS U,
   PICTURES P,
   POSTINGS P1
 WHERE
   U.EMAIL_ID = P1.EMAIL_ID AND
   P1.PIC_ID = P.PIC_ID AND
   P.CAPTION LIKE '%car%'
 GROUP BY
   U.NAME;

If I understand you correctly, you want to list to exclude duplicates on one column only, inner join to a sub-select

select u.* [whatever joined values]
from users u
inner join
(select name from users group by name having count(*)=1) uniquenames
on uniquenames.name = u.name

If I understand you correctly, you want a list of all pictures with the same name (and their different ids) such that their name occurs more than once in the table. I think this will do the trick:

SELECT U.NAME, P.PIC_ID
FROM USERS U, PICTURES P, POSTINGS P1
WHERE U.EMAIL_ID = P1.EMAIL_ID AND P1.PIC_ID = P.PIC_ID AND U.Name IN (
SELECT U.Name 
FROM USERS U, PICTURES P, POSTINGS P1
WHERE U.EMAIL_ID = P1.EMAIL_ID AND P1.PIC_ID = P.PIC_ID AND P.CAPTION LIKE '%car%';
GROUP BY U.Name HAVING COUNT(U.Name) > 1)

I haven't executed it, so there may be a syntax error or two there.


Your question is kind of confusing; do you want to show only one row per user, or do you want to show a row per picture but suppress repeating values in the U.NAME field? I think you want the second; if not there are plenty of answers for the first.

Whether to display repeating values is display logic, which SQL wasn't really designed for. You can use a cursor in a loop to process the results row-by-row, but you will lose a lot of performance. If you have a "smart" frontend language like a .NET language or Java, whatever construction you put this data into can be cheaply manipulated to suppress repeating values before finally displaying it in the UI.

If you're using Microsoft SQL Server, and the transformation HAS to be done at the data layer, you may consider using a CTE (Computed Table Expression) to hold the initial query, then select values from each row of the CTE based on whether the columns in the previous row hold the same data. It'll be more performant than the cursor, but it'll be kinda messy either way. Observe:

USING CTE (Row, Name, PicID)
AS
(
    SELECT ROW_NUMBER() OVER (ORDER BY U.NAME, P.PIC_ID),
       U.NAME, P.PIC_ID
    FROM USERS U
        INNER JOIN POSTINGS P1
            ON U.EMAIL_ID = P1.EMAIL_ID
        INNER JOIN PICTURES P
            ON P1.PIC_ID = P.PIC_ID
    WHERE P.CAPTION LIKE '%car%'
    ORDER BY U.NAME, P.PIC_ID 
)
SELECT
    CASE WHEN current.Name == previous.Name THEN '' ELSE current.Name END,
    current.PicID
FROM CTE current
LEFT OUTER JOIN CTE previous
   ON current.Row = previous.Row + 1
ORDER BY current.Row

The above sample is TSQL-specific; it is not guaranteed to work in any other DBPL like PL/SQL, but I think most of the enterprise-level SQL engines have something similar.


Examples related to sql

Passing multiple values for same variable in stored procedure SQL permissions for roles Generic XSLT Search and Replace template Access And/Or exclusions Pyspark: Filter dataframe based on multiple conditions Subtracting 1 day from a timestamp date PYODBC--Data source name not found and no default driver specified select rows in sql with latest date for each ID repeated multiple times ALTER TABLE DROP COLUMN failed because one or more objects access this column Create Local SQL Server database

Examples related to duplicates

Remove duplicates from dataframe, based on two columns A,B, keeping row with max value in another column C Remove duplicates from a dataframe in PySpark How to "select distinct" across multiple data frame columns in pandas? How to find duplicate records in PostgreSQL Drop all duplicate rows across multiple columns in Python Pandas Left Join without duplicate rows from left table Finding duplicate integers in an array and display how many times they occurred How do I use SELECT GROUP BY in DataTable.Select(Expression)? How to delete duplicate rows in SQL Server? Python copy files to a new directory and rename if file name already exists

Examples related to distinct

Using DISTINCT along with GROUP BY in SQL Server How to "select distinct" across multiple data frame columns in pandas? Laravel Eloquent - distinct() and count() not working properly together SQL - select distinct only on one column SQL: Group by minimum value in one field while selecting distinct rows Count distinct value pairs in multiple columns in SQL sql query distinct with Row_Number Eliminating duplicate values based on only one column of the table MongoDB distinct aggregation Pandas count(distinct) equivalent