How to Remove Duplicate Elements from a Python List

A list is a fundamental Python data structure that holds an ordered collection of items such as integers, strings, or custom objects. It’s the Python equivalent of an array in other languages.

Below are proven techniques to remove duplicate elements from a list while preserving order and readability.

• Remove duplicates using a set
• Remove duplicates with a temporary list
• Remove duplicates using OrderedDict (or dict.fromkeys)
• Remove duplicates with a simple for‑loop
• Remove duplicates via list comprehension
• Remove duplicates with NumPy’s unique()
• Remove duplicates using Pandas’ unique()
• Remove duplicates with enumerate() and list comprehension

Remove duplicates using a set

Python’s built‑in set() function eliminates duplicates by returning only unique elements. Convert the list to a set, then back to a list to preserve the collection format.

my_list = [1,1,2,3,2,2,4,5,6,2,1]
my_final_list = set(my_list)
print(list(my_final_list))

Output:

[1, 2, 3, 4, 5, 6]

Remove duplicates using a temporary list

This approach creates an empty list, iterates over the original list, and appends only items that haven’t been seen before.

my_list = [1, 2, 3, 1, 2, 4, 5, 4 ,6, 2]
print("List Before ", my_list)
temp_list = []

for i in my_list:
    if i not in temp_list:
        temp_list.append(i)

my_list = temp_list
print("List After removing duplicates ", my_list)

Output:

List Before  [1, 2, 3, 1, 2, 4, 5, 4, 6, 2]
List After removing duplicates  [1, 2, 3, 4, 5, 6]

Remove duplicates using OrderedDict (or dict.fromkeys)

For Python 3.5+, a plain dict preserves insertion order, so dict.fromkeys() can be used to keep unique values in order.

my_list = ['a','x','a','y','a','b','b','c']
my_final_list = dict.fromkeys(my_list)
print(list(my_final_list))

Output:

['a', 'x', 'y', 'b', 'c']

Remove duplicates with a simple for‑loop

Traverse the list, adding items to a new list only if they haven’t appeared before.

my_list = [1,2,2,3,1,4,5,1,2,6]
myFinallist = []
for i in my_list:
    if i not in myFinallist:
        myFinallist.append(i)
print(myFinallist)

Output:

[1, 2, 3, 4, 5, 6]

Remove duplicates via list comprehension

List comprehensions offer a concise syntax while maintaining readability.

my_list = [1,2,2,3,1,4,5,1,2,6]
my_finallist = []
[my_finallist.append(n) for n in my_list if n not in my_finallist]
print(my_finallist)

Output:

[1, 2, 3, 4, 5, 6]

Remove duplicates with NumPy’s unique()

NumPy’s unique() function returns the sorted unique elements of an array.

import numpy as np
my_list = [1,2,2,3,1,4,5,1,2,6]
myFinalList = np.unique(my_list).tolist()
print(myFinalList)

Output:

[1, 2, 3, 4, 5, 6]

Remove duplicates using Pandas’ unique()

Similarly, Pandas offers a lightweight unique() method that works on list-like objects.

import pandas as pd
my_list = [1,2,2,3,1,4,5,1,2,6]
myFinalList = pd.unique(my_list).tolist()
print(myFinalList)

Output:

[1, 2, 3, 4, 5, 6]

Remove duplicates with enumerate() and list comprehension

Combining enumerate() with a list comprehension allows removal of duplicates while preserving the first occurrence of each element.

my_list = [1,2,2,3,1,4,5,1,2,6]
my_finallist = [i for j, i in enumerate(my_list) if i not in my_list[:j]]
print(my_finallist)

Output:

[1, 2, 3, 4, 5, 6]

Summary

• Using set() is the quickest way to drop duplicates, though it doesn’t preserve order.
• Temporary lists or for‑loops keep the original order but are slightly less concise.
• Ordered dictionaries or dict.fromkeys() preserve order and are very efficient.
• NumPy and Pandas provide one‑liner solutions for numeric or DataFrame‑heavy workflows.
• List comprehensions with enumerate() offer a functional style that is both readable and order‑preserving.