Checking If A List Has Duplicate Lists

Given a list of lists, I want to make sure that there are no two lists that have the same values and order. For instance with my_list = [[1, 2, 4, 6, 10], [12, 33, 81, 95, 110], [1

Solution 1:

you could count the occurrences in a list comprehension, converting them to a tuple so you can hash & apply unicity:

routes = [[1, 2, 4, 6, 10], [1, 3, 8, 9, 10], [1, 2, 4, 6, 10]]
dups = {tuple(x) for x in routes if routes.count(x)>1}

print(dups)

result:

{(1, 2, 4, 6, 10)}

Simple enough, but a lot of looping under the hood because of repeated calls to count. There's another way, which involves hashing but has a lower complexity would be to use collections.Counter:

from collections import Counter

routes = [[1, 2, 4, 6, 10], [1, 3, 8, 9, 10], [1, 2, 4, 6, 10]]

c = Counter(map(tuple,routes))
dups = [k for k,v in c.items() if v>1]

print(dups)

Result:

[(1, 2, 4, 6, 10)]

(Just count the tuple-converted sublists - fixing the hashing issue -, and generate dup list using list comprehension, keeping only items which appear more than once)

Now, if you just want to detect that there are some duplicate lists (without printing them) you could

• convert the list of lists to a list of tuples so you can hash them in a set
• compare the length of the list vs the length of the set:

len is different if there are some duplicates:

routes_tuple = [tuple(x) for x in routes]    
print(len(routes_tuple)!=len(set(routes_tuple)))

or, being able to use map in Python 3 is rare enough to be mentionned so:

print(len(set(map(tuple,routes))) != len(routes))

Solution 2:

routes = [[1, 2, 4, 6, 10], [1, 3, 8, 9, 10], [1, 2, 4, 6, 10]]
dups = set()

for route in routes:
    if tuple(route) in dups:
        print('%s is a duplicate route' % route)
    else:
        dups.add(tuple(route))

Solution 3:

Not sure if you want an external library but I have one that contains a function explicitly made for this purpose: iteration_utilities.duplicates

>>> from iteration_utilities import duplicates

>>> my_list = [[1, 2, 4, 6, 10], [12, 33, 81, 95, 110], [1, 2, 4, 6, 10]]

>>> list(duplicates(my_list, key=tuple))
[[1, 2, 4, 6, 10]]

Note that this also works without key=tuple but that will have O(n*n) behaviour instead of O(n).

>>> list(duplicates(my_list))
[[1, 2, 4, 6, 10]]

It also keeps the order of appearance (with or without a key) if that's important:

>>> list(duplicates([[1], [2], [3], [1], [2], [3]]))
[[1], [2], [3]]

---

If you're only interested if there are duplicates you could use any on it instead of list:

>>> any(duplicates([[1], [2], [3], [1], [2], [3]]))
True
>>> any(duplicates([[1], [2], [3]]))
False

Solution 4:

for x in routes:

    print x, routes.count(x)

that will return you each list and how many times it appears. alternativaly you can only show if they appear > 1:

new_list= []

for x in routes:ifroutes.count(x)>1:if x not in new_list:new_list.append(x)for x in new_list:printx,routes.count(x)

hope it helps!

Solution 5:

defduplicate(lst):
    cntrin=0
    cntrout=0for i in lst:
        cntrin=0for k in lst:
            if i==k:
                cntrin=cntrin+1if cntrin>1:
            cntrout=cntrout+1if cntrout>0:
        returnTrueelse:
        returnFalse

Enjoy!