#### [SOLVED] Remove both duplicates in multiple lists python

By mb567

I have three lists X, Y, Z as follows:

``````X: [1, 1, 2, 3, 4, 5, 5, 5]
Y: [3, 3, 2, 6, 7, 1, 1, 2]
Z: [0, 0, 1, 1, 2, 3, 3, 4]
``````

I am trying to remove both duplicated set of values at the same index of the lists get a reduced list as follows, all three list will always have the same length initially and at the end as well:

``````X: [2, 3, 4, 5]
Y: [2, 6, 7, 2]
Z: [1, 1, 2, 4]
``````

I tried using the zip(X, Y, Z) function but I can't index it and the dict.fromkeys only removes one of the duplicates and leaves the other in the new list. I want to be able to remove both.

Any help is appreciated!

#### @Laurent H. 2018-06-13 15:20:48

Here is my solution without any import, but still short and easily readable:

``````X = [1, 1, 2, 3, 4, 5, 5, 5]
Y = [3, 3, 2, 6, 7, 1, 1, 2]
Z = [0, 0, 1, 1, 2, 3, 3, 4]

zipped = list(zip(X, Y, Z))
X, Y, Z = zip(*[i for i in zipped if zipped.count(i) == 1])
X, Y, Z = list(X), list(Y), list(Z)

print(X, Y, Z, sep='\n')
# [2, 3, 4, 5]
# [2, 6, 7, 2]
# [1, 1, 2, 4]
``````

#### @jpp 2018-06-13 14:31:02

Using `collections.Counter` and `zip`, you can count unique triplets.

Then remove duplicates via a generator comprehension.

``````from collections import Counter

X = [1, 1, 2, 3, 4, 5, 5, 5]
Y = [3, 3, 2, 6, 7, 1, 1, 2]
Z = [0, 0, 1, 1, 2, 3, 3, 4]

c = Counter(zip(X, Y, Z))

X, Y, Z = zip(*(k for k, v in c.items() if v == 1))

print(X, Y, Z, sep='\n')

(2, 3, 4, 5)
(2, 6, 7, 2)
(1, 1, 2, 4)
``````

Note if ordering is a requirement and you are not using Python 3.6+, you can create an "OrderedCounter" instead by subclassing `collections.OrderedDict`.

#### @jpp 2018-06-13 14:32:40

Downvoter care to comment?

#### @Jared Goguen 2018-06-13 14:39:54

correct me if I'm wrong, but this does not guarantee that the order of the triplets will be preserved - albeit, the triplets themselves will be. I'm not the down-voter, just an observation

#### @jpp 2018-06-13 14:41:43

@JaredGoguen, Good point: in Python 3.6+ it would, not in <3.6. You could construct an OrderedCounter (subclassing `collections.OrderedDict`), but not sure if order is a requirement and OP's Python version.

#### @Jared Goguen 2018-06-13 14:42:59

that is a side effect of the CPython implementation and is not, in general, guaranteed in 3.6. It will however be guaranteed in 3.7

#### @Jared Goguen 2018-06-13 14:44:07

of course, we don't even know if preserving order is required by the OP

#### @shanmuga 2018-06-13 14:38:15

Not the best possible approach

``````>>> from collections import Counter
>>> zipped_items = list(zip(x,y,z))
>>> counts = Counter(zipped_items)
>>> filtered_items = [item for item in zipped_items if counts[item] == 1]
>>> x1, y1, z1 = [ list(map(lambda x: x[i], filtered_items))
... for i in range(3)]
``````

#### @koPytok 2018-06-13 14:31:56

It's convenient to use pandas library for the task. Just create dataframe using the lists and apply `df.drop_duplicates` with `keep=False` (means remove all duplicated rows):

``````import pandas as pd

dct = {
"X": [1, 1, 2, 3, 4, 5, 5, 5],
"Y": [3, 3, 2, 6, 7, 1, 1, 2],
"Z": [0, 0, 1, 1, 2, 3, 3, 4],
}
d = pd.DataFrame(dct)
d.drop_duplicates(keep=False)
``````

### [SOLVED] How to get the number of elements in a list in Python?

• 2009-11-11 00:30:54
• y2k
• 2931678 View
• 1716 Score
• Tags:   python list

### [SOLVED] How do I check if a list is empty?

• 2008-09-10 06:20:11
• Ray Vega
• 2047613 View
• 3074 Score
• Tags:   python list

### [SOLVED] Finding the index of an item given a list containing it in Python

• 2008-10-07 01:39:38
• Eugene M
• 3067287 View
• 2535 Score
• Tags:   python list