Ansible: How to get duplicate items from list?

Question

How can I get duplicated items in ansible

Input:

- vars: 
    list1: 
      - a
      - b
      - c
      - d
      - d
      - e
      - e
      - e

Expected output:

list1: 
  - d
  - e

Answer 1

Looping in ansible throught large data lists can be very slow. It is much faster to create a custom_filter.

In your project root create folder filter_plugins. Inside the folder create a python file (custom_filter.py for example) Then use the following code:

#!/usr/bin/python

class FilterModule(object):
    def filters(self):
        return {'duplicates': self.duplicates}

    def duplicates(self, items):
        sums = {}
        result = []

        for item in items:
            if item not in sums:
                sums[item] = 1
            else:
                if sums[item] == 1:
                    result.append(item)
                sums[item] += 1
        return result

Then call the custom filter in your playbook:

    - name: "debug"
      debug:
        msg: "{{ [1, 2, 2, 4, 5, 1] | duplicates }}" 

ok: [localhost] => {
    "msg": [
        2,
        1
    ]
}

When you are processing data it is usually better to use custom filters.