Update ManyToManyField by calling view via ajax("Like" button)

Question

Not sure if this is more of a javascript question or a django question so please forgive me if it is in the wrong spot. I am trying to update a ManyToMany field representing "likes" on an image by calling a view via ajax. The "like" portion is working, the "unlike" portion is not.

I have an Image model with a users_like field:

class Image(models.Model):
    user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name = 'images_created', on_delete=models.CASCADE)
    users_like = models.ManyToManyField(settings.AUTH_USER_MODEL, related_name="images_liked", blank = True)
    # etc

The view:

def image_like(request):
    image_id = request.POST.get('id')
    action = request.POST.get('action')
    if image_id and action:
        try:
            image = Image.objects.get(id = image_id)
            if action == 'like':
                image.users_like.add(request.user)
            else:
                images.users_like.remove(request.user)
            return JsonResponse({'status': 'ok'})
        except:
            pass
    return JsonResponse({'status': 'ko'})

Url:

path('like/', views.image_like, name = 'like'),

html/js:

<a href="#" data-id="{{ image.id }}" data-action="{% if request.user in users_like %}un{% endif %}like" class="like button">
   {% if request.user not in users_like %}
       Like
   {% else %}
      Unlike
   {% endif %}
 </a>
<!-- other html, skipping down to the jQuery-->

{% block domready %}
  $('a.like').click(function(e){
    e.preventDefault();
    $.post('{% url "images:like" %}',
      {
        id: $(this).data('id'),
        action: $(this).data('action')
      },
      function(data){
        if (data['status'] == 'ok')
        {
          var previous_action = $('a.like').data('action');

          // toggle data-action
          $('a.like').data('action', previous_action == 'like' ? 'unlike' : 'like');
          // toggle link text
          $('a.like').text(previous_action == 'like' ? 'Unlike' : 'Like');

          // update total likes
          var previous_likes = parseInt($('span.count .total').text());
          $('span.count .total').text(previous_action == 'like' ? previous_likes + 1 : previous_likes - 1);
        }
      }
    );
  });
{% endblock %}

This code successfully allows a user to 'like' a post and the link subsequently changes to <a href="#" data-id="18" data-action="unlike" class="like button">Unlike </a>, so both the data-action and text are updating properly, but then if I click it again, it doesn't "unlike" the image. Not sure what is going here, as when the data-action attribute is equal to "unlike", the

else:
  images.users_like.remove(request.user)

portion of the view should go into effect, it seems like the view is not being called again but I am not sure. Thanks for any help.