SaveChangesAsync not updating navigation property

Question

I have users public virtual ICollection<File> Files navigation property. And im trying to remove file

var user = await _unitOfWork.UserRepository
                .GetUserByName(User.FindFirstValue(ClaimTypes.GivenName));
var image = user.Files.FirstOrDefault(x => x.Id == imageId);

user.Image = null;
user.Files.Remove(image);

if (await _unitOfWork.Complete())
{
    return Ok();
}

It's removing it from files but not updating in database. I tried this: await _unitOfWork.UserRepository.UpdateUser(user); and it didn't work

UpdateUser(User user):

_context.Users.Attach(user);
_context.Entry(user).State = EntityState.Modified;

UnitOfWork:

public async Task<bool> Complete()
{
    return await _dbContext.SaveChangesAsync() > 0;
}

It works fine with adding user.Files.Add(image) but i dont understand whats problem with removing