How to make R to read the date consistently

Question

The way that I use R to convert the a column to date format in a csv file is

df = read.csv(file)    
df %>% mutate(date_column = as.Date(as.character(date_column), format = "%m/%d/%Y"))) #or "%Y-%m-%d"

But after read the csv files, I found myself consistently need to change the format between "%m/%d/%Y" and "%Y-%m-%d" for different files. The NA will show in the date column if the format does not "match". But when I opened the files that work with "%m/%d/%Y" and the files that work with %Y-%m-%d", they all seem to have same format to me (month/day/year), so I just don't understand why I need to constantly change the format so that the date column will be read correctly.

Also, there are times that the same file was working fine with a format but the next time I read the file I have to switch the other format so that the date will read properly. I just wonder if there is a way I can keep format consistently? or is there anything I can do before the as.Date() so that I do not need to change the format all the time.