Trying to fix line-endings with git filter-branch, but having no luck

Question

I have been bitten by the Windows/Linux line-ending issue with git. It seems, via GitHub, MSysGit, and other sources, that the best solution is to have your local repos set to use linux-style line endings, but set core.autocrlf to true. Unfortunately, I didn't do this early enough, so now every time I pull changes the line endings are borked.

I thought I had found an answer here but I can't get it to work for me. My Linux command line knowledge is limited at best, so i am not even sure what the "xargs fromdos" line does in his script. I keep getting messages about no such file or directory existing, and when I manage to point it to an existing directory, it tells me I don't have permissions.

I've tried this with MSysGit on Windows and via the Mac OS X terminal.

Answer 1

The easiest way to fix this is to make one commit that fixes all the line endings. Assuming that you don't have any modified files, then you can do this as follows.

# From the root of your repository remove everything from the index
git rm --cached -r .

# Change the autocrlf setting of the repository (you may want 
#  to use true on windows):
git config core.autocrlf input

# Re-add all the deleted files to the index
# (You should get lots of messages like:
#   warning: CRLF will be replaced by LF in <file>.)
git diff --cached --name-only -z | xargs -0 git add

# Commit
git commit -m "Fixed crlf issue"

# If you're doing this on a Unix/Mac OSX clone then optionally remove
# the working tree and re-check everything out with the correct line endings.
git ls-files -z | xargs -0 rm
git checkout .

Answer 2

My procedure for dealing with the line endings is as follows (battle tested on many repos):

When creating a new repo:

• put .gitattributes in the very first commit along with other typical files as .gitignore and README.md

When dealing with an existing repo:

• Create / modify .gitattributes accordingly
• git commit -a -m "Modified gitattributes"
• git rm --cached -r . && git reset --hard && git commit -a -m 'Normalize CRLF' -n"
  • -n (--no-verify is to skip pre-commit hooks)
  • I have to do it often enough that I defined it as an alias alias fixCRLF="..."
• repeat the previous command
  • yep, it's voodoo, but generally I have to run the command twice, first time it normalizes some files, second time even more files. Generally it's probably best to repeat until no new commit is created :)
• go back-and-forth between the old (just before normalization) and new branch a few times. After switching the branch, sometimes git will find even more files that need to be renormalized!

In .gitattributes I declare all text files explicitly as having LF EOL since generally Windows tooling is compatible with LF while non-Windows tooling is not compatible with CRLF (even many nodejs command line tools assume LF and hence can change the EOL in your files).

Contents of .gitattributes

My .gitattributes usually looks like:

*.html eol=lf
*.js   eol=lf
*.json eol=lf
*.less eol=lf
*.md   eol=lf
*.svg  eol=lf
*.xml  eol=lf

To figure out what distinct extensions are tracked by git in the current repo, look here

Issues after normalization

Once this is done, there's one more common caveat though.

Say your master is already up-to-date and normalized, and then you checkout outdated-branch. Quite often right after checking out that branch, git marks many files as modified.

The solution is to do a fake commit (git add -A . && git commit -m 'fake commit') and then git rebase master. After the rebase, the fake commit should go away.

Answer 3

git status --short|grep "^ *M"|awk '{print $2}'|xargs fromdos

Explanation:

• git status --short

  This displays each line that git is and is not aware of. Files that are not under git control are marked at the beginning of the line with a '?'. Files that are modified are marked with an M.

• grep "^ *M"

  This filters out only those files that have been modified.

• awk '{print $2}'

  This shows only the filename without any markers.

• xargs fromdos

  This takes the filenames from the previous command and runs them through the utility 'fromdos' to convert the line-endings.

Answer 4

Here's how I fixed all line endings in the entire history using git filter-branch. The ^M character needs to be entered using CTRL-V + CTRL-M. I used dos2unix to convert the files since this automatically skips binary files.

$ git filter-branch --tree-filter 'grep -IUrl "^M" | xargs -I {} dos2unix "{}"'

Answer 5

The "| xargs fromdos" reads from standard input (the files find finds) and uses it as arguments for the command fromdos, which converts the line endings. (Is fromdos standard in those enviroments? I'm used to dos2unix). Note that you can avoid using xargs (especially useful if you have enough files that the argument list is too long for xargs):

find <path, tests...> -exec fromdos '{}' \;

or

find <path, tests...> | while read file; do fromdos $file; done

I'm not totally sure about your error messages. I successfully tested this method. What program is producing each? What files/directories do you not have permissions for? However, here's a stab at guessing what your it might be:

One easy way to get a 'file not found' error for the script is by using a relative path - use an absolute one. Similarly you could get a permissions error if you haven't made your script executable (chmod +x).

Add comments and I'll try and help you work it out!

Answer 6

okay... under cygwin we don't have fromdos easily available, and that awk substeb blows up in your face if you have any spaces in paths to modified files (which we had), so I had to do that somewhat differently:

git status --short | grep "^ *M" | sed 's/^ *M//' | xargs -n 1 dos2unix

kudos to @lloyd for the bulk of this solution

Answer 7

I had the same problem in one of my repos. If you are using both windows and linux systems for the same code repo and pulling and pushing simultaneously, try this:

First, set your git config as follows for windows:

git config --global core.autocrlf true

This will make sure to convert CRLF to LF when writing into the object database and then again replace LF with CRLF when writing out into the working directory. As a result, your repo will be safe with only one type of line endings and locally you'll have windows line ending on the windows system.

For linux/MAC set the git config as follows:

git config --global core.autocrlf input

This will make sure to convert CRLF to LF when writing into the object database but will not do the reverse, preserving LF which is needed for linux/MAC.

For the wrong line endings that are already there on your linux/MAC use dos2unix

For MAC:

brew install dos2unix # Installs dos2unix Mac
find . -type f -exec dos2unix {} \; # recursively removes windows related stuff

For Linux:

sudo apt-get install -y dos2unix # Installs dos2unix Linux
sudo find . -type f -exec dos2unix {} \; # recursively removes windows related stuff

Hope this solves your problem.