Unable to process strings in files with different encoding [Not accepting answers]

Question

I am trying to process the strings present in a particular file, The file is written in English. The problem arises when the encoding of the file differs from "UTF-8". But the file with encoding as "UTF-16-le" does not behave as expected. My main goal is manipulate the strings from the read file. For example the strings.TrimSpace() only works with the UTF-8 file,

I am aware that golang only supports UTF-8 by default, Any alternate approach would be helpful.

Personal Question

Also I would like to point out, many new programming languages, do process the strings irrespective of the encoding, And why does Go only support UTF-8. If at least there would be an alternative way to pass the encoding format to the reader, that might still help.

What I tried

1. I tried using utf-8 and utf-16 std packages

Code

(main.go)

sample code to show the difference.

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "strings"
)

func processFile(src string) {
    data, _ := ioutil.ReadFile(src)

    fmt.Println("--- original source ---")
    fmt.Println(string(data))
    fmt.Println(http.DetectContentType(data))

    fmt.Println("\n--- modified source ---")
    for _, val := range strings.Split(string(data), "\n") {
        fmt.Println(strings.TrimSpace(val))
    }
}

func main() {
    processFile("./utf-16-english.txt")
    processFile("./utf-8-english.txt")
}

File-1

(utf-8-english.txt)

Hello
    This is
        Sample
            Document

File-2

(utf-16-english.txt)

Hello
    This is
        Sample
            Document

EDIT

1. Seems that the only way to process strings in a better way is to convert them to UTF-8. Kindly refer the marked answer.

2. As per comments I have written the result from the program to respective files. And the special symbols are not present, but the process with strings, works fine with UTF-8

Answer 1

You have to decode the utf-16 encoded file. The decoding will convert the input to utf-8, after which you can use the string libraries to process the input.

You can use something like this:

import "unicode/utf16"

func processFile(src string, decode func(in[]byte) string) {
    data, _ := ioutil.ReadFile(src)

    fmt.Println("--- original source ---")
    fmt.Println(decode(data))

    fmt.Println("\n--- modified source ---")
    for _, val := range strings.Split(decode(data), "\n") {
        fmt.Println(strings.TrimSpace(val))
    }
}

func main() {
    processFile("./utf-16-english.txt",func(in []byte) string {
        return string(utf16.Decode(in)) })
    processFile("./utf-8-english.txt",func(in []byte) string {
         return string(in)})
}