[utf-8] Force encode from US-ASCII to UTF-8 (iconv)

Here's a script that will find all files matching a pattern you pass it, and then converting them from their current file encoding to UTF-8. If the encoding is US ASCII, then it will still show as US ASCII, since that is a subset of UTF-8.

#!/usr/bin/env bash
find . -name "${1}" |
    while read line;
    do
        echo "***************************"
        echo "Converting ${line}"

        encoding=$(file -b --mime-encoding ${line})
        echo "Found Encoding: ${encoding}"

        iconv -f "${encoding}" -t "utf-8" ${line} -o ${line}.tmp
        mv ${line}.tmp ${line}
    done

Examples related to utf-8

error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte Changing PowerShell's default output encoding to UTF-8 'Malformed UTF-8 characters, possibly incorrectly encoded' in Laravel Encoding Error in Panda read_csv Using Javascript's atob to decode base64 doesn't properly decode utf-8 strings What is the difference between utf8mb4 and utf8 charsets in MySQL? what is <meta charset="utf-8">? Pandas df.to_csv("file.csv" encode="utf-8") still gives trash characters for minus sign UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 23: ordinal not in range(128) Android Studio : unmappable character for encoding UTF-8

Examples related to character-encoding

Changing PowerShell's default output encoding to UTF-8 JsonParseException : Illegal unquoted character ((CTRL-CHAR, code 10) Change the encoding of a file in Visual Studio Code What is the difference between utf8mb4 and utf8 charsets in MySQL? How to open html file? All inclusive Charset to avoid "java.nio.charset.MalformedInputException: Input length = 1"? UTF-8 output from PowerShell ERROR 1115 (42000): Unknown character set: 'utf8mb4' "for line in..." results in UnicodeDecodeError: 'utf-8' codec can't decode byte How to make php display \t \n as tab and new line instead of characters

Examples related to iconv

Force encode from US-ASCII to UTF-8 (iconv) iconv - Detected an illegal character in input string How to write file in UTF-8 format? How do I remove accents from characters in a PHP string?