invalid byte sequence in UTF-8

This means that your file somehow contains incorrect characters that are not part of the UTF-8 character range. It could mean:

some of the records in the database are corrupted
you have manipulated the export file (for example trying to split it) and damaged one or more characters
you have exported the file in a different encoding than UTF-8
you have opened the file and saved it in a different encoding by mistake (for example in UTF-16 by Notepad)

To find out where exactly is the issue you can use a utility iconv that is easily available (most of the times in the basic package) of any Linux and MacOS X/OS X installation. You can install it on Windows if necessary as well.

If you will try to convert the file to UTF-8 with iconv then it will complain about all the incorrect characters it finds:

iconv -t UTF-8 inputfile.txt

Then you can fix them either in the file or in the database and create the export file once more.

You can also fix these characters by skipping them adding -c parameter

iconv -c -t UTF-8 inputfile.txt > exportfile.txt

Then you can compare both files to see the wrong characters and fix them in the source file.

Browser not supported