Today I’m going to take a look at a error that occurs when you trying to create a BigQuery table in GCP. It doesn’t matter whether you upload a file locally or choose the file from a Google Cloud Storage. You may face the same issue.
Error while reading data, error message: CSV table encountered too many errors, giving up. Rows: 1781833; errors: 1. Please look into the errors[] collection for more details. (error code: invalid)
Normally it will come with a link as well. But since the error code is invalid, it wont tell much.
The issue is actually the data encoding. BigQuery only support UTF-8 at the moment. So any other file type will not be processed. And that is the reason why the error does not tell much. Because the file actually hasn’t reach the point to process.
In term of fixing the problem. You should firstly download the file from GCS. And then check the file’s encoding. If you are using mac, just run the following command:
# file -I dec18.csv dec18.csv: text/plain; charset=us-ascii
You may get other charset, but as long as you don’t get utf-8. You need to run the fix.
iconv -f us-ascii -t utf-8 dec18.csv > dec18_utf8.csv
If you are using Windows, just use your favorite code editor open the csv. And save as encoding utf-8.
After the change, just reload the table. It should just work.
