I read in search terms from a simple text file to send to a search engine. It works fine in English, but gives me ???? for any Japanese text. Text with mixed English and Japanese does show the English text, so I know it's reading it.

What I'm seeing:

Input text:
Snow Leopard ???????????????

Turns into:
Snow Leopard ???????????????

This is in my POST field of an HTTP. If I set JMeter to encode the data, it just puts in the percent sequence for question marks.

Interesting note: In the example above there are 15 Japanese characters, and then 15 question marks, so at some point it's being seen as full characters and not just bytes.

About the Data:

The CSV file is very simple in structure.
There's only one field / one column, which I name TERM, and later use as ${TERM}
I don't really need full CSV because it's only one string per line.
There's no commas or quotes.
When I run the Unix "file" command on the file, it says UTF-8 text.
I've also verified it in command line and graphical mode on two machines.

JMeter CSV Dataset Config:

Filename: japanese-searches.csv
File encoding: UTF-8 (also tried without)
Variable names: TERM
Delimiter: ,
Allow Quoted Data: False (I also tried True, different, but still wrong)
Recycle at EOF: True
Stop at EOF: False
Staring mode: All threads

A few things I've tried:

• Tried Allow quoted Data. It changed to other strange characters.
• -Dfile.encoding=UTF-8
• Tried encoding the POST, but it just turned into a bunch of %nn for question marks

And I'm not sure how "debug" just after the each line of the CSV is read in. I think it's corrupted right away, but I'm not sure.

If it's only mangled when I reference it, then instead of ${TERM} perhaps there's some other "to bytes" function call. I'll start checking into that. I haven't done anything with the JMeter functions yet.