I recently ran a survey at work using FluidSurveys. Their survey building tools are excellent, and they have great support, but I ran into a time consuming issue when it came time to process the responses because they’re double byte unicode, UTF-16LE to be specific. Turns out knowing that is 90% of the battle.
The files on first inspection are a bit strange, because although they spring from a csv export button, they’re tab-delimited, but with CSV-style quoting conventions. That’s easy enough to work around, but R and Ruby both barfed reading the files. I cottoned on to the fact that the files had some odd characters in them, so I recruited JRuby and ruby 1.9 to try to load them, due to better unicode support, but still couldn’t quite get the parameters right.
Then I thought of iconv, the character set converting utility. Since in this case, the only special characters was the ellipsis character, I was happy to strip those out, and the following command does the trick:
iconv -f UTF-16LE -t US-ASCII -c responses.csv > converted_responses.csv
And, as they say, Bob’s your uncle
Hey James,
Would love to get your thoughts on how you’d ideally like to get the files… How do you use these files? feel free to drop me a line at my email address and we’ll see what we can do to make your life easier next time around 🙂