4502e7563b
Contrary to what a comment in wxTextInputStream::GetChar() said, it is actually possible to get more than one wide character from a call to wxMBConv::ToWChar(len+1) even if a previous call to ToWChar(len) failed to decode anything at all. This happens with wxConvAuto because it keeps returning an error while it doesn't have enough data to determine if the input contains a BOM or not, but then returns all the characters examined so far at once if it turns out that there was no BOM, after all. The simplest case in which this created problems was just input starting with a NUL byte as it as this could be a start of UTF-32BE BOM. The fix consists in keeping all the bytes read but not yet decoded in the m_lastBytes buffer and retrying to decode them during the next GetChar() call. This implies keeping track of how much valid data is there in m_lastBytes exactly, as we can't discard the already decoded data immediately, but need to keep it in the buffer too, in order to allow implementing UngetLast(). Incidentally, UngetLast() was totally broken for UTF-16/32 input (containing NUL bytes in the middle of the characters) before and this change fixes this as a side effect. Also add test cases for previously failing inputs. |
||
|---|---|---|
| .. | ||
| bstream.h | ||
| datastreamtest.cpp | ||
| ffilestream.cpp | ||
| fileback.cpp | ||
| filestream.cpp | ||
| iostreams.cpp | ||
| largefile.cpp | ||
| memstream.cpp | ||
| socketstream.cpp | ||
| sstream.cpp | ||
| stdstream.cpp | ||
| tempfile.cpp | ||
| textstreamtest.cpp | ||
| zlibstream.cpp | ||