When I started messing around, I found the culprit to be B frames - when I reduced them to 0 it works. CABAC also seems to not be supported.
However, B frames are useful to anime encodes and CABAC to any encode, so I had to up my bitrate a tiny bit to compensate.
Using Handbrake I'm doing a 480x272 (for widescreen - 432x320 was the calculated size for full screen items)
I do a H-264 2-pass encode (turbo first pass) @ 375 KBps
The H-264 settings are mostly default, but I do change the Motion Estimation to Uneven Multi-Hexagon - if your PC has a lot of CPU this won't be noticeable, but with a slower computer, this option can make encodes take longer. If you remove that, you probably will want to up your bit-rate a little.
Here's the full setting from Handbrake's H-264 tab:
level=30:bframes=0:weightp=0:cabac=0:vbv-maxrate=1500:vbv-bufsize=2000:analyse=all:me=umh:no-fast-pskip=1
sy-rd=0,0:subq=6:8x8dct=0:trellis=0