I have the same problem with my AK7802 netbook, even using the excellent TCPMP. I suspect the problem lies in the fact that the AK7802 doesn’t have a Digital Signal Processor (DSP) and only runs at 248Mhz.
I have a Maplin MID that uses a Telechips 480Mhz CPU + 480Mhz DSP and TCPMP performs very well indeed viewing the same files that the Anyka can’t.
You may be able to resample your files to reduce the demand on the CPU to make them play more smoothly but the quality will, no doubt, be reduced to.
I’d probably stick with AVI files and experiment to find a sample rate that the AK7802 can cope with.
Video manipulation is not my area of expertise, I’m afraid.