The VLC issue might also have to do with the program thinking it has access to hardware decoding capabilities when it really doesn't. On Windows you can switch to GDI video output but I don't think VLC for Mac has that option.

Maybe you can get it to work in an alternative player like Movist, but honestly I think the best solution is just to not watch videos in a VM. VLC for Mac and Windows are basically identical so you might as well watch the videos in your host OS.