Removing all keybindings as you suggested in your second patch is not an option, as this would make some keys, e.g. Home and Delete, unusable (zsh doesn't recognize them correctly by default).
Removing the other keybindings may cause problems, because I don't know how zsh determines which sequence the key sends (here this is done via the terminfo database).
So I would suggest to change the keybindings that are annoying and differ from what you would expect (e.g. k in vicmd mode is bound to up-line-or-history and arrow up in viins mode is bound to vi-up-line-or-history).
[[ -z "$terminfo[kcuu1]" ]] || bindkey -M viins "$terminfo[kcuu1]" vi-up-line-or-history
should be replaced with:
[[ -z "$terminfo[kcuu1]" ]] || bindkey -M viins "$terminfo[kcuu1]" up-line-or-history
and similar for the other Arrow keys.