i think it is lors!! Could you write a formal letter of complaint to anyone above your line manager?
Its harrassment and bullying to make someone feel intimadated this way and he knows that he is doing it. Is there noone else that you work with that will stand up to him and deny the accusations and ask for proof? as he probably feels he can do it as you are a woman and wont stand up to him again!