Taking permission from the ex-husband when taking the children to different places

I am a divorced woman with two daughters under the age of 9.

I live with my parents and travel with them. Is it necessary to seek permission from my daughter's father everytime I take them out of town. I let him know out of courtesy and I only travel with a mahram.

My children's father demands that I need to ask him before taking his daughters anywhere and when I do, he refuses the permission. He also wants to know exactly where I'm taking them, but I am afraid that he will show up there and harass ME. I know he's doing it just so he can control where I go and know where I am, but I am not his. I just don't want to be doing anything wrong Islamically whilst they're in my custody.