Look at it this way: you're there to record the day's events. As soon as you start "directing", you're no longer recording but dictating the day.
On the hand, photographer's job (even the so called "journalistic" photographers) is to stage manage shots. They are, by their very nature, staged for the camera.
Generally speaking you'll find the photographer will pop up at all the crucial moments. He or she won't consider getting in the way of your shots, but as long as you've chatted to the photographer and stressed what's crucial for your video, you'll be fine.
Key moments that I see as the photographer's domain are:
Arrival of the Bride
Signing of the Registry
Photo Shoot after the Ceromony
Photo Shoot of the B&G before the Speeches
Cutting of the Cake
I'd leave the photographer be here and work around them. But for the other parts of the day, it's YOUR turn to hog the best spots

.