COMET CENTRAL

I have a 65 Mercury Comet Caliente. Power brakes with single master cylinder with drum brakes all around. Far as I know it is a stock system. 289 automatic. My problem is that the power brakes are super touchy and then at other times it will feel like I don't have power brakes at all. It stops but you have to push fairly hard on the pedal. Most of the time though I have to very gently push on the brake. Any ideas? Thank you.

Seeing as the power assist is produced by vacuum from the motor I'd be checking the vacuum line, maybe pull it out and clear any possible obstruction, check for cracks.
I don't know how to check the diaphram.
Fred