A 1980s British government question

In the episode "The Official Visit" of Yes, Minister there is the following exchange between the Permanent Secretary of State for the Ministry for Administrative Affairs (Sir Humphrey) and the Permanent Secretary of State of the Foreign Office (Sir Frederick):

Sir Humphrey Appleby: Can you imagine her old man Millen doing a thing like that?

Sir "Jumbo" Frederick: Yes

Sir Humphrey Appleby: Yes, so can I, actually

Wikipedia is of no help, so I'm left wondering who this "her old man" could be a reference to. It's certainly not Prince Phillip.