Sargasso of Lost Starships by Poul Anderson

Again that thoughtful stare. She saw a tall young man, badly disarrayed, blood and dirt streaking his long, thin-carved, curve-nosed features, an old scar jagging across his high narrow forehead. The hair was yellow, the eyes were blue, the whole look that of an old and settled aristocracy. His bitter voice lashed at her: "We ruled Ansa well because we were part of it, we grew up with the planet and we understood our folk and men were free under us. That's something which no upstart Solar Empire can have, not for centuries, not ever to judge by the stock they use for nobility. When peasants command spaceships -- "

Her face grew a little pale, but she smiled and replied evenly, "I am the Lady Jansky of Torgandale on Valor -- Sirius A IV -- and you are now a commoner. Please remember that."