Brooklyn, 26, who lives in the U.S., was not present for the Highlands trip nor was he spotted at David’s lavish birthday bash in London over the weekend