Insecure television host listens in on private conversations in the office. How can I mess with him without risking my job?

First job working in the industry and my boss is.. phew.. something else. I've already been confronted twice about things I've said behind closed doors. I know it's not illegal but it's genuinely creepy and shows a deep level of insecurity.