I agree with him - the suspicion would fall on you. Even if Ema-dono is not a detective, she and Robin-dono are both determined to find out who are spies... For that reason we should avoid voting for Hisoka-shi as well, as our companion had initially tried to get him voted off.