Nope, you are incorrect. There is nothing wrong with the sentence. The subject of the sentence is "you" (implied) -- you "leave the message." Examine the clause "with whoever answers the phone." The verb is "answers." What word is doing the answering? It is "whoever" -- that is the subject. Therefore, the subject form -- whoever -- should be used, not the object form.