Max M. answered 07/14/19
Harvard Literature major with 20 years of coaching writers
A couple of things.
First of all, any questions about what characters in Shakespeare's plays, or really any literary work, should do differently are not going to have absolute, definitive answers. Only in the occasional Enlightenment-Era work are people going to behave like totally rational, coldly calculating actors, and even then, it's as a thought experiment. Otherwise, any decent work of literature is going to have characters who are flawed, and who make all kinds of poor decisions, much like regular people. As Hamlet himself says, "the purpose of playing...is to hold, as 'twere, the mirror up to nature." So the audience can learn what they can from the characters. If you were in Claudius's court, would you have figured out what the play was really about? Maybe so. Then what? What would you expect, say, Osric (who's probably around for it) to do? Jump up and denounce Hamlet? Denounce Claudius? Gasp and brace for the worst? Who's to say he doesn't notice? Maybe some people do and say nothing. Why might they hold their tongues? The best possible answer I can give to your question is to say, let questions like that deepen your experience of the play. Don't look at them as flaws, use them to explore what's given.
That said, Elizabethan drama, broadly speaking, was different from modern drama in a lot of ways. One of those is that it's only one evolutionary step from medieval English drama. Shakespeare and friends are taking the first steps towards more complex, sophisticated stories. What this has to do with your question is the way they handle lying, deception, and disguises. In a nutshell, they didn't want to run the risk of confusing their audience, so they always made it crystal clear when a character was lying or in disguise. Mission Impossible / M. Night Shyamalan twists just weren't a thing. We don't meet Cesario and get surprised when we find out he's a lady; we meet Viola and she tells us that she's going to disguise herself as a man, and we see her do it. Ditto Rosalind / Ganymede. And in Hamlet, we don't see Hamlet saying crazy stuff and then find out it's just an act, he tells us he's going to put on an antic disposition. Contrast that with Ophelia when she shows up mad, no trickery there. There are a lot of causes and effects of this reality, but one big one is that Shakespeare's audience was used to the idea that if a character was going to deceive another, the deception would work. Of all the questions Shakespeare's characters wrestle with, "is this guy for real?" wasn't really one of them. A Claudius who sees the play and thinks, "Hmmm...Hamlet must think he knows something...what kind of switcheroo can I pull on him" just wasn't the kind of thing characters did, or anyway, not without telling the audience. And besides, isn't that how guilty people often act when they're caught? They get mad, not scheming.
Lastly, and perhaps I should have lead with this, but I think the other stuff is really worth discussing, you may be overestimating just how much Hamlet is messing with the play. He didn't add the murder. Remember that he asks the Player if he knows "The Murder of Gonzago." He tells us he's adding a speech at the beginning, and maybe tweaking some of the details, but it's still a play about a murder of a king. By analogy, if Claudius were Richard Nixon, Hamlet might have called for a stage version of The Italian Job, but making the booty political documents instead of gold. Would people have seen what he was up to? Maybe, but The Italian Job is a well-known movie that just happens to be about a break-in, so he does have some legitimate cover.
So back to point one--it's a good question, but it should be one to explore, not just answer.