Actually there is reference to this being a wrongful act.

Playing with dolls is for children, before menzies and there are passages that suggest you're not supposed to have sex with them.

Ayshas testimony says she had a doll in hand when her mother got her from her swing to present her to the prophet.

So this explicity says that she didnt have her menzies and the consumation shouldnt have taken place. This adds even more to the crime of which muhammad did.