That isn't quite correct. If you start in Lorule, attach yourself to the wall, leave to Hyrule, you can pop off the wall, merge back and finally void out in Lorule, you will respawn where you first merged with the wall in Lorule.
That is Hyrule -> Lorule without touching the ground, spawn at your entrance into Kakariko in Hyrule
Stand on Lorule -> Hyrule -> Lorule without touching ground, spawn where you merged.
Well, if the game remembers the spot you merged at as the last position you stood on, it'd make sense for it to put you back there if you void out in that same area, wouldn't it?
However, if you were to void out in Lorule but haven't actually stood anywhere in lorule, it would want to send you back to where you last stood in Hyrule, correct?
However, this doesn't seem to be the case. It's the same when you clip through portals and void out.
So I did some testing.
Warp to Link's House, enter and exit house, then portal clip. You'll notice that you respawn at your last position.
Now go down south and then west to the portal which leads you to the desert. Portal Clip. You'll notice that you respawn at the entrance, not where you were standing.
I did the same at Death Mountain Mines, I did a portal clip, link spawned where he was previously standing. I went to Lorule, did the clip again, Link respawned where he was standing.
So, now for the theory. Zones are flagged to behave differently. Some zones use your last position, while others just use the last entrance. Kakariko may very well be a zone that use the entrance.
All zones which have voids or swimmable water seems to use position instead of entrance. The only exception that I know of is outside Link's House. You appear at the position you were standing instead.
So this "wrong-warp" could very well just be returning you to the proper position from voiding, and in kakariko that would be the entrance.
It's a theory at least.