I got ranked quite high in top new racing and I am also on the trending list in that category.
I am wondering to switch to different category. Now it’s in games/racing and I want to go for casual or arcade since these categories are a bit more relevant and provide more downloads.
What happens to my ranking if I do that? Do I have to start from the bottom or the rank is somehow magically casted from old to new category?
There is no magic formula, otherwise we would all be doing it 
Of course there is an actual formula that only Google knows, but that’s subjective to constant “improvement” in any case.
Generally your stats cross over, but there is no guarantee. It pretty much comes down to your own experience.
I would say that you keep your stats but they will be fit to new category. For example if you are in top 10 of Word games you probably won’t be even in first 100 in Action or Arcade games.
But remember that this is speculation and as Alkaline said - noone knows except Google:)
Well,My app was always thrown out when I have changed my game category.when It was in top new free 50
Can you share what was your rank before and after changing. How big was the gap? From which and to which category did you migrate your app?