What I'd say is that perhaps the exfoliating cloth — or it could also mean that this general method of physical exfoliation — isn't suitable for you. Bumps that appear (only) after exfoliation and disappear shortly (of if they're persistent) are signs of skin irritation and a sort of red flag/defense mechanism to signal a stop.
If you feel that besides the initial bump that goes away fairly quickly isn't a big deal to you, in my humble opinion, I'd just continue since it sounds like you're happy with the results. If you'd like to explore other types of exfoliation methods, chemical exfoliation is extremely effective and can be very gentle (despite how it sounds, i.e. chemicals must be harsh).
Goat's milk soaps contain natural Lactic acid which is great! Exfoliate gently and daily while showering. But if you're thinking soap bars consider getting one with a low pH (less irritating as bar soaps tend to be more acidic and drying/moisture stripping). OR you can opt for an exfoliating lotion, I've great results with a Glycolic acid-based one. As you can see, I'm lazy and like my exfoliation products to be efficient and 2-in-1 :)
* Both Lactic and Glycolic acids are AHA - they are both exfoliating and humectant (locks in moisture)! Hope this helps!