Neuron numbers can change both directions.
Concerning how the brain grows for a mammal I dimly recall some spine area (I
hope I recall that right) forming into sort of a tube, and from the upper part
then grows the brain, and also there was something with sort of long "lines"
(glia?) along which neurons moved to their later brain positions.
If I recall right using an area a certain way a lot and feelings of success
might increase cell numbers in the brain (but I forget for what cells that all
went), while non-use and feelings like the opposite of success can decrease.
... But don't nail me down on parts that I wrote; it is a long time that I read
about that.