If Sauron was defeated and people could feel safer, then people could start colonizing new lands, discover new animals and plants in the fourth era? By the way, they couldn't get to Aman. The elves and dwarves slowly disappeared and the humans were left alone.