Yeah lol it’s straight up being told in the show White Walkers are the last method of revenge of people being genocided to extinction and knowing they are the last of their kind. Official companion book to ASoIaF are also pretty clear on what happened during the First Men and Andal migrations.