I have fail2ban setup pretty tight and it does a good job on bad bots. I’m talking more about hrefs bot, bing, goggle, etc who are indexing my instance. They will respect the file and optimistically a mirror bot with a public site should too. Without robots configured you don’t have as good of a case against their unwanted retrieval.