I like seeing them try and then thinking "begone thot! There is no entry for you"

In fact, I might make a honeypot that issues exactly that

But an IP can have multiple websites and even not return anything on plain IP access. How do crawlers find out about domains and unlinked subdomains? Do they even?

Servers which are meant to be secure usually are configured to not react to pings and do not give out failure responses to unauthenticated requests. This should be viable for a authenticated only walled garden type website op is suggesting, no?

It's not as simple as "only security through obscurity". You could say the same thing for an encryption key of a certain length. The private key to a public key is still technically just an obscurity, but it's still impractical to actually go through the entire range

IPv6 is big enough where this obscurity becomes impractical to sweep. But of course, as you said, there may be other methods of finding your address

Almost certainly. There are only 4,294,967,296 possible IPv4 addresses, i.e. 4.3ish billion, which sounds like a lot but in computer terms really isn't. You can scan them in parallel, and if you're an advanced script kiddie you could even exclude ranges that you know belong to unexciting organizations like Google and Microsoft, which are probably not worth spending your time messing with.

If you had a botnet of 8,000 or so devices and employed a probably unrealistically generous timeout of 15 seconds, i.e. four attempts per minute per device, you could scan the entire IPv4 range in just a hair over 93 days and that's before excluding any known pointless address blocks. If you only spent a second on each ping you could do it in about six days.

For the sake of argument, cybercriminals are already operating botnets with upwards of 100,000 compromised machines doing their bidding. That bidding could well be (and probably is) probing random web servers for vulnerabilities. The largest confirmed botnet was the 911 S5 which contained about 19 million devices.

I don't know exactly how they do it, but probing every ipv4 address isn't that hard

If it's https it's discoverable by hostname.

https://0xffsec.com/handbook/information-gathering/subdomain-enumeration/#certificate-transparency

Certificate Transparency (CT) is an Internet security standard and open-source framework for monitoring and auditing digital certificates. It creates a system of public logs to record all certificates issued by publicly trusted CAs, allowing efficient identification of mistakenly or maliciously issued certificates.

DNS only translate a string address (www.mywebsite.com) to its IP address (xxx.xxx.xxx.xxx) so that it is easier to remember.

Bots just try a range of address and they don't need to know your domain name. You could have the most unintelligible domain name in the world, bots would still ping your website because they use direct IP addresses.