I can only say that for my personal computers I would have it enabled if present, and if stable. With XMP off it will default to a "JEDEC standard" profile. The standard JEDEC profiles for DDR4 are listed in a table here - DDR4 SDRAM - Wikipedia Note there's no DDR4 speeds over 3200 MT/s in the JEDEC standards, so anything over that will be an overclock. These profiles are safe, but will not give peak performance to people who want that. XMP may be technically an overclock, or may not - it depends. For example, I have come across DDR4-2666 which was installed in combination with an i5-9x00 CPU, and the memory had an XMP profile you could enable. It basically caused it to run slightly "tighter" memory timings than the JEDEC standard profile (which was also set at 2666 MT/s), but did not increase voltage over the standard (1.2v), or cause it to run faster than officially supported by that CPU, therefore as far as I'm concerned, it's not an overclock.
As another example, you could have DDR4-3600 RAM with a Ryzen 5xxx series CPU - its a very common combination. If XMP is enabled it will likely bump the voltage up to 1.35v and run the memory at 3600 MT/s. This is a higher speed than officially supported by the CPU (3200), and also a higher voltage than officially used by DDR4. Will it cause any damage? Pretty unlikely I'd say, but each user has to make their own decisions. Sometimes there can be stability issues with attempting to run XMP profiles, particularly if they are very aggressive, or it can work fine with 2 modules installed but become unstable with 4. If it's something you're not interested in at all, and you just want the best stability, with no complications, you can buy memory modules without any XMP profile embedded - it's the standard OEM style plain DIMM's (often green, plain modules) sold by, for example, Kingston and Crucial. Example DDR5.