hello magentoexpert,
A XML sitemap is much helpful to index website URLs. I think it is the most reliable methods to get index website URLs. Please make sure that sitemap URLs are proper with HTTP status code. Also, ROR sitemap are best for this purpose.
____________________________
Event Management Companies in Delhi Ncr