Headless Chrome Driver Not Working For Selenium

I am current having an issue with my scraper when I set options.add_argument('--headless'). However, it works perfectly fine when it is removed. Could anyone advise how I can achie

Solution 1:

This screenshot...

...implies that the Cloudflare have detected your requests to the website as an automated bot and subsequently denying you the access to the application.

---

Solution

In these cases the a potential solution would be to use the undetected-chromedriver in headless mode to initialize the google-chrome-headless browsing context.

undetected-chromedriver is an optimized Selenium Chromedriver patch which does not trigger anti-bot services like Distill Network / Imperva / DataDome / Botprotect.io. It automatically downloads the driver binary and patches it.

• Code Block:

  import undetected_chromedriver as uc
  from selenium import webdriver
  
  options = webdriver.ChromeOptions() 
  options.headless = True
  driver = uc.Chrome(options=options)
  driver.get(url)
  

---

References

You can find a couple of relevant detailed discussions in:

• Selenium app redirect to Cloudflare page when hosted on Heroku
• How to bypass being rate limited ..HTML Error 1015 using Python