Crawl HTTPS Websites with A1 Sitemap Generator

A1 Sitemap Generator allows the user to select a HTTP solution in Scan website | Crawler engine.

The default setting is Auto detect which translates to:

• Windows: The setting HTTP using Windows API
• Mac: The setting HTTP using Mac API

It is also possible to select HTTP using Indy library which is an alternative solution.

Tip: If you have problems getting crawling working, be sure to check if A1 Sitemap Generator is getting blocked by firewalls or similar solutions.

Note: This section is only necessary if:

1. Your website uses HTTPS.
2. You use Indy in Scan website | Crawler engine.

Configuring OpenSSL / Configuring LibreSSL for use with A1 Sitemap Generator will help for all HTTPS / SSL based websites.

To add support for this, see General options and tools | Tool paths.



Clicking the button at the right will show a menu with information and links.

In newer versions of A1 Sitemap Generator the menu will also show which version you should download for your computer system.

While this will usually work out of the box, you may sometimes need to do some configuration, especially on older systems

This will mainly be in Tools | Internet Options | Advanced | Security:



Windows 11:

1. Download and apply all Windows updates, e.g. by using Windows Update.
2. Enable TLS 1.1, TLS 1.2, TLS 1.3 and newer if available in Windows internet settings at Tools | Internet Options | Advanced | Security.

Windows 10:

1. Download and apply all Windows updates, e.g. by using Windows Update.
2. Enable TLS 1.1, TLS 1.2, TLS 1.3 and newer if available in Windows internet settings at Tools | Internet Options | Advanced | Security.
3. If crawling using the embedded system browser option, download the Edge / Chromium a.k.a. WebView2 runtime from Microsoft:
   • https://developer.microsoft.com/en-us/microsoft-edge/webview2/consumer/

Windows 8.1 without Internet Explorer 11:

1. Download and apply all Windows and IE updates, e.g. by using Windows Update.
2. Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.
3. If crawling using the embedded system browser option, download the Edge / Chromium a.k.a. WebView2 runtime from Microsoft:
   • https://developer.microsoft.com/en-us/microsoft-edge/webview2/consumer/

Windows 8 without Internet Explorer 11:

1. Download and apply all Windows and IE updates, e.g. by using Windows Update.
2. Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.

Windows 7:

1. Download and apply all Windows and IE updates, e.g. by using Windows Update. You at minimum need to use SP1 / service pack one.
2. Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.
3. If crawling using the embedded system browser option, download the Edge / Chromium a.k.a. WebView2 runtime from Microsoft:
   • https://developer.microsoft.com/en-us/microsoft-edge/webview2/consumer/

Windows Vista:

1. Download and apply all Windows and IE updates, e.g. by using Windows Update.
2. Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.
3. Experts: If you do not see these options then check these guides:
   • https://johnhaller.com/useful-stuff/enable-tls-1.1-and-1.2-on-windows-vista
   • https://msfn.org/board/topic/176902-enabling-tls-1112-support-in-vistas-internet-explorer-9/

Windows XP and derivatives:

1. Download and apply all Windows and IE updates, e.g. by using Windows Update.
2. Experts: Investigate if Windows Embedded POSReady 2009 is applicable to you and that you are eligible.
3. Experts: A registry setting essentially turns your system into a different edition for which updates were released until 2019 April.
4. Experts: You can get started learning more here:
   • https://www.pcworld.com/article/2310301/windows-xp-registry-hack-keeps-the-security-updates-rolling.html
   • https://www.zdnet.com/article/registry-hack-enables-continued-updates-for-windows-xp/
5. Download and apply all Windows and IE updates again, e.g. by using Windows Update.
6. Enable TLS 1.1, TLS 1.2 and newer if available in Windows / IE internet settings at Tools | Internet Options | Advanced | Security.
7. Experts: If you do not see these options then check this guide:
   • https://support.microsoft.com/en-us/help/4019276/update-to-add-support-for-tls-1-1-and-tls-1-2-in-windows

Note: Windows systems with IE 8 or less will still not work with websites using elliptic curve cryptography.

Alternative solution: Use Indy as the chosen crawler engine and configure as explained further above.