URLertBot

Automated Security Scanner Policy

URLertBot is the automated security scanner for URLert.com. We analyze URLs to detect malicious content, including phishing, scams, and malware distribution.

Purpose & Mission

The primary purpose of URLertBot is Public Safety. We scan websites to identify threats that endanger internet users, organizations, and platforms.

  • Category: Phishing, Scam, & Malware Detection
  • Benefit: Our data protects the internet ecosystem by:
    • Protecting Users: Verifying links in emails or messages (e.g., detecting fake banking login pages) before a user clicks.
    • Securing Platforms: Helping public sharing services and platforms identify if users are hosting malicious data or bad links.
    • Assisting Owners: Alerting domain owners if their infrastructure has been compromised or injected with malicious code.

Identification

URLertBot identifies itself clearly in the HTTP User-Agent header.

User-Agent: Mozilla/5.0 (compatible; URLertBot/1.0; +https://urlert.com/bot)

Verification (IP Addresses)

To verify that traffic is genuinely from URLert, checking the User-Agent is not sufficient as it can be spoofed. We provide a strictly defined list of IP addresses that our official bot operates from.

You can validate our bot using our Public IP List:

Any request claiming to be URLertBot that does not originate from an IP in this list should be considered spoofed.

Behavior & Ethics

We strive to be a "Good Citizen" of the internet. Our scanning behavior is designed to be non-intrusive.

  • Security-First Scanning: URLertBot is a security scanner, not a search engine crawler. Because threat actors routinely add robots.txt disallow rules to conceal malicious content from detection, following those rules would directly undermine our mission to protect users. URLertBot therefore scans only the specific URLs submitted for analysis, regardless of robots.txt directives.
  • Traffic Volume: We do not index entire websites. We only fetch specific URLs that have been submitted for security analysis.
  • Abuse Prevention: Our bot is not used for credential stuffing, inventory scalping, or aggressive scraping. It is strictly a read-only analysis tool used to detect threats.

Controlling URLertBot

Because URLertBot is a security tool rather than a general-purpose crawler, robots.txt disallow directives are not honored. This is intentional — malicious actors routinely use robots.txt to hide harmful content from automated scanners. Honoring these rules would allow attackers to evade detection and put your users at risk.

If you would like to opt your domain out of URLertBot scanning entirely, please contact us directly at support@urlert.com. We review opt-out requests on a case-by-case basis.

Contact

If you believe URLertBot is malfunctioning, accessing sensitive data, or causing issues for your infrastructure, please contact our engineering team immediately.