Why is the True Anthem beacon getting rate-limited on my site?

If you've noticed delays in content appearing in your True Anthem queue or seen 429 (Too Many Requests) errors in your server logs, it means your website's security systems are unintentionally throttling our platform. This article explains how our content ingestion works and gives you the exact technical steps to resolve the issue.

What the True Anthem "Beacon" Actually Does

Our "Beacon" isn't a traditional web crawler; it's our JavaScript analytics tag that runs on your site. Its job is simple: to efficiently monitor and feed new content back into our system for social scheduling, including verifying we have all correct meta-data elements required to post in real-time.

💡 TIP: Unlike a standard web crawler that crawls your entire site, the True Anthem system only monitors pages that have recently been viewed by a human user. It issues just one hourly request per active page to check for updates. This means it requires whitelisting, as robots.txt files won't manage it.

Why Your Server is Blocking Our Requests (The Problem)

Your website uses smart rate-limiting and security settings to prevent excessive traffic, which is a great safeguard against malicious activity. 

However, these settings sometimes mistakenly identify True Anthem's legitimate, high-frequency hourly checks on popular pages as a malicious bot or attack. This causes your server to reject our request with a 429 (Too Many Requests) response.

⚠️ IMPORTANT: When this rate-limiting happens, you will experience:

• Content Delays: New or updated articles will take significantly longer to show up in your True Anthem queue.
• Post Failures: The system may fail to publish old metadata because it couldn't perform the final check before going live.

Resolving the Rate-Limiting Issue (Technical Steps)

Resolving this issue requires a small technical adjustment on your server. We have two recommended paths, starting with the immediate fix:

1. Immediate Fix: Whitelist Our User Agent

The quickest and easiest way to stop the 429 errors is to have your technical team add our platform's unique identifier to your server's whitelist.

Action: Please whitelist the following User Agent string in your security/rate-limiting rules:

Mozilla/5.0 (compatible; trueAnthem/1.0; +[http://www.trueanthem.com](http://www.trueanthem.com))

For more context, please reference our technical article here: True Anthem Scraping Issues Due to High Server Traffic and Rate Limiting

2. Definitive Solution: Implement Site Change Syndication (JSON Feed)

For the long-term, most efficient solution that will drastically reduce the request volume you see, we recommend setting up Site Change Syndication.

Instead of our system having to check all active pages every hour, your Content Management System (CMS) tells us exactly when a page is new or updated. This cuts down on unnecessary requests and significantly lowers server load.

Action: Set up a JSON Feed that pushes updated content links to True Anthem.

For technical guidance on implementation, please refer to: Setup Site Change Syndication for Optimized Web Page Scraping

By following either of these steps, you ensure uninterrupted content flow and optimal performance from the True Anthem platform.