Site Change Syndication for Optimized Web Page Scraping

Updated September 11, 2023 19:16

Goal

Site Change Syndication is used to efficiently update True Anthem's index of a customer's web pages. Links to these web pages may be used when posting to social networks.

Without Site Change Syndication, True Anthem's system scrapes a web page's URL every 60 minutes if the URL has been viewed by a browser within a given hour in order to keep True Anthem's index current. This approach can create a large amount of HTTP requests to a customer's web server.

Site Change Syndication allows the customer's web site infrastructure to identify any newly published or updated web pages to True Anthem's system while dramatically reducing the number of HTTP requests made by True Anthem to a customer's web server. Site Change Syndication also provides customers greater control over what web pages can be posted.

Instructions

The Site Change Syndication URL content's format follows the JSON Feed Version 1.1 specification. Other syndication formats, RFC 4287 - The Atom Syndication Format and RSS 2.0 Specification, may be supported in future releases.

The True Anthem implementation will request the Site Change Syndication URL every 5 to 60 minutes. The exact interval is determined during setup in conjunction with a customer's preference.

The JSON Feed's items array only needs to contain additions or modifications to web pages made in the prior 24 hours. The 24-hour period provides redundancy in case a technical issue occurs during a request interval.

Required Top-level JSON Feed Fields

version (required, string) is the URL of the version of the format the feed uses. This should appear at the very top and must match "https://jsonfeed.org/version/1.1".
title (required, string) is the name of the feed and used by monitoring and troubleshooting.
feed_url (optional but strongly recommended, string) is the URL of the feed, and serves as the unique identifier for the feed.

Required Item JSON Feed Fields

id (required, string) is unique to each item for that feed over time. If an item is ever updated, the id should be unchanged. New items should never use a previously-used id. Ideally, the id is the full URL of the resource described by the item, since URLs make great unique identifiers.
url (required, string) is the URL of the resource described by the item. The value must be the permalink. This may be the same as the id — but should be present regardless.
date_published (required, string) specifies the date in RFC 3339 format. (Example: 2010-02-07T14:04:00-05:00.) While the JSON Feed Version 1.1 specification makes this field optional, True Anthem's change detection implementation requires this field.
date_modified (required, string) specifies the modification date in RFC 3339 format. While the JSON Feed Version 1.1 specification makes this field optional, True Anthem's change detection implementation requires this field.