Product feed monitoring: catch empty or stale exports

Why a product feed fails silently

For many merchants, the feed is the link to channels that carry a large share of their reach. In a representative survey, 85 percent of online shoppers said they often shop on large online marketplaces, and 21 percent buy via comparison portals (Bitkom). The survey covered 1,120 internet users aged 16 and over in Germany, including 1,072 online shoppers (Bitkom). On the seller side, 45 percent of EU enterprises with web sales sold via an online marketplace (Eurostat); the count covers enterprises with at least 10 employees (Eurostat), reference year 2024.

The feed itself is inconspicuous. A scheduled job reads the products from the shop database, writes an XML or text file and places it at an address that the portal fetches on its own schedule. There are several steps between database and portal, and each one can fail without anything showing on the website:

• The job does not start. A changed crontab or a move to a new server without the schedule, and the export simply stops.
• The job runs but finds nothing. A filter on a renamed category or a query that comes back empty after an update produces a file with a header and no products.
• The job aborts. Memory, a time limit or a single broken record ends the run halfway. What remains is half a file or the old one.
• The file is not replaced. Missing write permissions or a full disk, and the feed URL keeps serving yesterday's version.
• The URL serves something else. A maintenance page, a login page or an error page sits where the portal expects a file.
• A cache serves an old version. The new file is on the server, but an older copy is delivered.

What all these cases share is that the usual availability check stays green. The website responds, and the feed URL often responds with status 200 as well, just with the wrong content. You know the pattern from the article on silent failures of cron jobs: if you only ask from the outside, you do not see the process behind it.

The 30-day expiry is not an alert

How long a portal keeps working with a stalled export is up to the portal. For Google, the Merchant Center help describes a fixed limit: items expire 30 days after the last refresh (Google Merchant Center Help). The limit applies to items that come from a data source such as a feed file; according to the same page, items added directly in Merchant Center do not expire (Google Merchant Center Help).

Items are refreshed by resubmitting the data source or by scheduling regular data source uploads (Google Merchant Center Help). In addition, Google checks the website itself and refreshes items automatically if the product data can be validated, for example through schema.org structured data on the product pages (Google Merchant Center Help). If the feed sets its own expiration date for an item, the earlier of the two applies: the date set or 30 days after the last refresh (Google Merchant Center Help).

For monitoring, the main takeaway is this: waiting for the expiry is not a plan. The help page does not say explicitly whether an unchanged file counts as a refresh on every scheduled fetch. If the export stalls and the feed URL keeps serving the old file, a portal can fetch it for days without anything expiring. Prices, stock levels and new products then stay at the state of the last successful run. If the URL stops serving anything at all, refreshes stop, but weeks still pass before the period runs out. As a warning, the expiry comes too late.

Three places where a feed fails

Sorting the failure patterns leaves three places: the export job that builds the file, the file itself with its item count, and the address from which the portal fetches it. Each place has its own signal. The table shows which failure pattern triggers which signal and where only the portal itself can help.

The heartbeat covers the first five rows because it sits inside the job. The API check covers the next two because it fetches the same address as the portal. The last two remain invisible to both; more on that in the section on limits.

The export job reports in with its item count

A heartbeat turns the check around: instead of a checker asking from the outside, the job reports in after every run. If the report does not arrive, an alert goes out. With the order heartbeat, a small job counts the orders of the last hour. With the feed, the export job counts what it has just written, and that is the key difference: the count comes from the finished file, not from the database. Only then does it show when the query is right but something gets lost while writing.

#!/usr/bin/env bash
# Builds the product feed, checks it and reports the result to a heartbeat.
# Usage: feed-export.sh <heartbeat-slug> <min-items>
set -uo pipefail

HB="https://api.xictraq.com/api/heartbeat/$1"
MIN="${2:-1}"
TARGET="/var/www/shop/public/feeds/products.xml"
TMP="$TARGET.new"   # temporary file in the same directory

# Report the start: XICTRAQ® derives the runtime from it.
curl -fsS -m 10 "$HB/start" >/dev/null || true

# Placeholder for the export command of your shop system.
/opt/shop/bin/feed-export --format=xml --output="$TMP"
RC=$?

# Count in the written file, not in the database.
N=0
if [ -s "$TMP" ]; then
  N=$(grep -o '<item>' "$TMP" | wc -l)
fi

# Only a plausible file replaces the one being served, in a single step.
if [ "$RC" -eq 0 ] && [ "$N" -ge "$MIN" ]; then
  mv -f "$TMP" "$TARGET"
fi

# Report the result, even if the new file was discarded.
curl -fsS -m 10 "$HB/ping?exit_code=${RC}&items=${N}" >/dev/null

Three parts carry the script. The call to /start records the beginning so that XICTRAQ® calculates the runtime itself when the final ping arrives. The export writes to a temporary file in the same directory, and only when exit code and item count check out does mv replace the file being served. Within one file system this is a rename during which the name points either to the old or to the new file (POSIX). A portal fetching at exactly that moment therefore does not get half a file.

The last line reports both to the heartbeat: the exit code of the export and the counted items. The report goes out even if the new file was discarded. It is exactly this ping with items=0 or an error code that triggers the alert, while the portal keeps fetching the last good version. What a portal would do with an empty file is up to the portal; it is safer not to serve an empty or truncated file in the first place.

For tab-separated text files, wc -l counts the lines and the header line is subtracted. For XML, the script counts the product elements; depending on the format they are called <item> or <entry>. Which other parameters a ping can carry, such as a short note via msg, is described in the heartbeat documentation.

Setting schedule, minimum count and runtime

A few settings on the heartbeat decide when a run counts as faulty. The table shows them for an export that starts every night at 02:15 and usually takes eight minutes.

The schedule describes when the export starts. From the last ping, XICTRAQ® calculates the next time in the schedule; the next ping has to arrive by that time plus the grace period. If the export starts at 02:15, the heartbeat gets 02:15, and the grace period covers the runtime including fluctuations. With the end time 02:25 instead, a ping at 02:23 would make 02:25 on the same day the next due time, and from 02:55 an alert would follow even though the export ran. XICTRAQ® allows the grace period between 0 and 3,600 seconds, so at most one hour (XICTRAQ® documentation). If the ping has not arrived by the due time plus grace, the alert follows. A gap before the portal's fetch makes sense: without a custom setting, Merchant Center fetches a linked file every 24 hours (Google Merchant Center Help), and frequency and timing can be chosen in the data source schedule (Google Merchant Center Help). If the fetch happens a few hours after the export, there is time after an alert to rerun the job manually.

The minimum count is a fixed value per heartbeat. XICTRAQ® compares the reported number against it; there is no expectation for the item count that adapts to the history on its own. A useful starting value is the lowest normal run of the past weeks minus a buffer for usual fluctuations. The expandable history on the heartbeat shows the numbers per run. If the product range changes significantly, for example at the change of season, the minimum count is adjusted along with it. If you also enable expect_output, items=0 triggers an alert even when no minimum count is set.

Checking the feed URL: what the portal finds

The heartbeat proves that the job ran. It does not prove that the file arrives at the address the portal fetches. That is what the API check is for: it fetches the feed URL at the configured interval and tests the response against fixed expectations. Three of them fit a feed:

1. Status. A status between 200 and 299 is expected. Status 200 means that the request has succeeded (RFC 9110); it says nothing about the content.
2. Response time. A threshold well above the normal value shows when the server only delivers the file sluggishly. Read the normal value beforehand with "Run now".
3. Content. A text assertion looks for a string that only appears in a real feed, such as the <item> element or, for text files, the start of the header line.

[
  { "type": "status_code", "operator": "in_range", "value": "200-299" },
  { "type": "response_time_ms", "operator": "lt", "value": 5000 },
  { "type": "text_contains", "value": "<item>" }
]

The third expectation is the most important one. The Merchant Center help requires the address to point directly to the file; if it points to an HTML page of the website, processing fails (Google Merchant Center Help). That is exactly what happens when a maintenance page is served with status 200 during an update or when a redirect leads to the login. The status check is green, the text assertion is not. How a maintenance page sets the right status is covered in the article on maintenance windows with 503 and Retry-After.

Two notes on setup. A feed file in Merchant Center can be up to 4 GB in size (Google Merchant Center Help), and for very large responses the check only evaluates the beginning. The string you look for should therefore appear early in the file; whether all items are in it is counted by the heartbeat, not by the API check. If the feed URL is password-protected, the check sends the credentials as a header. How to set thresholds for response time and status is covered in more depth in the article on assertions and thresholds. API checks are included from the Starter plan; the pricing page has the overview.

Stale despite a green fetch: the age of the served file

One case slips through both signals: the export runs cleanly, the heartbeat is green, the feed URL responds with a valid feed, but it is the feed from the day before yesterday. This happens when a cache holds the file longer than intended or when the export writes to a different directory than the one the web server serves from. The Last-Modified header field reveals which version arrives: it states the date and time at which the server believes the file was last modified (RFC 9110). How long a cache treats a response as fresh is controlled by, among other things, the max-age directive (RFC 9111). More on these traps in the article on stale content after a deploy.

The API check compares fixed expectations and does not calculate an age. A second heartbeat can: a small script on another machine fetches the file the way a portal does, reads the age from Last-Modified, counts the items in the served version and reports both. If the file is older than allowed, it reports a failure with a reason.

#!/usr/bin/env bash
# Fetches the feed the way a portal does and reports age and item count to a second heartbeat.
# Usage: feed-delivery-check.sh <heartbeat-slug> <feed-url> <max-age-in-hours>
set -uo pipefail

HB="https://api.xictraq.com/api/heartbeat/$1"
URL="$2"
MAX_HOURS="${3:-26}"
TMP=$(mktemp)
trap 'rm -f "$TMP"' EXIT

fail() { curl -fsS -m 10 --get --data-urlencode "msg=$1" "$HB/fail" >/dev/null; exit 1; }

HEADERS=$(curl -fsSL -m 120 -D - -o "$TMP" "$URL") || fail "Fetch failed"

LAST_MODIFIED=$(printf '%s\n' "$HEADERS" | tr -d '\r' \
  | awk -F': ' 'tolower($1)=="last-modified" {v=$2} END {print v}')
[ -n "$LAST_MODIFIED" ] || fail "No Last-Modified in the response"

AGE_HOURS=$(( ( $(date +%s) - $(date -d "$LAST_MODIFIED" +%s) ) / 3600 ))
[ "$AGE_HOURS" -le "$MAX_HOURS" ] || fail "Served file is ${AGE_HOURS} hours old"

N=$(grep -o '<item>' "$TMP" | wc -l)
curl -fsS -m 10 "$HB/ping?items=${N}" >/dev/null

Run hourly, the script fires as soon as the served file is older than one export cycle plus a buffer. Because it fetches from the outside, it also covers caches, redirects and the wrong directory. It does not replace the heartbeat in the export job, which reports the error code and runtime that only the job itself knows.

What monitoring does not see

Heartbeat and API check verify whether a feed is built, how many items it carries and whether it can be fetched. They do not check the content of individual items. Four gaps remain:

• Outdated prices and stock levels. An export can run on time and still write wrong values, for example if it reads its prices from a different source than the shop. Only a comparison with the product page shows that.
• Disapproved items. Missing required attributes, invalid identifiers or policy violations are reported by the portal in its own error list.
• Blocked fetchers. The Merchant Center help asks you not to block the user agents "Googlebot" and "AdsBot-Google" from the directory containing the data source (Google Merchant Center Help). Overly strict bot protection may block exactly these fetchers while the API check gets through with its own identifier.
• Gradual shrinkage. The minimum count is a fixed lower bound. If the item count drops from, say, 4,800 to 4,100, that stays above a minimum count of 4,000 without an alert. The heartbeat does not detect a drop compared with the previous day.

For these gaps, a look at the portal's diagnostics remains a must, ideally on a fixed rhythm such as every Monday. The search in the shop itself can fail just as quietly without any page showing an error; how to check it is covered in the article on monitoring shop search.

Several feeds, several recipients

Many shops build not one feed but several: one for shopping ads, one for price comparison, one per marketplace, often in different formats and with their own schedules. Each feed gets its own heartbeat and, if it is fetched from its own address, its own API check. A naming scheme with channel and cadence, such as "Feed price comparison nightly", keeps the alert list readable. Heartbeats have their own quota per plan, independent of the number of websites.

Where the alert goes is set by the notification channels: email, Slack, Teams, Discord, Telegram or webhook. A feed alert at 03:00 only helps if the right person reads it in the morning. Whether the chain reaches the phone can be checked beforehand in an alerting drill.

Set up feed monitoring before the sale weeks

The weeks before Black Friday are a bad time for a stalled feed. In the Black Friday 2025 survey, roughly one in two (52 percent) wanted to take advantage of the reduced prices, most of them online (Bitkom). If you change promotional prices on a schedule, you need a feed that keeps up; otherwise a feed from the day before shows a price in the portal that the shop no longer offers. Setup is manageable:

• Create one heartbeat per feed with cron expression, time zone and grace period.
• Switch the export script to temporary file, check and rename, and ping with exit_code and items.
• Derive the minimum count from recent runs and enable expect_output.
• Set the fixed runtime threshold well above the usual runtime.
• Create one API check per feed URL with status, response time and text assertion.
• Optionally set up a delivery script that checks Last-Modified as a second heartbeat.
• Schedule the portal's fetch a few hours after the export.
• Cross-check: send a ping with items=0 by hand and wait for the alert in the channel.

The cross-check is the most important item on the list. Only when an artificial empty run triggers an alert that reaches the recipient is monitoring more than a setting. The server returns the anomaly text in German. The next healthy ping from the real job clears the alert again.

Monitor the feed as a process, not as a file

A product feed rarely fails loudly. It becomes empty, old or incomplete while the shop and homepage work normally. If you monitor it as a process, with a report from the export job, a check of the address and a look at the age of the served file, you learn about it the same morning rather than from the portal. The same principle works for backup runs and for any other job whose result is a file. Both can be set up with XICTRAQ® heartbeats and API monitoring in the same account.