Processing Data Feeds
The first thing anybody does with a data feed is count the rows and compare the number to Analysis Workspace.
It never matches. The raw count is always higher, sometimes by a few per cent and sometimes by a great deal. The usual reaction is to assume the feed is wrong, or that the report is wrong, or that somebody has misconfigured something.
Nothing is wrong. Adobe does not count every row, and the rows it discards are marked in the file. Working out which ones, and why, is most of what processing a feed involves.
The rows you have to throw away
Four filters do nearly all the work of making raw rows agree with reporting.
Keep only rows where exclude_hit is 0. This is the big one. Adobe marks rows it excluded from reporting for a range of reasons, including bot filtering and IP exclusions, and it hands them to you anyway. Reporting never counted them.
Keep only rows where customer_perspective is 1. Unless you specifically want mobile background hits, which most analysis does not.
Drop rows where hit_source is 5, 7, 8 or 9. These are data source uploads rather than hits your site sent, and mixing them with live traffic double counts.
For orders, drop rows where duplicate_purchase is 1. Adobe already identified these as repeat submissions of the same purchase, usually from somebody refreshing a confirmation page.
Apply those four and the gap between your count and the report closes most of the way. Skip them and every number you produce will be quietly inflated, in a way nobody downstream can detect, because an inflated number looks exactly like a good month.
Counting the things Workspace counts for you
Standard metrics have to be rebuilt from the rows, and the recipes are worth knowing even if you never write one, because they show you what a metric actually is.
Page views. Count rows that have a value in post_pagename or post_page_url. This one is close to what people expect.
Unique visitors. Concatenate post_visid_high and post_visid_low, then count distinct values. Both halves, always. One half alone will merge unrelated people.
Visits. This is where it stops being obvious. Concatenate the visitor ID pair with visit_num and visit_start_time_gmt, then count distinct combinations. The visit start time is in there because visit numbers can repeat in rare cases, and without it two separate visits occasionally collapse into one.
Events. The post_event_list column holds comma separated numbers. Split it, then join each number to event.tsv to find out which event it was. Counting an event means counting rows whose event list contains its number.
Revenue. Parse it out of post_product_list, which packs product, category, quantity and price into one delimited string, then sum after removing duplicate purchases.
This is the idea worth carrying out of the whole module. There is no visit stored anywhere in Adobe Analytics. A visit is what you get when you group rows by visitor and visit number, and that grouping was decided by rules about inactivity timeouts and session boundaries that somebody at Adobe wrote down. The same is true of unique visitors, of revenue, and of every other number in every report you have ever presented. None of them exists until something counts. Seeing that once is what changes how you argue about numbers, because it moves the question from whether a number is right to which rule produced it.
Your first count will still not match, and that is informative
Even with the filters applied and the metrics built correctly, small differences remain. The common causes are worth knowing, because each one sends you somewhere different.
Late-arriving hits. Adobe keeps updating a day as stragglers come in. Your file was cut at a moment. Recent days will differ and older days will settle. This is the first thing to check and it explains most small gaps.
Time zones. The feed uses GMT. Your report suite has its own time zone. A day boundary in one is not a day boundary in the other, so a daily comparison can be off by several hours of traffic at both ends.
Virtual report suites. Feeds run against the parent suite, never a virtual one. If the dashboard you are comparing against sits on a virtual report suite, you are comparing two different populations. This is covered in Virtual Report Suites.
Raw columns instead of post_ columns. If somebody built the pipeline on the unprefixed columns, persistence and processing rules were never applied, and the numbers will disagree in ways that look random.
Every feed pipeline should be reconciled against Adobe reporting before anybody trusts it, on a closed period rather than on recent days, for two or three metrics people actually use. Doing that once takes an afternoon and gives you a documented gap you can explain. Skipping it means the first person to notice a discrepancy discovers it in front of stakeholders, at which point the whole warehouse loses credibility rather than the one metric that was wrong. Record the reconciliation and the residual difference somewhere the next person will find, because they will ask the same question and nobody will remember the answer.
Where feeds actually go
Three destinations account for nearly all feed pipelines, and the shape of the work differs in each.
A warehouse or lake of your own. The rows are loaded into tables and queried alongside everything else the business holds. This is where joining Adobe behaviour to orders, customers and cost data actually happens, and it is the main reason feeds exist.
Customer Journey Analytics. Adobe's own newer product consumes hit-level data rather than finished reports. Worth noticing: when Adobe built the next generation, it went back to this layer rather than to the reporting layer above it.
Data science. Sequence and propensity work needs individual events in order, which is what the file is. The filters above still apply, and they matter more here, because a model trained on excluded hits learns from traffic that was never real.
Follow along: reconcile one day
This is the exercise that makes somebody trustworthy with feed data. Pick a closed day at least a week old so late hits have settled.
- Part one, the naive count
- Count every row in one day's hit data file, with no filtering at all. Write the number down. It is going to be too high.
-
Build the same day in Workspace with
Occurrencesand note that number too. The gap between the two is what the filters are about to explain. - Part two, filter properly
-
Keep only rows where
exclude_hitis 0 andcustomer_perspectiveis 1, and drophit_sourcevalues of 5, 7, 8 and 9. Count again after each filter rather than all at once. You learn which one was responsible. - Part three, rebuild two real metrics
-
Count distinct
post_visid_highpluspost_visid_lowand compare to Unique Visitors for that day. -
Count distinct visitor ID plus
visit_numplusvisit_start_time_gmtand compare to Visits. Try it without the visit start time as well, and see whether it changes. On most data it barely does, which is why people leave it out and are occasionally wrong. - Part four, explain the remainder
- If a gap remains, check the report suite time zone against GMT, and confirm the dashboard is not on a virtual report suite. Those two explain most of what survives the filters.
- Write down the final numbers, the filters used, and the residual difference. This document is what lets the next person trust the pipeline without repeating the exercise.
There is nothing to configure. Four filters, two metrics, and one written record of the result. That record is the deliverable, more than the pipeline is.
Nothing here is free
Raw rows do not become numbers by themselves. Adobe discards rows before it counts, and it marks them for you in exclude_hit, customer_perspective, hit_source and duplicate_purchase. Miss those and every figure you produce is inflated in a way nobody downstream can see.
The metrics people take for granted have to be rebuilt. Unique visitors is a concatenation of two columns. A visit is a grouping of three or four. Revenue is parsed out of a delimited string. Each of them is a rule somebody wrote, applied to rows, and none of them exists in the data until something counts.
That is the reason to know this layer even if you never build a pipeline. Every number in every report is the output of rules like these, applied to rows like these. Once you have seen a visit assembled out of four columns, you stop treating reported numbers as facts and start treating them as results, which is the more accurate and more useful way to hold them.
It also puts the earlier modules in their place. The tracking call decided what arrived. Processing rules and VISTA decided what the post_ columns say, and the difference between the two versions of a column is those rules made visible. If you want to change what lands in a feed, that is where the change is made, and What Are Processing Rules is where those decisions live.
Nothing in this section happens in the Adobe interface. The work is in whatever reads the file: a warehouse, a notebook, a pipeline.
The one Adobe screen worth having open is Analytics > Admin > Data feeds > Jobs, which confirms which deliveries actually ran. Before investigating a number, confirm the day you are reconciling was delivered completely.
This article focuses on the concepts, architecture, and practical guidance behind the topic. For the latest UI walkthroughs and step-by-step implementation instructions, use the links below. They leave this site and open Adobe's own documentation in a new tab.