Data Feed Contents
The first time somebody opens a data feed delivery, two things surprise them. It is not one file, and the data file has no column names in it at all. There is just a wall of tab separated values, several hundred per line, and no way to tell which is which.
That is not a fault. The names are in a different file, and understanding why is the beginning of reading the format properly.
What actually lands in the bucket
A delivery has three parts.
The hit data file. Named hit_data.tsv, tab separated, encoded in ISO-8859-1. One row per server call. This is the data.
The lookup files. A compressed archive of small tables that translate numeric codes into readable values. Adobe stores a browser as a number, a country as a number, an operating system as a number. Without the lookups, a large part of the row is unreadable.
The manifest. A small file listing the file names, their sizes, their MD5 hashes and the number of records in each. It is delivered last, after everything else, which is what makes it useful. If the manifest is present, the delivery finished.
Cloud storage shows a file the moment it starts being written, not when it finishes. A pipeline that triggers on the appearance of the hit data file will sometimes read a partial file and load a partial day, and it will do so without any error, because a truncated tab separated file is still a valid tab separated file. The manifest exists precisely to prevent this. Trigger on the manifest, then read the files it names, then compare the record count it states against the rows you actually loaded. Two of the worst data quality incidents in any feed pipeline are prevented by that one rule.
The file has no header row
The hit data file starts with data on line one. The column names live in column_headers.tsv, inside the lookup archive, and the order of names in that file is the order of columns in the data.
This matters more than it sounds. Column order is not fixed across feeds, because it follows the column list you selected when you configured the feed. Two feeds from the same report suite, built by two people, will have different column orders. So a pipeline must read the header file and map by name, never by position.
Hard-coding column positions is the single most common way a feed pipeline breaks. It works perfectly until somebody adds a column to the feed, at which point every value after the insertion point silently shifts one place to the left and the data continues to load without error.
post_ is where the processing became visible
This is the most useful thing in the whole format, and it takes a moment to appreciate.
Many fields appear twice. There is page_url and there is post_page_url. There is evar1 and there is post_evar1. The column without the prefix holds the value exactly as it arrived from the browser. The column with the prefix holds the value after Adobe finished with it.
Everything that happens to data between collection and reporting shows up in the gap between those two columns. VISTA rules. Processing rules. eVar persistence carrying a value forward from an earlier hit. Currency conversion. Any other server side logic your implementation has.
Adobe's guidance is to use the post_ columns, and it is correct. Those are the values reporting is built on, so a warehouse built from the raw columns will disagree with every dashboard in the business.
The raw columns earn their place in one situation, and it is a good one. When you need to know what the browser actually sent, as opposed to what Adobe decided it meant, the raw column is the only honest answer. That makes the pair the best implementation debugging tool in the product. A processing rule you are not sure about becomes visible: compare the two columns on the same row and you can see exactly what it did. The rules themselves are covered in What Are Processing Rules and What Are VISTA Rules.
The columns worth knowing by name
There are several hundred and you do not need most of them. These are the ones that appear in nearly every piece of work.
| Column | What it holds | Why it matters |
|---|---|---|
hit_time_gmt | When the hit was received, as a Unix timestamp | Ordering. Everything sequential depends on it |
post_visid_high / post_visid_low | The visitor ID, in two halves | Together they identify a visitor. Always use both |
visit_num | Which visit this is for that visitor | Part of identifying a visit |
visit_start_time_gmt | When that visit began | Guards against duplicate visit numbers |
visit_page_num | Position of the hit within the visit | Entry, exit and path work |
post_event_list | Events on the hit, as comma separated numbers | Join to event.tsv for names |
post_product_list | Products, categories, quantities and prices | Revenue is parsed out of this string |
exclude_hit | Whether Adobe excluded this row | Rows Adobe did not count |
hit_source | How the hit reached Adobe | Separates live traffic from uploads |
The identity columns are worth one extra note. The visitor ID is split across two columns because it is a large number, and using only one half will silently merge unrelated visitors. Concatenate both, always.
What the feed does not contain
Two absences catch people, and both are by design.
Classifications are not in the feed. If your campaign codes are classified into friendly names, the feed carries the code and not the name. The classification tables live elsewhere and are applied at reporting time. So a warehouse built from feeds shows tracking codes where the dashboard shows campaign names, which looks like a bug and is not. Data Warehouse does carry classified values, which is one of the clearest reasons to choose it instead. The behaviour is covered in Classifications Overview.
Nothing is summarised. There is no visit count column, no unique visitor column, no revenue total. Those are all things Adobe calculates when a report runs, and reproducing them is the subject of Processing Data Feeds.
Follow along: read one row end to end
Twenty minutes with a real delivery and a text editor teaches more about Adobe Analytics than any amount of reading about it.
- Part one, get your bearings
-
Unpack a delivery and open
column_headers.tsvfrom the lookup archive. This is your map. The order of names here is the order of fields in every row. -
Take a single line out of
hit_data.tsvand pair it with the header names. A short script is easier than a spreadsheet. Most of the fields will be empty. - Part two, read it
-
Convert
hit_time_gmtto a readable time, and note the visitor ID and visit number. You now know who, and when. -
Look up the numeric browser and country values in
browser.tsvandcountry.tsv. This is what the lookup files are for, and it is the same pattern for every coded column. -
Split
post_event_liston commas and match each number inevent.tsv. Now you know what happened on this hit rather than just where it happened. - Part three, see the processing
- Find any eVar where the raw column and the post_ column differ, and work out why. Empty raw and populated post_ usually means persistence. Different values usually means a processing rule.
-
Check
exclude_hitandhit_sourceon the same row. These decide whether Adobe counted this row at all, which is where Processing Data Feeds starts.
There is nothing to configure. A header file, one row, and three lookups. Doing it once makes every column reference afterwards obvious rather than abstract.
The raw row is the source of truth
A delivery is three things: the hit data with one row per server call, the lookup tables that decode its numeric columns, and a manifest that arrives last and tells you the transfer completed. Trigger on the manifest, never on the data file.
The data file carries no column names. They are in column_headers.tsv, and the order follows the column list configured for that feed, so a pipeline must map by name rather than by position.
The most valuable convention in the format is the post_ prefix. One column holds what the browser sent and the other holds what Adobe made of it, and everything server side sits in the gap between them. Use post_ for anything that has to agree with reporting, and reach for the raw column when you need to know what really arrived.
What is not in the file is as important as what is. No classifications, and nothing summarised. There is no visits column and no revenue total, because those are not stored anywhere. They are produced by counting, and doing that counting yourself turns out to be considerably harder than it looks. Processing Data Feeds covers what it takes to make raw rows agree with the numbers everybody is already looking at.
There is no screen for this. The contents are whatever your feed delivers to its destination, and the column list is set in Analytics > Admin > Data feeds on the feed itself.
The full column reference is Adobe documentation rather than an interface, and it is the page to keep open while building anything on a feed. Your own column_headers.tsv is the authority for what your feed actually contains.
This article focuses on the concepts, architecture, and practical guidance behind the topic. For the latest UI walkthroughs and step-by-step implementation instructions, use the links below. They leave this site and open Adobe's own documentation in a new tab.