One of the goals of Adalytics is to give marketers and curious users greater transparency into the digital ad tech ecosystem and allow them to inspect how ads are served. Previous Adalytics research examined whether ad tech platforms are able to deliver the often alluded-to ‘relevant’ ad experience for consumers, and explored why individual consumers see specific ads. 

There are two methodologies for auditing ad tech platforms: 1) access to proprietary log level data from ad servers and other platforms, or 2) via crowd-sourced ad impression data from consumers.

This analysis details preliminary results from an ongoing Adalytics crowd-sourced audience research pilot that uses a convenience sample of 25 volunteers. The study’s small sample size is not statistically representative, but is meant as a proof-of-concept for the types of insights that could be derived with greater statistical rigor provided more participants.

This write-up discusses several interesting observations, such as:

1. Amazon, Progressive, Vrbo, and American Express serve ads to a wide range of users, whereas brands such as Toyota, Mastercard, and CVS appear to have significantly narrower audience reach with their ads.
2. Zappos and PayPal are segmenting customers based on the conversion funnel, but PayPal appears to be overly optimistic with prospective customers’ intent.
3. Pharma companies such as Pfizer, Novartis, Sanofi, and Bristol Myers Squibb are incorrectly serving drug ads intended for doctors to non-healthcare professionals.
4. Nike appears to be 93% accurate with regards to targeting users by gender, while Merino Shoes and Santoni Shoes serve a majority of their women’s footwear ads to men and men’s footwear ads to women.
5. Some VC funded Silicon Valley startups appear to be using lookalike audiences that are too loosely defined.

1. Methodology
2. Which brands have the widest reach in their ad campaigns?
3. Calibrating the marketing funnel
4. Auditing demographic & behavioral ad targeting
1. Gender-based ad targeting
2. Behavioral ad targeting for healthcare professionals
3. Interest-based ad targeting
4. Lookalike audience ad targeting
5. Conclusion
1. Caveats & limitations
2. Discussion
3. Take away points & recommendations

Methodology

25 audience research volunteers were asked to read a study description and fill out a consent form. Participants had the option of self-reporting a number of demographic attributes, such as gender, race, profession, and age.

Once a volunteer had filled out the study consent form, they were asked to use the Adalytics browser extension, which allows them to track the ads they are shown while browsing the internet.

The Adalytics extension has a number of features such as letting consumers review their ad history in a personalized dashboard, as well as see how much advertisers are paying per programmatic ad impression in certain contexts. After collecting data for at least two weeks, some study participants received personalized surveys to help contextualize some of the ads they were being shown.

The study participants consisted of 14 males and 10 females (1 did not report gender), with ages ranging from 24 to 63. The participants were primarily based in the United States, although four participants were in other OECD countries. The participants included digital marketers, ad tech professionals, physicians, lawyers, software engineers, retirees, journalists, academic researchers, and one C-Suite business executive. Six participants were small business owners or early-stage startup founders.

Which brands have the widest reach in their ad campaigns?

Digital marketers often have to make a trade off between reach and frequency when configuring their ad campaign strategy. Reach is the total number of people who are exposed to an ad, whereas frequency is the number of times a given consumer sees a particular brand’s message.

In the Adalytics audience research dataset, one can observe which brands are able to reach the widest number of different consumers (see AirTable below). Unsurprisingly, ads for various Amazon services were shown to the largest number of different users (64%). Various eBay hosted product ads were seen by 32% of participants, while retail giant Walmart’s ads were only seen by 4% of consumers (Walmart is currently looking to expand its own advertising business to compete with Amazon). 

Amazon’s new online pharmacy in the US was seen by 28% of participants in the panel. For comparison, the pharmacy chain CVS’ ads were only seen by 16% of participants, and Walgreen’s ads were also only seen by 16% of participants. These differences are not statistically significant due to the small sample sizes, but could be re-evaluated with a larger audience panel.

The insurance company, Progressive, was able to reach more than half the participants in the study by running ads on 117 different websites. Geico, a competitor in the insurance industry, was only able to reach 20% of participants in the study, and State Farm Insurance group’s ads were served only to 16% of participants.

The vacation rental online marketplace Vrbo was able to achieve high reach amongst the study participants (48%), running prospecting ads targeting potential US travelers across 35 different websites.

Whereas American Express was able to reach 50% of the study participants (including those in 3 different countries) with its credit card ad campaigns, Apple Card reached 32% of the audience, Mastercard reached 24%, and Visa’s credit card ads were only able to reach 8%. In the ongoing video streaming wars, ViacomCBS’ Paramount+ and Discovery’s Discovery+ took the lead with 40% and 36% audience reach, which they achieved by running ads on the largest spread of different websites. NBCUniversal’s Peacock reached 32% of the audience, HBO Max reached 28%, and Disney Plus reached a quarter of participants. Hulu reached 16% and Apple TV+ only reached 12%.

In the automotive market, Volvo was observed running prospecting, awareness, and consideration targeted ad campaigns for both used cars and hybrids, which reached 36% of the audience. Ford’s car ads reached 28% of the audience, Honda reached 20%, Mercedes Benz sedan ads reached 16%, General Motors reached 12%, and Toyota reached 8%. Hyundai and Chrysler ads were both only seen by one participant.

Many news sites are running ads to promote specific articles or to attract new subscribers. The New York Times is advertising to get new subscribers on a number of their competitors' websites, including vox.com, forbes.com, gizmodo.com, businessinsider.com, and buzzfeed.com, in addition to on their own website. Wall Street Journal was able to reach 28% of the audience, primarily through Twitter ads. Bloomberg reached 16% of the audience with its subscription offers, running ads on 18 different websites.

Brandable Box, a 5-employee division of Pratt Industries  based out of Atlanta, Georgia, deserves a special shoutout. They were the only company that was able to reach 40% of the audience whilst running ads on only three websites - wsj.com, washingtonpost.com, and nytimes.com. Brandable should receive an award for great digital marketing tactics by an SMB!

Calibrating the marketing funnel

Some marketers analyze and stratify potential customers for their products via the “conversion funnel”.  In this abstract model, a potential consumer takes a “customer journey” through various digital advertising or search touchpoints, navigates to an e-commerce website and eventually makes a purchase. The metaphor of a funnel is used to describe the way users are guided towards the purchase goal with a narrower range of product options at each step. Using this metaphor, advertising efforts can be aimed at "upper funnel", "middle funnel", or "lower funnel" potential customers. These are occasionally abbreviated as TOFU (top-of-funnel), MOFU (middle-of-funnel), or BOFU/LOFU (bottom of funnel or lower funnel). 

According to this theory, different types of promotions, content, and messages are needed to nudge a potential customer at each stage of the funnel. The Top-of-the-Funnel represents an “awareness” stage, during which blogs or social media posts may be recommended. The Middle-of-the-Funnel is an “evaluation”, at which point various comparison guides or videos may help a potential customer evaluate competing solutions for a given need. The Bottom-of-the-Funnel, potential customers are thought to be actively looking to execute a purchase, and may be enticed to convert with a free trial or coupon promotion.

In this Adalytics audience research study, it is possible to infer how individual consumers are being categorized in the marketing funnels of various brands. This is done by examining the UTM query string parameters of various ad clickthrough URLs. For example, one Adalytics user was being targeted by the online apparel retailer Zappos, where the “utm_campaign” query string parameter labeled them as being in the “mainlowerfunnel”.

Another Adalytics user was targeted with ads from the cybersecurity company ExtraHop, who was running an awareness campaign and placed this particular user in the top of the funnel.

After Adalytics analyzed each individual ad clickthrough URL for informative UTM parameters, individual study participants were surveyed in order to gauge whether a given brand was accurately classifying each user’s stage in the purported customer journey.

For the Zappos ad above, the given user was asked “Were you actively seeking to purchase new shoes in the last few months? If you made any shoes purchases recently, where did you buy those shoes?” The given user confirmed that they were actively looking to buy shoes, and they listed Zappos as a merchant from whom they had purchased shoes recently. Therefore, one can conclude that Zappos was appropriately targeting their ad to this individual user and categorizing them as Lower Funnel.

Another (male) Adalytics participant was targeted with ads for women’s wedding tennis shoes and bridesmaids’ footwear from Keds, where he was labeled as being ‘mid-funnel’. This male user was asked if he was “Looking to buy any female shoe-ware?” or “Have any significant others who may be looking for female wedding shoes?”. The volunteer responded that he was generally looking at shoes, but not bridal shoes specifically, and his wife was also not looking to buy or learn about women’s bridal shoes. In this case, one could reasonably conclude that the given Adalytics user was not ‘mid-funnel’ for bridal shoes.

One Adalytics user was served digital adverts for Ross University School of Medicine’s 4-year degree course. In this programmatic ad campaign, the user was labeled as being “LowerFunnel''. The user confirmed in a survey that they were not even remotely considering applying to any medical school programs, and they are also not in the US. They also confirmed they do not work in, or plan to switch to, the healthcare industry from their current profession.

Wyrmwood Gaming, a 21-employee Massachusetts-based manufacturer of dice and gaming supplies, was observed targeting the same Adalytics user with ads 39 times across several different websites. Wyrmwood is running a retargeting campaign, and labeled this panelist as being in the “lowerfunnel”. Yet when surveyed “Are you (or were you in the past few months) considered purchasing gaming-related items such as dice or tabletops?”, the given user indicated that they were not at all in the market for such items. Furthermore, the user indicated that they tried to click and opt-out of seeing the particular ad further, but the opt-out button apparently had no effect as they noticed more Wyrmwood ads being re-targeted to them afterwards.

Paypal was observed running private marketplace (PMP, also referred to as “programmatic guaranteed”) ads for their seller QR codes on buzzfeed.com, washingtonpost.com, wsj.com, and other websites. These ads were observed by 4 different Adalytics volunteers, who were each surveyed with several questions to gauge where they were in the customer journey (all 4 were served with ads that said “mid_funnel”.

Out of these 4 panelists, only two are sellers who have their own businesses. Neither of these panelists said they have a need to accept payments in-person (i.e., they are both online businesses), and one of them responded that they have been using PayPal for client billing for many years, but purely online. All 4 of the panelists were asked if any of them were currently researching or looking for a point-of-sale or in-person payment solution for a business (now or recently), to which all 4 responded in the negative.Therefore, it is not clear if any of these 4 panelists who were served PayPal ads for in-person QR payment solutions were indeed ‘mid-funnel’. Lastly, the New York City based luxury mattress company Saatva was observed running display ads using Google Display & Video 360 on several websites targeting one specific Adalytics user. The company appears to be utilizing “psychographic targeting”, and labeled this particular user as being “mofu”, or Middle-of-the-Funnel.

When the given user was surveyed, they indicated that they were not in the market for mattresses. Interestingly, they indicated that they had visited Saatva’s website recently because they had received an older Saatva mattress from a friend, and were looking to check the dimensions of the item. However, they were not in the market to purchase a new mattress.

In light of these previous observations, advertisers may take care from drawing too many inferences about a consumer’s level of interest in a given product or brand simply by virtue of a consumer’s browsing history. Just because an individual has visited a given e-commerce site does not necessarily indicate that they are considering purchasing a brand’s products.

Auditing demographic & behavioral targeting

Harvesting and re-selling demographic and behavioral data about individuals is estimated to be a $200 billion business. Many companies, including tech giants such as Twitter and Facebooks, aggregate offline datasets (such as public records) and online browsing behavior across websites, to classify individuals by age (“Ages 25-54), gender, profession, and interests (e.g., “coffee lovers” or “in-market for shoes”). 

Many marketers, wishing to deliver the right message at the right place and time to right person, rely on these data brokers to help target ads more precisely to a specific audience. These third party data purchases can impact large portions of advertiser’s media buying budgets - in some cases doubling the advertising costs for a particular campaign.

However, recent research has raised doubts about the quality of third party data and inferred demographic segments. The former CEO of the New York Times previously asked “When we say a member of the audience is a female fashionista aged 20 to 30, what’s the probability that that’s actually true?”

Previous research published by Professors Nico Neumman of Melbourne Business School and Catherine Tucker of MIT found the average accuracy of gender segments from data brokers for classifying males was only 42.5%, which is less than the ~50% natural chance of identifying men. When two attributes, such as gender and age, were utilized from third party data brokers, the average accuracy drops to 24%. 

Furthermore, an internal Facebook email that was released as part of an ongoing lawsuit revealed that Facebook’s own employees raised doubts about the accuracy of their ad targeting: “interest precision in the US is only 41%—that means more than half the time we’re showing ads to someone other than the advertisers’ intended audience. And it is even worse internationally. We don’t feel we’re meeting advertisers’ interest accuracy expectations".

Gender-based ad targeting

In this present Adalytics study, digital ad clickthrough URLs were examined for indicators suggestive of gender-based ad targeting. For example, the online clothing retailer Banggood was observed serving shoe ads, where the UTM Gender string contained values of “Female” or “Male”.

If a given ad contained UTM parameters or product descriptors suggestive of male or female focused targeting, the gender of the given Adalytics user was cross-referenced to determine if the ad was accurately served. For example, Nike was observed running ads for both men’s and women’s shoes, which were seen by different study participants. Of these 78 Nike shoe ad impressions, 73 were ‘appropriately’ matched based on their gender. In 5 instances (6.4%), an ad for women’s shoes was served to a man, or an ad for men’s shoes was served to a woman. This would suggest that in this small sample-sized dataset, Nike was accurately targeting their ads based on gender 93.6% of the time (see table below).

Other apparel companies were not as precise as Nike in their gender targeting. San-Diego based Merino Shoes served 90.5% of their shoe ads to the opposite gender relative to what they were advertising. Santoni, an Italian luxury shoes manufacturer, was found to be serving their ads to the wrong audience 70% of the time.

Several Adalytics users who were consistently shown ads for products intended for the other gender (i.e., men served women shoe’s ads), were surveyed to see if they were perhaps shopping for items for significant others or other family members. As with the previously mentioned example of a male user shown ads for women’s wedding tennis shoes and bridesmaids’ footwear from Keds, each panelist responded that they were not searching or looking to buy any such apparel.

One of these male Adalytics volunteers observed that, when he visited Twitter’s profile settings page, the given page showed that Twitter had labeled his gender as “Female”. This user reported that he had never volunteered his gender to Twitter; the platform must have either purchased that label from another data broker or determined it based on inferential machine learning. Previous analyses have documented how Twitter often infers user’s gender without their input, and in many instances makes labeling errors through those inferences.

Behavioral ad targeting for healthcare professionals

In addition to examining gender based ad targeting, this study also tried to examine instances of profession based targeting. The clearest examples of this are pharmaceutical company drug ads that are intended to be served to healthcare professionals (HCP) in the US. Most pharmaceutical companies maintain a separate, drug-specific website for patients and one for healthcare professionals. For example, the cancer drug Keytruda (made by Merck) has a website for patients - keytruda.com, and one for healthcare professionals: keytrudahcp.com. The latter website has a warning banner that clearly indicates the site is intended only for medical professionals.

In this Adalytics audience research study, one can observe a number of pharmaceutical companies serving ads that direct a user to a drug information website that is intended for medical professionals. For example, one Adalytics was repeatedly served ads for Sanofi Genzyme’s site about metastatic breast cancer for healthcare professionals. This particular user was also served ads for Bristol Myers Squibb’s oncologist-focused and ulcerative colitis researcher sites, Karyopharm Therapeutics’ physician site for Xpovio (a multiple myeloma drug), and Pfizer’s HCP website for Bavencio (a treatment for urothelial carcinoma). These ads were shown on many non-medicine related websites: for example, the Xpovio ad was shown to the user on the celebrity news and entertainment website Celebuzz. This user was also shown ads for the healthcare professional web page for MicroPort’s orthopedic knee medical devices on Breitbart.com.

Somewhat unexpectedly, this Adalytics user reported that they were not a healthcare professional or biomedical researcher. It is unclear why this particular user was being shown so many advertisements that were intended for medical specialists - in fact, this user had a higher propensity for receiving professional-focused pharmaceutical drug ads than the other Adalytics study participants who reported that they were medical professionals.

Two other Adalytics users were served ads for a Novartis website for healthcare professionals regarding TIM-3, a potential drug target for Acute Myeloid Leukemia. Both volunteers confirmed they are not remotely involved in the healthcare industry. One of these participants was also served an ad from AstraZeneca for the healthcare professional oriented site for FluMist (a Influenza vaccine).

This analysis focused exclusively on pharmaceutical drug ads intended for healthcare professionals; general awareness ads or ads intended for patients, such as Takeda’s corporate mission website, Janssen’s information site about retinal disease, or Novartis’s patient-focused, private-marketplace ads on the nytimes.com for the dry-eye drug Xiidra, were excluded from the analysis. Additionally, the analysis excluded any ad impressions served on biomedical focused websites, such as the New England Journal of Medicine’s nejm.org, as those ads were likely placed through direct deals or contextual targeting rather than through behavioral or demographic targeting.

Interest-based ad targeting

Certain ad tech platforms and data management platforms (DMPs) allow advertisers to target ad campaigns to consumers with specific interests, such as “coffee lovers” or “cycling enthusiasts”.  However, as mentioned previously in the studies by Professors Neumman and Tucker, and internal Facebook emails, the quality of interest-based targeting can vary greatly.

This study observed a small number of cases of what appears to be interest-based advertising (a future Adalytics study will revisit this topic once more data becomes available). The National Rifle Association (NRA), a gun rights advocacy group in the US, was observed serving ads repeatedly to two Adalytics users. These NRA cost-per-click ads were for a “National Gun Owners Survey”, and appeared to be running for lead generation.

In several of the ad clickthrough URLs shown on different websites, one can observe the mention of “luxury_vehicle_enthusiasts”.

Each of the two Adalytics users targeted with NRA ads was surveyed if they had any interest in or own “luxury vehicles''. Both responded in the negative. They also responded that neither owns any firearms or plans on affiliating with the NRA.

Energy drink company Redbull was observed serving ads targeting “A18-34 Competitive Gamers'' to two Adalytics users. Both Adalytics users were in that age range, but neither was a competitive gamer.

Lookalike audience ad targeting

Several digital advertising platforms and data brokers enable marketers to target users who are in “lookalike audiences''. In this modality, a marketer can upload a list of emails belonging to existing customers (or an email list purchased from a data broker). Then, the ad tech platform  will try to identify potential customers online who are likely to share similar interests and behaviors with the existing customers, and target those “lookalike” individuals with ads.

Several companies were observed using UTM query string parameters in this Adalytics study that mentioned lookalike audiences. For example, MainStreet, a venture-funded Silicon Valley startup that “qualifies software, hardware, and venture-backed startups for 200+ unclaimed tax credits”, was observed serving ads that mentioned “Lookalike a16z mafia”. A16z is an abbreviation for Anderseen Horowitz, a leading Silicon Valley venture capital fund that has invested in many startups. ‘Mafia’ may be a reference to the so-called “PayPal Mafia”,  group of former PayPal employees and founders who have since founded and developed additional technology companies. It is possible that Mainstreet generated or purchased a list of emails belonging to employees at startups financed by Anderseen Horowitz, and is trying to target advertisements to a lookalike audience trained on this list.

10 different Adalytics users were served ads from Mainstreet. Of these users, seven do not work for startups or in the software and hardware industries (one of the seven is fully retired). Two of the users served MainStreet’s ads were indeed venture-funded, tech startup founders, and one was a bootstrapped software startup founder. All three of these Adalytics volunteers reported that they were not interested in the services Mainstreet has to offer startups. One of these surveyed startup founders said they had done consultations with similar services before, but that their startup does not pay enough in payroll to qualify for any of these tax credits, and secondly, hardly have enough taxes for it to be worth them signing up for such programs.

Lookalike audiences might offer reduced return on ad spend (ROAS) to small companies or startups because of the small sample size of their existing audience, which would inevitably lead to insufficient data drawn from the current audience and interference based on outliers. 

Another Adalytics user was served ads for the cybersecurity company Recorded Future that were based on a US Twitter follower lookalike audience. In this case, the particular Adalytics user was indeed a cybersecurity professional.

Conclusion

Caveats & limitations

This study was intended as proof-of-concept for the types of insights about ad targeting that could be generated through crowd-sourced ad impression datasets. The sample size of 25 users was not statistically significant or demographically representative. To accurately measure the precision of various brands’ ad targeting, one would need access to a large pool of volunteers and a more demographically diverse dataset. If you are a digital marketer interested in working together to create such a panel, please contact me here to discuss a sponsorship!

The measurements were conducted through a desktop browser extension, and would thus be unable to draw any conclusions about mobile or CTV targeted ad campaigns.

Furthermore, the analysis draws inferences from various URL query string parameters that are encoded in digital ad clickthrough URLs. These query string parameters may simply be arbitrary character strings with no functional significance, or their meaning could have been entirely misinterpreted. 

Lastly, this analysis did not have access to any proprietary information from advertisers about how they were targeting their ad campaigns. Many advertisers may not be using demographic or behavioral targeting, or they could be iteratively adapting their ad targeting strategies over time.

Discussion

John Wanamaker, a 19th century American business leader, famously quipped that “I know half the money I spend on advertising is wasted, but I can never find out which half.” The digital marketing and ad tech industries have promoted the idea the internet allows for very finely tuned ad targeting. However, this and previous Adalytics studies have shown that an empirical examination reveals mixed results for the accuracy of ad targeting.

This study observed examples of pharmaceutical companies serving ads intended for healthcare professionals to non-doctors. While Nike was observed to be precisely serving men’s apparel ads to men and women’s apparel ads to women, other advertisers had much lower observed accuracy, potentially because data brokers or tech companies were erroneously inferring and labeling specific users. Several advertisers were potentially labeling individual consumers as being middle or bottom of the conversion funnel, though in reality they were not in the market at all for those advertisers’ wares. One mattress company appeared to be using “psychographic targeting” and labeled an individual as being “in market” for their products, when in reality, the particular user reported no interest in that product category at the time.

Marketers may want to carefully evaluate the benefits of paying for audience segments for ad targeting. In some cases, the marginal cost of the targeting may outweigh the purported benefits. This is particularly true if the segments are low quality - in these scenarios, a marketer may be better off carefully curating a list of websites frequented by their target audience, and simply placing direct placement orders on those websites. This would have the added benefit of not relying on data gathering that may be harmful to consumer privacy.

If marketers or ad tech intermediaries have a particular need to rely on audience segments for ad targeting, they should take care to audit the quality of the data. If they are paying extra for a particular platform or DMP to show ads to healthcare professionals, “luxury car enthusiasts” or females, it would be beneficial to check if this data is indeed accurate. Marketers should also require information about the provenance of specific audience segments. 

Many data brokers and ad tech platforms do not disclose how particular audience segments are generated, oftentimes for competitive reasons. This creates an opaque market, where marketers cannot evaluate whether a particular audience segment merits a particular level of resource investment. The data management platform Lotame purged 400 million user profiles from its exchange in 2018 after identifying them as bots or fraudulent accounts.

For example, an advertiser should ask if a particular audience segment was:

• Self-reported by individual users (i.e., user’s birthdays on Facebook)
• Acquired from another data broker or platform (i.e. email lists acquired from other sources
• Inferred based on user behavior (such as a user’s gender on Twitter)

Such sourcing labels would help marketers more judiciously allocate ad spend. It would also have the advantage of checking if the data was ethically sourced. If audience segments are derived without informed user consent, they may violate consumer’s privacy rights (and be liable to forced deletion by regulators such as the US Federal Trade Commission).

This study examined how various brands were able to reach different proportions of audience members in a small cohort. While brands such as Amazon, Progressive Insurance, and Vrbo were able to reach a significant chunk of users in the cohort, other major companies such as Visa, CVS, Walgreens, and Toyota, did not. Brandable Box, a 5-employee organization, was able to reach 40% of the study audience whilst running ads on only three websites - wsj.com, washingtonpost.com, and nytimes.com. Whilst the significance of these results is obviously limited by lack of statistical power in the study, it does highlight the need for advertisers who are seeking to build brand awareness to look for orthogonal data about how many distinct people are actually seeing their ads. This is particularly true with the advent of the iOS App Tracking Transparency, future depreciation of third party cookies in Google Chrome, and regulation in Europe and the US on user tracking. 

One way to achieve that last point is to simply pay a sample of individuals who saw a particular ad for feedback - are these exposed individuals actually in the target audience for a particular ad? Was the ad appropriately targeted?

Take away points & recommendations

1. Crowd-sourced ad impression data and surveys can help consumers understand who is targeting them with ads, and help marketers understand how effective their ad campaign targeting is.
2. Some ad campaigns appear to have incorrect audience segmentation - this could be related to low quality third party data sources. Marketers should audit the quality of 3rd party data segments and ask questions about the provenance of those segments.
3. Marketers who are looking to build brand awareness should consider independent, empirical ways to monitor for reach and frequency
4. Advertisers with significant media investments may want to consider paying people who see their ads to provide (anonymous) feedback to help gauge if their ad campaigns are working.