Introduction to Product Data Syndication
In today’s digital commerce landscape, product data syndication has become a critical component for manufacturers to sell their products outside of their own websites. The COVID 19 pandemic didn’t create this need, but COVID did accelerate it for B2B and B2C manufacturers as we all became locked in smaller silos. As selling on digital channels ramped up, so did the need to transmit product data to more endpoints to continue to sell products.
Accurate and consistent product data is fundamental for both operational efficiency and customer satisfaction. Discrepancies or inaccuracies in product information can lead to lost sales, increased return rates, and a diminished brand reputation. Therefore, leveraging robust product data syndication mechanisms helps mitigate these risks by facilitating the synchronization of product data across all platforms where products are listed. This is particularly vital in a multi-channel retail environment where different platforms may have varying requirements for product information.
Product data syndication began in spreadsheets and still resides in spreadsheets for most retailers. Although APIs are available for companies like Amazon and Walmart, the complexity required to manage an API connection and build the externally available endpoints is beyond the desire of many big box retailers that have functioning syndication systems based on email and spreadsheets.
Therefore, simply getting your product data in whatever format the retailer requires is a challenge, let alone dealing with 20 or 30 retailers at a time. COVID pushed companies to find more channels, but in doing so increased the complexities for the teams managing syndication. And this is only 1 of many challenges facing manufacturers selling to retailers and marketplaces today.
The Complexity of Diverse Retailer Requirements
One of the central challenges of product data syndication is navigating the diverse requirements posed by different retailers. Each retailer often demands unique attribute sets, which makes it incredibly difficult for brands to maintain consistency and accuracy across multiple platforms. For instance, while one retailer may prioritize detailed descriptions and high-resolution imagery, another might focus more on specific attributes such as dimensions, weight, and technical specifications.
This is not just restricted to category-specific attributes. Most retailers use product data syndication to create (or birth) a SKU. There are unique system requirements to start that birthing process, as the ERP and Ecomm systems used by retailers vary in their requirements. Aligning something as simple as a GTIN (Global Trade Item Number) can be difficult, as some retailers utilize the GDSN framework while others don’t. Understanding how to map a UPC between GTIN and SKU varies by retailer. This is the first out of hundreds of attributes retailers require to set up an item in their system, all with similar complexities.
Then there is the issue of category-specific attributes. A fashion retailer might ask for a detailed description of fabric type, washing instructions, and size charts, whereas an electronics retailer could require intricate specifications such as battery life, compatibility data, and warranty information. These mismatched requirements necessitate customized data sets for each retailer, complicating the syndication process significantly. Imagine the number of templates that a manufacturer needs to maintain to sell their products on a discount retailer like Target, which sells in thousands of categories, and then imagine the same activity for Walmart, Best Buy, Home Depot, Lowes, Menards, etc.
Then imagine the same activity, but with each of those retailers making changes daily to their categories and the category-specific attributes within those categories. I have previously worked for one of those retailers, and we changed our entire template structure so often that we had to have a tool built just to output the new templates daily. Multiply this rate of change by all the retailers a manufacturer sells through and syndication starts becoming a challenge to keep up with.
Additionally, multimedia content often poses additional obstacles. Retailers may have different criteria for product images, videos, and augmented reality elements. Some demand images with a pure white background, whereas others might prefer lifestyle or in-context photos. Additionally, while a few retailers integrate videos demonstrating product use, others might ignore these completely or have specific video specifications. Meeting these varied demands requires brands to create multiple versions of the same product content, adding to the logistical burden.
This complexity becomes especially pronounced when dealing with international markets. Retailers in different regions may have unique compliance requirements, language translation needs, and cultural considerations that further complicate the data syndication process. As a result, brands must constantly adapt their product data strategies to align with the preferences and regulations of each retailer, often incurring significant time and resource investments.
The lack of standardization is the primary culprit, driving brands to adopt a fragmented approach to product data management. Without a universal set of guidelines, brands are left to navigate a labyrinth of individualized requirements, causing inefficiencies and increasing the likelihood of errors. Consequently, the syndication process becomes a bottleneck, thwarting efforts at seamless and effective product data dissemination.
Why Isn’t There an Industry Standard?
The simple solution to this would be for all the retailers to agree on a single specification for how they collect data. With a single file format, a single set of categories in a hierarchy that everyone agrees upon, and attributes that are standard based on industry, this problem would suddenly become highly manageable. The length of an item should be the same no matter where you sell it, and most technical specifications don’t worry by retailer. Yet we have not seen this elusive industry standard arise.
The reasons for this are complex, but we will attempt to explain the evolution of this through 3 different points: The Competitive Effect, the Google Effect, and the Amazon Effect.
The Competitive Effect: As early as 2006 there were attempts to normalize to a set of categories and attributes to solve this issue. It fell apart because of a desire among manufacturers to exploit the system for their own gain. I was part of a group that attempted to standardize the attributes in the durable goods categories as a way of kick start this process, but every B2C manufacturer wanted their own proprietary information included in both the categories and the attributes. Color alone caused such problems, as appliance manufacturers spent a decade building up their own proprietary color spectrums, that the entire process fell apart.
I know of 3 other attempts to reign in competitive effects, both on the manufacturer side and the retailer side, all of which failed for the same reason. Manufacturers have spent enormous amounts of resources differentiating their product lines to build brand loyalty that they have no desire to normalize and lose those differentiations.
The Google Effect: Then there is RankBrain. We could go into how RankBrain functions, but other experts have created better explanations than we could here. (Moz.com, SearchEngineLand.com, SEMRush, and even Google Themselves) But the impacts and effects of RankBrain are vital to winning the SEO game: You need unique content, you need an internal and external linking strategy, and you need to get product data to market quickly to own your share of the SEO space for any set of product types.
RankBrain actually penalizes search results with duplicate copy. If you use the same copy on two retailers’ websites for a single product, both search results will be lower in the rankings than a site with unique copy. Add to that the difficulty in establishing content in the keyword domains Google sets out to differentiate between informational, commercial, and navigational intent types can mean the difference between someone using your site to find other sites, educate themselves on products, or buy products. Sometimes you need to own two or three of these intents to win.
The Amazon Effect: Finally, there is the Amazon effect, which is essentially just a variation of the Google Effect. Amazon invented the concept of A+ content, which is the content than is below the first window (or first fold as it is called) on an Amazon listing page. This added content gives vendors wanting to sell on Amazon extra places to put unique content to make their listings stand out, but also to give more opportunities for keyword placement to drive both internal Amazon searches and external Google search listings.
This was so successful that many retailers have added Below-the-Fold content options to their websites. Target, Walmart, Home Depot, and many others use this content space to own both the commercial and informational intent spaces to clarify their positioning within Google’s search index. This means manufacturers not only have to compete on the tradition romance copy elements, but also on these additional copy pieces that have no standard loading mechanisms and take extreme amounts of resources to get right across all channels.
A secondary impact from Amazon is the rise of the marketplace, where manufacturers and distributors both fight for placement of the exact same product lines across multiple marketplaces. A manufacturer that sells direct may be competing for the exact same listing as a distributor to whom that manufacturer sells those products. This channel conflict has forced a new phenomenon called disintermediation, where manufacturers are attempting to sell direct to avoid this channel conflict with their distributors. It allows them to better understand their customers while increasing profit margins, all while cannibalizing sales to their channel partners.
File Formats: A Babel of Tech Specs
In the realm of product data syndication, one of the most significant challenges brands encounter is the diversity of file formats demanded by different retailers. Retailers, each with their unique system requirements, often necessitate product data to be delivered in specific formats such as CSV, XML, JSON, or even proprietary formats. This variation represents a technical maze for brands looking to seamlessly share their product data across multiple retail platforms.
CSV (Comma-Separated Values) and Excel files, for instance, offer a straightforward way to list product attributes in a tabular form. However, the simplicity of CSV comes at a cost — they can be prone to errors during data import and export processes, especially when dealing with complex product data. Yet, the intricacies involved in writing and parsing XML files can be a deterrent for brands lacking advanced technical capabilities.
XML (eXtensible Markup Language) files present another common format, renowned for their flexibility and ability to handle hierarchical data. JSON (JavaScript Object Notation), on the other hand, is increasingly favored due to its lightweight nature and ease of use, especially in web applications. However, systems are required to build XML and JSON outputs, as these are highly structured and not human-readable formats. JSON is also the native format of APIs, but the truly functioning APIs are few and far between.
The technical complexities of converting product data to meet these diverse file format requirements cannot be understated. Each conversion process involves meticulous mapping of data attributes to ensure accuracy and completeness. Brands must invest in robust data management and transformation tools to efficiently tackle these challenges. Furthermore, maintaining data integrity throughout these conversions is paramount, as inconsistencies or errors can lead to significant ramifications for product visibility and sales performance on retail platforms.
In essence, the lack of standardization in file formats represents a critical hurdle in the landscape of product data syndication. Overcoming this Babel of tech specs requires investment in technology, expertise, and continuous adaptation to the evolving demands of retailers.
Template Variations: One Size Doesn’t Fit All
Every retailer has its unique template for submitting product data, demanding specific fields, distinct formats, and unique ordering. These variations, although necessary to some extent, create a complex and labor-intensive scenario for brands. They are tasked with continually adapting their data to meet each retailer’s distinct requirements, which is no small feat. This process is often fraught with potential for errors and inconsistencies due to the diverse nature of these templates.
For instance, one retailer might prioritize product descriptions and images in a specific sequence, while another might focus on pricing and SKU details upfront. Additionally, the required data fields can vary significantly, with some retailers demanding more granular details on products than others. This necessitates that brands tailor their syndication efforts meticulously, aligning their product data to the specifications of multiple templates across different retail platforms.
Moreover, the repetitive and detail-oriented task of reformatting and reordering product data amplifies the potential for human error. Even a minor inconsistency in data formats can result in product information being displayed inaccurately, ultimately impacting customer satisfaction and sales. Brands must constantly ensure that product data is up-to-date and synchronized across all platforms, a process further complicated by the lack of a standardized approach.
The absence of standardized templates means that brands often employ substantial resources in time and labor to manually modify product data for each retailer. This not only hampers efficiency but also diverts attention from other strategic business activities. Furthermore, the inconsistencies in product data presentation across various retailers can lead to consumer confusion, eroding brand trust and loyalty.
In conclusion, the diverse template requirements of retailers highlight the pressing need for standardization in product data syndication. Streamlining these processes can significantly reduce the burden on brands, ensuring more accurate, consistent, and reliable product information reaches consumers, thus enhancing overall market efficiency.
The Lack of Industry Standardization
The absence of a unified system for product data syndication across the retail industry poses significant challenges for all stakeholders involved. A primary factor contributing to this disorder is the diverse and sometimes conflicting interests of retailers. Different retail sectors, ranging from grocery to electronics and fashion, have unique requirements for product data. This variability complicates the creation of a one-size-fits-all model that can serve the entire industry. The customization needed to meet specific requirements often leads to fragmented and incompatible systems.
Another barrier to standardization is the technology gap among retailers. While some retailers operate on advanced platforms capable of handling complex data sets and seamless integration, others may use outdated systems that struggle to manage even basic product information. These technological disparities make it challenging to establish a universal standard that accommodates both ends of the spectrum. Furthermore, technological incompatibility often results in increased costs and time investments as businesses attempt to adapt their systems to varying demands.
Competitive interests also play a significant role in hindering standardization efforts. Retailers may feel that unique modes of data syndication give them a competitive advantage, deterring them from adopting common standards that could potentially erode their market position. This competitive fragmentation results in a landscape where mutually beneficial cooperation is limited, further entrenching the problem.
Lastly, the regulatory environment can be both a barrier and a driver for standardization. Different countries and regions often have differing regulations regarding consumer information and data privacy, making it difficult to create a universally accepted standard. Compliance with these various legal frameworks adds complexity, as product data needs to align with diverse regulatory requirements.
Collectively, these barriers create an ecosystem where product data syndication remains siloed and inefficient. The need for a standardized system is evident, yet overcoming these obstacles requires coordinated efforts and a willingness to collaborate that, at present, the industry lacks.
The Impact on Brands and Retailers
The fragmented nature of product data syndication has profound consequences for both brands and retailers. One of the most significant impacts is the increased costs incurred due to the lack of a standardized approach to data exchange. Brands often need to adapt their product information to meet the specific requirements of various retailers, necessitating additional resources and expertise. This process not only demands significant manpower but also escalates expenditure on technology and systems designed to cater to diverse data formats.
Time delays are another critical issue stemming from non-standardized product data syndication. The necessity to tailor product data for distinct retailer specifications means that launching new products or updating existing ones takes considerably longer. Consequently, brands may miss crucial market windows, facing lost sales opportunities and diminished competitive advantage.
Potential for errors increases exponentially in the absence of uniform data practices. When product information must be manually adjusted for different retailers, the likelihood of inaccuracies rises. These errors can range from minor inconsistencies to major discrepancies that misrepresent the product. Such mistakes can lead to customer dissatisfaction and increased return rates, further straining the relationship between brands and retailers.
Consumer satisfaction and brand integrity are also at stake due to the disjointed nature of data syndication. Discrepancies in product information across various retail platforms can confuse consumers, leading to a fragmented customer experience. When product details do not align across channels, it erodes consumer trust and can damage the brand’s reputation. This trust erosion frequently results in lower customer loyalty and reduced sales over time.
Overall, the lack of standardization in product data syndication poses substantial challenges. By exacerbating costs, delaying time-to-market, increasing the potential for errors, and impacting consumer satisfaction, it hampers the effectiveness and efficiency of both brands and retailers. Recognizing these implications highlights the urgent need for standardized practices in the industry to foster smoother operations and enhanced market performance.
Current Solutions and Workarounds
In the intricate landscape of product data syndication, numerous solutions have emerged to mitigate the challenges faced by retailers. Prominent among these are Product Information Management (PIM) systems, data aggregators, and third-party syndication services. Each tool comes with its distinct approach, aiming to bring some semblance of order to the chaotic process of synchronizing product information across various platforms.
PIM systems are often hailed as a comprehensive solution to manage product data syndication. These platforms centralize all product-related information, enabling retailers to maintain consistent and accurate data. A PIM system allows for the integration of product data from multiple sources, ensuring a single point of truth. Retailers benefit from streamlined workflows and improved data quality, which ultimately enhance customer experiences. However, despite these advantages, PIM systems can be expensive to implement and maintain, and they might require significant customization to fit the unique requirements of different retailers.
Data aggregators represent another critical tool in the domain of product data syndication. These organizations collate data from various sources, standardizing it to facilitate easier distribution to multiple platforms. By acting as intermediaries, data aggregators play a pivotal role in reducing the complexity that retailers face when managing product data. The primary advantage of using a data aggregator is its ability to curate vast amounts of data efficiently. Nonetheless, issues such as data accuracy, timeliness, and dependency on third-party synchronization still pose challenges. Retailers often find themselves scrutinizing the aggregated data for errors, which can be both time-consuming and labor-intensive.
Third-party syndication services offer a more specialized approach, focusing on the distribution of product data across specific channels. These services handle the nuances of each platform’s requirements, ensuring that the data presented is in the correct format and meets the necessary standards. Third-party syndication services can significantly ease the burden on retailers by automating the distribution process. However, they often come with their own limitations, such as the potential for data mismatches and a lack of direct control over the syndication process.
The Future of Product Data Syndication: Towards Unity?
The future of product data syndication is poised at a critical juncture, with the industry steadily moving towards greater standardization. As retailers demand more comprehensive and accurate product information, the need for unified syndication practices becomes increasingly apparent. Emerging technologies are set to play a pivotal role in this evolution, offering promising avenues for tackling the current challenges.
The main focus of unity at this time is data pools. There are two essential types of data pools: The GDSN system and Distributed Data Pools. GDSN relies on a single standard for validating logistics data to pass between data sources and data recipients. Distributed Data Pools are companies that harvest product data across all available sources and then sell access to that data.
Unfortunately, most GDSN data pools have extra data attributes they maintain on top of the GDSN standard. This “Top Off” data allows them to align to a vertical, like 1WorldSync’s focus on grocery, or specific retailers. This push towards a unifying standard has actually lead to de-standardization, as this Top Off data cannot be passed between pools. In attempting to solve these issues GS1, the owner of the GDSN standard, has allowed their data pools to defy the need for unification.
Similarly, Distributed Data Pools have fragmented into non-standard data pools without a comment data format, standardization, or even a common data set. They believe that one day they will be the last pool standing, therefore becoming the defacto standard. However, they have become sources of data with poor normalization, poor data quality, and massive gaps in their listings.
So what is the solution? Unfortunately, there are too many competing factors to see a straight line to standardization. I personally have tried resurrecting the industry standard concept, only to find that the technology and buy-in required is beyond the grasp of the manufacturers and retailers from a commercial perspective. Although I will continue to advocate for this standardization and look for partners with the reach to influence the markets, industry standardization is a long ways away.