2016 Honor Roll Methodology

The 2016 Online Trust Audit has evolved over the past 8 years and includes a composite analysis focusing on three major categories; a company’s consumer protection, data and site security and adherence to responsible privacy practices, including over three-dozen criteria.  One of the major changes for 2016 involves scoring sites which fail any major component of the site security assessment (normally equating to a “C” or lesser grade) as an automatic fail for the overall Audit. This change was based on the fact that a site’s security is only as strong as its weakest link and that the steps required to earn a “B” or better grade are simple, and it is unacceptable to leave the vulnerability unresolved.  

2016 Online Trust Audit Report

2016 Honor Roll Methodology Press Release — 3/4/2016

2016 Honor Roll Methodology Webinar — 3/23 (Presentation Deck)

2016 Honor Roll Methodology Webinar — 3/23 (YouTube video)

2016 Honor Roll Methodology

Sites are eligible to receive 300 total base points, including up to 100 points in each category. Bonus points are available for implementation of emerging best practices and penalties are assessed for vulnerabilities, breaches and regulatory settlements. Audits in the last year evaluated over 900 websites across multiple sectors including; the Internet Retailer Top 500, the FDIC Top 100 Banks, the Top 50 Federal Government sites, the Top 50 Social Networking sites, the Top 50 News/ Media sites, the Top 50 IoT manufacturers, 2016 Presidential Candidates, IRS approved free e-File sites and OTA Members.

Each sector will be scored in three categories:

1. Domain, Brand & Consumer Protection
2. Site, Server & Infrastructure Security
3. Data Protection, Privacy & Transparency

The 2016 scoring has been expanded and enhanced with additional weight and granularity given to key practices.  To qualify for the Honor Roll sites had to receive composite score of 80% or better and a score of at least 55 in each of the three categories. 

Data sampling of survey sites, their DNS, server infrastructure, email and privacy policies is planned to be completed between April 20 and May 15, 2016. In total, it is estimated more than 500 million email headers and approximately 100,000 web pages will be analyzed. It is important to note a site’s policies and practices may have change after the sampling and the data only reflects findings based on this period of time.

Addressing the ever changing security and privacy landscape as well as regulatory requirements, criteria continues to evolve with the bar raising in all areas. This year as in years past, criteria that were previously considered bonus points are now part of baseline requirements. Examples include adoption of DMARC, increased TLS/SSL granularity, as well as Do Not Track (DNT) disclosures and upgrading from SHA1 certificates. Addressing increasing consumer and regulatory concerns over the blurring of lines between advertising and editorial, the 2016 report will examine the use of native advertising and how such disclosures may impact consumer trust.

With the goal to drive adoption of best practices allowing all companies the ability to access their status and optimize their scores, OTA published the 2016 criteria in late January 2016 on the OTA website, blog and external facing newsletters.

It should be noted that this research is based on a “slice of time” and individual companies may have adopted or change their security and privacy practices after the Audit. We recognize that the sites examined might be using other technologies (which our tools or research did not detect) to authenticate domains or subdomains, secure their infrastructures, track users on their sites, etc. Due to the sensitivity of this data and risk of disclosing vulnerabilities, individual organization’s scores and data are not publicly available. Information will be provided to site owners upon written request and verification.  For details, including reporting fees, please send an email to admin @ otalliance.org.

COMPONENTS OF THE COMPOSITE SCORES

DOMAIN, BRAND & CONSUMER PROTECTION

• Email Authentication (SPF & DKIM) – The report will analyze more than 500 million emails and the respective DNS infrastructure of leading sites and subdomains. Email authentication assesses efforts to protect users from domain and email spoofing via the adoption of two industry leading protocols – Sender Policy Framework (SPF) and Domain-Keys Identified Mail (DKIM). Sites receive maximum scores by implementing both SPF and DKIM authentication at the top level domain (i.e. yourdomain.com) as well as on their respective delegated subdomains (i.e. email.youremail.com). Verification of DKIM-signed email requires review of the email headers of individual emails via sampling providing by Agari, Microsoft and Return Path. Augmenting previous year’s methodology, OTA subscribes to marketing email newsletters and / or submitted inquiries to sites, to review responses providing for increased granularity of email data. Results are integrated into the composite scoring and factored as a component of the baseline points required to qualify for the Honor Roll. Verification of SPF records will be completed using the OTA DNS record lookup tool.  
• Updated – Domain-based Message Authentication, Reporting & Conformance (DMARC) – DMARC standardizes how email receivers perform email authentication using the SPF and DKIM mechanisms.  Sites that have published DMARC records receive a positive score. This year publishing a “reject” or “quarantine” policy is part of the baseline scoring for email authentication.  Due to growing adoption and success, additional weighting will be given to sites publishing a DMARC record this year.  DMARC Information >.  Verification will be completed using the OTA query tool.
• Domain Locking – Domain locking is a security enhancement offered by most registrars to help prevent unauthorized transfers of your domain to another registrar or web host by locking your domain name servers.  When your domain is locked, you’ll be substantially protected from unauthorized third parties who might try to redirect your name servers or transfer your domain without your permission. Sites receive negative points if their domain is not locked.
• New – Transport Layered Security (TLS) for Email – New for 2015, sites which have implemented ” New in 2015, sites which implement “opportunistic TLS” will receive bonus points. TLS helps prevent eavesdropping on email as it is carried between email servers that have enabled TLS protections for email. Just as TLS can be used to secure web communications (HTTPS), it can secure email transport. To maximize the content security and privacy, TLS is required between all the servers that handle the message including hops between internal and external servers.
• Domain Name System Security Extension (DNSSEC) – Testing for DNSSEC will be completed by Verisign and IDD. Sites adopting DNSSEC will receive bonus points.

SITE, SERVER & INFRASTRUCTURE SECURITY

• Updated – Server and SSL Configuration – Sites will be evaluated using a combination of data and tools from DigiCert, GlobalSign, High-Tech Bridge SA, Qualys Labs, SiteLock and Symantec. These tools provide visibility into the server architecture, configuration and digital certificate. Testing checks for weak keys, protocols, algorithms, and server misconfigurations that can enable attackers to exploit system vulnerabilities and compromise SSL communications.   Refinements to the SSL scoring include applying negative scoring is the following are observed:
  • Servers that use SHA1 certificates, (Including intermediate certificates)
  • Sites vulnerable to POODLE
  • Servers that don’t support TLS_FALLBACK_SCSV
  • SSL 3 is supported
  • If RC4 is supported
  • If the chain is incomplete
  • Fail servers that have SSL3 as their best protocol
• New – Bonus point for use of Organization Validation (OV) and EV-SSL certificates; neutral scoring for Domain Validated (DV) and self-signed receive a negative score). Organization Validation (OV) or Extended Validation (EV) practices contain the verified name of the entity that controls the website.  Certification authorities (CA) issuing these certificates check with third parties to establish the official name of the organization and where they are located. In contrast, Domain-Validated certificates are typically verified through automated processes. A DV certificate contains no identifying information in the organization name field. Typically, this value just re-states the domain name or simply says “Not Validated.” Although the certificate supports transaction encryption, the end-user cannot trust the certificate to confirm who is on the other end.
• Extended Validation SSL Certificates (EV SSL) – EV SSL offers trust mechanisms visibly confirming the identity of the site to the user. The 2016 analysis will focus on all sites with SSL connections, not limiting the evaluation to consumer facing e-commerce or banking sites.  Cybercriminal target business-to-business, social networking and government sector sites with non-EV Certificates. Acquiring an Extended Validation certificate requires extensive verification by the certificate authority.  Sites that have implemented EV SSL Certificates received additional bonus points.
• Always On SSL (AOSSL) – Sites are evaluated for the adoption of AOSSL and /or HTTP Strict Transport Security (HSTS) as best practices to secure sensitive data between a user’s device and a web site. With the advent of widely available tools, criminals can “sidejack” cookies and data packets from unsuspecting users. Sidejacking allows hackers to intercept cookies (typically used to retain user-specific information such as username, password and session data) when they are transmitted without the protection of SSL encryption. Sites supporting AOSSL receive additional points, with added weight from previous years. This capability is measured using the Qualys SSL Server Test and other tools to look for Strict Transport Security and verified by auditors accessing the sites.
• Malware, Malicious Links & Cross-Site Scripting – Sites will be scanned for malware, malicious links and susceptibility to cross-site scripting. Sites with vulnerabilities will receive penalty points.
• Bot and Botnet Protection – Sites will be checked for basic protection against web scraping, vulnerability scanning, scripted form completion, and other common bot-driven activities. Sites which have such protection will be awarded bonus points.

DATA PROTECTION, PRIVACY & TRANSPARENCY

• Privacy Policy & Tracking Score  – Sites will be analyzed for their privacy policy and data collection practices. The privacy policy score evaluates privacy risk based on a website’s published policies about protection of personal data and the privacy qualifications of third-parties seen to be collecting data on the site. Website privacy policies regarding sharing, deletion, retention policies, disclosure notices and vendor confidentiality are reviewed by analysts. Scoring for third-party tracking companies (reflecting policies on anonymity, boundaries, choice, retention and oversight) are weighted based on their prevalence in site scans. Sites are crawled for a period of at least 96 hours to observe data collection and onsite tracking. It is important to note that scores are dynamic and can change based on the mix of third-party tracking and revisions to privacy policies. Results are integrated into the composite scoring and factored as a component of the baseline points required to qualify for the Honor Roll. 
• Notice If Data Is requested & Shared Based on Court Orders – Addressing the increased legal disclosures being made by third parties and governments, site were evaluated for disclosure polices.  Recommended copy to maximize scoring includes “To the extent we are legally permitted to do so, we will take reasonable steps to notify you in the event that we are required to provide your personal information to third parties as part of legal process.”
• Do Not Track Browser Settings (DNT) – In response to the State of California disclosure requirement for a site regarding Do Not Track, Websites privacy policies will be evaluated for compliance as part of the core scoring.  In addition sites which publicly disclose they are honoring the browser-based DNT setting received bonus points.  Such an assertion would be in addition to any such notice a user is presented when visiting a site and does not preempt any such notice.  A DNT signal asserts a user’s request to not collect and share their online data.  Sites that Honor the DNT signal will receive additional bonus points. For an example of a DNT notification best practice review the OTA Privacy Policy.  For additional information see the updated California Guidelines
• Public vs. Private WHOIS registration – Sites that are registered by proxy or private registration will receive a negative score, reflecting a lack of transparency. While it is recognized that sites may choose to opt-in for private domain name registration, public facing sites are discouraged from doing so and consumers should exercise caution when interacting with sites that have made their domain information private.  Results will be integrated into the composite scoring as a negative score for sites with private registrations and factored as a component of the baseline points required to qualify for the Honor Roll.
• Tag Management System or Privacy Solution – Sites that implement a tag management system or a standalone privacy solution receive bonus points. A tag management system provides a site operator visibility of data collection practices and how to proactively manage third party activities. A standalone privacy solution monitors third party activity site for the site operator. They enhance the ability to manage analytics tools, marketing tags, and other tag-based technologies that may collect and share data. Websites will be scanned using tools from Ensighten and other data providers.
• Data Breach & Loss Incidents – Companies who have experienced a data breach or a data loss incident since January 1, 2015 will receive negative points. See 2016 OTA Data Protection & Breach Readiness Guide
• FTC / FCC / State Settlements – Companies which have been in violation of the FTC Act including settlements and judgments since January 1, 2015 receive negative points.  The FTC Act and related FCC regulation focuses on consumer protection, including but not limited to deceptive advertising, privacy and data security practices. 
• Updated – Layered Short Notices – Shifted in 2016 to be integrated into baseline scoring. OTA’s Privacy Policy as an example.
• Privacy Policies with icons.  Building on short layered notices, site with the use of consumer friendly icon receive additional bonus points.  See example of Publishers Clearing House 

SECTORS TO BE EVALUATED

• Internet Retailer Top 100 (IR100) & Internet Retailer Top 500). Ranking based on revenue as reported by Internet Retailer Magazine, produced by Vertical Web Media. Ranking as of April 1, 2016.
• FDIC top 100 banks (FDIC 100). Based on net assets as reported by the Federal Deposit Insured Corporation as of December 31, 2015.
• Top 50 Federal Government sites (Fed 50). Based on a combination of consumer traffic and recent cybercriminal targeting of Federal Government sites including forged email campaigns and phishing sites. Includes Cabinet level agencies at risk of such exploits.
• Top 50 Social Networking and sharing sites (Social 50). Includes social networking, dating, entertainment, gaming, document storage, photo sharing and collaboration sites.
• Top 50 News and Media sites (News 50). Includes top ranked news, content and media sites, (non-ecommerce or social).
• Top 50 News and Media sites (News 50). Includes top ranked news, content and media sites, (non-ecommerce or social).
• Top 50 IoT Device Manufacturers (Connected Home & Health / Wearables)
• OTA Member Companies (OTA Members). Includes commercial members including consumer and business to business sites. Does not include academia, law enforcement, professional members, public sector, non-profits or members companies who joined since May 1, 2016. https://otalliance.org/about-us/members.