An interview with Himaja Motheram, security researcher at Censys

Censys: In cybersecurity, few things are as critical as maintaining the integrity of digital certificates, which serve as the backbone for secure communications across the web. In August 2024, DigiCert, one of the largest and most trusted Certificate Authorities (CA), faced a significant compliance issue related to a bug in their Domain Control Verification (DCV) process. This bug affected over 83,000 certificates, forcing DigiCert to revoke them within 24 hours. Many organisations scrambled to replace their certificates or faced severe operational and security consequences.

To gain deeper insights into this incident and its broader implications, we at 4iMag spoke with Himaja Motheram, a Security Researcher at Censys, a company renowned for its comprehensive internet-wide security monitoring and one of the most trusted sources for certificate-related data. Censys has built a reputation as the global leader in monitoring certificates and tracking vulnerabilities across the web. During our interview, Himaja shared her expertise on how the bug unfolded, the critical role Censys plays in detecting such vulnerabilities, and the lessons organisations can learn from the DigiCert incident.

The DigiCert bug: a breakdown

Certificates form the foundation of trust on the internet, enabling secure communication between users and websites. When a website requests a certificate, a Certificate Authority (CA) like DigiCert must verify that the entity requesting it owns or has authorisation over the domain. This process is known as Domain Control Validation (DCV).

In DigiCert’s case, a bug was discovered in their DNS-based DCV process. Himaja explained, “The issue was due to a missing underscore character in the verification process. While it might sound like a small oversight, this bug had significant security implications. Under certain conditions, it allowed potential attackers to bypass verification and gain control over certificates they weren’t authorised to use.”

This vulnerability could enable malicious actors to generate certificates for domains they didn’t own, leading to a breakdown of trust between websites and users. The bug was particularly concerning for large organisations with many subdomains, as it created opportunities for widespread exploitation.

Small mistakes that lead to significant impacts

The fallout from the DigiCert bug was swift and severe. DigiCert, the fourth most active trusted Certificate Authority, reported that 83,267 certificates were affected by the bug. As a result, organisations were given just 24 hours to revoke and reissue these certificates to remain compliant with industry standards. This led to operational chaos, with companies racing to replace their certificates before their services were interrupted.

At the time of writing, Censys observed 26,373 impacted certificates still in use on public-facing hosts, nearly all of which had already been revoked. Himaja highlighted the strain on organisations: “For companies managing thousands of certificates, this incident was a nightmare. They had to manually replace certificates across their infrastructure, risking service outages and data security issues. It meant days of manually replacing certificates, all while their websites were vulnerable and potentially down. One firm even went as far as filing a legal restraining order against DigiCert to extend the 24-hour revocation window.”

One particularly notable case involved Alegeus, a financial technology company in the healthcare sector. The company filed a court order to delay revocation, citing severe operational disruptions. Their plea emphasised the gravity of the situation, with millions of healthcare participants potentially unable to access funds due to revoked certificates.

Insights from Censys: data and monitoring

At Censys, monitoring the global certificate landscape is a daily operation. With over 9 billion certificates in their dataset, Censys tracks expired, revoked, and improperly issued certificates on a large scale. Their data was instrumental in analysing the broader impact of the DigiCert bug.

“We don’t just monitor certificates in isolation,” Himaja explained. “We track how certificates are used in the wild, which gives us a more accurate picture of the impact. At the peak of the incident, we identified over 33,000 certificates in use on the public web, and within a week, that number dropped to around 26,000 as organisations scrambled to replace their revoked certificates.…We also track how certificates are used in the real world by combining our certificate data with our host data. Not every certificate gets used, but knowing which certificates are active on the public internet gives us a clear picture of how incidents like this affect businesses.”

This capability allows Censys to provide real-time insights into security events, helping organisations understand the scope of incidents and take swift action. In the case of the DigiCert bug, Censys tracked the number of affected certificates and the industries most impacted by the vulnerability. When the bug was discovered, DigiCert was required to revoke the affected certificates within 24 hours, as mandated by industry standards. However, this created chaos for many organisations. Websites went down, data was unprotected, and businesses scrambled to replace their certificates.

The critical role of the underscore

At the heart of the DigiCert bug was a missing underscore in the DNS record used for verification. While this might seem like a minor issue, the underscore plays a crucial role in preventing domain collisions and ensuring the integrity of the DCV process. Himaja explained the broader security implications: “The underscore is not just to prevent name collisions; it’s also a security measure to prevent attackers from creating certificates for domains they don’t own.”

In cases where users have access to control DNS records, such as with dynamic DNS services, the missing underscore could allow attackers to exploit this oversight and obtain valid certificates for domains they do not control. This underscores the need for stricter enforcement of DNS verification standards to prevent such vulnerabilities from being exploited.

Industry response and future solutions

The DigiCert bug was a stark reminder of the fragility of the digital trust infrastructure. Organisations must balance compliance requirements from Certificate Authorities with the practical challenges of replacing certificates at scale. Though necessary, DigiCert’s 24-hour revocation window proved too short for many businesses to respond adequately.

“DigiCert’s swift response was necessary, but it came at a cost for many companies,” Himaja noted. “This incident has shown that compliance issues like this can have huge ramifications, and it’s a wake-up call for organisations to ensure they have robust processes in place for managing their certificates.”

One potential solution to avoid future incidents like this is the implementation of automatic certificate renewal. Himaja pointed out the importance of automation in certificate management: “Automatic renewal wouldn’t have prevented the bug, but it would have made the mitigation response much easier. Instead of manually replacing certificates, organisations could have automated the process, reducing the risk of downtime and security breaches.”

Proactive measures: what can companies do?

While it’s impossible to fully eliminate the risk of bugs in the certificate issuance process, companies can take several proactive steps to minimise their exposure. One critical measure is monitoring Certificate Transparency (CT) logs, which provide visibility into all issued certificates. This allows organisations to spot improperly issued certificates and act before they are exploited.

“Monitoring CT logs is a must for any organisation that relies heavily on web-facing assets,” Himaja says. “It gives companies the ability to detect unauthorised certificates in real-time and revoke them before they cause damage.”

Additionally, companies should conduct regular tabletop exercises to prepare for incidents like the DigiCert bug. Himaja emphasised the importance of preparedness: “It’s vital for companies to simulate worst-case scenarios. If all of your certificates were revoked tomorrow, would you be able to respond in time? Preparing for incidents like this is critical to maintaining security and business continuity.”

The lessons learned and the road ahead

The DigiCert DCV bug highlights the growing complexity of certificate management and the increasing importance of compliance in the digital trust ecosystem. Himaja predicts that stricter compliance standards will be implemented as the industry moves forward to ensure that similar incidents are less likely to occur.

“We’re going to see more stringent requirements for Certificate Authorities, and that’s a good thing,” Himaja concluded. “The internet runs on trust, and maintaining that trust is only going to become more challenging as the web continues to evolve. But with better monitoring, automation, and preparedness, we can reduce the risk of incidents like this in the future.”

The DigiCert bug was a stark reminder of the importance of certificate management and the potential consequences of even minor errors in the verification process. As organisations continue to rely on digital certificates for secure communication, proactive measures like automatic renewal and monitoring of Certificate Transparency logs will be essential to safeguarding their digital assets. The industry must evolve to meet these challenges, ensuring that the trust that powers the internet remains intact