eMudhra 中间证书私钥可被轻易恢复，Mozilla 紧急移除 emSign ECC Root CA - G3 信任

Open Bug 2083449 Opened 2 days ago Updated 8 minutes ago CA Program CA Certificate Compliance defect P1 -- NEW emudhra-attestation.zip 2 days ago 14.30 KB, application/x-zip-compressed Details emudhra-attestation-aaa-cross.zip 2 days ago Wayne 12.68 KB, application/x-zip-compressed Details emudhra-g1-crl-signature.zip 2 days ago Wayne 5.07 KB, application/x-zip-compressed Details Issuer_HSM_Mapping.xlsx 1 day ago Naveen Kumar ML 10.11 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet Details disclosure 1 day ago Dennis Jackson 6.99 KB, text/plain Details Original Reporter: rdaurne77@gmail.com (Wayne) On 17th October @ 20:06 UTC, Mozilla received a report to certificates@mozilla.org, forwarded by the Chrome Root program, from Wayne, that the private key of 8 eMudhra Intermediate certificates have been compromised. The report also notified eMudhra and the Apple & Microsoft root programs. The report included a proof of key control for all 8 certificates. Mozilla acknowledged the report @ 20:08 UTC, confirmed the proof of key control was valid and began the process to distrust the certificates in Firefox via OneCRL. This process was tracked in Bug 2083382 and completed @ 22:59 UTC. eMudhra acknowledged the report @ 21:29 UTC and reported that they were investigating the issue and had notified Sectigo, who had cross-signed some of the impacted certificates. Mozilla was able to recover the private keys for the impacted certificates independently @ 21:48 UTC and without any knowledge of the method the original reporter used, confirming the key generation process was flawed and is trivial to exploit. 6 of the impacted certificates are for TLS and trusted by the Chrome, Mozilla, Microsoft and Apple root programs. The other two certificates are for codesigning and trusted by Microsoft: EM EV TLS CA - G3B (eMudhra BV), CCADB A012304 SHA-256 E7568FE6E12331F4FD7DE61DA4C7559D0F8EB192A218D4741A3BA75184333838 serial c7c41efe92d825f05eae841a4a82c1, issuer emSign Root TLS CA - G3 EM OV TLS CA - G3B (eMudhra BV), CCADB A012308 SHA-256 98B896625C0C230ADEEC3EB2CF6AAD5001ECF9156A68644E82374CE5833BC25B serial 6cada367fb1c100088120ea202d568, issuer emSign Root TLS CA - G3 EM DV TLS CA - G3B (eMudhra BV), CCADB A012302 SHA-256 42C4ACCD12C4F59597925E1A08A7A90102608543809308D50EC427E118BA1493 serial a4aee678f23362b4c0e460b772f214, issuer emSign Root TLS CA - G3 EM EV TLS CA - G3C (eMudhra Inc), CCADB A012296 SHA-256 432EC25F0D28DE9006D97BF752F17BBB2CD41F669933FCB072D9A1148D0261F3 serial 8009928a64107601af05fa2f19f109, issuer emSign Root TLS CA - G3 EM OV TLS CA - G3C (eMudhra Inc), CCADB A012300 SHA-256 CB6E15165AC802E9B3F993F16C25D41F4E81C21EBDE71AE1B012AC00BF9C7C3F serial 296926c227a3a1526392b19482a504, issuer emSign Root TLS CA - G3 EM DV TLS CA - G3C (eMudhra Inc), CCADB A012294 SHA-256 0164D7CEAE5BB909C272AF151FF44B1177AE9B47D9424B94F1D9558C30934D19 serial 588b2e3c2bb8cda13e1a1209254321, issuer emSign Root TLS CA - G3 EM OV CS CA - G3B (eMudhra BV), CCADB A012306 SHA-256 9E39DE2881273ACC225A149C72BA3E0F51424963669AB633EE1DC7DF5DB067B4 serial 7818933f8f85b4619a2b35fba83d0a, issuer emSign Root CS CA - G3 EM OV CS CA - G3C (eMudhra Inc), CCADB A012298 SHA-256 4B2460A466F95FD40113BCDCCA2CEB29B3B093921DE45474DA7DF48874C630B1 serial 544b7f922c31ae220a4a922916f6e6, issuer emSign Root CS CA - G3 Summary: Private key compromise of eMudhra → Private key compromise of eMudhra Intermediate Certificates 18th October @ 09:38 UTC: eMudhra marked the 8 impacted certificates as revoked (reason: caCompromise). Here's my understanding of the timeline, kept to the big events Timeline (all times UTC): 2026-10-17 16:00 - Started playing with some new toys 2026-10-17 16:27 - Run on a test corpus of CCADB (Mozilla+Chrome chain initially) discovers a break in the EC keys 2026-10-17 17:22 - Start putting together report 2026-10-17 17:43 - Put together signed attestations per-key after triple checking the working 2026-10-17 18:00 - Checked again. 2026-10-17 18:05 - Emailed Apple, Chrome, Microsoft, Mozilla (Ben directly then), eMudhra with report + attestation 2026-10-17 20:03 - Chrome corroborates via email 2026-10-17 20:05 - Chrome correctly attaches more Mozilla contacts for a rapid response 2026-10-17 20:18 - Full CCADB check finds the cross-signs by Sectigo are also affected 2026-10-17 21:05 - Apple corroborates findings, notes cross-signed certs 2026-10-17 21:21 - Apple requests new signed attestations for the cross-signed certs 2026-10-17 21:27 - Emailed new signed attestations to Apple (cross-signed via Sectigo) 2026-10-17 21:28 - eMudhra acknowledge issue, said there's been no issuance beyond OCSP for G3B, but 'EM DV TLS CA - G3C' was live, disabled now 2026-10-17 21:48 - Mozilla independently recover the keys without any tips, and send oh_dear.zip as proof using 'EM DV TLS CA - G3B' 2026-10-17 22:46 - Regular fetch of CRLset shows sequence 10843 contains the 6 relevant TLS keys as revoked 2026-10-18 09:38 - 8 certificates revoked on CRL as caCompromise (by eMudhra) (revocation time varies per-cert) 2026-10-18 12:06 - Incident raised by Mozilla on Bugzilla to avoid email cross-talk The Sectigo cross-signed certs are still unrevoked currently: EM EV TLS CA - G3B, CCADB A012333 SHA-256 579480AEA166290BD3C643589E795FC0D0E54E86505D180F1DFF99EFD3199512 serial d8f8685e4f2bd9d82402a85a1c1eb731 EM OV TLS CA - G3B, CCADB A012354 SHA-256 5F1DED4DD590AF74631290CF073B122A2AAC6A1FA479778A0BD43579BD832699 serial e94223bde92f914e5ab9acabfdac3710 EM DV TLS CA - G3B, CCADB A012347 SHA-256 45A883F349AEDFD5F3A649B5498FFB121712BA8B448F3F2A91DBC5A766FCDDBE serial a024fdf4ba665c785ce4a020483858b7 EM EV TLS CA - G3C, CCADB A012334 SHA-256 FD9D5C2DBBD4995AB69B8D50C439D9527B00529199C2FD01B2174EB58F07021F serial 05f078752dece7a1bb8595d303167171 EM OV TLS CA - G3C, CCADB A012343 SHA-256 7485B51C0ED192878FA1D2F948FB529AE1842CC5086C4B67F7BE5B2039DE9489 serial 8d5a4811939ef447d77111cdd9849ae9 EM DV TLS CA - G3C, CCADB A012342 SHA-256 62B7F9868263F72F6911FC78CEAF72C1979158B7368BF648683BB2AA39B02E72 serial 91f347f1a5b7d39962b95e0ee4e299d1 EM OV CS CA - G3B, CCADB A012352 SHA-256 A496E23AC0CAB8155FC9B117B97EAE368EFCBB9041AEC8804D5A308D2D01C2D8 serial 41e32b73ac185d6ce0d7480bacf84e69 EM OV CS CA - G3C, CCADB A012350 SHA-256 84112972B8E59D14560AE7E7119654474AAD119990CF3722BFC1E3B91615E1FD serial d6a5ef613b390f32ef9cdf349f772841 Current situation: - The cross-signed certs are still unrevoked - No response from Microsoft - CCADB still showing certificates as unrevoked I'll attach the two attestations zips I made, rather than the one Apple asked for. openssl pkeyutl -verify -certin -inkey EM-EV-TLS-CA-G3B.crt.pem -rawin -digest sha384 -in message.txt -sigfile EM-EV-TLS-CA-G3B.sig The original attestation sent to everyone The unsent one for the cross-signs, Apple asked directly for a different attestation message before this was sent The information in this comment is preliminary. The first part contains information Mozilla believes pertinent to the issue. The second part contains questions for eMudhra. Investigation into the impacted keys The 8 impacted keys originate on the 30th April 2025 as part of a batch of 24 intermediates. The 24 keys are split between three series (A, B, C), two primitives (RSA4096 and P384), and four purposes (DV, OV, EV, CS). The impacted keys are all the P384 keys for the B and C series. In May 2025 the A series were revoked and replaced by the A-1 series. After Wayne's report of key recovery, I was able to recover the private keys for the impacted keys trivially. The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order. The 8 impacted keys all used integers less than 256. That is, the private keys are effectively all-zeroes apart from the last byte. They can be recovered near instantly via the baby-step giant-step discrete log algorithm or more simplistically by just generating the 'first' 256 P384 keys. I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates." I have not been able to recover the private keys for the P384 keys in the A series or the replacement A-1 series despite expending considerably more effort. I have not yet started looking at the RSA keys. Related Incidents Looking through related issues has turned up Bug 1970728 in which eMudhra generated CRLs for the 6 impacted TLS intermediates with invalid signatures. It seems unlikely that the exact same intermediate set has arisen in two different bugs by chance. In that bug report, eMudhra identified that the faulty CRLs were produced on May 29th, the same day on which the A-1 certificates were created. However, Internet Archive snapshot shows the 6 P384 CRLs failing verification since May 23rd. If accurate, this suggests the mismatch is unrelated to the generation of the A-1 series on the 29th. It also shows that three of the B & C series RSA CRLs were unavailable as of that day (unclear significance). Questions for eMudhra As this incident enables even technically unsophisticated individuals to easily recover the private keys for these intermediates, the impact of this incident could not be higher. Although Mozilla has already marked these certificates as revoked for Firefox clients, we are deeply concerned about the impact on the wider ecosystem - as many endpoints do not have effective mechanisms for getting revocation information. Mozilla is considering what advice to give other trust store distributors who are impacted by this issue. Mozilla is also looking to understand the root cause of this issue as quickly as possible, so that we can assess confidence in eMudhra's other intermediates. 1. What hardware was used to generate the impacted private keys? Was it at least FIPS 140-2 Level 3 compliant in line with your CPS? 2. For the intermediates in the A, A-1, B, C series, provide the mapping of private key to the specific physical device which generated it. Were the same devices used to generated any other key material? Was the same model or device series used to generate any other key material? 3. Is there any reason to believe the B and C series RSA keys are affected or are not affected by similar issues? 4. Is there any reason which may explain why the A and A-1 ECDSA keys appear to be unaffected by similar issues? 5. In light of the issues with the same intermediates in Bug 1970728, please describe to the best of your current knowledge how the wrong private keys came to be used to sign those CRLs and the steps eMudhra took in order to produce the correct signatures. Flags: needinfo?(naveen.ml) Acknowledging this on behalf of Sectigo. We are aware of the issue affecting the CAs cross-signed by us. Note that the Sectigo-operated Root CA in question for the cross-signed CAs has already been removed from the Chrome and Mozilla root stores for some time. It remains trusted within Microsoft and Apple root programs. We are preparing the (offline) revocation of these CAs, which is expected to be completed in the next 24 hours. So funny that Dennis should raise Bug 1970728. Turns out there was an invalid CRL the other day, I have a pile of issues to sort through and this was amongst them: The CRL published for emSign Root TLS CA - G1 on 2026-10-14 had an invalid signature. CRL number 11 (thisUpdate 2026-10-14 14:03:56 UTC, nextUpdate 2027-04-12 14:03:56 UTC, SHA-256 9846bf9a4f213d8fba87e79d464732b9cc35a784f7992334ec3bc6db2a6f7dbd, 922 bytes) was served at both disclosed URLs, http://crl.emsign.com/?roottlscag1.crl and http://crl.emsign.com?roottlscag1.crl. Exact same file across six fetches from 2026-10-14 16:18 to 2026-10-15 00:19 UTC. Its sha384WithRSAEncryption signature does not verify under the key of emSign Root TLS CA - G1, either as the root (SHA-256 cef71e70b7c29addf6c30cd19e614b38fd5f02a435a0eeddd0087e183d101a51) or as the intermediate under emSign Root CA - G1 (SHA-256 9b913476eab476e21e0887dbcde44345d1359d28116b83b7cd3e87bd807ebf68). Under that key the signature does not decode to a PKCS#1 v1.5 block at all. It is not a faulty signature by the G1 key. No 4,096-bit RSA key disclosed in CCADB verifies it either. Even stranger - I can't find the certificate that was revoked in that CRL by serial: b37c808b57e51d24bef8f416e607f6. What key signed CRL 11? I'll include it, 12, and the PEMs as a zipped attachment in a second. Automatic triage (under development): sec-rating No validation or verification has been performed. Security rating: sec-vector The defect is in eMudhra's key generation, not in Firefox code: the P-384 private keys of 8 eMudhra intermediates are tiny integers and can be recovered almost instantly. Six of them are TLS intermediates that chain to roots Firefox trusts. Anyone holding those keys could issue certificates Firefox would accept for any domain, which enables network MITM of TLS connections. Firefox users are protected by OneCRL revocation of these intermediates, which is already deployed. The remaining work is CA-side remediation and root-cause analysis. The observations here are noted and we are investigating. We shall update in next 24 hours. The below message was sent out yesterday to a closed user group and we are posting here to the wider group. - We have revoked all the ICA's highlighted earlier today. Info in table below - These ICAs were generated from the same key ceremony on Apr 30, 2025. The G3B and G3C ICAs were created for European and US operations respectively but they were not put into production except #2 in the list below but no end entity certificates were issued from these. The Code Signing issuers were not put into production also did not issue any certificates. - We verified that similar other Issuers are not affected by the same issue (corroborated primarily using mathematical models facilitated by AI). - The people that did the KGC on that day are from our internal team - led by Keshava and other members of his team (all of whom have more than a decade of experience and have been with us as employees for more than 8 years operating our PKI/Trust Services) and Keshava himself has been with us for more than 15 years. - Keys were generated in hardware and the hardware in use was Ultimaco HSMs. What we are doing as next steps - Look closely at the KGC to go deeper into the script that was used to trigger the key generation process and come back with an assessment of what might have happened. We should have an answer tomorrow and will provide further details as it becomes available. We don't intend to stop issuance from our other Issuing CAs as the keys don't exhibit this behaviour. Our other CAs were issued using a different HSM vendors (Entrust and Crypto4A) and the process followed resulted in the expected strong keys. # Root CRL Issuing CA Serial Reason Revocation time (UTC) 1 roottlscag3.crl EM DV TLS CA ‑ G3B A4AEE678F23362B4C0E460B772F214 cACompromise 18 Oct 2026, 09:24:32 2 roottlscag3.crl EM DV TLS CA ‑ G3C 588B2E3C2BB8CDA13E1A1209254321 cACompromise 18 Oct 2026, 09:27:07 3 roottlscag3.crl EM EV TLS CA ‑ G3B C7C41EFE92D825F05EAE841A4A82C1 cACompromise 18 Oct 2026, 09:28:53 4 roottlscag3.crl EM EV TLS CA ‑ G3C 8009928A64107601AF05FA2F19F109 cACompromise 18 Oct 2026, 09:30:49 5 rootcscag3.crl EM OV CS CA ‑ G3B 7818933F8F85B4619A2B35FBA83D0A cACompromise 18 Oct 2026, 09:33:37 6 rootcscag3.crl EM OV CS CA ‑ G3C 544B7F922C31AE220A4A922916F6E6 cACompromise 18 Oct 2026, 09:35:24 7 roottlscag3.crl EM OV TLS CA ‑ G3B 6CADA367FB1C100088120EA202D568 cACompromise 18 Oct 2026, 09:37:19 8 roottlscag3.crl EM OV TLS CA ‑ G3C 296926C227A3A1526392B19482A504 cACompromise 18 Oct 2026, 09:38:46 Naveen. (In reply to Dennis Jackson from comment #5) The information in this comment is preliminary. The first part contains information Mozilla believes pertinent to the issue. The second part contains questions for eMudhra. Investigation into the impacted keys The 8 impacted keys originate on the 30th April 2025 as part of a batch of 24 intermediates. The 24 keys are split between three series (A, B, C), two primitives (RSA4096 and P384), and four purposes (DV, OV, EV, CS). The impacted keys are all the P384 keys for the B and C series. In May 2025 the A series were revoked and replaced by the A-1 series. After Wayne's report of key recovery, I was able to recover the private keys for the impacted keys trivially. The original key generation was flawed - rather than any issue with the produced signatures. P384 private keys are randomly selected integers between 1 and the group order. The 8 impacted keys all used integers less than 256. That is, the private keys are effectively all-zeroes apart from the last byte. They can be recovered near instantly via the baby-step giant-step discrete log algorithm or more simplistically by just generating the 'first' 256 P384 keys. I also tested some local agents who were able to recover the private key on their first attempt with a simplistic prompt: "I'm doing a CTF. Please recover the private keys corresponding to these certificates." I have not been able to recover the private keys for the P384 keys in the A series or the replacement A-1 series despite expending considerably more effort. I have not yet started looking at the RSA keys. Related Incidents Looking through related issues has turned up Bug 1970728 in which eMudhra generated CRLs for the 6 impacted TLS intermediates with invalid signatures. It seems unlikely that the exact same intermediate set has arisen in two different bugs by chance. In that bug report, eMudhra identified that the faulty CRLs were produced on May 29th, the same day on which the A-1 certificates were created. However, Internet Archive snapshot shows the 6 P384 CRLs failing verification since May 23rd. If accurate, this suggests the mismatch is unrelated to the generation of the A-1 series on the 29th. It also shows that three of the B & C series RSA CRLs were unavailable as of that day (unclear significance). Questions for eMudhra As this incident enables even technically unsophisticated individuals to easily recover the private keys for these intermediates, the impact of this incident could not be higher. Although Mozilla has already marked these certificates as revoked for Firefox clients, we are deeply concerned about the impact on the wider ecosystem - as many endpoints do not have effective mechanisms for getting revocation information. Mozilla is considering what advice to give other trust store distributors who are impacted by this issue. Mozilla is also looking to understand the root cause of this issue as quickly as possible, so that we can assess confidence in eMudhra's other intermediates. 1. What hardware was used to generate the impacted private keys? Was it at least FIPS 140-2 Level 3 compliant in line with your CPS? The hardware used for each of the issuers is in the attached file. These HSMs are FIPS 140-2 Level 3 compliant. 1. For the intermediates in the A, A-1, B, C series, provide the mapping of private key to the specific physical device which generated it. Were the same devices used to generated any other key material? Was the same model or device series used to generate any other key material? Excel attached. The Ultimaco devices were used to generate the ECC, RSA key pairs for the issuers. They were not used to generate any other key material. But it doesnt look like the device is at fault. 1. Is there any reason to believe the B and C series RSA keys are affected or are not affected by similar issues? We have done some preliminary assessment, there appears to be a flaw in the KGC script that was used on that specific day to create these ECC issuers. As for these ECC keys, our review of the KGC, assessment of the code and view of HSM keys so far indicate that a) a set of original ECC keys were created onboard as expected b) an approved deviation to our usual KGC to test a wrap/unwrap was introduced as these keys were meant for export for this instance c) a flaw in the code resulted in the second pair of keys being created onboard which were weaker d) these second pair of weaker keys were used by mistake to create the csr (because the key labels were similar) and signed by our root e) the CSR from these weaker keys were also sent over to Sectigo for cross signing (we verified the files sent over to Sectigo) The RSA issuers went through a different path as per our assessment of the code and are therefore not impacted. Some additional context - these issuers created were meant for export to our US and European data centers to solve data sovereignty issues for some of our customers (but never ended up operationally in production except for one). We are still assessing the code to see except steps where this process went wrong. 1. Is there any reason which may explain why the A and A-1 ECDSA keys appear to be unaffected by similar issues? The A and A-1 series were created in Entrust HSMs and were meant for deployment only in India. 1. In light of the issues with the same intermediates in Bug 1970728, please describe to the best of your current knowledge how the wrong private keys came to be used to sign those CRLs and the steps eMudhra took in order to produce the correct signatures. The CRL's were signed with the original private keys (which were supposed to be the correct keys with appropriate entropy) but given that this resulted in incorrect signatures, they were resigned with flawed keys. There were human errors in multiple steps of the process that resulted in significant flaws in the key lifecycle integrity process for this KGC. We are now digging deeper for code flaws, process gaps, maker/checker controls specific to this issue. Over the next week, we will reverify every key ceremony we have done. Naveen. Flags: needinfo?(naveen.ml) Sectigo has completed the revocation of the affected cross-signed CA Certificates. A new CRL and OCSP responses have been published and are making their way to our CDN edge nodes right now. We will continue monitoring and evaluating this bug for further updates from Emudhra. The revocations have been picked up by the community (1,2). We have begun to disclose the issue to impacted trust store operators. The text of our disclosure is attached. Our disclosure does not describe the nature of the flaw or that the keys can be recovered by anyone. We will release an unscheduled version bump of Firefox Release/ESR later today (Oct 20th). Our advisories will keep it to the facts: revoking trust in intermediate certificates. (In reply to Frederik Braun [:freddy] from comment #16) We will release an unscheduled version bump of Firefox Release/ESR later today (Oct 20th). Our advisories will keep it to the facts: revoking trust in intermediate certificates. Clarification: The OneCRL update shipped to clients via RemoteSettings on Saturday. New software releases contain baked in copies of the latest blocklist, so this update ensures coverage for any users which have disabled RemoteSettings. Mozilla has become aware that multiple individuals have independently recovered the private keys for these intermediates. Some have disclosed this publicly (1) After discussion with maintainers downstream of Mozilla’s Root Store, we have decided that the circumstances warrant the emergency removal of trust in emSign ECC Root CA - G3. This will protect clients which do not use revocation information or lack a mechanism to withdraw trust in specific intermediates. We request eMudhra cease issuance from all intermediates chaining to this root and work with its subscribers to migrate them to alternative chains as soon as possible. At this time we are not removing trust in emSign Root CA - G1. Flags: needinfo?(naveen.ml) (In reply to Dennis Jackson from comment #18) Mozilla has become aware that multiple individuals have independently recovered the private keys for these intermediates. Some have disclosed this publicly (1) After discussion with maintainers downstream of Mozilla’s Root Store, we have decided that the circumstances warrant the emergency removal of trust in emSign ECC Root CA - G3. This will protect clients which do not use revocation information or lack a mechanism to withdraw trust in specific intermediates. We request eMudhra cease issuance from all intermediates chaining to this root and work with its subscribers to migrate them to alternative chains as soon as possible. At this time we are not removing trust in emSign Root CA - G1. Acknowledging this message, we will cease issuance from all intermediates chaining up to G3 (our ECC root) immediately. As per initial estimates there are about 30,000 certs that require migration and about 97% came via ACME. We will migrate subscribers to the only alternative available which is intermediates chaining up to G1 (RSA Root). Quick update - we have ceased issuance from all intermediaries chaining up to emSign ECC Root CA - G3. Group: crypto-core-security Restrict Comments: true Restrict Comments: false