First things first: an overdue apology
We’ve done a terrible job on comms over the past three weeks. We wanted to prioritize completeness in the form of a comprehensive post-mortem, but we should have led with directness. While the LayerZero protocol remained unaffected, our own internal RPCs that the LayerZero Labs DVN used were attacked by the Lazarus Group and had their source of truth poisoned while our external RPC provider were simultaneously DDOS’d.
We believe developers should choose their own security configurations, but we made a mistake by allowing our DVN to act as a 1/1 DVN for high-value transactions. We didn't police what our DVN was securing, which created a risk we simply didn't see. We own that. Moving forward, LayerZero Labs will be more active in educating developers and monitoring how applications should build on the protocol to ensure they are configured safely.
This impacted a single application (0.14% of total applications) and roughly 0.36% of the value of assets on LayerZero.
We have been working with our external security partners for the past few weeks and will deliver our official post-mortem as soon as their work has concluded.
Additionally, three and half years ago, one of the signers on our multisig used their multisig hardware wallet to perform a personal trade when they intended to use their own personal hardware wallet. This is obviously not ok. This signer was removed from the multisig, wallets rotated, and we’ve since updated our security practices around signing devices, added localized anomaly detection software on each device, and created a custom-built multisig called OneSig.
The LayerZero Protocol
LayerZero was built as a response to every existing bridge having a single point of failure / systemic risk: if one asset is at risk, they’re all at risk. It was built so that every single application can completely own their security end-to-end and do not need to rely on LayerZero Labs at all.
Our thesis was that this is the only method that institutional assets and institutions themselves would adopt long term. This architecture is exactly why it has been chosen by the largest asset issuers in the world and has been used to move more than $260B. It’s the only architecture that completely removes systemic risk, and it’s the only architecture that allows an application to fully own their own security end-to-end with no reliance on external parties.
There’s been a ton of claims floating around over the past few days that are worth addressing.
Are assets on LayerZero safe?
Yes. No other application has been affected and more than $9B has been moved across LayerZero since April 19th.
What does LayerZero Labs recommend?
- Pin all of your configurations so you are not relying on defaults controlled by LayerZero Labs.
- Set your block confirmations for each chain to a level where block reorganization is nearly impossible.
- Configure your DVNs to include at least two parties, though three to five is better for security.
- Consider running your own DVN and setting it to "required" so you are actively participating in your own security.
Does my asset have any exposure to LayerZero Labs, and if so how much?
There are four areas where LayerZero Labs may be involved in a transaction. Ordered from most-reliant to least:
Trust Assumptions
- If your application points to the defaults (block confs, messaging library, DVN selection) then you are fully reliant on the LayerZero Labs multisig. This multisig sets defaults for all pathways, chooses which messaging library to point to, and sets the default DVNs. This is intended to be for testing only and should not be used for production applications, unless the goal is to explicitly delegate all security assumptions to LayerZero Labs.
- If your application uses the LayerZero Labs DVN as one of the DVNs, then you rely on our DVN’s attestations as part of your security stack. You should never select any DVN as 1/1 as it creates a single point of failure, so if you are including it in a multiDVN setup it is 1 piece of the N-piece security assumption for your messaging.
Liveness Assumptions
- If the DVNs that you select use Essence as a gas relaying service. This has absolutely 0 impact on your trust assumptions, although if Essence failed to relay gas for the transaction the DVN would need to do it itself, and it may cause a temporary liveness failure.
- The LayerZero Labs executor is used for gas abstraction and execution. Again this has zero impact on your trust assumptions. However, if the executor does not execute a transaction, the user (or anybody, since this is permissionless) would need to pay the gas and manually execute it on the destination chain.
Do LayerZero Labs use their multisig for trading?
LayerZero Labs uses a multisig for Endpoint ownership. This gives LayerZero Labs the power to connect new chains, add (append only) new validations libraries, and update defaults configurations (for testing). A properly configured application is completely unaffected by the powers the LayerZero Labs multisig has over the Endpoint.
Three and half years ago, one of the signers on our multisig used their multisig hardware wallet to perform a personal trade when they intended to use their own personal hardware wallet. This is obviously not ok. This signer was removed from the multisig, wallets rotated, and we’ve since updated our security practices around signing devices, added localized anomaly detection software on each device, and created a custom-built multisig called OneSig.
What is LayerZero doing to make all asset issuers more secure?
Actions Since 4/19
- the LayerZero Labs DVN no longer services 1/1 DVN configurations
- All defaults on all pathways are being migrated to 5/5 where possible and no less than 3/3 on any chain where only 3 DVNs are available
- We are developing a second DVN client written in Rust (client diversity)
- We created a more robust RPC quorum configuration to allow DVNs to select granular quorums of internal, dedicated-external, and shared-external RPCs
Ongoing Product Innovation for Better Security Management
- We’ve developed OneSig, a specialized multisig that allows for more secure signing across all chains LayerZero supports.
- We will be updating our own multisig threshold and signers from 3/5 to 7/10 across all chains where OneSig exists as well as externalizing a productionized version to external parties.
- OneSig allows the signer to download transactions, then merklize and hash them locally to sign the root hash. This prevents the backend from slipping in unauthorized transactions since the hashing happens on the user's side.
- Every OneSig signer has developed their own private security checker to look for anomalies or transactions that seem out of place. They keep these checkers private on their specialized signing machines and do not share their specific criteria with the company or other signers.
- We’ve spent the past few months building Console, which will live as a unified platform for issuers to configure, deploy, and manage asset issuance and security from one place.
- Console comes with an automated anomaly detection: unknown DVNs, changes in ownership, changes in block confirmations, unsafe configurations, use of defaults and more.
- Built in OneSig integration.