✉️

Day one post mortem to the attack

Day one post mortem to the attack

Reviewing the events that transpired during the Tableland Discord hack.

Originally published on @July 22, 2022

Last night, an external account took control of our Discord server, blocked all users from posting, and posted a malicious website to our announcements channel that advertised some new token claim. The site prompted users to hand over permissions to their wallets, allowing any existing NFTs to be transferred out.

We've worked since early this morning to make our server functional and safe. I'll outline below the steps we've taken to protect the server (and you) from hackers in the immediate future.

Additionally, we've pieced together all the events that unfolded at the end of the day yesterday that left our server exposed to attack and how they perpetrated the takeover. We want to share those details with you so other server hosts can understand the attack vector and deploy immediate steps to protect themselves and their community.

An excellent community member has stepped forward to help us dig into the on-chain behavior of these criminals. We hope to share a short report to help inform you and other communities about who these people are that keep attacking web3—more on that in the coming days.

We've reviewed our internal security policies and published a summary of our team practices. We encourage you to read these and consider ratifying something similar with your team.

Finally, we're reviewing the affected wallets and are considering the best options to move forward. We need to check this from multiple angles before making any decisions. Unfortunately, these hacks are happening regularly across the wider web3 community, and the attackers are succeeding. We've seen projects from The Nouns to ZenAcademy, and many others hit. Zenica's thoughts about these situations is a highly recommend read, twitter thread.

What happened

On Tuesday (July 19th), some members of our team were approached about a potential project collaboration. The communication happened in Discord, and the solicitors requested two of our team members join their server to continue the discussion.

On the Wednesday morning (July 20) the team members joined the external Discord server. In the onboarding process, they clicked a verification step from a bot named "Dyno". This was a spoof of the well-known Dyno bot. When the members went through the verification step, they clicked a bookmark button with malicious javascript and then were prompted to interact with the bookmark, triggering that script to run. They continued in the 3rd party Discord server in a normal way.

The Javascript code seems to have granted access to the team members' Discord sessions. Unfortunately, one of the team members was the owner of the Tableland server.

Late in the evening of July 20th, the attackers began by banning the entire Tableland team. New Roles were created, and other discord accounts were granted admin access. One of our team members, along with a couple of community members, attempted to unwind from the attack to no avail. Unfortunately, the true server owner was attending to family issues and was unreachable for several hours.

The attacker posted a message to the announcements channel with a link to a fake website claiming some new assets could be minted by community members. This site _was not_ on a Tableland domain but was made to look similar.

Any individuals who clicked on the link and followed through with the wallet instructions granted the attackers access to any NFTs held in their account. If this was you, or you ever interact with a suspicious website, immediately revoke all access on https://revoke.cash/. Over the next few minutes, we believe the attacker stole 16 Tableland Rigs and other NFTs from community members.

At 2:30AM the server owner was online and able to reset sessions and passwords to begin taking the server back over. By about 3:00AM the attack had been contained.

Remarks for other servers

The root cause of the unauthorized server access was a team member using the account that owned the Tableland server as their primary account. Every server should employ an account in cold storage to own their server. Very few members should have restricted access to this account only for use in critical moments or for server changes.

Another crucial point for other server admins: while many people early on pointed to 2FA as a potential cause, it's important to note that this was not an effective security measure. The attack in question was designed to circumvent 2FA. The account owner, and all Tableland teams, use 2FA for all critical accounts.

The attack vector has been around, but not every project is familiar with it. Here's just one record of the same hack using the same bot name being deployed in other communities https://twitter.com/giveaway_boat/status/1528292710452723713

Server security measures

Starting at 3:00AM we began dedicated work to reconfigure our server and ensure clear security policies going forward. First and foremost, we adopted the cold stored owner account model. Second, we thoroughly reviewed and updated all bots and security measures. We've had fantastic input from community members in several focused conversations today to adopt best practices from other servers that do it well.

Web3 is getting slammed by scammers, and we must band together to plug the holes and protect the good people in our community. We will continue reviewing our changes with community members and ultimately publish our setup for review by you and other communities to improve or adapt. Finally, we are looking into a 3rd party security audit for the server.

In addition to Discord, we reviewed our full stack for possible other compromises. We operate very securely when building technology and are disappointed that we missed the gaps in our Discord setup and we've published our Security Practices for all to review. We hope others adopt and/or publish theirs as well.

Tableland team security notes

  • The highest privilege in the server should be a cold account which:
    • Stored in the password manager in a vault only accessible to a restricted set of people in Tableland.
    • As a policy, never “remember password” when used or similar.
    • Only used on trusted devices (e.g. never on someone’s else computer)
    • Consider rotating password every 3-6 months (or more if paranoid for good reasons).
  • Every Tableland member should have 2FA. Never SMS recovery since SMS are unsafe.
  • Every Tableland member should be highly suspicious of any DM with a link, even if it comes from a Tableland member.
  • Every Tableland member should never copy-paste any secret stored in the password manager into Discord (no DM or DM group allowed). Paste a link to the stored password.
  • Every Tableland member should never store any secret stored in .env or similar files in 1Password on their local machines. If, for some reason, this is needed by your development flow, be mindful of how to avoid leaving it there longer than required.
  • Every Tableland member should consider any conversation in Discord potentially revealed to the public. Even if it is a 1:1 DM or DM group. There’s no forward secrecy in Discord. For sensitive chats, consider a safer medium.
  • Every Tableland member should tune their Discord user “Settings → Privacy & Safety” to “Keep me safe.”
  • Be incredibly suspicious about any link that would make the browser suggest “Opening the Discord app”. This is a red flag. If you made a mistake, please try to record which was the URL and tell anyone in the team to know how to proceed. It might be worth assuming you got hacked and taking recommended measures.
  • If any sensitive Tableland Discord user got compromised, consider the following actions:
    • Reset your Discord password.
    • Reset your GH password since the attack could escalate to important repos.
    • If you are paranoid or suspect your computer got compromised, wipe out your laptop and install everything from scratch.
  • Keep a paranoid mindset to sensitive information on your computer, like: “Am I comfortable giving my computer with my personal password to some random user in Discord?”. If the answer is “No”, probably you already know what to do. [This is hard to achieve, but it’s good to know the risks you’re taking].
  • Every GH repo should have as mandatory have at least 1 reviewer and enforce the rule even for administrators of the repo. This is a GH repo configuration.

Gratitude

The day has been a long one for many of us. We couldn't have pulled it off without the positivity that was flowing in from our community members. Many of you stepped up to join us in private channels to coordinate changes and improvements. Others helped us in the middle of the night to help notify and try and stop the attack. We are honored to be building with all of you. We'll continue working with you to implement further improvements and educational programs and use this moment to educate as many other communities as possible. I can't thank you enough for seeing through the garbage and focusing on the improvements we are all trying to make on the Internet. I hope everyone has a much more restful evening.

Originally published: @July 21, 2022

Updates:

  • @July 22, 2022 changed title to reflect that this is only part one of ongoing work, covering the first day’s post mortem.
  • @July 24, 2022 the post was moved to https://docs.tableland.xyz/security.