Unexpected Google admins: HD Moore, Chairman & Founding CTO at runZero

Unexpected Google admins: HD Moore, Chairman & Founding CTO at runZero

For those of you who don't know HD Moore (which would be surprising if you are in security), he is a pioneer of the cybersecurity industry. He has dedicated his career to vulnerability research, network discovery, and software development since the 1990s. He is most recognized for creating Metasploit and is a passionate advocate for open-source software and vulnerability disclosure.HD currently serves as the CTO and co-founder of runZero, a provider of cutting-edge cyber asset management software and cloud services. Prior to founding runZero, he held leadership positions at Atredis Partners, Rapid7, and BreakingPoint. HD has also been a frequent speaker at industry events such as Black Hat and DEF CON.We can also add to his resume: Google Super Administrator.

What is your role/company?

I started Rumble (now runZero) back in 2018. It is a network discovery and asset inventory platform. I was the only employee for a couple of years until I brought in a chief architect and convinced Chris Kirsch (now CEO) to join at the end of 2020. We are now at about 100 people.

Why/How did you get designated as the Google Administrator? How long did you hold this role?

I was the first employee. I had a choice: Google or Microsoft. G Suite at the time had all kinds of quirks and bugs and issues, but I felt they were the lesser of the two evils, especially for email deliverability. Everything was just more straightforward and familiar with Google.Now our IT team owns the day-to-day administration work like onboarding, and offboarding. I still jump in and help out with things like DNS management or spot check on the security side

What is the strangest thing you encountered using Google Workspace?

Oh, man. There's so much weird stuff. The two biggest gotchas I've run into are both related to domains.First, Google domains are tied to individual user accounts, not tied to the organization. Let’s say you create your main admin user called admin@ and then you register a bunch of domains using Google Domains and DNS. Now, later on, I need to remove that user, admin@ and re-add it as HD@. You can no longer see (or manage) your domains! They're gone once you removed admin@! There's pretty much no way to get them back without restoring the user account or using an email-based domain transfer process (assuming transfer lock was disabled when you deleted the user). You want to make sure there's more than one person added as an admin to each domain you manage.Second is moving everyone to the new company domain name - for instance we were Rumble and then we became runZero. You start with creating aliases for all of your users in the secondary domain on the primary domain. After you set that up, you say I'm ready to change my primary domain.To be clear: We did dry runs. We set up a temporary organization. We created a giant 40-page plan. We did all these things to make sure we wouldn't cause any issues. We cleared our schedules at the end of the day Friday and we were ready to change the domains. We went to hit the button and Google says, “oh, you can't do that.” If you want to move your primary domain to a secondary domain, it won’t allow you to unless you remove every single alias of the secondary domain first.So now we have to delete sales@runzero, admin, help, etc., but in the process, you would be bouncing emails for those aliases. So to reduce the time of non-functioning email, the IT team and I had 15 screens open and we're like, “Okay, GO!” And we furiously start removing all the aliases and creating new entries one at a time.Then we hit the transfer button and of course, something else broke.So what should have been like a five-second “Hit the button, do the migration” turned into two to three hours of frantic clicking and hunting around the Google interface trying to find dangling references to old domains.And of course, during that period, no one could contact us by email on the new domain.

Any funny mishaps you want to share?

More annoying than funny.The integration between our payroll provider and Google Workspace can create some policy wrinkles. When our HR staff adds a new employee we have an automated process that calls out to Google and creates the new email account. It will automatically send the temp password link on their start date.Between when the account is created and the user actually starts, we show as being out of compliance on our SOC2 dashboard because they haven’t set MFA yet because they haven't logged in yet. We sorted that out with a policy override.But the most frustrating thing is when you terminate a user in our payroll platform; it immediately deletes the user from Google, which is not the way you want to offboard. You want to be able to move over calendars and Google Drive, etc.. So we have to go back into Google, undelete the user, do the transfer, and then delete the account again. It adds another step to our offboarding process.

What is one thing you wish you had set up / knew earlier in your (short) IT career?

Google Workspace has changed quite a bit in the last four years. But the UI is still terrible and you have to click 15 times to get things done.Google has APIs to do many things that the UI does not, which is highly annoying because it's hard to discover these very useful functionalities, especially if there is an emergency or you just want to automate repetitive stuff. There are two areas you have to watch for updates: Google Workspace and Google Cloud Platform (GCP). They overlap in functionality, and it's unclear where to look for what and when.GAM is awesome. [GAM is a command line tool for Google Workspace admins to manage domain and user settings quickly and easily.] You can get on the command line and do actions. GAM also has functionality that they don’t have in the Google Workspace UI. Like it is the only way to do a silent delegation of someone’s inbox (let’s say - if they are a salesperson leaving).To set up GAM to manage your Google Workspace, GAM needs to be set up as a service account in Google Cloud Provider (GCP), and then GAM needs permissions for various services, including GWS, which is controlled by GCP. So, it's all spaghetti. So, my advice is: Set up GAM in advance and automate repetitive stuff for emergencies.

Which parts of Google Workspace did your org outgrow first?

The things that were non-starters were Google Meet and Chat. None of that worked for us. There are participant limits that are smaller than what you need and threading was atrocious.

Did you ever find active accounts for users that did not exist or had left the company?

Earlier in runZero’s history, instead of deleting folks' accounts and doing transfers, we were just suspending them. And then we ended up with a bunch of suspended users that we'd forgotten about. Their email aliases couldn't be used by anybody else and they were still, costing us money.But now we are pretty neurotic about user offboarding. We start running down our offboarding checklist the second the HR phone call ends with an employee. The user gets removed from Google’s IDP, we lock them out of their laptops and then we go through the long list of accounts you need to clean out. Finally, you recover licenses and stuff - which is less important from a security standpoint.

What security advice do you have?

Make sure you get a Google Workspace plan that has Vault enabled [Business Plus, Enterprise]. Vault allows you to retain, hold, and search your users’ Google Workspace data.You should set up your [vault] retention policies on day one. It has been a lifesaver to figure out what’s going on from a people or security standpoint. If someone leaves a company or somebody reports harassment, you can go look at the email. So even if an employee deletes the email from their inbox, you can still review what happened.Google’s authentication has been pretty good. We feel like out-of-the-box works well. We send employees a Yubikey. We enforce MFA. I think that we have a three day or 48 hour grace period for folks to get MFA set up and then we nag them until they set it up.