Notifying administrators about unhashed password storage

Google’s policy is to store your passwords with cryptographic hashes that mask those passwords to ensure their security. However, we recently notified a subset of our enterprise G Suite customers that some passwords were stored in our encrypted internal systems unhashed. This is a G Suite issue that affects business users only–no free consumer Google accounts were affected–and we are working with enterprise administrators to ensure that their users reset their passwords. We have been conducting a thorough investigation and have seen no evidence of improper access to or misuse of the affected G Suite credentials.

How Google Stores Passwords for Consumers & G Suite Enterprise Customers
If you have a Google account, Google’s core sign-in system is designed not to know your password. How, then, can we verify your password when you sign in to your Google account again? The answer lies in a bit of cryptography: when you set your password, instead of remembering the exact characters of the password, we scramble it with a “hash function”, so it becomes something like “72i32hedgqw23328”, and that’s what we store with your username. Both are then also encrypted before being saved to disk. The next time you try to sign in, we again scramble your password the same way. If it matches the stored string then you must have typed the correct password, so your sign-in can proceed.

The effectiveness of the hash function lies in its one-way nature: it is simple to scramble your password, but nearly impossible to unscramble it. So, if someone should obtain the scrambled password, they won’t be able to recover your real password. The downside of password hashing is that if you forget your password, we cannot show you what it was; there’s nothing we can do other than reset it to a temporary password (valid one time only) and then require you to pick a new one.

G Suite enterprise accounts
In our enterprise product, G Suite, we had previously provided domain administrators with tools to set and recover passwords because that was a common feature request. The tool (located in the admin console) allowed administrators to upload or manually set user passwords for their company’s users. The intent was to help them with onboarding new users; e.g., a new employee could receive their account information on their first day of work, and for account recovery. The functionality to recover passwords this way no longer exists.

The issues discovered with G Suite enterprise accounts
We made an error when implementing this functionality back in 2005: The admin console stored a copy of the unhashed password. This practice did not live up to our standards. To be clear, these passwords remained in our secure encrypted infrastructure. This issue has been fixed and we have seen no evidence of improper access to or misuse of the affected passwords.

In addition, as we were troubleshooting new G Suite customer sign-up flows, we discovered that starting in January 2019 we had inadvertently stored a subset of unhashed passwords in our secure encrypted infrastructure. These passwords were stored for a maximum of 14 days. This issue has been fixed and, again, we have seen no evidence of improper access to or misuse of the affected passwords. We will continue with our security audits to ensure this is an isolated incident.

Has my company been impacted?
We recently notified G Suite administrators to change those impacted passwords. Out of an abundance of caution, we will reset accounts that have not done so themselves. Our authentication systems operate with many layers of defense beyond the password, and we deploy numerous automatic systems that block malicious sign-in attempts even when the attacker knows the password. In addition, we provide G Suite administrators with numerous 2-step verification (2SV) options, including Security Keys, which Google relies upon for its own employee accounts.

We take the security of our enterprise customers extremely seriously, and pride ourselves in advancing the industry’s best practices for account security. Here we did not live up to our own standards, nor those of our customers. We apologize to our users and will do better.