Warehouse CA and credentials upgrade, early 2024

What's Happening?

Two security changes are occurring to your Benchling data Warehouse:

1. The certificate that enables secure SSL/TLS connections is being updated - CA Certificate Bundle Update
2. The credentials system has been updated to use a more secure standard - Warehouse Credentials Upgrade

Actions you will need to take by July 27th, 2024:

1. Update the root CA (Certificate Authority) certificate bundle on tools connecting to Warehouse.
2. Get each Warehouse user to create themselves a new credential pair. Once this is set up, have them delete any existing credentials. This should afford for zero downtime.
3. Double-check your Warehouse tools are set to connect using sslmode=verify-ca – this may not be the default!

Users of Benchling Insights (from within the Benchling web UI) are all set: no action required.

Please read on for further details!

Dates to Remember

Key dates, at a glance:

• March 6th - credentials made after this date are retained; before this date will be deactivated.
• June 27th - recommended date to make your changes by
• July 27th - due date: Benchling will make changes to your Warehouse:
  • The SSL/TLS certificate on your Warehouse will change
  • Old user credentials (created before March 6th) are deactivated
• August 22nd: final changes happen to your Warehouse:
  • AWS changes its root certificates; extensions cannot be granted past this date
  • Old user credentials (created before March 6th) are deleted

CA Certificate Bundle Update

CA = Certificate Authority, the "root of trust" for encrypted communications using SSL/TLS.

What’s happening?

AWS is updating their root CA (Certificate Authority) for RDS. This affects the SSL/TLS certificate that is used on your Warehouse. We will be changing your warehouse to have a certificate signed by this new root authority on July 27th, 2024.

Why do I need to take action?

Updating to the new root CA certificate bundle will ensure your Warehouse tools continue to make a secure connection over SSL/TLS. Some hosted tools, for example Tableau Cloud, may do this for you, but if you are responsible for configuring a custom tool or local program, you will need to update the certificates yourself.

What do I need to do?

If your cloud-hosted tool doesn’t give you any options on the SSL/TLS settings used, then there’s nothing you need to do!

Otherwise, you need to download the new CA certificate bundle from AWS and place it in the correct location for your tool. Please see our getting started with Warehouse guide for instructions on how to do this and where to place the bundle.

In both cases, we recommend you double-check that it’s correctly using SSL/TLS and using a mode like verify-ca to check the identity (certificate) of the Warehouse server on the other end. This is crucial to establishing a secure connection to your warehouse.

We also recommend you switch to the new CA bundle as soon as possible: it contains both the present and future root certificates and will ensure your tools stay up and running. Please set an internal target of June 27th, about a month early, to give time for any unforeseen issues.

What will happen on July 27, 2024?

We will change the certificate on each Warehouse to be signed by the new root CAs (Certificate Authorities) from AWS RDS. If you are still using the old certificate bundle on this date, SSL/TLS connections by your tools will begin to fail. 

Note: July 27th is the same due-date as the credentials change below.

What will happen on Aug 22, 2024?

AWS is set to expire the old root CAs – a date that cannot be changed – and only use the new root certificates to sign the individual certificates on the RDS instances. AWS will also publish a new bundle soon after this date with just the new root certificates, completely deprecating the old set.

Warehouse Credentials Upgrade

Warehouse Credentials are unique username-password pairs (created for authorized users) that are used by tools to access a Benchling Data Warehouse.

What’s happening?

PostgreSQL, the database server Benchling uses for Warehouse, previously stored (“hashed”) passwords using an older standard and now supports a much stronger standard (SCRAM-SHA-256). Benchling changed Warehouse to apply the new standard as the default on or after Midnight UTC+0 on March 6th, 2024 (2024-03-06T00:00:00Z), but have retained old credentials using the old standard to ease the transition. These old credentials now have a shelf life, and will be deactivated on July 27, 2024. Please try to recreate credentials as soon as possible, preferably by June 27th, to give time for any unforeseen issues.

See this PostgreSQL article for additional technical information on SCRAM-SHA-256, and this article by 1Password about password hashing more generally. The new SHA-256-based standard is many orders of magnitude harder to crack than before – and therefore much more secure.

Why do I need to take action?

Passwords stored under the old standard can’t be automatically converted to the new standard (SCRAM-SHA-256). We can’t read the old password, by design. The only secure way for you to get a Warehouse password is for you to personally create it in the Benchling web UI, which is why we need you to take action.

Technically speaking: The only time Benchling has that plain password is the instant when it’s displayed in the web UI. What Benchling’s Warehouse actually stores is a password hash, consistent with widespread best-practices. There’s no tractable way to take the stored hash and figure out what the actual password was – this is normally how hash functions work, but in this case it is especially intractable because the password was randomly-generated.

What do I need to do?

Re-create your Warehouse tools’ credentials if they were created before Midnight UTC+0 on March 6th, 2024 (2024-03-06T00:00:00Z).

Note for Validated Cloud customers: it is okay to create new credentials any time after March 6th.

See below on how to do this, depending on your role. Users of Benchling Insights (from within the Benchling web UI) need not make any changes.

As a user, how do I know which of my Warehouse credentials need changing?

Note for Validated Cloud customers: this display feature will be in the 2024.2 release.

If you go to your user settings in Benchling (click the bottom-left profile icon > Settings), you will see a warning message above each credential (a.k.a. “login”) created before Midnight UTC+0 on March 6th, 2024 (2024-03-06T00:00:00Z) – look for the unique username in your tools (e.g. u$lpasteur_s4 below).

As a user, how do I recreate my warehouse credential?

Note for Validated Cloud customers: users may safely re-create their credentials prior to the 2024.2 release, but the warning indicating the credential was made before March 6th below won't display until then.

To generate a new Warehouse credential for your tool:

1. Log into the Benchling web interface.
2. Click on your profile in the bottom left, then Settings.
3. On the top right should be displayed a list of Warehouse credentials (logins), if you’ve previously created any. If there are none, you’re done!
4. If you see this warning, it needs to be changed because it was created before Midnight UTC+0 on March 6th, 2024, but don’t delete it just yet: 
5. First, create a new credential and be sure to capture the username & password in your tool or in your password manager (please follow your organizations’ best practices for security).
6. Update your tool to use the new credentials. Old credentials will remain active so you can gradually update your tools.
7. Once your Warehouse tools are using the new credential, click the trash-can icon beside the old ones to permanently Revoke it. It will then display Revoked and won’t be usable. Once Revoked, Benchling cannot recover the credential.

Now is a good time to ensure that your warehouse connection is using the most secure verify-ca mode. This is not the default setting for most tools! See our Warehouse Getting Started guide on how to configure this yourself, or if you are an IT Administrator, some ways to configure it for all users.

As an Administrator, how can I recreate a user’s credentials?

Since we generate a password for a user, it’s best that they generate and capture their own username & password following the instructions above. This is more secure than sending it to them over chat, email, or similar. As such, we don’t provide a direct way for Tenant Admins to create these credentials for a user.

As an Administrator, how do I see which users still need to recreate their credentials?

Note for Validated Cloud customers: this feature will be in the 2024.2 release.

Log into Benchling then go to the Tenant Admin Console (see here on how to access it)

• Alternatively, you could use the Users ReST API (see below).

From there, go to the Users tab then Export all Users on the top right

This will give you a CSV report. The Oldest Warehouse Credential Created At column has the date of the oldest credential. Credentials made before 2024-03-06T00:00:00Z will be removed on July 27, 2024. The Number of Active Warehouse Credentials column gives the number of active credentials a user has. Please note the export also expresses the time in UTC+0 (+00:00).

As an Administrator or Auditor, can I tell when users change their credentials?

Note for Validated Cloud customers: this feature will be in the 2024.2 release.

Yes! Both Creation and Deletion events are sent to the audit log. The events are named “Warehouse Login: Created” and “Warehouse Login: Deleted”

I’m using the Benchling API to automate user management, how can I see the status of user warehouse credentials?

Note for Validated Cloud customers: this feature will be in the 2024.2 release.

The Users ReST API has been updated to have more information about individuals’ credentials. See https://benchling.com/api/reference#/Users/getUserWarehouseLogins for complete details.

In short, /v2/users/<user_id>/warehouse-credentials can be used to retrieve details about the user’s Warehouse credentials:

• username – what the user’s tool will be using to log into warehouse, e.g., u$lpasteur_s4
• createdAt – (an ISO8601 timestamp) if this is before 2024-03-06T00:00:00.000000+00:00 then the credential will need to be changed
• label – what the user entered as the custom label at creation time, which may hint at the tool the credential is being used in.

What will happen on July 27, 2024?

Benchling will deactivate, but retain, credentials created before Midnight UTC+0 on March 6th, 2024 (2024-03-06T00:00:00Z). If you discover your tool suddenly isn’t working, and can’t change it or its configuration, please let us know and we may be able to extend it until Aug 22, 2024.

Please try to have your re-creation of credentials done by June 27th, a month early, so there’s plenty of time to react to unforeseen issues. 

Note: July 27th is the same due date as the CA bundle change above.

What will happen on Aug 22, 2024?

On or after Aug 22, we will permanently delete any previously deactivated credentials (with possible requested exceptions). Also on or after Aug 22, we will begin to change our servers to reject all attempts to connect using the old standard as a layer of protection for all Benchling customers. This will similarly be delayed by any exemptions.

I tried to update my tool, but it’s unable to authenticate. What can I do?

If you’ve created a new credential and a previously-working connection now doesn’t work, let us know immediately through your support contact. There are two probable causes:

1. The verify-ca mode isn’t set up correctly and it’s failing to connect. Check for SSL or TLS error messages in your tool. See our getting-started with Warehouse guide on configuring this. DO NOT DISABLE SSL or TLS to get this working or you risk exposing your data.
2. The tool is not able to support SCRAM-SHA-256 authentication. Check the documentation for the tool you’re using to confirm.

In the latter case, please let your CSM or support@benchling.com know immediately if you run into this issue. We can work together on a possible exemption and timeline so your connection stays working.

I’m a Validated Cloud customer, what does the timeline look like for me?

Validated Cloud customers also need to complete the same two changes for their Warehouse tools: update the certificate bundle and re-create credentials.

Presently: if a user makes a new credential (or has already made one after Midnight UTC+0 on March 6th, 2024 (2024-03-06T00:00:00Z)), it will be retained post-July-27th.

By the next 2024.1 patch to preview & production, you can begin to change the root CA certificate bundle on tools connecting to your Warehouse. However: if you want to change tools to use the new certificate bundle earlier, it will work just fine. The presently downloadable bundle contains both sets of roots and will ensure your tools can continue to connect post-July-27th. This is planned to enter preview on Apr 3rd, then production on Apr 10th.

By the 2024.2 release all of the features Administrators & users need to support credential re-creation will also be available. This is planned to enter preview on May 17th, then production on June 14th.

We recommend aiming to accomplish both changes (certificates & credentials) roughly 2-4 weeks before the July 27th due date. If you cannot meet the July 27th due date, please reach out to your CSM or support@benchling.com, and Benchling will consider a possible exemption/extension. 

For certificates, extensions past Aug 22nd will be impossible. This is not in Benchling’s control since this is a change AWS is doing.