[PasswordBoss] Investigating backend services issue

Incident Report for CyberFOX

Postmortem

Issue Overview

On February 17, PasswordBoss underwent scheduled maintenance to deploy a system update to address several key issues related to performance, user experience, and system functionality with our partner.passwordboss.com website. Despite careful planning to minimize disruptions, some users may have experienced temporary difficulties accessing services during the update window.

The technical details of the update included improvements to:

  • Companies Query Performance: Optimizing how companies' data is queried in paginated batches to improve efficiency and speed.
  • Subscription Layout Improvements: Enhancing the user interface for subscription-related pages.
  • License Validation Update: Using previously cached data for validation.
  • Removal of User Roles: Streamlining user permissions by removing outdated or unnecessary roles.
  • Environment Adjustments: Adjustments such as page size and batch processing.

Actions Taken

February 13, 2023 (11:00 AM EST): A new version of the partner portal was created to address the identified issues. This marked the beginning of the update's development and testing phase. The team worked diligently to ensure all improvements were thoroughly tested before deployment.

February 17, 2023 (16:44 EST): PasswordBoss announced the scheduled maintenance and informed users about the potential impact of signing in during the update window. This proactive communication ensured transparency and prepared users for any temporary inconvenience.

February 17, 2023 (16:45 EST): The scheduled maintenance officially began. The engineering and cloud engineering teams started deploying the updates and closely monitored the system to ensure a smooth transition.

February 17, 2023 (18:45 EST): The maintenance was completed ahead of schedule. All updates were deployed without major issues, and full functionality was restored. The team conducted post-update checks to confirm that all improvements were functioning as intended.

We apologize for any inconvenience and thank you for your understanding.

Posted 15 days ago. Feb 26, 2025 - 11:15 EST

Resolved

This incident has been resolved.
Posted 1 month ago. Feb 03, 2025 - 17:39 EST

Monitoring

We have identified the root cause of the service degradation impacting some users’ ability to sign into Password Boss. We are currently monitoring the changes we made to make sure everything runs smoothly. Thank you for your patience.
Posted 1 month ago. Jan 31, 2025 - 15:45 EST

Investigating

We are investigating an issue with some of our backend services being degraded, you may experience a temporary interruption in Signing into Password Boss. We apologize for any inconvenience this may cause and assure you that our team is working diligently to minimize the impact on your user experience.
Posted 1 month ago. Jan 31, 2025 - 11:06 EST
This incident affected: Password Boss (Cloud Services, Password Boss Portal, Password Boss Partner Portal, Mobile Apps, Desktop Clients, Browser Extensions).