Cisco Systems' DEMA
Company: Cisco Systems, San Jose, CA
Company Description: Cisco is the worldwide leader in IT that helps companies seize the opportunities of tomorrow by proving that amazing things can happen when you connect the previously unconnected. At Cisco customers come first and an integral part of our DNA is creating long-lasting customer partnerships and working with them to identify their needs and provide solutions that support their success.
Nomination Category: New Product & Service Categories - Software
Nomination Sub Category: New Product or Service of the Year - Software - Systems Management Solution
Nomination Title: Cisco Systems' DEMA
Tell the story about this nominated product or service (up to 650 words). Describe its function, features, benefits, and performance to date:
As Cisco IT continues it’s journey towards digitization bandwagon, speed with quality of delivery has become the key focus. ‘Intensely Focus On Customers’, is one of the Cisco Values which we look to imbibe in our efforts to fulfill our perpetual dream of being the number one IT Company in the world. Without a robust and innovative mind-set, it’s impossible to fulfill the ever-growing customer demands.
Cisco is a giant IT enterprise which houses around 360 production, 1400 non-production Oracle databases, 159 Oracle ERP, and 56 Oracle fusion middle-ware environments. On an average, it takes 8 manual hours to perform Disaster Recovery (DR) Resiliency Testing for a single Database. Achieving Operational Excellence in performing Disaster Recovery Testing in such a vast, diverse and complex enterprise seemed like a Herculean task. We had to overcome major hurdles to enable FAST IT and continuous delivery, to achieve the end goal of operational excellence for our clients. In resonate with our objective of client wait time reduction, we, The Database and Replication Services team developed our own state of art Automated Disaster Recovery Testing Tool - "DEMA – Data center Emergency Management Administration” to support digitized IT initiatives.
In the past, The Disaster Recovery Testing/Fail over process was time & resource intensive. It required too many manual activities & resources which added up to overall Infrastructure Operations Cost. And in a real disaster scenario, 8 hours of wait time to business is not acceptable and this will cause severe impact on our client experience. Huge amount of time was spent on planning effort as the process was complex and manual. Also the process was error prone as execution was manual and required human intervention during the firefighting situations.
With DEMA, the complete Disaster Recovery Testing/Fail over process has been automated.
Here is the high level architecture of DEMA:
• DEMA will do the pre-validations and makes sure the primary and standby (DR) databases are in sync. In case any issues this report to the support team for corrective actions if the self-heal is not fixing the issues.
• At the time of DR testing/fail over check point, with a click of button, DEMA will start all the fail over activities and make the standby as primary DB, do the complete application validations and reports back to support teams in case of any failures.
• DEMA will auto notify our clients’ and our own support teams when resiliency testing is completed and provides a comprehensive report on the checks it performed.
• Once the fail over testing is successful, it resets the DR to it’s original state and continues to monitor the health of the primary and DR.
In summary, DEMA will help Cisco IT database team to significantly optimize the execution time for Disaster Recovery (DR) Resiliency Testing. This completely eliminates the manual activities performed by DBA earlier leading to overall effort optimization of 95%. In the past we used to spend 8 hours to perform one single DB DR testing, now are able to do the same with just in 23 mins. Considering the fail over scenarios at the enterprise level we are able to perform.
In Q2, we performed a Big-Bag DR testing in 28 databases with 254 applications. We used DEMA, 2 DBAs performed and supported the entire activity, in the past the same was handled by 9 DBAs.
Overall we gained 75% of downtime reduction/execution time.
• For Core Database, 23 mins/DB with tool vs 90 mins/DB without tool
• For ERP Database, 120 mins/DB with tool vs 285 mins/DB without tool
In bullet-list form, briefly summarize up to ten (10) of the chief features and benefits of the nominated product or service (up to 150 words).
Here are the business benefits we have achieved using DEMA.
• With DEMA, disaster management on Database platforms are simplified.
• Error free.
• Overall DR testing efforts are reduced by 95%, 8 hours to 23 mins.
• Big-Bang DR testing performed by 2 DBAs compared to 9 in the past.
• Overall we gained 75% of downtime reduction/execution time.
** For Core Database, 23 mins/DB with tool vs 90 mins/DB without tool, **For ERP Database, 120 mins/DB with tool vs 285 mins/DB without tool