Staging environment is something which is suggested as best practice but considered as a burden. Many of us feel pounded with the thought of extra investment and effort involved to upkeep it. It happens very often that a company in spite of having a Staging environment ends up failing in reaping proper results from it. Which makes us ponder on what went wrong in our QA environment? Why is a change which performed so well in QA, happened to walk south after migrating to Production?
My previous blog aimed towards the importance of having a dedicated staging environment for QA in all companies. If you think you are better off Staging environment, give it a quick read and think again!
This is my attempt to help everyone understand that Staging environment is not to be blamed for poor Production results. Poor Production results are a reflection of mismanagement conducted in terms of using the QA environment.
This is why your stage has been failing you!
Lack Of Constant Monitoring
You reach out to Stage only to test a recently deployed change in it. Monitoring helps you to prevent any code deployment above the threshold limit offering state stability, ultimately preventing the QA from going erratic. Don’t simply rely on monitoring tools! They tend to exert a significant overload on the monitored server and it’s not wise to just leave a complete environment in the hands of a 3rd party service provider. Especially, if you have invested a large amount of money on it. Staging being the closest of all environments to Production needs a continuous monitoring.
Running Stage At 11th Hour
This is a very common let down on management part. Pressure of RAD(Rapid Application Development) leads to hasty deployment through pipeline. With a large number of requests from end users bug logging, RCA(Root Cause Analysis), bug fixing, validation, outage management and other responsibilities often overshadow the presence of Staging Environment. As a result, the pipeline to QA is established when the release date starts dawning upon the horizon. You need to provide your testers enough time to test the product thoroughly in this environment, otherwise this is no different than pushing change from DEV to Production.
Cross Browser Testing
There are times when a major outage disrupts the whole working atmosphere of a team, bringing everyone on call. The clients, the managers, the developers and even testers. Everybody brainstorming on what went wrong and on whose behalf. When the services are down, customers are onto you like a 911 emergency & you need to deliver a quick fix ASAP. In this rush we often provide a workaround or even deploy a minor fix in the Production immediately to calm the scenario but we forget to deploy that tiny fix in the Staging too. We forget to Update it in the next couple hours or the next couple days due to handling a lot on the plate. Efficient management is needed to make sure that even a micro level fix is migrated to all associated environments especially to QA.
Your QA Is Not As Identical To Production As You Assume It To Be
This is related to previous point. You deploy an immediate fix into Production but miss out on QA. The next release cycle draws upon you. Your QA validation passes perfectly but the moment you migrate to Production the code goes haywire. This could happen due to that small bug which was missed out between the 2 environments.
Obsolete Testing Practices
Many companies follow outdated testing practices where they have an isolated QA team which fails to completely integrate with Dev. You will notice a constant argument between the testers and developers in this scenario. A fix will go to QA, another bug related to that fix gets logged reverting the pointer back to Developer who will then redeploy hastily and the vicious cycle continues. By the time release date pops up, you end up deploying a lot less fixes as compared to the number you planned or the number that your clients expected.
Missing a Common Goal
This is something that has been a problem from as long as i can remember. Separate teams working on a same project — task focused only towards their goals and being ignorant when asked for cooperation. United we stand, divided we fall. This motto needs to be followed to reach the pinnacle stage of customer centric deliver & efficient resource utilization.
Live Data From Production Is Missing
You will have a hard time in believing two people are twin if one of them weighs double than the other. This is exactly how it is for Staging. Remember, the aim of staging is to replicate as much of production on it. So customer data is not something that you can dummy out. Rather than running test on empty tables, you need to fill staging database with as much data as your production database in handling to have an estimate of your latest schema capabilities. There are tools available in the market for supporting you with this.
Performance Parameters Are Not Accurately Simulated
Often the result of test cases is not as fast in Production as it is in Staging. This happens because your Staging is not facing the real time internet traffic, but your Production environment does. So you can’t encounter Race conditions, Load handling, performance tracing and deadlocks in QA environment. Good news is that there are tools for that purpose too.
Isolated Staging Server Is Missing
Hosting your Staging server on the same place as your Production is a major mistake that not only brings confusion but also risks data leakage & disruption as whole.
Missing Out On Exploratory Testing
We are too determined on testing our knowns test scenarios that we forget about the unknowns. By Unknowns I am referring to scenarios that are unforeseeable by a team of engineers and testers but are exposed when the product is made available to thousands of your customers. Performing exploratory testing is crucial for eliminating the risk of unknowns.
Introduction of Agile has been well executed in many companies. However, some companies find it very hard to transit from Waterfall to Agile. The whole team from developers to testers and even managers are baffled when it comes to handling QA environment on regular and scheduled basis. I understand how hard it can be with so much on plate, but managers need to plan ahead of time for organizing the appropriate data that needs transferring to QA in the right amount.
Difficulties in Deployment & Management of Microservices
Microservices are something that everybody is looking to apply for in their organization for reliable and smooth expansion. It is believed that microservices and staging servers are not meant for each other. Reason being with so many independent teams providing connectivity to numerous 3rd party application simultaneously. It becomes very challenging to map all external & internal microservices with latest versions that are running in Production. This is tough but is something which is relevant for ensuring a reliable quality product in the market.
Why not dump Staging and save yourself the hassle?
Well you need to consider few things before you make that decision.
- Migrating change directly to Production by skipping QA can have some major loss involved in the aftermath. Are you prepared for accepting losses that may come from the unknowns.
- Testing after deploying in Production is pretty different than doing it in QA. You have a very small time window to test a compound system and not being able to test every component may lead to outage or even worse as you may end up losing a valuable customer. Planning and organizing will be the key if done well, otherwise it can also be the doom.
- Approach for testing in Production? Feature flags serves as a helpline as you can deploy your change in Production and activate it only when you feel ready. However, be ready to encounter noise in your system’s codebase. You may even end up with a code that may look absurd to you in many way. Feature flags need proper cleaning as a good practice. Make sure to retire old feature flags.
- Feature Flags won’t guarantee you against changes where the code is reflecting on a large number of existing components. Also, your testers are going to take a fair amount of time to get used to feature flags.
To build a robust Staging or QA environment make sure that you replicate it as close to Production as possible in terms of hardware and software.
Develop a code reviewing process. Make sure that a code generated which is deployed by one team in QA gets thoroughly reviewed by another development team.
If you are working in Agile then schedule your code migration to QA on weekly basis. Follow best practices for software testing, they will serve you as a bible for successful implementation of QA.
Staging is a crucial element for small startups and big enterprises alike. Consider It is an expensive vase that you need to maintain & handle with care. Although, when tidied up properly, it is bound to bring more grace to your Production environment.
Originally published at Lambdatest