Shift Right Testing: Learning from Software in the Real World | Amber Insights #008
Updated: Sep 8
By Amber Idumwonyi
In the previous issue of Amber Insights, I explored Shift Left Testing and why quality thinking should begin much earlier in the software delivery lifecycle.
We discussed reviewing requirements before development, identifying risks early, designing for testability, and creating faster feedback throughout development.
But there is another side to modern Quality Engineering.
No matter how thoroughly we test before release, something changes when software enters production.
Real users arrive.
Real data flows through the system.
Real integrations behave under real conditions.
Traffic patterns change.
Infrastructure responds to genuine demand.
And people use our software in ways we may never have anticipated.
This leads to an equally important principle:
Quality learning should not stop when software is released.
That is where Shift Right Testing comes in.
What Is Shift Right Testing?
If Shift Left moves quality activities earlier in the software development lifecycle, Shift Right extends quality activities beyond deployment and into production.
The objective is not simply to test software after releasing it.
It is to use information from the live environment to understand how the system actually behaves.
That distinction matters.
Before production, we work hard to simulate reality.
We create test environments.
We prepare test data.
We simulate integrations.
We execute expected and unexpected scenarios.
We conduct performance, accessibility, security, functional, integration, and other appropriate forms of testing.
All of that is valuable.
But a test environment is still a representation of reality.
Production is reality.
Shift Right allows teams to learn from it.
Why Pre-Production Testing Is Not Enough
This does not mean pre-production testing has failed.
It means complex software systems contain variables that are difficult to reproduce completely before release.
Consider an application that has successfully passed functional, integration, regression and performance testing.
It goes live.
Thousands of users begin interacting with it.
Some use older devices.
Some have slow network connections.
Some follow journeys the team expected.
Others behave in ways nobody anticipated.
External services experience delays.
Data volumes grow.
Traffic suddenly increases.
An unusual combination of events occurs for the first time.
The application may technically have passed every planned test and still reveal new information in production.
This is one reason I believe we need to be careful with the phrase:
"Testing is complete."
A particular phase of planned testing may be complete.
But learning about the quality of a live system continues for as long as that system is being used.
Production Is a Source of Quality Evidence
Traditional testing asks:
Does the system behave as expected under the conditions we have tested?
Production gives us another set of questions:
How is the system actually behaving?
Are users successfully completing important journeys?
Where are failures occurring?
Are response times changing?
Which errors are appearing?
Are certain services becoming unreliable?
Are users abandoning particular journeys?
Did a recent release change system behaviour?
Are there patterns we did not observe before deployment?
These questions turn production information into quality evidence.
And this is where Shift Right becomes much broader than simply checking whether the application is "up."
Observability: Understanding What the System Is Telling Us
One of the concepts closely associated with Shift Right is observability.
Monitoring can tell us that something has happened.
Observability helps teams investigate why it happened.
Modern systems may contain multiple services, APIs, databases, cloud components and third-party dependencies.
A user may report:
"I clicked Submit, and nothing happened."
From the user's perspective, that is the problem.
From an engineering perspective, the cause could exist almost anywhere.
Did the request reach the application?
Did an API fail?
Did a downstream service time out?
Was an authentication token rejected?
Did the database operation fail?
Was the transaction successful but the response never returned?
Without sufficient visibility into system behaviour, teams may spend considerable time trying to reconstruct what happened.
Logs, metrics, traces, alerts and other telemetry can provide the evidence needed to investigate.
For Quality Engineers, this creates an important question that should actually begin before production:
If this functionality fails in the real world, will we have enough information to understand what happened?
That is a quality question.
Shift Right Does Not Mean "Test in Production and Hope"
This misconception is worth addressing directly.
Shift Right should never become an excuse for inadequate pre-production testing.
It does not mean:
"We can release it and see what happens."
That is not a quality strategy.
The purpose of Shift Right is to complement earlier quality activities, not replace them.
High-risk functionality should still receive appropriate testing before release.
Critical business rules should still be validated.
Security, accessibility, integration and other relevant quality considerations should still be addressed.
Shift Right acknowledges something different:
There are things we can only fully understand when software meets its real operating environment.
The objective is to learn from that environment safely and deliberately.
Monitoring Is More Than Watching Dashboards
Teams can easily collect enormous quantities of production data.
That does not necessarily mean they are learning from it.
A dashboard containing hundreds of metrics may look impressive.
But what decisions does it help us make?
Useful monitoring should connect technical behaviour to meaningful outcomes.
For example:
A server may be running.
An API may be returning responses.
CPU utilisation may look normal.
Yet customers may still be unable to complete the organisation's most important journey.
Technical health does not always equal service health.
Quality-focused monitoring therefore asks questions such as:
Can users complete critical journeys?
Are error rates increasing?
Has performance deteriorated?
Are particular user groups experiencing failures?
Are external dependencies behaving normally?
Did behaviour change after the latest release?
The purpose of monitoring is not simply to collect data.
It is to identify signals that tell us something meaningful about quality.
Real Users Will Always Teach Us Something
One of the most fascinating things about production systems is how creative users can be.
As testers, we deliberately explore unusual scenarios.
We challenge assumptions.
We test boundaries.
We ask, "What happens if...?"
But millions of real interactions can create combinations that even strong testing teams may not anticipate.
Users may navigate backwards unexpectedly.
They may leave sessions open for hours.
They may double-click buttons.
They may use browser configurations we rarely encounter.
They may abandon a journey and return days later.
They may enter data combinations that seemed improbable during test design.
This does not mean testers failed to think creatively.
It means the real world contains enormous variability.
Shift Right allows that behaviour to become another source of learning.
Controlled Releases Can Reduce Risk
Another useful Shift Right practice is controlling how new functionality reaches users.
Instead of exposing a major change to every user immediately, organisations may use techniques such as:
Feature flags to enable or disable functionality without requiring another full deployment.
Canary releases to expose a change initially to a small proportion of users.
Phased rollouts to increase exposure gradually.
A/B testing where appropriate to compare different experiences or behaviours.
These approaches can help teams observe how changes behave before expanding their reach.
The principle is simple:
Reduce the blast radius while increasing what we can learn.
However, controlled releases still require careful planning.
If teams cannot monitor the right signals or respond quickly when something goes wrong, gradually releasing faulty functionality does not automatically make the process safe.
Resilience Is Easier to Understand When Things Actually Fail
Quality is not only about whether software works when everything around it behaves correctly.
It is also about what happens when dependencies fail.
What happens when an external API becomes unavailable?
What happens when a downstream service responds slowly?
What happens when connectivity is interrupted?
What happens when demand suddenly increases?
What happens when part of the infrastructure fails?
Resilience testing can and should occur before production.
But production incidents also provide valuable evidence about how resilient the system truly is.
The important question after an incident is not simply:
"How quickly did we fix it?"
We should also ask:
"What did this incident teach us about the system?"
Production Incidents Should Feed Back Into Testing
This is one of the most valuable aspects of Shift Right.
Imagine a production incident reveals a scenario nobody had previously considered.
The team resolves the immediate problem.
But if that is where the process ends, an important opportunity has been missed.
The incident should create new knowledge.
Could a test reproduce the failure?
Should regression coverage be updated?
Was an assumption incorrect?
Was monitoring insufficient?
Could the problem have been identified earlier?
Should a requirement or design standard change?
Does another part of the system contain the same vulnerability?
Production learning should flow back into development and testing.
In other words:
Shift Right should eventually feed Shift Left.
That creates a continuous quality feedback loop.
We learn earlier.
We learn during delivery.
We learn in production.
And what we learn in production improves how we build and test the next change.
Quality Engineering Does Not End at Deployment
In some organisations, the Quality Engineer's involvement reduces dramatically once software reaches production.
The mindset can become:
"Testing signed it off. Operations owns it now."
Modern Quality Engineering challenges that separation.
Quality Engineers do not necessarily need to become production support engineers.
But understanding production behaviour can make us significantly better testers.
Production defects reveal gaps in our assumptions.
User behaviour reveals scenarios we may not have considered.
Operational data can help identify areas requiring stronger coverage.
Incident patterns can influence risk-based testing.
Performance behaviour can improve future test design.
This information is extremely valuable.
If Quality Engineers never look at what happens after release, we lose access to one of the richest sources of evidence about the software we test.
Shift Left and Shift Right Are Not Opposites
It is tempting to think of Shift Left and Shift Right as two competing approaches.
They are not.
They answer different questions.
Shift Left asks:
How can we identify quality problems earlier?
How can we receive faster feedback?
How can we prevent defects before they become expensive?
Shift Right asks:
How does the software behave in reality?
What can production teach us?
How can we detect problems quickly?
How can operational evidence improve future delivery?
Together, they create something much more powerful than either approach alone.
A continuous quality model.
Quality begins before code exists.
It continues through development and testing.
It continues through deployment.
And it continues through production.
The Feedback Loop Matters More Than the Direction
Perhaps this is the most important point.
The industry talks about moving left.
We talk about moving right.
But mature Quality Engineering is not really about choosing a direction on a diagram.
It is about building continuous feedback loops.
Requirements inform testing.
Testing informs development.
Development informs automation.
Production informs testing.
Incidents inform design.
User behaviour informs requirements.
And the cycle continues.
The faster reliable information can move through that system, the faster teams can learn.
That is ultimately what modern quality practices should enable.
Where AI Could Change Shift Right
Artificial Intelligence also has significant potential within production quality.
Modern systems generate enormous amounts of telemetry.
Logs.
Metrics.
Traces.
Alerts.
Incident histories.
User behaviour.
Performance information.
For humans, identifying meaningful patterns across this volume of information can be difficult.
AI-assisted analysis may help teams identify anomalies, correlate events, detect emerging patterns, support incident investigation and prioritise signals that deserve human attention.
But just as I discussed in Issue 007, AI introduces another quality challenge:
We must be able to trust the conclusions being generated.
An AI system identifying an anomaly does not automatically mean it has correctly identified the cause.
Human judgement, technical expertise and contextual understanding remain essential.
This relationship between AI, automation, observability and Quality Engineering deserves its own deeper discussion later in the Amber Insights series.
Final Thoughts
Shift Right changes an important assumption about software testing.
Release is not the end of quality assurance.
It is the beginning of a new source of evidence.
Production shows us what happens when our assumptions meet reality.
It tells us how users actually behave.
It reveals how systems respond to genuine demand.
It exposes unexpected interactions.
It teaches us about resilience.
And sometimes, it shows us things our test environments simply could not.
The objective should never be to replace careful pre-production testing with experimentation in production.
The objective is to recognise that quality learning continues throughout the life of the product.
Shift Left helps us learn sooner.
Shift Right helps us learn from reality.
And mature Quality Engineering connects those lessons into one continuous feedback loop.
Because ultimately, the question is not whether quality belongs on the left or the right.
Quality belongs everywhere.
About the Author
Amber Idumwonyi is a Senior Quality Engineering Leader with extensive experience delivering quality engineering solutions across UK Government, defence, healthcare, and enterprise technology programmes. She has led Quality Engineering initiatives, developed Test Strategies, contributed to modern testing and automation practices, spoken at industry conferences, and founded LEMP, an organisation focused on practical technology education, mentoring, industry experience, and career development.
Amber is passionate about Quality Engineering, AI-assisted testing, technical leadership, and developing the next generation of technology professionals.
Read More:
Amber Insights | Issue 005 – Risk-Based Testing: Focusing on What Matters Most
Amber Insights | Issue 006 – Building a Quality Culture Across Delivery Teams
Amber Insights | Issue 007 – Shift Left Testing Explained: What It Really Means in Practice
Explore LEMP's Software Testing programme and learn how you can get started.
Comments