Continuous Delivery: Removing manual scenarios | Javalobby

Continuous Delivery: Removing manual scenarios | Javalobby

• We’d be able to deploy multiple times a week while still having a reasonable degree of confidence that the application still works. 
• To reduce the number of manual scenarios that our QAs need to run through.

Pain Driven Development

Pain Driven Development at Mark Needham

• Following the ideas of pain driven development we delayed automating any bits of the application if we couldn’t work out how to automate them easily.

Essence of Testing: To test or not to test ... do you ask yourself this question?

This inherently happens at the expense of ignoring / de-prioritizing other equally important activities like Manual (exploraratory / ad-hoc) testing. 

I have found a few strategies work well for me when I approach Test Automation:

• Take a step back and look at the big picture.
• Ask yourself the question - "Should I automate this test or not? What value will the product get by automating this?"
• Evaluate what test automation will truly provide good and valuable feedback.
• Based on the evaluation, build and evolve your test automation suite.

To start off, analyze your tests and categorize them in the different quadrants of the above graph.

1. First automate tests that provide high value, and low cost to build / maintain = #1 in the graph. This is similar to the 80/20 rule.
2. Then automate tests that provide high value, but have a high cost to build / maintain = #2in the graph.
3. Beyond this, IF there is more time available, then CONSIDER automating tests that have low value, and low cost. I would rather better utilize my time at this juncture to do manual exploratory testing of the system.
4. DO NOT automate tests that have low value and high cost to build / maintain.

IMVU’s Approach to Integrating Quality Assurance with Continuous Deployment « IMVU Engineering Blog

we have implemented:

• A continuous integration server (Buildbot) to run and monitor our automated tests with every code commit
• A source control commit check to stop new code being added to the codebase when something is broke
• A script to safely deploy software to our cluster while ensuring nothing goes wrong. We wrote a cluster immune system to monitor and alert on statistically significant regressions, and automatically revert the commit if an error occurs)
• Real-time monitoring and alerting to inform the team immediately when a bug makes in through our deployment process
• Root cause analysis (Five Whys) to drive incremental improvements in our deployment and product development processes

We’re definitely not perfect: with constrained QA resources and a persistent drive by the team to deliver value to customers quickly, we do often ship bugs into production or deliver features that are imperfect. The great thing is that we know right away what our customers want us to do about it, and we keep on iterating.