My Journey from Scriptless to script based automation

I have been working as a QA specifically on automation since quite a time. I have used various tools to automate GUI, WebServices and Mobile apps. And over the period of time realised whats the best approach based on the requirements and the nature of the application under test.

As a beginner i chose a ‘record and playback’ tool to automate my application. I chose a typical basic user flow. I just had put the recording on in the tool and perform all the actions I wanted to automate. Bam..!! My user flow is automated..!! It was a wow moment for me. I did this as part of my regression test and resting on a build. It did not take any time for me to understand/learn the tool and i am able to automate my repetitive tasks.

Some insight to the Application Under Test :

An configurable ERP system available in three configurations namely a backed and a corresponding front end to it and the mobile native app for the outside sales rep.

This system was integrated to multiple freight service providers, accounting systems, shipping tracking systems and the user specific pricing engines maintained as separate systems.

So lot of dependencies when i have to even test a simple Quote-Cart-Sales Order- Purchase Order flow in terms of configurations, pricing , taxes applied based on shipping address and the discount applicable baed on the discount rules.And all of this changes user to user account to account. Interesting..?

We followed Agile and a 2 week sprint and the builds to clients were every 2 week with some user/account specific enhancements , integrations and new features for the entire client base.

My first Regression run :

I got a new build and I was ready with my regression suite with all the high priority user flows.

Bam…!! All failed. Where did i go wrong? At that time i did not spend time on it to figure out what had happened. All i knew is this approach will not work for me.

I continued with manual testing and build was sent on the production, in retrospective this was noted down.

While analysing , i learnt the following about the record and playback tools.

• As the name itself says , its just record and playback where all the user actions such as mouse movements and clicks are recorded and part of test script development and played back when required once or multiple times.
  • This is more suitable when your application is static , no data dependancy , and no frequent changes to the UI elements coming on the way.
• The test data is hard coded.
  • Lets take an example : Say “User creation” was automated. The username (has to be chosen by the user and should be unique) , first name, last name, address line 1, address line 2 , city, state, country and the pincode were the fields captured and entered data for. Data entered was “Sugale”, “Shama”, “Ugale”,”Flat no 1″,”Fatima Nagar”, “Pune”,”Maharashtra”,”India”,”4110040″. Now I play back this test,which fails , with error popping “Username already exists”.
• Easy for beginners. Takes less to no time to get started.
• Record and playback test automation tools allow all the stakeholders – including Business Analysts and Domain Experts – to participate in and contribute to the automation testing effort.
• Although the test development is faster, the test data, objects/UI elements and actions are hard coded, hence eventually maintenance and debugging time would over-weigh this advantage.
  • The tests are very fragile. Just the simplest change to the application often breaks a dozen tests, even though the application still works as intended. Say in the above example “User creation” , if the username element id changes , object identification fails and all the recorded tests around user creation flow would fail.
  • To fix the failed tests, you will have an option to re-recordall the flows again. Not the only the failed actions but the entire flow and all the flows where this change has to be applied. Lots of overhead..?

Now if not record and playback tool is the approach , whats next in there for me?? Its time for me to evaluate tools available for the kind of application i was supposed to automate. Which i should have done earlier. Anyways, its never too late and i have learnt a lesson for my life. Don’t jump onto a solution. Evaluate what best suits for you. Have the pros and cons very well known before proceeding.

First in place i noted down all my requirements from the Application under test.

1. I should be able to automate end-to-end user flows.
2. My Test automation tool that i would select should help me accommodate my GUI tests and integrations tests which are webservices.
3. My mobile native app and mobile web app should be automated with the same flows to validate it works on mobile as same as it does on the web.
4. I should be able to do complex calculations for cart/quote values based on pricing , freight charges, discounts and taxes applied. And validate if these parameters are applied correctly for the given logged-in user.
5. I should be able to verify the results against DB and validate my persistence layer.
6. I should be able to debug.
7. Should be able to produced sufficient logs to analyse the failures. Capture the screenshots for the failed tests/steps.
8. Publish the automation execution summary reports to the management.
   1. Total tests executed.
   2. The actual flows that were automated.
   3. Pass and failed %
   4. Categorisation of failures – Blockers, P1 P2 and so on.
   5. Compatibility testing against browser and platform combinations
9. And all of this should be faster , so that i get the feedback as soon as possible after the build is deployed.
10. Also , great if i can trigger the tests as soon as the build is done. So ,need to make this as part of my CI/CD pipeline.
11. Good to have – Integrate this with out tests management tools and bug tracking tools as the management is heavily dependant the reports generated by these tools.

Evaluated various tools based on what i was looking for. The parameters i considered were:

1. Cost
2. Support
3. How active is the community
4. How Frequent are the releases with newer features
5. Learning curve
6. Maintenance efforts/cost
7. Flexibility

Based on my own research and evaluation i realised that there is no such tool which can give all the features i am looking for and I came down to creating my own automation framework so as to address all the needs and keep it scalable and more flexible for the future. Also generalise the libraries around test data sources,reporting , support for TDD/BDD, DB interactions, detailed loggings, integrations, cross bowser/platforms/devices support, have flexibility ti fire API calls in UI tests for test data generation or validations, integrate it with the CI/CD build process. All this with the minimal maintenance efforts by leveraging all the benefits of OOPS and reusing the code with no duplicity,  and assuring for any given change in the application or data or elements there should be a single point of change.

This definitely doesn’t happen over night, lots of experience on the table to be bought on , lots of thought process in terms of design, development, lots of testing is to be done to validate it works as expected. Over the years i have been working to get to the desired one and finally after working on various domains including Banking , E-commerce, ERP, HealthCare , Finance i have been able to successfully design and implement the automation solution satisfying most of the requirements form these domains – TEAL. One can simply start to automate their end-to-end user flows with TEAL and get started.