Archive

Archive for the ‘QA’ Category

Testcast – QA Market around the world

Hello everyone,

I participated in a Podcast called TestCast ūüėÄ

We talked about our different experiences in QA Market around the world. Sadly, the podcast is 100% in Portuguese with no transcription.

Cucumber report with protractor tests

Hi guys, today I will show what you can do with the module cucumber-html-report that I am currently using in my protractor automated tests.

So, it is a node library that reads the Json reports and convert them into a nice html report. It should show the coverage of your scenarios and if you want you can have pictures attached as well.
I am pasting here some snippets that you guys can use and customise as you want.
To install you need to add the cucumber-html-report in your dependencies like:
{
    "name": "Rafazzevedo",
    "version": "1.0.0",
    "description": "Rafazzevedo",
    "private": true,
    "dependencies": {
        "cucumber-html-reporter": "^0.4.0",
        "protractor": "^5.1.1"
    },

    "devDependencies": {
        "chai": "^3.5.0",
        "chai-as-promised": "^6.0.0",
        "chai-string": "^1.3.0",
        "cucumber": "^1.3.2",
        "protractor-cucumber-framework": "^1.0.2"
    }
}
Then you create an After in your Hooks class and Add the creation of the report:
import { browser } from 'protractor';
import reporter from 'cucumber-html-reporter';
import _ from 'lodash';
import Navigation from './helpers/navigation';

const API = automation.API;

const Hooks = function () {
    const api = new API(browser.params);
    const navigation = new Navigation(browser);
    const credentials = browser.params.credentials;

    this.registerHandler('BeforeFeatures', () => {
        api.authenticate(credentials);
        browser.api = api;
        browser.navigation = navigation;
        browser.get(browser.baseUrl);
    });
    this.registerHandler('AfterFeatures', () => {
        browser.getProcessedConfig().then((config) => {
            const jsonFile = config.capabilities.cucumberReportPath;
            browser.getCapabilities().then((capabilities) => {
                const reportName = createReportName
(config.capabilities, capabilities);
                const htmlFile = jsonFile.replace('.json', '.html');
                const options = {
                    name: `Automation (${reportName})`,
                    theme: 'bootstrap',
                    jsonFile,
                    output: htmlFile,
                    reportSuiteAsScenarios: true,
                };
                reporter.generate(options);
            });
        });
    });
    function createReportName(configCapabilities, browserCapabilities) {
        const browserName = configCapabilities.browserName;
        let deviceName = 'desktop';
        let browserVersion = '';
        if (!_.isUndefined(configCapabilities.chromeOptions)) {
            browserVersion = ` ${browserCapabilities.get('version')}`;
            if (!_.isUndefined(
configCapabilities.chromeOptions.mobileEmulation)) {
                deviceName = configCapabilities.chromeOptions.
mobileEmulation.deviceName;
            }
        }
        return `${browserName} ${deviceName}${browserVersion}`;
    }
};

module.exports = Hooks;
After adding these 2 functions about creating the report for multiple browsers and adding the version and the device name (desktop or mobile) in the report, you can add pictures in case of the scenario fails:
this.After((scenario) => {
        if (scenario.isFailed()) return browser.takeScreenshot().
then(screenshot => scenario.attach(new 
Buffer (screenshot, 'base64'), 'image/png'));
    });
 
You can always customise and improve the functions, if you feel that you need a picture after all the scenarios even they passed, you can just remove the condition in the after. Also you have different available themes and other options on https://www.npmjs.com/package/cucumber-html-reporter
See you guys !

Bad practices when writing your BDD scenarios

I usually write about the best practices to follow when writing your BDD scenarios. This time I will do different and write some examples that I found of how to not write your BDD scenarios.

 Example 1 РToo many steps :

  Scenario: Valid data entered
    Given a user exists
    When I visit the details access page
    And I fill in email with "test@email.com"
    And I select "read only" from "Permissions"
    And I fill in "Valid until" with "2010-03-01"
    And I press "Grant access"
    Then I should see "Access granted till 2010-03-01."
    And I should be on the details page

 

 Example 2  РUI elements dependency:

   Scenario: Adding a picture 
     Given I go to the Home page 
     When I click the Add picture button 
     And I click on the drop down "Gallery"
     And I click on the first image
     Then I should see the image added on the home page

 

 Example 3 РExcessive use of tables :

   Scenario: Adding a new data user 
     Given I am on  user details page
     When I select an existent user
     And I send some new user data
     |name |age|country|language|address |telephone|
     |James|20 |england|english |street 1|123456789|
     |Chris|30 |spain  |spanish |street 2|987654321|
     Then I should see the correct previous registered data
     |gender  |married|smoke|
     |male    |true   |true |
     |male    |false  |false|

 

 Example 4  РCode and data structure:

   Scenario: Include attachment
     Given I go to the Attachments page 
     When I click the Add attachment button with css "#attachment-button"
     And I click on the first csv file with class ".file .csv"
     Then I should see the file attached with the id ".attachment"

 

  • Write declarative¬†scenarios.
  • Write at a higher level of abstraction, instead of being concerned with clicking widgets on a page.¬†Surfacing UI concerns on a feature can make it fragile¬†to ordinary¬†design changes.
  • Try to¬†stick to having not¬†more than 5 steps per scenario.¬†It’s easier to read.
  • Avoid code or data structure¬†like xpath selectors, css class, regex in your scenario.

 

Test Reporting in Agile

When working in an Agile team, the feedback loop should be as quick as possible. If you don’t send the feedback at the right time, future¬†bugs could be costly as the feature has already a big amount¬†of code.

What is this feedback loop ?

If you have implemented Continuous integration and have automated tests running after a new commit on your dev environment, you need to report back as soon as possible the result of these tests. This is the feedback loop and you need to know the correct time to report the issue to the dev team.

If your automation is taking too long to run the tests after a new commit, this is a sign that you need to improve your smoke tests, maybe your scope is too long or maybe your automation is taking too long to run for other reason, sleeps, not scalable, etc.

The feedback loop influences how your agile process works and if you are saving or wasting time when developing. Tight feedback loops will improve performance of the team in general, give confidence, save time and avoid costly bug fixes.

Feedback loops are not only about the continuous integration, it is about pair programming and unit tests as well, but this time we will focus on Continuous integration tests.

When you are implementing a new scenario in your automated tests, you want to know ASAP¬†if something you implemented is breaking some other scenario or the same scenario. Same situation when you are developing something related to that feature and you want to know if this new implementation is breaking the tests. It is easier to fix, it is fresh in your mind, you don’t need to¬†wait 30 minutes to know there is a bug when you changed the name of a variable…

In my personal opinion,¬†if you don’t have parallel tests to check in multiple browsers or mobiles at the same time if something is broken, it is better you focus on the most used browser/mobile, since this is first priority in all the cases.

Use case: 90% users are on Chrome on Desktop, other 5% users are on Firefox mobile, other 5% are on Safari mobile. What is the best strategy ?

After commit:

  1. Run Smoke tests and all the browsers, take 15 minutes to receive feedback ?
  2. Run Regression tests and all the browsers, take 40 minutes to receive feedback ?
  3. Run Smoke tests and only the most used browser, take 5 minutes to receive feedback, leave to run on all the browsers every hour ?
  4. Run Regression tests and only the most used browser, take 10 minutes to receive feedback, leave to run on all the browsers every hour ?

There is no¬†rule to follow, since¬†in this case¬†you don’t have parallel tests, I would¬†go for the third option. Then, you can focus on the most used browser and leave running the other browsers in a dedicated job each hour. Why not fourth option ? Because you need to keep in mind the business value.

Of course that we need to delivery the feature¬†on all used¬†browsers, but when the time is tight (very often) and you need to deliver as fast as you can, you go for the most business value option and implement the other browsers after. Don’t forget when you automate, you don’t think only about helping the development, but also you think about helping the end users.

If you are wondering how long each type of test should take to give feedback, you can build your own process basing on this graph:

Screen Shot 2017-04-06 at 19.18.44.png

For how long should the team keep the test reports ?

Depends of how many times you run the tests through the day. There is not a rule for that, so you need to find what is the best option for you and your team. In my team we keep the test reports until the 15th run on Jenkins and after this we discard the report and the logs. In most of the cases, I’ve found that if something goes back more than 3 major versions, look¬†for more resolution is a waste of time.

If regressions are reported as soon as they’re observed, the reporting should include the first known failing build and the last known good build. Ideally these are sequential, but this isn’t necessarily the case. Some people like to archive the old reports outside Jenkins. I didn’t feel the¬†need for this until now, but is up to you¬†to keep these reports outside Jenkins.

 

Resources:

http://istqbexamcertification.com/why-is-early-and-frequent-feedback-in-agile-methodology-important/

https://www.infoq.com/news/2011/03/agile-feedback-loops

Webinar: Using Espresso for Fast and Reliable Feedback

How to make automation tests more effective

As a QA Engineer you often need to do regression tests and you know this is a really waste of time since every time is the same test. Then, you keep in mind that you need to automate the regression tests in order to save time with the old things that could be broken with the introduction of the new features.

The development of the automation is usually done by the developers in test, people who has QA ¬†and programming knowledge. One completes the other, so you know how to automate the right scenarios with the right prioritisation. Don’t spending time automating scenarios with no or small¬†value or with wrong prioritisation for the regression pack.

You need to include the repetitive scenarios, the critical ones and the happy path but try to avoid the edge cases. Why ? Because edge cases are often scenarios with minimum or no value, where it is a specific flow that should be test once when developing the feature not every time when doing the regression. Regression pack ensures the happy path is working and the end user will be able to do what he needs to do. When you spend time implementing automated edge cases, you actually waste your valuable time implementing scenarios without real business value.

Although the product owners may be able to immediately suggest points to automate, it also depends on developers working on the detailed code. So, for this reason you need to have a good analyse before about the scenarios that should be implemented and if they will change very often.

Here some tips about how to create your automation tests more effectively:

 

Developing Automation

Automation code is quite similar to¬†the development code, but it should be more simple and readable. Automation is not meant to be complex. The business rules should be implemented¬†as part of the BDD scenarios, so the code is clean and doesn’t have anchors of complexity.

ROI (Return of Investment), you need to guarantee the value of the scenarios when automating them. For example, a scenario that tests the function of a feature is far more important and valuable than a scenario that tests if the buttons have the expected colour. Layouts are important, but you will spend more time implementing a scenario asserting the colour than opening the page and manually checking in milliseconds, also it is not a critical issue, the user will be able to finish the scenario despite the colour of the button. Measure the time before and after the automation is implemented, so you can have an idea of the time and effort saved.

Optimizing time

We have a common problem in the agile environment, when we rush to finish the sprint and forget quality. This is because we close the tickets, but we create a backlog of bugs every time and it keeps growing every sprint. These sprint backlogs make it difficult to devote time for the development, debugging and testing of each iteration.

For this reason it is really important to save a good amount of time for testing. To help saving time with the test, you can run the automation in parallel to the development, so any regression bug would be caught as soon as it is merged in a master branch. This gives more scope to the QA engineers to develop efficient tests through exploratory search.

Client Test Coverage

The ideas that come just after the brainstorming help testers to identify different scenarios and create a better test coverage for the feature. So you need to have this time to mature the idea of the feature and think about the flow and possibilities.

It is important to think more broadly when talking about test automation and not think only about the test cases. The planning and brainstorming can lead to breakthroughs that change the testing pattern altogether.

Regression Pack

When you implement the automated regression tests you need to keep well maintained with the development of the features. If not, your regression will be not up to date and you will not be able to trust on your automation anymore. Make sure your regression pack guarantee the functionality of the system and monitor the performance of the tests so as soon as you have some failure you can identify if it was a bug on the system or if is something you need to update with the current development code.

Regression tests should run smoothly and without human intervention. The only thing you need to worry is adding new features to the package.

Visibility

As I have described before, you need to keep it simple. This is the key to have a smooth automation. You need to be sure the stakeholders will understand what is being tested. Show the statistics of how long is taking to run the regression pack, how much time you are saving, the percentage of tests coverage vs time before and after automation, the overall quality improved.

Sharing this data will show a positive thinking about automation and how much you have improved automating the tests. This makes it simpler to frequently update test scripts, and guarantees collaborative effort through mutual cooperation.

Stay well aligned with Developers

It is essential to be aligned with the development work. Understand all the flow and how something they have changed could impact on another completely different, for example. This will help you to anticipate and be one step ahead when maintaining the scenarios. Also, it is good all the teams work with the same environment, using the most similar tools when it is possible.

Understand the functionality of the current environment, in order to successfully perform root-cause analysis that yields in constructive solutions. This will help you to find bugs more efficiently and build your automation focusing on your actual environment. Remember you need to align your needs with your project and the current development cycle. Companies/projects/teams are not equal and there is no formula, but some tips of how can you take the best for your situation.

 

Webinar – Quality Metrics (Sealights)

Hey guys, today I am going to post this quick webinar with some QA metrics to use in your project.

Really simple and good presentation about percentages of integration/code coverage and other metrics.

 

Categories: QA Tags: , ,
%d bloggers like this: