Integrating TestRail and Gitlab CI/CD

TestRail

Tools: Gitlab CI/CD, TestRail, Node.js, GOT HTTP request library.

Perhaps you are using Gitlab CI/CD at your project. Every project requires some test cases for regression testing, and Gitlab actually provides this feature, but it's only available in Ultimate version that is more expensive. TestRail is another popular platform for managing your test suite that provides way more extensive capabilities and options than Gitlab's own test case management feature. So the chances are that you are still willing to use TestRail for your acceptance and regression testing efforts. Why not combining the best of two worlds - the flexibility of Gitlab CI/CD and rich test case management capabilities of TestRail? In the following example I'll demonstrate how this goal could be achieved with ease.

Let's assume that we need to create a new Milestone in TestRail that contains two test runs - the one with Acceptance tests, and another one with Regression tests. The step in your .gitlab-ci.yml configuration file would look like this:

prepare-tr-milestone:
  stage: regression-tests
  allow_failure: false
  only:
    - /^v\d+\.\d+\.\d+$/
  script:
    - npm install
    - export TESTRAIL_MILESTONE="My Application ${CI_COMMIT_REF_NAME}"
    - export TESTRAIL_TITLE_MANUAL="${TESTRAIL_MILESTONE} - Manual Regression Testing"
    - export TESTRAIL_TITLE_ACCEPTANCE="${TESTRAIL_MILESTONE} - Acceptance Testing"
    - node ./scripts/prepare-tr-milestone.js
    - exit $?

This step is reading ${CI_COMMIT_REF_NAME} environment variable which is the tag name for which the project is built. /^v\d+\.\d+\.\d+$/ regular expression in the only field allows this step to be executed only for the tags which have this particular format, e.g. v15.1.0. Thus our TestRail milestone will be named My Application v15.1.0. We are also setting two more environment variables for Acceptance and Manual Regression test run names: My Application v15.1.0 - Acceptance Testing and My Application v15.1.0 - Manual Regression Testing. prepare-tr-milestone.js script is where the magic happens, let's review it in detail.

(async function main() {
  const TestRailAPI = require('testrail-api');
  const GitlabClient = require('gitlab-api');
  const GITLAB_DOMAIN = 'gitlab.com';

  // Initializing TestRail and Gitlab API clients

  const testrail_api = new TestRailAPI(
    'https://'+process.env.TESTRAIL_DOMAIN,
    process.env.TESTRAIL_USERNAME,
    process.env.TESTRAIL_APIKEY);

  const gitlab_api = new GitlabClient.API(
    'https://'+GITLAB_DOMAIN,
    process.env.CI_PROJECT_ID,
    process.env.GITLAB_PRIVATE_TOKEN
  );

  await testrail_api.createNewMilestone(process.env.TESTRAIL_PROJECTID, process.env.TESTRAIL_MILESTONE);

  await testrail_api.createTestRun(process.env.TESTRAIL_TITLE_MANUAL, process.env.TESTRAIL_PROJECTID, process.env.TESTRAIL_MILESTONE, "Regression");

  var issueIds = gitlab_api.getIssueIdsFromTag(process.env.CI_COMMIT_REF_NAME);

  await testrail_api.createTestRun(process.env.TESTRAIL_TITLE_ACCEPTANCE, process.env.TESTRAIL_PROJECTID, process.env.TESTRAIL_MILESTONE, "Acceptance", issueIds);
})();

This script begins with initialization of TestRail and Gitlab API clients. Valid access credentials have to be used. You can see the implementation of createNewMilestone method below, along with all the helper methods. It only creates a new TestRail milestone if it doesn't exist yet, and skips creation of a milestone otherwise. getMilestones method also uses pagination because TestRail API limits the maximum number of results which are returned for /api/v2/get_milestones endpoint.

// Create a new project milestone if it doesn't exist yet
async createNewMilestone(projectId, milestoneName) {
  var milestones = await this.getMilestonesByName(projectId, milestoneName);
  if (milestones.length == 0) {
    await this.addMilestone(projectId, milestoneName);
    console.log(`Created "${milestoneName}" milestone`);
    milestones = await this.getMilestonesByName(projectId, milestoneName);
  } else {
    console.log(`"${milestoneName}" milestone already exists`);
  }
  return milestones;
}

// Filter the list of project milestones by name
async getMilestonesByName(projectId, milestoneName, isCompleted = false) {
  var milestones = await this.getMilestones(projectId);
  return milestones.filter((ms) => ms.name == milestoneName && ms.is_completed == isCompleted);
}

// Add a new milestone to the project
async addMilestone(projectId, milestoneName) {
  var url = `${this.base_url}/api/v2/add_milestone/${projectId}`;
  var res = await request.post(url, {
    headers: this.headers,
    json: { name: milestoneName }
  });
  if (res.statusCode >= 300) console.error(res.body);
  var milestone = JSON.parse(res.body);
  return milestone;
}

// Get the list of milestones for the project
async getMilestones(projectId) {
  var url = `${this.base_url}/api/v2/get_milestones/${projectId}&is_completed=0&limit=250`;
  var milestones = [];
  var next_page_lnk = null;
  do {
    var res = await request.get(url, { headers: this.headers });
    if (res.statusCode >= 300) console.error(res.body);
    var page = JSON.parse(res.body);
    milestones = milestones.concat(page.milestones);
    next_page_lnk = page['_links'].next;
    url = `${this.base_url}${next_page_lnk}`;
  } while (next_page_lnk != null);
  return milestones;
}

The next part of the script creates a test run with Regression test cases (this is a Test Case Type) and assigning this test run to the milestone that we've created. You can additionally filter test cases by references, but this is only needed for Acceptance test runs that will be explained in the next paragraph.

async createTestRun(runName, projectId, milestoneName, caseTypeName, referenceList = null) {
  var milestones;
  if (milestoneName !== null) {
    // Getting a milestone if needed
    milestones = await this.getMilestonesByName(projectId, milestoneName);
    if (milestones.length < 1) {
      console.log(`Could not find "${milestoneName}" milestone in the project`);
      return null;
    }
  }
  // Checking if run already exists
  var existingRuns;
  if (milestoneName !== null) {
    existingRuns = await this.getRunsByName(projectId, runName, milestones[0].id);
  } else {
    existingRuns = await this.getRunsByName(projectId, runName);
  }
  // Getting all test cases for the project
  var cases = await this.getCases(projectId);
  // Getting the required case type
  var caseTypes = await this.getCaseTypesByName(caseTypeName);
  if (caseTypes.length < 1) {
    console.log(`Could not find "${caseTypeName}" case type in the project`);
    return null;
  }
  // Filtering by case type
  cases = cases.filter((tc) => tc.type_id == caseTypes[0].id);
  if (cases.length > 0) {
    // Filtering by references if needed
    if (referenceList !== null) {
      var casesByReferenceList = [];
      for (var i = 0; i < referenceList.length; i++) {
        var re = new RegExp(`#${referenceList[i]}(\s+|,|$)`);
        var caseByReference = cases.filter(tc => re.test(String(tc.refs)));
        casesByReferenceList = casesByReferenceList.concat(
          caseByReference
        );
      }
      cases = casesByReferenceList;
    }
    // Getting case ids and preparing a payload
    var caseIds = cases.map((c) => c.id);
    // Removing duplicate ids
    caseIds = Array.from(new Set(caseIds));
    if (caseIds.length > 0) {
      var payload = {
        name: runName,
        include_all: false,
        case_ids: caseIds
      };
      var run;
      if (existingRuns.length > 0) {
        console.log(`Test run with the name "${runName}" already exists, updating it.`);
        await this.updateRun(existingRuns[0].id, payload);
        return existingRuns[0];
      } else {
        // Creating a test run and assigning it to a milestone
        run = await this.addRun(projectId, payload);
        console.log(`Created "${caseTypeName}" test run with the name "${runName}"`);
      }
      if(milestoneName !== null) {
        await this.updateRun(run.id, { milestone_id: milestones[0].id });
        console.log(`Assigned "${runName}" test run to the "${milestoneName}" milestone`);
        return run;
      }
    } else {
      console.log(`No test cases found which satisfy the filter parameters, skipping the test run creation.`);
      return null;
    }
  } else {
    console.log(`No ${caseTypeName} test cases found in the suite, skipping the test run creation.`);
    return null;
  }
}

// Filter the list of runs by name
async getRunsByName(projectId, runName, milestoneId = null) {
  var all_runs = await this.getRuns(projectId);
  var runs = milestoneId
    ? all_runs.filter(
        ps =>
          ps.name == runName &&
          ps.milestone_id == milestoneId &&
          ps.is_completed == false
      )
    : all_runs.filter(ps => ps.name == runName && ps.is_completed == false);
  return runs;
}

// Filter the list of case types by name
async getCaseTypesByName(caseTypeName) {
  var caseTypes = await this.getCaseTypes();
  if (caseTypes.length == 0) {
    console.log(
      'No case types found in the project, you need to have at least one'
    );
    return null;
  } else {
    return caseTypes.filter(ct => ct.name == caseTypeName);
  }
}

Generic methods of the TestRail API client which get the list of runs, test cases and test case types for your project. Note the use of pagination for runs and cases, the responses could contain thousands of entries. Case Types are account-specific, not project-specific, this is why the number of case types is limited and pagination is not needed.

// Get the list of runs for the project
async getRuns(projectId) {
  var url = `${this.base_url}/api/v2/get_runs/${projectId}&is_completed=0&limit=250`;
  var runs = [];
  var next_page_lnk = null;
  do {
    var res = await request.get(url, { headers: this.headers });
    if (res.statusCode >= 300) console.error(res.body);
    var page = JSON.parse(res.body);
    runs = runs.concat(page.runs);
    next_page_lnk = page['_links'].next;
    url = `${this.base_url}${next_page_lnk}`;
  } while (next_page_lnk != null);
  return runs;
}

// Get the list of test cases for the project
async getCases(projectId) {
  var url = `${this.base_url}/api/v2/get_cases/${projectId}`;
  var cases = [];
  var next_page_lnk = null;
  do {
    var res = await request.get(url, { headers: this.headers });
    if (res.statusCode >= 300) console.error(res.body);
    var page = JSON.parse(res.body);
    cases = cases.concat(page.cases);
    next_page_lnk = page['_links'].next;
    url = `${this.base_url}${next_page_lnk}`;
  } while (next_page_lnk != null);
  return cases;
}

// Get the list of case types
async getCaseTypes() {
  var url = `${this.base_url}/api/v2/get_case_types`;
  var res = await request.get(url, { headers: this.headers });
  if (res.statusCode >= 300) console.error(res.body);
  var case_types = JSON.parse(res.body);
  return case_types;
}

These are the generic POST methods which create and update TestRail runs. There's no need to delete and create a run again if it can be updated with a new set of tests.

// Add a new run to the project
async addRun(projectId, entryJSON) {
  var url = `${this.base_url}/api/v2/add_run/${projectId}`;
  var res = await request.post(url, {
    headers: this.headers,
    json: entryJSON
  });
  if (res.statusCode >= 300) console.error(res.body);
  var run = JSON.parse(res.body);
  return run;
}

// Update an existing run
async updateRun(runId, entryJSON) {
  var url = `${this.base_url}/api/v2/update_run/${runId}`;
  var res = await request.post(url, {
    headers: this.headers,
    json: entryJSON
  });
  if (res.statusCode >= 300) console.error(res.body);
  var run = JSON.parse(res.body);
  return run;
}

In order to add test cases to Acceptance test run we would need to additionally filter test cases by references to avoid including test cases which had been created for older releases. This filter is already included into the createTestRun method, but it's only used when the list of references is provided as the last parameter.

// Filtering by references if needed
if (referenceList !== null) {
  var casesByReferenceList = [];
  for (var i = 0; i < referenceList.length; i++) {
    var re = new RegExp(`#${referenceList[i]}(\s+|,|$)`);
    var caseByReference = cases.filter(tc => re.test(String(tc.refs)));
    casesByReferenceList = casesByReferenceList.concat(
      caseByReference
    );
  }
  cases = casesByReferenceList;
}

So where do we get those references from? References are Issue IDs, and we can get them from a tag in Gitlab that is created for every product release. This is where we need to use our Gitlab API client. Gitlab API has a special /repository/tags/ endpoint for tags, we just need to parse the response result, converting it into the list of Issue IDs.

// Read a comma-separated tag message and convert it to the list of issue IDs
// it should be in the format: #[issue1], #[issue2], ...
async getIssueIdsFromTag(tag_name) {
  var gitlabTagMsg = await this.getTagMsg(tag_name);
  var issueIds = gitlabTagMsg
    .replace(/#/g, '')
    .replace(/ /g, '')
    .split(',');
  return issueIds;
}

async getTagMsg(tag_name) {
  var res = await request.get(
    `${this.api_endpoint}/projects/${this.project_id}/repository/tags/${tag_name}`,
    { headers: this.headers }
  );
  var json = JSON.parse(res.body);
  return json.message;
}

Note that we use a human friendly GOT HTTP request library for Node.js for TestRail and Gitlab API clients above. We also retry failed requests when we receive 409, 429 or 500 codes. You could also use any other open-source libraries to wrap the API, but they might not be so easy to update in case of breaking API changes.

const got = require('got');

const request = got.extend({
    retry: {
        limit: 10,
    methods: ["GET", "POST", "PUT"],
    statusCodes: [409, 429, 500]
    }
});

The initialization of Gitlab API client (you have to provide valid Project ID and Gitlab private token):

class API {
  constructor(domain, project_id, private_token) {
    this.api_endpoint = `${domain}/api/v4`;
    this.project_id = project_id;
    this.headers = {
      'Content-Type': 'application/json',
      'PRIVATE-TOKEN': `${private_token}`
    };
  }
}

The initialization of TestRail API client (you have to provide valid username and API key):

class TestRailAPI {
  constructor(domain, username, apikey) {
    this.base_url = `${domain}/index.php?`;
    this.auth = Buffer.from(`${username}:${apikey}`).toString('base64');
    this.headers = {
      'Content-Type': 'application/json',
      Authorization: `Basic ${this.auth}`
    };
  }
}

The demonstrated integration of Gitlab and TestRail helps us to make our testing efforts more efficient, bringing together the best features of both tools. Feel free to use these tips to create a test run for your automated tests, but this would also require the implementation of a TestRail reporter. Depending on your test automation framework, this reporter could be implemented in many different ways.

Artem Nagornyi (Monex Insight - Test Automation Engineer)