During the last decade, methodologists and software engineers have been pushing for a paradigm shift in IT focusing on new roles and creative intereaction among them. The background of those driving the change has conditioned the roles and responsabilities in development teams, forgetting about some of the key actors in more traditional processes:

• QA: Quality Assurance Engineers, mainly functional testers
• Project Manager.
• Project Office.

The challenge in introducing an agile culture in traditional corporations (or just change-resistant ones) requires a review of the role of these and some other actors in software development teams. In this first article on corporate agilism we will provide our vision on the new areas of responsibility for quality assurance engineers.

QA in agile teams

As opposed to heavyweight methods from the 80′s, agile frameworks seem to have reserved a marginal role to QA guys. Formal documentation on functional testing for agile teams is hard to find and the introduction of techniques such as TDD or Continuous Integration may be confused with functional validation.

We firmly believe all agile teams should have specific roles in charge of functional testing and quality assurance.

QA as Product Owner

Product Backlog in Scrum is the ordered list of functional requirements sorted by business value (or ROI). Each item in the list (Product Backlog Item) must contain at least the following data:

• Feature: A functional description (user story).
• Business Value: Relative estimate of value for the client/user.
• Cost: Relative estimate of effort for the team.
• Acceptance Criteria: Definition of Done, Validation and/or regression tests.

Leaving aside estimates for business value (it’s sole responsibility lies on the user) and cost (it’s sole responsibility lies on the development team), data related to a PBI belongs to the area of responsibility of Functional QA testing!

Consider appointing a Funtional tester as Product Owner in order to enhance communication of user stories and maximize quality in every Sprint.

This approach has also, some added benefits:

• User Stories expressed in a more developer-friendly language
• Definition of Done for every user story
• Product Quality in every release, which provides greater reliability on the team and project advance.

Those companies where product quality is critical or software development driven teams are ideal candidates for this layout.

QA as part of the development team

Frequently, rigid organizations do not allow the flexibility and selft-organization required by agiles teams (see Agile Ecosystems). In some cases, responsibility for functional requirements lies in departments outside development or technology.

When it is not possible to have a Quality tester act as Product owner, the team should enfoce product quality in the following practices:

• Planning session. Quality should contribute to the understanding of user stories and Definition of done as these information may have an impact in the cost/effort estimate.
• Tracking. Continuous Integration (on different environments) must be a valuable tool for functional validation.
• Sprint Review. Review sessions should focus on validating that committed features have been delivered completely, satisfying the acceptance criteria (for more info see Sashimi).

Trust in the team (and therefore, in the new paradigm) rely heavily on the quality of every bit of functionality released in a sustained rithm. Thus, product quality must be top priotity for the team.

Agile Ecosystems

As we have mentioned, Technology departments on many traditional corporations are not ready for a collaborative environment. To make thing worst, frequently responsiblity for product development is highly distributed: Requirements Manager, Architecture Manager, Sfotware Development Manager, Quality Manager, Project Office…

Sometime the only way to introduce agile in these environments is to create an agile ecosystem.

An agile ecosystem is a protected environment for developers and QA in which conditions for self-organization are set to allow organic growth processes.

A word of warning! An agile ecosystem’s goal is to implement itereative and incremental proceses in organizations that do not allow such models explicitly (as they operate in a closed scope or closed budget approach). This modus operandi may be the only one feasible at a given time but best results can only be obtained when the new paradigm is ambraced by the whole corporation.

QA in Agile Ecosystems

The concept of Agile ecosisystems is not part of any de facto standard or body of knowledge accepted by the community. The following suggestions are just our particular outlook on this subject as a result of several years helping companies embrace the agile paradigm:

• Planning. QA must contribute to cost/effort estimates providing feedback on Acceptance criteria for every user story. Besides, an explicit Definition of Done will be provided by QA for every user story planned for the next sprint.
• Continuous Integration. Sustained product integration along the sprint must be used for automation of funtional tests (whenever possible). QA will have access to testing environments and automation tools to ease continuous delivery on them.
• Tracking tools. Tracking toos and charts (burndown chart, kanban, etc.) will take into account validation as a new stage to evaluate progress. A user story will only be finished once it has made it from the validation stage to the closed status.

Kanban & QA in Greenhopper

By using tools such as Kanban boards and CFD the team must reflect on the optimum number of resources to reduce WIP

CFD in Greenhopper

• Sprint Review. Sprint review sessions must be aimed to analyzing the fulfillment of commitments. We can’t stress this enough; one of the main goals of an iterative and incremental process in to strengthen the reliability on the team (as well as its self-esteem) which means that any deviation from commitment must be dealt with in the retrospective session.
• Sprint retrospective. In the first sprints development and QA will hold retrospective sessions to analyze in a collaborative fashion the impediments that are stopping the team from increasing velocity as well as structre and process changes that may improve productivity and overall quality.
• Scrum Master. Agile ecosystems introduction must be driven by a great deal of common sense and, above all, by an expert facilitator that can pull the best results in a constrained environment. Degree of success (and time required to achieve it) depend heavily on the experience of this role.

In next entries we will deal with some other factors affecting the introduction of agile in conventional organizations. Keep tuned!

¿What is Ofbiz?

Apache Ofbiz (The Apache Open For Business Project) is a java framework by the Apache Foundation that provides corporate tools automation. It offers corporate tools such as: ERP, CRM, etc. So, what does this have to do with JIRA? Traditionally, JIRA has used OfBiz as a Database abstraction layer to access records as map of fields (as an instance of the class GenericValue). The main benefit of this approach is keeping forward compatibility even if the database schema changes.

The matching betweern logical model and database schema is performed by defining OfBiz Entities (tables) as a set of Fields (columns) with a strong data type. This model map is stored as a set of XML documents (for different DB servers) in a folder named entitydefs inside a JIRA installation directory. The file entitymodel.xml stores all entities available via Ofbiz. This is a little sample from this file:

    <entity entity-name="Issue" table-name="jiraissue" package-name="">
        <field name="id" type="numeric"/>

        <field name="key" col-name="pkey" type="long-varchar"/>

        <field name="project" type="numeric"/>

        <field name="reporter" type="long-varchar"/>
        <field name="assignee" type="long-varchar"/>

        <field name="type" col-name="issuetype" type="long-varchar"/>
        <field name="summary" type="long-varchar"/>
        <field name="description" type="extremely-long"/>
        <field name="environment" type="extremely-long"/>
        <field name="priority" type="long-varchar"/>
        <field name="resolution" type="long-varchar"/>
        <field name="status" col-name="issuestatus" type="long-varchar"/>
        <field name="created" type="date-time"/>
        <field name="updated" type="date-time"/>
        <field name="duedate" type="date-time"/>
        <field name="resolutiondate" type="date-time"/>
        ...
    </entity>

Ofbiz is a complex tool and it’s API is out of the scope of this article so we will focus on an interface used by JIRA when accessing data: Delegator. This interface provides all required methods to search for different criteria.

Accesessing OfBiz from JIRA

JIRA public API contains interfaces to access OfBiz from plugins. com.atlassian.jira.ofbiz.OfBizDelegator is a wrapper interface to hide OfBiz Delegator. In the following snippet you can see how to retrieve an object of this interface:

OfBizDelegator delegator = new DefaultOfBizDelegator(CoreFactory.getGenericDelegator());

This interface provides methods to explore entities and fileds from JIRA datamodel. Bear in mind that values for fileds are usually indexes from some other tables. For instance, in order to retireve all issues of type Bug (assuming the numerical value for Bug is 1) we can use this code:

final List<GenericValue> bugs = delegator.findByField("Issue", "type", (Integer)1);

Every GenericValue object returned by delegator provides a simple interface to get field values from name:

String key = bugs.get(0).getString("key");

OfBizDelegator interface include some other methods for creating complex queries:

• findByLike
• findByPrimaryKey
• findByCondition
• findByAnd
• findByOr

See JIRA API for more info on OfBizDelegator.

Conditional queries

So far we have created simple queries using OfBizDelegator but there is no way to use complex logic using these methods. In order to combine different fields and logical conditions we will need some other classes:

• org.ofbiz.core.entity.EntityCondition: Stores a logical condition consisting of 1 field, 1 operator (instanceof EntityOperator) and a value.
• org.ofbiz.core.entity.EntityExpr: Stores a logical expression between openrands or conditions (OR, AND, etc)
• org.ofbiz.core.util.UtilMisc: Helper class to assist in creating complex conditions.

This is a sample of how to use the method findByCondition from OfBizDelegator to get a list of GenericValues using conditional statements.

java.sql.Timestamp fromTime = new Timestamp(new java.util.Date().getTime() - 365*24*3600);
List conditions = UtilMisc.toList(
                                  new EntityExpr("created", EntityOperator.GREATER_THAN, fromTime),
                                  new EntityExpr("field", EntityOperator.LIKE, "status"),
                                  new EntityExpr("fieldtype", EntityOperator.LIKE, "jira"),
                                  new EntityExpr("newvalue", EntityOperator.LIKE, (Integer)5) // "Resolved"
                                 );
final EntityCondition condition = new EntityConditionList(conditions, EntityOperator.AND);

OfBizDelegator delegator = new DefaultOfBizDelegator(CoreFactory.getGenericDelegator());
final List<GenericValue> resolved = delegator.findByCondition("ChangeGroupChangeItemView", condition, EasyList.build("issue"));

 JQL WAS operator

From version 4.3 on, JIRA provides a new operator to search in the past using JQL: WAS. It is identical as operator IS except it also checks in the issue‘s histoy. For example, the following JQL sentence would return all issues ever assigned to the current user:

assignee was currentUser()

In the next section we will try to replicate this functionallity using a JQL function based on OfBiz.

¿IS the user the actual assignee?

Operator IS works as a subset of WAS but schema-wise, queries affect completely different entities. This way, the current assignee for a particular issue is stored in the Issue entity (shown previously).

final List<QueryLiteral> literals = new LinkedList<QueryLiteral>();
OfBizDelegator delegator = new DefaultOfBizDelegator(CoreFactory.getGenericDelegator());
final List<GenericValue> assignedToGVs = delegator.findByField("Issue", "assignee", userId);

for (GenericValue changeItem : assignedToGVs) {
    try {
        literals.add(new QueryLiteral(operand, changeItem.getLong("id")));
    }
    catch (NumberFormatException e) {
    }
}

Now, let’s look into the past…

However, in order to know if a certain user has ever been assigned to a particular issue we need to use another entity that stores previous changes for all issues: ChangeGroupChangeItemView (see the example for Conditional queries). In this case, the helper class EasyMap comes in handy to create a map using constructor arguments.

final List<QueryLiteral> literals = new LinkedList<QueryLiteral>();
Map<String, Object> paramsItem = EasyMap.build("field", field, "fieldtype", "jira", "newvalue", newValue);
final List<GenericValue> changeItemsForFieldGVs = delegator.findByLike("ChangeGroupChangeItemView", paramsItem);
for (GenericValue changeItem : changeItemsForFieldGVs) {
    try {
         literals.add(new QueryLiteral(operand, changeItem.getLong("issue")));
    }
    catch (NumberFormatException e) {
    }
}

The final test

We have already seen in previous posts how to write and bundle JIRA plugins as well as JQL functions so we will skip this step. The XML descriptor for the new JQL funcion will be as follows:

<jql-function key="issue-assigned-to-jql-function" name="Issue Ever Assigned To Function"
 system="true">
    <fname>everAssignedTo</fname>
    <list>true</list>
</jql-function>

So let’s deploy and test our JQL function everAssignedTo.

Let’s double check with operator WAS.

References

Besides JIRA and OfBiz API links shown in this post, there is detailed information on JIRA’s datamodel in this page.

As with the rest of entries of our blog, you can download the source code from the repository publicly available at bitbucket.org: https://bitbucket.org/novagenia/elmanifiesto.git.

Let us know what you think about this post.

¿What is JQL?

JQL (JIRA Query Language) is a query language for JIRA’s data model very similar to SQL that allows complex searching from version 4.0. A JQL query is made up of a logic combination (OR, AND) of conditions in the form field + operator + operands. The operands can be one or more values or a JQL function (returning one or a list of values). For instance:

project = "New office" AND status = "open"

This JQL query returns all issues in project “New office” in status “open”. The next exaple shows the use of a JQL function:

assignee != currentUser()

Which returns all issues assigned to a user different from the one executing the query, returned by the JQL function currentUser().

¿Can I extend the JQL sintax?

It is not possible to create new operators. JQL can use any custom field, though. And we can create new JQL functions that provide complex operands that cannot be expressed with the built-in logic. In the next lines we will try to show you how to create a JQL function to display all blocked issues (identified in JIRA by the ‘blocks‘ issue linking type).

JQL Functions in a plugin

In order to use our JQL function in JIRA we will create a plugin as described in our previuos entry: Atlassian plugins with eclipse & maven.

1. Create base class BlockedIssuesJqlFunction.java

Every JQL function must implement the interface JqlFunction provided by JIRA’s API. For the sake of simplicity we will use an abstract class that provides common functionality: AbstractJqlFunction. The following methods are abstract and needs to be implemented:

• getMinimumNumberOfExpectedArguments(): returns the number of arguments our JQL function needs, in out example this is 0.
  
• getDataType(): returns the function’s return type, for issues this should be JiraDataTypes.ISSUE.
• validate(…): validates arguments passed to our JQL function.
• getValues(): returns a list of values as objects of class QueryLiteral.

So now we create the class BlockedIssuesJqlFunction.java to extend AbstractJqlFunction:

public class BlockedIssuesJqlFunction extends AbstractJqlFunction {

    @Override
    public JiraDataType getDataType() {
        return JiraDataTypes.ISSUE;
    }

    @Override
    public int getMinimumNumberOfExpectedArguments() {
        return 0;
    }

    @Override
    public List<QueryLiteral> getValues(
        QueryCreationContext queryCreationContext, FunctionOperand operand,
	TerminalClause terminalClause) {
	// TODO Auto-generated method stub
	return null;
    }

    @Override
    public MessageSet validate(User searcher, FunctionOperand operand,
	TerminalClause terminalClause) {
	// TODO Auto-generated method stub
	return null;
    }
}

 2. Now Implement validate(…)

In the method validate(…) we need to check arguments passed to our Jql function, generate error messages (using an object of class MessageSet) if needed and store the values of the arguments to parameterize our search. In our example we will check that the following conditions apply:

1. JIRA’s issue linking is enabled.
2. There is at least one issue using ‘is blocked by’ linking type. We will use object of class IssueLinkTypeManager and IssueLinkManager provided during construction.

public class BlockedIssuesJqlFunction extends AbstractJqlFunction {

    private final IssueLinkTypeManager issueLinkTypeManager;
    private final IssueLinkManager issueLinkManager;
    private final static String LINK_NAME = "is blocked by";

    public BlockedIssuesJqlFunction(IssueLinkManager issueLinkManager,
    		IssueLinkTypeManager issueLinkTypeManager) {
		super();
		this.issueLinkTypeManager = issueLinkTypeManager;
		this.issueLinkManager = issueLinkManager;
	}

	@Override
	public MessageSet validate(User searcher, FunctionOperand operand,
			TerminalClause terminalClause) {
        MessageSet messageSet = new MessageSetImpl();
        if (!issueLinkManager.isLinkingEnabled())
        {
            messageSet.addErrorMessage("Invalid argument for " + getFunctionName());
            return messageSet;
        }

        final List<String> args = operand.getArgs();
        if (args.isEmpty())
        {
            messageSet.addErrorMessage("Invalid argument for " + getFunctionName());
            return messageSet;
        }

        return messageSet;
    }
...
}

 3. getValues(…): Return candidates as literals

Using a similar code we will retrieve all issues with a linking of type “is blocked by”. Finally, convert issue objects into literals (objects of class QueryLiteral).

@Override
public List<QueryLiteral> getValues(
        QueryCreationContext queryCreationContext, FunctionOperand operand,
	TerminalClause terminalClause) {
    notNull("queryCreationContext", queryCreationContext);
    final List<QueryLiteral> literals = new LinkedList<QueryLiteral>();

    Collection<IssueLinkType> linkTypes = issueLinkTypeManager.getIssueLinkTypesByOutwardDescription(LINK_NAME);
    if (
         null == linkTypes ||
         linkTypes.isEmpty() ||
         null == issueLinkManager.getIssueLinks(linkTypes.iterator().next().getId())
       ) {
           return null;
    }

    Set<Issue> linkedIssues = new LinkedHashSet<Issue>();
    for (IssueLinkType linkType : linkTypes)
    {
        List<Issue> issues = new LinkedList<Issue>();
        Collection<IssueLink> links = issueLinkManager.getIssueLinks(linkType.getId());
        for (IssueLink link: links) {
            issues.add(link.getSourceObject());
        }

        if (issues != null)
        {
            linkedIssues.addAll(issues);
        }
    }
    for (Issue issue : linkedIssues)
    {
        literals.add(new QueryLiteral(operand, issue.getId()));
    }

    return literals;
}

4. Plugin descriptor

In order for our class to be available in JIRA we must package it in a JIRA plugin with the appropiate OSGi descriptor. The following lines should be included in the file src/main/resources/atlassian-plugin.xml.

<jql-function key="blocked-issues" name="Blocked Issues Function"
  class="com.novagenia.jira.plugins.jql.function.BlockedIssuesJqlFunction">
  <!--The name of the function-->
  <fname>blockedIssues</fname>
  <description>Provides a JQL function to returned issues with is blocked by link.</description>
  <!--Whether this function returns a list or a single value-->
  <list>true</list>
</jql-function>

Now you can deploy the plugin in JIRA and use the JQL function as follows:

issue in blockedIssues()

And you should see some like…

The final touch

Some suggestions to enhance your code.

1. Add log traces

There are several ways to debug JIRA plugins but you should, at least, provide runtime log traces to easy debugging and maintenance. By adding the right entries in the file log4j.properties we can adjust the verbose level of our our plugins from JIRA administration interface.

This is how we create an object to use logs from code:

private final Logger log = Logger.getLogger(BlockedIssuesJqlFunction.class);

Calls to this object should specify the severity of the trace:

log.warn("Linking is not enabled!");

2. Refactor to use more linking types

With little effort we can extend the functionality in our previous example to make it reusable. By using eclipse’s built-in refactoring tecnique “Extract Superclass” we will create a superclass with the linking type as a parameter.

Now we add the new JQL function in the plugin descriptor:

<jql-function key="linked-issues" name="Linked Issues With Function">
    <!--The name of the function-->
    <fname>linkedIssuesWith</fname>
    <description>Provides a JQL function to get all issues with an outward link.</description>
    <!--Whether this function returns a list or a single value-->
    <list>true</list>
</jql-function>

3. Keep it private (sanitize)

To end with, we would like to talk about a mechanism (sanitising) that ensures that parameterized JQL functions keep information private. Although not mandatory, it is highly convenient if your JQL function is accesible from different projects. The goal is to keep all information regarding fields and values private. This is important when using a JQL function is filters that might be shared with people in different projects and, probably, with different security or visibility levels. If you think this is the case with your Jql fucntion then you may need to implement the interface ClauseSanitisingJqlFunction.

You will find more details in the following link.

Share

The code included in this entry (as well as examples for all entries in The manifest), can be accessed from bitbucket.org: http://git.novagenia.com.

Any comment that helps us improve would be hilghly appreciated.

Best regards.

1. Get ready for Atlassian SDK.

Atlassian provides an SDK for plugins (and other artifacts as well ) creation that simplifies the setup of a complex environment. Platform requirements: Java Development Kit (JDK) version 6 you can download here.

The Atlassian SDK comes with a Maven instalation with extra tools that helps jumpstarting a plugin. Download the latest SDK from this location and uncompress on a local path without spaces. Finally add the Maven’s bin folder to your path.

Then we will install the IDE, making sure we user eclipse Helios for Java development. Important! The latest eclipse version (when this entry was written) is Indigo which is incompatible with Atlassian SDK Atlassian and you won’t be able to use eclipse pluginj for Maven: M2Eclipse. Eclipse Helios can be downloaded from this link. Again, avoid spaces in eclipse installation path.

2. Configure eclipse

Eclipse plugin for Maven won’t run on a JRE so you need to make sure it starts with a Java Development Kit (JDK). In order to avoid trouble  you can modify eclipse configuraion file eclipse.ini and specify the path to the installed JDK:

-vm
C:/Program Files/Java/jdk1.6.0_29/bin

Eclipse Helios does not came bundeled with M2Eclipse plugin so we will add it manually. This way we can resort to a plugin version compatible with Atlassian SDK: http://m2eclipse.sonatype.org/installing-m2eclipse.html. Add the Update repository and select Maven Integration for Eclipse for installation.

Sonatype’s plugin version includes an embedded Maven installation that must be replaced for the one bundled in the Atlassian SDK. In the plugin configuration Window->Preferences->Maven->Installations we create a new entry for the Maven installation.

To finish eclipse configuration we set the maven’s reposotory path as an environment variable. We can do that invoking the following command in a terminal:

atlas-mvn eclipse:add-maven-repo "-Declipse.workspace="

On windows, it is important to use double quotes in order for the workspace pth to be resolved correctly. If the command executed successfully you should see some like:

By executing this command a new variable path is created in eclipse with the name M2_REPO that refers to the repository folder.

3. Create your first plugin

We are now ready to start writing plugins. There are several alternatives depending on the affected product and the kind of artifact you are building. We will use the SDK tools for the sake of simplicity.

Using the command-line on a terminal, Execute the following command in the workspace path:

atlas-create-jira-plugin

During the command execution you’ll be prompted for JIRA’s version, GroupId, ArtifactId, plugin version, etc. The expected output is something like:

This will create a Maven project and the pom.xml file. Now we create an eclipse project skeleton by running the following command inside the plugin’s folder:

atlas-mvn eclipse:eclipse

This command will update current dependencies depending on the product and version. Now our plugin is ready to be imported into eclipse either as an eclipse or as a Maven project.

E voilà! The project is ready to generate our first plugin.

4. Build, depoly and test

This project will build automatically. In order to create a plugin package we will resort to the maven tools. Select the project in eclipse and click on Run As->Maven package.

Upon successfull run a  test-plugin-1.0-SNAPSHOT.jar file will be created in the target subdirectory. Now it is time to deploy.

Install JIRA if you haven’t already. You can download the latest version installer from here. You can acquire a comercial license for just 10$ …

Log in JIRA with administrator privileges click on Administration->Plugins. On the Install tab, click on Upload plugin and select the jar file we just created from eclipse.

Once the process finishes, the plugin is loaded and ready to play. Improve it, extend funcionality, change the plugin version and generate a new jar file as explained. You can deploy newer versions of the plugin without the hassle or restarting JIRA.

For simple debugging add your plugin’s groupid to the log4j.properties file and restart. From the administration page you can further customize the logging levels for every entry in the file.

log4j.logger.com.novagenia = DEBUG, filelog
log4j.additivity.com.novagenia = false

6. Explore new possibilities

Explore the SDK to get a glimpse of all you can do by creating plugins. There is a site devoted to developers regarding Atlassian products: http://developer.atlassian.com.

When your plugin is production-ready, register it in the Atlassian plugin repository and share with the community: http://plugins.atlassian.com.

I hope this was helpful. Tell us what you think..

Regards.

Novagenia  is a spanish company specialized in  Agile Methods and Tools applied, mainly, but not exclusively, to creating software solutions.

From a personal perspective,my experience on agile methods dates to ancient history, back to 1999, when I stumbled upon an interesting refection on new ways of facing software development methods. The author was a Mr Kent Beck and it was published in the first wiki site (created by Ward Cuningham who, besides, came up with the name wiki from a Hawaian word for  fast although that is another story…) You can still access a modern version of the site in this link. Once I was test infected there was no cure; ¡it was impossible to accept waterfall paradigm dogma!

Some time later, after co-founding Novagenia in 2004, I decided that besides using the best software develoment process, I needed the best tools available in the market. I tested dozens of applications for months (Saas computing was not in fashion at the time) from open source solutions to comercial tools our budget could buy. Until I found the tool suite that I have been using since: JIRA + Confluence. For those of you who still haven’t used them, there are important open source projects using these tools: Apache, JBoss, Spring, Alfresco, etc. Then in 2006 Novagenia started providing consulting services on Agile Methods and we became the first Atlassian partner in Spain.

In the next articles on The Manifest we’ll try and share our take on agilism from a pragmatical outlook, writing about experiences and lessons learned during these exciting years providing our small contribution to the community. I hope you will share with us the journy that starts today…

Eduardo Mayor