Test Utopia
The QA Revolution is here

If you want to automate these days you have two choices: you can do it for free and go with open source tools or you can pay the GDP of a small country and fill the coffers of some big evil corporation. Recently I’ve been looking at a tool that is potentially a middle ground (plus the corporation whose coffers you fill isn’t really evil - more well-intentioned and known for innovative ideas).

One of the biggest challenges we face in the open source automation tool arena is the lack of any good test case management. Part of the reason Mercury/HP has been able to gain the lion’s share of the automation market is because they have been able to sell companies the concept of an entire system of tracking and managing requirements that can be traced through to defects. This is an area where open source tools have simply not caught up. When I walk into an organization that is struggling to convert from expensive bloated mercury-based automation to open source automation, the biggest barrier to entry that I hear repeatedly from QA professionals is that there is no existing open source automation tool that provides an easy means for managing and tracking test cases and tying them directly to requirements (Some of you will say Fit and Fitnesse is the tool of choice for that, but the pros and cons of that option is another blog post). Until now, the general philosophy was that applying the kind of rigid control that test case management requires to open source automation would kill the “agile” spirit of the tools.

I’ve recently been evaluating TWIST a new testing IDE built by Thoughtworks using eclipse as the underlying platform. To be clear, this is not an open source tool. It is a commercial tool but will be selling at affordable prices accessible to small businesses etc. I have to admit, at first I was extremely skeptical about using an IDE paradigm for a new automated test tool, but I’ve been looking at the beta version of the product for the past few days and I’m convinced of the power of this tool to help solve some of the biggest hurdles we face with TCM in open source tools. This tool leverages the power of domain specific languages in a manner I’ve never seen before and the potential of this is staggering. Mark my words when I say this is the way of the future. Okay, so in the interest of full disclosure I should mention that I’m currently an employee of Thoughtworks. But anyone who knows me would tell you that I will not hesitate to openly criticize a bone-headed move by Thoughtworks (and believe me there are plenty of those to choose from). I’m a big believer in free expression and open discussion. Perhaps to my detriment, fear of losing my job (or alternatively benefitting in my job) is not really a motivating factor in my blogging decisions J

So here’s the story of my journey from skeptic to fan of TWIST:

• I saw a super early demo of it at a Thoughtworks away day in Denver last year. I wasn’t terribly impressed. It just seemed like we were using eclipse to write test scripts. I can use notepad or scite or any of a thousand existing applications for that. There was nothing extraordinary about what I saw.
• Recently, I began consulting with a client where my role is to help them identify support structures and practices that will help them put sustainable test automation in place across the organization. The client has low expectations of my success rate because every automation project they’ve attempted has either been abandoned or ended in failure. I talk a lot about the idea of developing a domain specific language as that seems to be the ideal solution for this specific client. They have no idea what I’m talking about and nod glassy-eyed while I extol the virtues of domain specific languages. I realize I just need to show them. I start writing Ruby scripts.
• Coincidentally Thoughtworks sends out an announcement that TWIST is available for internal review. I request a license.
• Installing was quick and painless, although without the twist license you just have eclipse and nothing else
• Finding the TWIST tutorial was a little awkward but I managed to find it within the first few minutes of running the tool
• Before going through the tutorial I decided to jump right in and run the hello world sample script. The script opens the browser (using selenium), goes to the google homepage, types in “Thoughtworks” and gets the results. It works like a charm. So far so good.

• Then I decide to examine the script. Here’s the snippet of code:

At Google: search for “Thoughtworks”

Really? I change the word “Thoughtworks” to “Yellow” and re-run the script. It goes to google, types the world “Yellow” and returns the results.

Wow! As I go through the rest of the tutorial and start experimenting on my own I start to get really excited because TWIST uses the power of an IDE to create a tool that will allow BA’s/QA’s to write their acceptance criteria and tests in plain English and then they have the means to convert those scripts to automated tests on the spot by either selecting from a pre-existing library of automated functions or writing a new one. In this way every script is driven by the requirements and needs of the business and is permanently associated with a living record of those requirements (which can of course be easily modified along with the code it describes). An IDE is really the perfect choice for this because it’s a mature and well-adopted tool for managing code. It makes reuse and refactoring of scripts as easy as a few key strokes or the click of a mouse. As for test case management, since you’re using an IDE you can connect your scripts directly to the code they test and as you build them out, the documentation for the automated scripts is built into the domain language so code will be delivered with the written requirements, acceptance tests, and automated system tests all built in. At least that’s potentially what could happen. Ultimately, like everything else out there, this is just a tool that makes it easier to be more effective in test automation. Whether it is used that way depends on the person wielding it.

In the immediate future, though, this is exactly what I need to demonstrate the potential of domain-specific languages to my client. Now I’m working on a kick-ass demo using twist to build them a mini DSL so they can see what I’ve been talking about. Thoughtworks Studios has made my job worlds easier.

A few years ago I fell in love with an Indian national. As the relationship progressed we both eventually realized that in order to be together we were going to have rearrange our lives. I started to do research on what we needed to do to live together in the United States. Unfortunately, within a few hours a growing awareness of the hopelessness of our situation began to sink in.

We have no rights under archaic US federal immigration laws as a same sex couple. I cannot bring my partner to live with me in the United States, and since she’s an Indian national, we of course can’t live in her home country either. This means our lives together are endless visa applications and strategic moves to find ways to stay together. Thank goodness we both have degrees and some skills and experience in the IT profession because if we didn’t, we wouldnt have any options at all.

Although this deviates quite a bit from the theme of this blog, I’ve chosen to tell this story here because I’ve learned that many people are unaware of this social injustice and I’m taking shameless advantage of the public sphere available to me here (forgive me if you came here looking for ways to write intelligent automated tests in ruby..we’ll get back to that later). When I tell people about our story, most people express surprise and disbelief that this is happening in the United States in modern times. This is a country founded on the principles of tolerance and open-mindedness. How ironic, that my family came to the United States two hundred-odd years ago fleeing persecution for their religious beliefs. But now, I have had to live in England for the past year because my partner and I are being persecuted by those same religions that were granted clemency and freedom from persecution for their beliefs and lifestyles by the constitutional edicts that gave birth to our country.

Last year, I gave up my job, my home, my car and proximity to my loving family in order to be with the person I love. Over the holidays when I brought my partner with me to the US to visit my family, I stood anxiously on the other side of the immigration line worrying that they wouldn’t let her through as they questioned her sternly,
took her finger prints and admonished her for failing to give them an exit card from her last visit. I wanted to run back through the immigration line and yell
at the officer that this woman is with me. She’s my family, my partner, my life and how dare you treat her like a second class citizen. But all I could do was wait helplessly and hope they let her through.

I’m not alone in this. There are hundreds of thousands of same-sex couples who are displaced and like my partner and I, without a country to call home because of these cruel and outdated laws. The United States is increasingly alone amongst nations in the western hemisphere with regards to this issue and I believe we, of all nations, should be champions for human rights, not perpetrators of abuse.

If you want to help bring about change,the good news is: you can. I’m writing this here because I believe that most people are not aware of this issue, and I have to believe that if they were, the educated and compassionate people of my home country would not approve of this treatment of any human being. If you want to know more please read other people’s stories and if you want to take more definitive action, please write to your local congress rep and express your support for the Uniting American Families Act (UAFA).

Next week: back to regularly scheduled programming (i might even post a script or two)

I once kept a running statistical tally of the test cases I was executing, passing and failing. It consisted of a total number of test cases, and then a ratio of executed, passed, failed which was then converted to a percentage. Sounds pretty simple eh? Senior management absolutely loved it because it provided the illusion of visibility to them. At the time,however, I considered the whole exercise a miserable failure. Because the numbers shifted so wildly, it became apparent after less than a week that we couldn’t get a feel for the actual progress being made in testing with those measurements. One day we were 75% passing and the next day we’d regressed to 50% and then maybe back up the following day. Also, because the total number of test cases kept shifting upwards, our total percentage of cases executed kept changing downward.
Looking back I realize two things: firstly, the wildly fluctuating numbers were a marker of big problems with the development process, quality of code, ill-defined requirements, etc and not problems in the testing process (which is, sadly, how they were interpreted by senior management). Secondly, the exercise in and of itself uncovered an interesting phenomenon.

In addition to the pre-defined core test cases we tracked I had directed my staff to write a new test case for every defect they found, if that case was previously non-existent. This is standard practice in any testing process and even in test-driven development unit tests or acceptance tests should arise from serious defects. But where do those cases come from?

When I bothered to document these cases which I call unspoken test cases, our library of test cases more than quadrupled. A technocrat may look at that statement and say, well that’s because you didn’t define the test cases well enough in the first place and it’s a failure of the development team. I admit, this was not a stellar development team and there were definite issues with quality. However, I believe that if you actually documented every test case that was executed even on really solid high-performing teams, you’d see a similar phenomenon. I can’t really prove that theory, though, because not only do we not have time to document every case run, but because of the intuitive nature of testing, we would still miss some of the cases that were run totally unconsciously. These unspoken cases that arise seemingly out of nowhere, are critical to systems and development testing because they define the human factor. We may execute them based on a conversation we had with the business, or a vague feeling,  or maybe just assumptions about how the program should behave.

They add an element of life and soul to the final product that no machination or automation could replicate. They shouldn’t be interpreted as a failure on the part of developers, but more as a realization of the value that a good test team provides. I firmly believe that measurements and documentation are merely incidental tools and not the life-blood of a good project. But in this case, if I’d been wiser, I could have interpreted my measurements more holistically and offered sound advice for organizational change to senior management and perhaps reassured them in some ways.

I work for a consultancy which sells software development services to firms around the world and it has recently come to my attention that our sales team is afraid to sell QA as a standalone service and in fact they are reticent about even selling it as part of a full-service package. “Analysis”, they say, is always a hard thing to sell. I’m issuing an open challenge to the sales people in my company who truly believe that you can’t sell QA or testing services. If you’re not selling QA, there’s a very simple reason: You, yourself do not believe that QA provides value. I know what you’re going to say, because I’ve heard it all before. I understand perfectly the value that QA provides, it’s just that clients aren’t willing to pay for that. Well, Mr and Ms. Sales Person are you a sales person or are you a mouse? You can’t just accept the first no you get. I thought that was one of the tenets of selling. People don’t always know what they need and part of your job is to help them figure that out. The only way you can do that is if you believe in what you’re selling. So, to gird your loins a bit, I’m listing twenty things you should be able to package up and sell as a consultancy service that your local QA professional can and would love to do. Note that none of these involve writing and executing test plans and test cases. That’s for the birds. If you still feel this stuff isn’t sellable, then I have a proposition for you. Let me help you sell it. I want to sit in on your sales calls. I want to go with you when you go to the client. I’ll observe at first and then I’ll offer you what I think are ways you can sell this, and I’ll give you honest feedback on why you tried to sell it and failed. If you ignore this, trust me when I say, you’re missing out on a fantastic source of revenue. I might just go into business selling this myself.

1. Strategic analysis
2. Systemic analysis - identify and analyze interdepencies in a system
3. Project plan risk Identification and recommendations
4. measure testability in development tasks and requirements
5. define risks and resolutions for defects
6. test data design and implementation
7. manage deployment and maintenance of internal test environments
8. manage deployment and maintenance of User Acceptance Testing environments
9. manage User Acceptance Testing feedback cycle
10. manage release process and expectations
11. Analyze automation requirements
12. Implement and deploy automation solutions
13. Identify strategic and technical solutions for legacy code that is plagued with regression issues
14. Analyze performance testing requirements
15. Implement and deploy performance testing solutions
16. identify risks in scoping and sizing of work being done
17. identify logical inconsistencies, usability issues and gaps in designs and requirements
18. Identify testing process inefficiencies and propose solutions
19. Knowledge transfer of system functionality to users
20. Troubleshoot problems found in production and process issues causing production problems

Most organizations assume that test strategy should be defined by test managers, tech leads or executives. Perhaps because anything involving the word ’strategy’ is open to interpretation giving it a mystique-laden abstract quality, therefore it is erroneously offered up to the intellectual elite (said tongue-in-cheek) i.e. upper management as a job function.

The chairman of Greyhound once said: ‘a strategist’s job is to see the company not as it is but as it can become’(JW Teets)

The truth is, most of the test strategy documents produced by said upper management, tech leads and the like are really just templated after-thoughts that regurgitate and rubber-stamp the currently accepted development process (hereinafter referred to as the status quo). They’re written to appease some organizational bureaucrat or to give the illusion of productivity from an administrative role but ultimately they are neither read, nor referred to nor in any way applied to the organization in a manner that justifies the half-day management salary it cost to format the headings and line up the dots in the table of contents. Clearly these type of ‘test strategy’ documents miss the whole point of the game.

So what type of test strategy would openly defy the institutional silver-back templates and provide real value in a fast-paced, roller-coaster web application agile environment? I’m not talking fifty page tome with boldfaced fonts and anchored sub-sections. I’m talking dog-eared worn-from-use concept that’s carried around lovingly by team members and incorporated into the tasks of daily development life.

What can this organization become and how can we get there?

If you think about it, the idea of a strategy is really pretty simple. It’s taught to us from the time we’re old enough to play checkers and we implement it every day whether we’re consciously aware of it or not. What makes a strategy seem complicated are the multiple levels of engagement. To make it less complicated, I break it down into three basic angles of approach:

1, identify risks and form a response
2. make efficient use of resources
3. affect change or realize goals

When it comes to a software development environment the test strategy should be an ongoing endeavor. It redefines itself iteratively throughout the life of the project. No wonder no one reads those test strategy documents that PMO’s mandate. They’re useless in the real world because the idea of regenerating a 4 page table of contents and reformatting all those headings in Word every time you iteratively change your strategy is not scalable if your strategy changes every day.

HINT: your strategy changes every day.

Think about it. In my little model above, the first approach to strategy is to identify and respond to risks (which incidentally is the only reason a testing role exists in any organization). Newsflash: risks don’t line up passively at the beginning of a project and wait around for you to pick them up, hug them and adopt them for your very own. They’re risks because they contain an element of the uknown. You can only identify them as they become visible. In other words, they change over the course of the project and they are about as predictable as a pit viper.

In case you weren’t paying attention to my first paragraph, what I’m getting at here is that any thinking person can formulate strategy. The problem with creating an over-arching test strategy is that there is an assumption that the strategic part of testing is taken care of and no further thought is given. There is no way a single document can encompass the dynamic nature of strategy. Nor is there any way a single person can control the shifting aspects of strategy for everyone in the organization. The best strategies arise from a shared vision and are implemented and challenged collaboratively by each member of the team.

I believe test strategy should be an organic coordinated effort that is produced by the testers themselves in concert with developers and designers. Managers obviously strategize too. Part of that strategy should be to give the team members the tools they need to strategize effectively in their daily tasks. This might involve a way to communicate risks to the team, a daily evaluation of resources and a willingness to respond to changing landscapes. Unfortunately there are so many large organizations out there where a testing function is divided into hierarchical levels where you have the strategizers who do nothing but write tests cases and PMO documents and you have test lackeys who go through and check off tasks given to them by the strategizers and they are never to deviate from the given set. Not only is this inefficient but, man, that test lackey job totally sucks. Testers are generally intelligent creative people, and it’s a total waste of resources not to give them the creative freedom and the business knowledge to take a well-thought out strategic approach to their work on a daily basis.

My advice if you’re a victim of test strategy documents: steal it and burn it. And if another one gets written. Steal that one and burn it too. And if your boss fires you for that, call me. I’ll hire you.

I was recently asked an intriguing question at a talk I gave for skills matter on automated testing. The question was simple: do you have to be a programmer to write automated tests? But I struggled a bit to answer the question because to me the answer is not as simple as yes or no. But I have a feeling that well-considered thought to the question is at the heart of what happens to the future of automated testing. Many career testers are feeling the pressure of picking up ‘automated testing’ skills in today’s job market because every IT company wants it. On the one hand, it’s not fair to put someone into the position of architecting and building or even maintaining an automated test system without having any education\understanding of the sound programming skills and practices that are necessary. On the other hand, why would anyone with good programming skills choose to enter into the thankless, dead-end, glass-domed, low-paying and under-respected testing career path? Let’s be honest, they wouldn’t unless they recognized it as an entry point into a much more lucrative development career path.

The latter is exactly the reason multi-billion dollar corporations have recognized that there’s a huge market in creating easy to use automated test tools that require special skills to operate but no programming is necessary to be up and running and fairly successful in a short period of time. I think mercury, silktest and IBM haven’t done us any favors though. They’re marketing these ‘record-and-playback’ tools (in a way that’s reminiscent of Sun Microsystems early Java marketing) as entry level tools and creating a misperception in the market-place that deep technical knowledge and understanding of software systems is not necessary for automated testing.

It sounds as if I’m leaning strongly towards a ‘yes you pretty much do have to be a programmer to write automated tests.’ doesn’t it? So this is where my brain starts to spin in circles as I contemplate the paradox we’re in. If you think about it, Automated testing is right now about where we were in 1995 with web application development. (and yes I am actually old enough to speak from experience about that). There was a tremendous amount of buzz around then as people started realizing the potential of the world wide web and anyone who knew a few lines of HTML got hired. But as the requirements got more complicated, so did the systems and the crappy beginner code that people were writing to get stuff out fast started to crumble. Now people are much smarter about hiring for the necessary skills. I think we’re starting to see a little of that backlash with automated testing, as a few progressive companies start thinking a bit more carefully about the skills that are needed to do it properly, but the industry hasn’t reached the moment of truth yet. Partly because, there truly aren’t enough people with expert-level experience to get us to that moment. So given all that, how do we solve the problem of creating the right balance between automated testing and strategic testing and how do we define the set of skills needed for the testing industry to move forward? Well, let me say that firstly that a few of those people who got hired because they knew a few lines of HTML turned into really excellent programmers. Correspondingly some of the automated test developers out there are also turning into really excellent programmers and are thinking about the right things when it comes to architecting an efficient automated test system.

I believe automated testing has absolutely tremendous potential and we haven’t even cracked the egg yet. There are so many frontiers to explore in this arena such as artificial intelligence, service-oriented testing, intelligent dimensional data analysis…the list goes on.

So how do we convince the bright, talented innovative thinkers in IT that automation can be a destination too, not just a path to another better job? I really don’t know any quick answers as the problem is a systemic one and touches on the mindset of the whole industry. My two big suggestions are to: 1. raise the bar in how we define the skillset needed to do this work and realize that an automated test developer is actually a programmer and 2. open up research and development projects that make automated testing a destination instead of a half-way house.

Recently I’ve had the opportunity to participate in a project that is experimenting with a lean software development environment and so have made a few observations about where QA fits into a lean methodology. Not being an expert in lean methodology, it’s possible (and likely) that the process my team is following isn’t strictly lean but some subjective interpretation of it. The central idea of lean is that you never do anything until the moment when it is needed and since there is no concept of iterations, schedules tend to unfold organically. Priorities for tasked out work are set daily and decisions about what gets done is made the day it is worked on. Like agile, documentation takes a back seat to conversations about design and requirements. This means that when decisions get made, we must rely on the frailty of human memory or a scribbled one-line note on a sticky as documentary evidence.

I’ve often said that the role of QA is to be the memory for the organization. We spend alot of time conversing with various stakeholders in various departments. We take these conversations and turn them into scenarios that form the basis of our criteria for testing. The result of all of this is that the QA engineer has a more holistic perspective of the end product. Our soul mates in the development process are the business analysts and architects of the system.

I was excited to get the opportunity to be a part of a lean project. I’ve been reading about it and I like the underlying concepts but mostly I wondered just how it would incorporate QA into the process. So far my conclusion is that it doesn’t. When I say “it doesn’t”, I don’t mean this in a “the developers are so good they test all their own work and deliver perfect software” kind of way. I mean, in my opinion the biggest weakness of the lean process we’re following is that in encourages piecemeal delivery to such a degree that those working on the project completely lose sight of the big picture. The code is wonderfully modular, and the bits that get delivered to me for testing work very well in their own right, but in the end they don’t play well together. This team’s implementation of lean doesn’t encourage a collaborative environment but instead forces a handful of people into a role of systemic analysis. And as the project progresses the team becomes increasingly dependent on the few people who have a systemic understanding of the system to clarify requirements and tell them how it should work.

Can the Lean concept work with QA? One of our team members made the observation that the process we were following tended to create a much higher burden on QA because developers had no impetus or opportunity for identifying the interdependencies. A big advantage of Lean is that you get to a mature stage of development extremely quickly and this can be very useful in situations where an organization has been paralyzed in the requirements analysis stage (which was the case with the client we’re working with). The client has been very impressed with short turnaround time. We’ve been able to deliver a working system in about half the time it took our predecessors just to hash through the requirements process. I think for this particular client, it works very well because they know they will change their minds about what they want at any stage of the process. It was less important for them to have a top quality system. That said, we found an unusually high number of major issues post-release. Part of the legacy of our ‘lean’ philosophy was that we had no automated acceptance tests and since the original testers and business analysts are no longer on the project, we lost some of the original understanding (or memory) of how the system was supposed to behave. This problem was exacerbated by a lack of documentation. Even the stories we developed from were simply keywords with no acceptance criteria and no elaboration as to what those keywords meant. As I tested the requirements, I spent alot of time talking to developers about what they had built, talking to the business about what they had asked for and trying to manage the large gaps between the two.

So back to the original question. Can a lean environment support a QA or testing role? I believe it can and I believe the problems with this lean project are rooted in the interpretation of the philosophy. I don’t think lean means: don’t do anything that takes time. And that was the tendency here. I think the underlying philosophy is fantastic. Basically the idea is to be smart about what you spend your time on and when. In our case, the lean philosophy was more like bare bones. We accomplished a great deal but in the end I believe it was a skeleton of what we needed. Spending a little more time in analysis and offering the developers more of an opportunity to understand the context of their work could have given us a little more flesh on those bones.