Carl Chilley - Executive Consultant, Xansa UK Ltd

Of the many movements and drivers that have emerged over the last few years in the computing space, there are two that have affected my life in more ways than others.

The first is the burgeoning use and impact of Open Source code and deliverables, free stuff that is well crafted and well priced - like free!

The second is the so-called "Agile" movement. I have to be honest here and say that when I first heard about eXtreme Programming, Scrum and other approaches that effectively empowered the programmer to work directly with the user I was more than a little sceptical. Like Rapid Application Development (RAD) before it, Agile felt like this was yet another way to deliver legacy systems in weeks rather than years. Where was the discipline? Where were the hoards of folk needed to architect, design and govern the process? Where was the process?

However, I discovered that Agile did promote and deliver on something that I had felt was missing in the computing space for a long time: the socialisation of the problem and solution with key players in the game. Open source also socialises the problems and solutions as well, and the ability to get to grips with the nub of the issue with those directly affected by it just has to be right, doesn't it?

In the area I tend to occupy there are a lot of "food chains" involved in the decision making process on both the producer and consumer sides of the "space", as is the way with large and complex systems. Food chains have to be fed, people have to feel as though they are involved and have made contributions that have been acknowledged by the "great and the good", giving rise to a degree of bureaucracy and "administrivia" that is sometimes overwhelming. And this is where another other ideal espoused by the Agile movement that I really agree with, the notion of "just enough", comes into its own.

So, here we have a powerful combination of "socialisation" and "just enough", principles that, when used judiciously, increase the effectiveness and efficiency of the software delivery process. Seventh heaven, right?

Well, not quite.

Call me naïve but I had assumed that the practitioners of XP et al would be applying the principles end to end, from understanding the problem to be solved and the context in which it has to fit right through to delivered systems that were fit for purpose. But it appears that a lot of the principles are applied in the context of developing and testing code and do not appear to address the larger concerns of architecture and governance, the overall context in which the solution has to sit.

Now I would be the first to agree with the importance of code in the delivery of solutions involving computers: no code, no program. However, code is just one of many elements that, taken together, define the solution proffered to the consumer, even though the diversity of the kinds of code required is large in and of itself. It is almost as though some of the proponents of Agile have decided that by empowering the carpenter and the brick layer to work with the home owner that this is sufficient to deliver the "dream home", albeit over several iterations.

It is true that for certain projects and problems their scope does not demand much more than the experience of the brick layer and carpenter artisan equivalents in the computer world. Such projects typically have a well-defined and bounded scope and easy and clear accessibility to the consumers of the results of the process. I am thinking here of the equivalent to the delivery of out-buildings and extensions to existing dwellings.

But what about the delivery of the dream house itself? Or the street of dream houses? Or the neighbourhood, the suburb, the residential conurbation? How do you involve the real-life food chains that exist on both the producer and consumer side that have to be satisfied in order for anything to happen?

This is the world that I am often called in to develop solutions for, the larger-scale projects and programmes that are inherently more complex in the scale of the problem that they are addressing, that are not "green field" systems and where rules and regulations apply in droves. Politics is an essential element of this landscape and the need to be able to satisfy often conflicting demands is part of the day job. There are stakeholders aplenty to satisfy on both sides of the "divide", those stakeholders often changing over the life of some of the larger programmes and, more often than not, changing their needs and requirements over time.

This politicised arena is not one in which the carpenter and bricklayer are ideally suited to or best employed to deal with.

So here you see my dilemma: how do I apply some of the basic and useful tenets of the Agile movement to these large and often complex programmes without demanding that such programmes openly embrace the totality of the Agile creed?

The good news for me is that there is a lot that we can and should learn from the Agile ideals. I mentioned socialisation earlier. Traditionally, stakeholder management was often caught between a rock and a hard place because of the often conflicting needs that were presented by the various stakeholders. The tensions are often palpable especially with "strategic" systems that will change the way the organisation operates, the tensions occurring not just between consumer and supplier (or suppliers as is more often the case) but also within the consumer and supplier teams themselves.

Misinformation plays a key role in a lot of these tensions, especially when there is a feeling that "knowledge is power". Actively promoting the true understanding of the problem and its ramifications is critical in all areas: no point in an executive seeing one problem while the users see a totally different one. Worse still, no point in the architects and business analysts seeing a neat solution when the programmers and testers see nothing but chaos!

These inter- and intra-team tensions are a fact of life on large programmes but can be mitigated with good communication and good sponsorship on all sides. Moreover, there is a need to ensure that everyone is aligned to the same problem and solution vision while accepting the inevitable degree of change that occurs over any project with a half life of more than one day: change is the only constant here. The horizontal communication between the consumer and supplier must be augmented by clear and concise communication within the respective teams.

And here is the rub: we not only know about all of these things but have been using techniques and approaches - albeit on a small scale - to mitigate them.

I have to be honest and state that I was very anti the whole XP thing because of its emphasis on the artisans, the empowering of the brick layers and carpenters at the expense of the overall integrity of the solution space. But XP, Scrum and all of the other approaches in this arena work - at some level of scale. However, scaling the approach delivers the very antithesis of Agile in that bigger teams destroy the social fabric which makes Agile so powerful. Equally, with more work being undertaken in an off-shore environment, the powerful notion of working alongside the users becomes more problematic.

For me, the two killer principles of Agile - socialisation and "just enough" - provide a "mantra" that can be applied elsewhere in large-scale projects and programmes. For sure, there is a need for some degree of structure and control - but "just enough". I firmly believe in the need for understanding requirements and for requirements gathering - but only enough to kick-start the process, accepting that the requirements themselves will change as more understanding of the problem space unfolds and the impact of the proposed solutions becomes more apparent.

On large-scale programmes you need architecture - the blueprint for the overall solution context - but "just enough" to ensure good communication and understanding with the consumers and the supplier workforce. Architectural decisions constrain choices further down the delivery food chain so we best make sure that those affected by them can live with them and influence them.

But more than all of this you need to ensure that everyone working on the programme is in tune with each other, regardless of which side of the consumer/producer space they sit. Socialising the problem and the solution, understanding the ramifications of the decisions made and being able to say "Hang on! This is not the best way to do this!" should be part of any programmes social fabric.

The Agile movement has shown us how to do a lot of things much better. Now we need to take some or all of those principles and apply them to large-scale problems without losing the overall impact of the approach. We not only need to empower the bricklayers and carpenters but also everyone else involved in the game, and that may prove to be an interesting challenge as empowerment can also lead to more open dialogue, something we have not always lived well with!