Thanks for that Mike.  I like it when you filter the HBR!

Some people, wrongly, may think of agile teams as self-sufficient, like maybe a subsistence farm is.  Life on such a farm, although romantic to the western bourgeois, would actually be a nightmare.  There would be no light, medicines, starvation would be common, etc.

Agile teams actually are a function of specialization.  We have the building people, they are specialists.  The receptionists.  Then we have the IDE writers, the open source tools, we have you, specializing in knowledge and knowledge transfer, we have the computers we work on (each of which are built from components made my specialists).  The freedom to work in teams like we do is a side-effect of specialism.  Any self-sufficiently we may show is because the foundations are set already.  Are freedom is an illusion, as all freedoms in complex systems are.

I don’t think that it’s a question of agile been compatible with specialization, I think the former is a function of the latter.  The more specialisation there is, the more agile we will become - what a liberating idea.

I think working together as a team is the future. What we’ll have to learn as “specialits” is to work together as a globally dispersed team. The collaboration will not be defined by artifacts and handoffs but by remote communication. We have a long way to go, but I hope we will eventually get there.

Hi Jamie—
There are always great articles in HBR. It’s one of the monthly subscriptions that I really look forward to reading. It will be interesting to see how the growth of agile intersects with the trend toward hyperspecialization. My first reaction in reading the article was, “Uh-oh, this is very counter to what I like to do and do.” And then looking at the process on TopCoder, that was very specific, waterfall steps. It will likely be years before we see what the result is.

Hi David—
You could be right. Maybe what this collision of trends leads to is us learning even newer ways of working together.

Interesting post, Mike.

But I believe the trend toward hyper-specialization is mostly applied in the manufacturing arena.  That model has been tried in the software world—IMO it was one of the motivators behind the “software engineering” trend—and the consensus seems to be it was found wanting.  Further, it seems to me agile was/is a reaction to the heavyweight processes imposed by this mindset.

On the one hand, people lament that software development has not yet reached the level of an engineering discipline.  But on the other hand, people expect to be able to change their minds about how the software should behave.  This is in contrast to the creation of a pin or a gear or even an ASIC where the specs are tightly nailed down well in advance.

In some cases—e.g. controller firmware for medical devices—the specs for software may be very well defined and nailed down.  But for most software (i.e. your typical line-of-business application) the stakeholders seem to want to have some wiggle room built in to the process.  That makes developing software as much art as science.

Hi David—
Yes, a lot of this has been tried in software before but that doesn’t on its own mean it won’t succeed in the future. I’m not saying it will and I’m not hoping it will. But I can definitely see the appeal of it in a lot of cases. Reading that article and then reading about TopCoder and their massive waterfall process were enough to make me wonder about the future of agile. That’s the first time that’s happened.

A counter argument to “Hyper-specialization” is the “Total Football” concept (see Wikipedia).  That is, an outfield player on a soccer team can take over the role of any other outfield player.  This ranges from dribbling, passing, shooting, and so on so that given the right scenario and opportunity, the team can “out produce” and beat a team that is specialized – “Defender, Striker, Midfielder, etc.”.

I certainly don’t see specialization to the point of “code reviewer”, “unit-tester”, “duplicate an issue on my dev machine for me”, “refactor the change”, etc.

Good point, Kyle. You give some interesting examples at the end. I wonder what the world would be like if we did have those specialists. What if before I started coding something I could write a list of the unit tests I wanted and then send that list to someone to write for me. I could offshore it so when I come in tomorrow my tests are written (first). Or I could have the tests done in parallel to speed time to market. Or what about “Refactoring Specialist”? Suppose we’ve got lots of tests in place. But over the years some sloppiness has crept in so we send a chunk of code to Refactoring Specialists, Inc. They refactor the code. Send it back with some metrics that show the improvements. That should be feasible to outsource like that—no real domain knowledge is necessary if it’s just refactoring. And the tests are there for safety.

I don’t imply those as predictions but my initial reaction was “you’re right, we won’t hyperspecialize that far.” But then I realized we could—and that there could be some benefits to it.

Mike,

Thanks for another insightful post!

I think that Agile really outperforms when clarity and uncertainty are high. One of its primary benefits is that it quickly and efficiently brings to the surface any issues about which there is confusion or lack of clarity.

My guess is that hyper-specialization will outperform when most or all of the requirements of production are known and fixed. When there are gaps in knowledge, requirements or approach (as is the case with most software), Agile will tend to beat specialization. Just a guess, but would like to see studies in the future going forward.

There ARE some rare software projects where requirements are extremely well known (porting a system to a new platform comes to mind). It would be curious to see which approach outperforms in that case.

Michael

Hi Michael-
I’m glad you like the post. Yes, I’m sure there are some projects where we can lock things down enough in advance that this hyperspecialized approach could work. Similarly, I suspect that even on a project with high uncertainty, there are some parts we can lock down well enough. For example, we may be doing some highly innovative website but it includes a whole portion of things like generic user account management. Maybe that gets sent off to a variety of hyperspecialists while the novel part of the application is done by an agile team.

Great article and this is something I’ve been thinking about for some time as well. I think outsourcing is great where it makes sense for work outside of your core competency. However, there is definitely a limit to hyperspecialization and when it’s effective. I think you would really find the following article relevant here: http://www.startuplessonslearned.com/2011/09/power-of-small-batches.html

Thanks, John. I’ll check out the article.

Hi,

I am not confortable with the analogy done between manufacturing activities and IT industry, which leads to reduce developers to “simple” workers that could work in a production line (cf. Ford)...that’s a point that is totally missed with hyper-specialisation.

I like a new vision called “mentofacturing”, see http://www.mentofacturing.com/ (read pdf…and for those understanding french: http://bit.ly/pTuyM2) that is more suitable to our industry.
Particularly, the reasons why Adam Smith advises divison of labor was based on different parameters than those that caracterize work today.
I like also the vision of Craig Larman (first book o norganizational tools http://bit.ly/oFtSAM).

I don’t promote zero-specialization: I don’t want to install my workstation and servers supporting VMs, nor administrate network, nor install databases clusters…but hyper-specialization leads to an extrem where your are finally responsible of something so tiny, that you become responsible of nothing…and a land of everyone responsible of nothing, it becames a nightmare!
To illustrate, I give you an organisation like that involved for one application development :
- users (they don’t know what they want exactly),
- business analyst team (produce specs not related to something that could be done and not in phase with user “real” needs),
- level 1 support (don’t want to elimante “waste” their work are based on -> see http://bit.ly/pE8h8N),
- X project dev team (“just” write code for new features),
- a maintenance dev team (“just” write code for bug fixes, far more cheaper !)
- level 2 support (same bias as level 1)
- test team (will not try to automate tests -> same bias as support…)
- dev tools organization (as complex as previous one for each tool !) -> impossible to customize anything according specifities (standards & norms are the rules!)
- many teams for operations (often too specialized themselves also…)
...and I don’t see how you could have only one team with one member specialized on each function…how do you feed constantly each function with the same amount of work ? How do you do when one of the function is overloaded ? It seems to me completely counter-productive and against lean spirit.
I think generalizing specialists (http://bit.ly/9AOqRL) are a good way to benefit from existing specialities of some people and to leverage skills of others (http://en.wikipedia.org/wiki/Learning_organization).

The way we apply this in our team is : everyone is responsible from business need elicitation (with users directly, proxy of users - Marketing - exactly) to delivery for bugs and features, and supporting at level 2 (level 1 is done by). So each member work on business analysis, dev, part of tests (unit, integration & functional ones automation). Giving the problems encountered with dev tools organisation, we have our own operation (not so hard and secured at best we can). With this organisation, we are often slowed by some counter-productive activities (like support level 2, manual tests, regressions due to bad tests…), so we are motivated too reduce them to improve our velocity.

Clément

Hi Clément—
I’m not familiar with mentofacturing so I’ll check into that.

The world is clearly past an age when we do everything ourselves. Your example of installing your servers, etc. is a good one. There will always be a balance in the amount of specialization and the transaction costs involved. I, too, certainly have a preference for whole-team responsibility. But if we rewind 36 years to Fred Brooks’ The Mythical Man Month, we can see suggestions like from Harlan Mills of IBM saying that the best way to build a program is to pattern things on a surgical team—one chief surgeon and a supporting cast. The program should essentially all fit in one person’s brain. Our heads would explode at the size of today’s programs! So 36 years ago the idea of specializing even to the extent of programmer, DBA and tester would have been foreign. That’s quite normal today and a good Scrum team might have people in those roles even though each shares a whole-team responsibility and is a team member before whatever specialist they are. 

I’m starting to think that 50 or 100 years from now (and likely far less) we could be looking at divisions of labor that seem unfathomable now. Above there was the somewhat sarcastic possibility that we end up with specialists like “Unit Tester.” While that borders on the nutty in 2011, will it in 36 years? I’m honestly not sure. I think the key for those of us who see the tremendous benefits of teams and teamwork will be to find new ways of working together such that we get the benefits we’re seeing today with the likely greater specialization in the future.

Thanks for the reply.

Just to precise, I don’t do sarcasm with “Unit tester”, just talk about Tester team (rather functional tests, but heavily manual with no montivation to automate them since it would remove work on which team existence relies -> same thing for support…).

The problem is that very few people see hyper-specialization in combination with “a good Scrum team might have people in those roles even though each shares a whole-team responsibility and is a team member before whatever specialist they are”...it is rather lots of separated teams (called services) that interact through unmanageable workflows.
And by the way, if it was possible to specialize inside a single team, can you answer to these questions “how do you feed constantly each specialist with the same amount of work ? How do you do when one of the function is overloaded ?” ?

Hi Clément—
I wasn’t referring to you referencing “unit tester” as a new specialist. I was referencing Kyle mentioning that. He was saying he didn’t think we’d get that far. My initial reaction was “no, of course not.” But then I started to think, “could we…” and I really can see that happening.

As with so many of these posts, I am being misunderstood. I am not advocating that we all go out and hyperspecialize. I am saying this is 300 year trend. I’m not big on fighting 300 year trends. I’m one of the biggest advocates of agile you’ll ever find. So I read an article that says “here’s a trend” and realized that strong trend goes against what I believe and how I make my living. I started to react with “uh-oh, this isn’t good.” Now I’m trying to think through ways that the future may transpire such that agile’s emphasis on teamwork survives a centuries long trend toward greater specialization.

Mike,

This is a very interesting subject.  One of the keys for me when I set up software projects is to minimize dependencies external to the team.  I find that semi-autonomous teams are quite good at managing their dependencies themselves (“Can you get that new build to QA by 3pm today so I can finish functional testing by the end of the sprint?”), but things break down rapidly when you have inter-team dependencies.  It seems to me that assembling the right set of specialists who have the attitudes of generalists (“I intend to do whatever I have to do for the team to meet its commitments”) is the sweet spot.

-John M.

Hi John—
What you describe is exactly what I’m always after as well. A few well-chosen specialists can be a big boon to a team. But no matter how specialized someone is, I always want that person thinking, “What is the best thing I can do to help my team today?” and to be willing to work outside their specialty.

Hey Mike,
I was thinking along the same lines after recently attending your scrummaster course but there is a difference.
The traditional model of defined sequential roles and specialisation works well in a production environment where you cannot build z until component parts x and y are complete - we catagorically know what the end product is. I think it was the success of this that meant it was then applied to the creative process (what we know as waterfall) but here is fails as what we’re aiming for isn’t so rigidly defined. We are creating something and so what we’re making can and will change over time - and it is here the benefit of agile comes into its own as we can adapt.
I think you could (and should) apply agile to the R&D processes of something that is in production so it can improve over time but the actual production process I would argue the waterfall method does work well.

Hi Chris-
It’s good to hear from you. I think you’ve done a good job of describing the sweet spot for where agile works best. My hope is that sweet spot stays of a significant size rather than being reduced by this trend toward hyperspecialization.

Firstly I read your post and thought “bullocks” - software development is nothing like a pin assembly line. But then I got thinking (thanks for that!) and yes, developers are quite specialized. And if you’re building a system with heavy focus on - say - DB performance you’d better bring along a least one SQL specialist. That said, a developer specialist will often be able to contribute to other areas of the solution - he’s just better at this one thing.

The real difference between an assembly line and an agile team is then, in my view, the communication between and the responsibility shown by each team member. The best agile teams I have been part of, are the ones where every member helps solving every task. Even if it’s out of his/her knowledge the team member should still care for completing the task and reaching the goals of the sprint - and feel the pain if the goals are not met! If the team member only cares for his own (specialized) tasks, the team starts failing.

Come to think of it, this “caring for the complete product” is exactly one of the keystones in the Toyota Production System, where one guy can stop the entire assembly line if he thinks there is a critical error. 
TPS  fostered Lean, which in recent years has influenced agile thinking… And now I think I’ve come full circle!

Hi Jonas-
I think you’re right that the attitude of caring for the whole product is one of the things that makes agile (and lean) unique and so valuable. On traditional projects I used to hear architects say things like, “I don’t know if they’ll be able to implement the architecture as I’ve documented it but that’s the best architecture for this project.” (Huh? The best architecture is one that may not be implementable?) So whole-team caring is something that I suspect is missing with hyperspecialization (and one of many reasons I hope it’s not fully where we’re headed).

Hi Mike, thanks for the referencing of TopCoder in the above article. There are a few things I’d like to point out wrt the TopCoder model. Thanks for the opportunity to do so.

One quick point on terminology is the use of “bids” ... There are no bids inside TopCoder because no one individual is doing work at any set rate. Multiple individuals compete at all stages of every project - whether that be a front end wireframe contest or a Java component competition or an Assembly contest. Instead of bidding on work based on competency and hourly compensation, individuals self-select which aspects they want to compete on. They register, see in a fully transparent manner who else in the Community is also registering and then make the decision to do the work and submit. So no one individual is selected to do the work through bids prior to the work being done. The work gets done by those who want to do it and the best in breed takes the lion’s share. This eliminates any single point of failure because we will have multiple quality submissions.

Also worthy of noting here. Often in our Community you will have a specialist who is seeking to become more of a generalist. By selecting to compete in competitions where they perhaps know they will not win, they gain access to winning solutions - the thought process behind the solutions and the very active forums and discussions surrounding that particular competition. They also receive their peer review score-cards which means 3 TopCoder members who are reviewers on that piece of work didactically rate the submission and showcase where it could have been sharper or improved. Because of all of this, they view competing as the single best way to improve themselves in a specific proficiency. Knowledge is shared throughout and though the individual may not have won, they certainly got better and advance themselves as programmers, specialists and generalists. 

To your point about these individuals not working as a team. I would welcome a discussion to showcase the role of the project co-pilot who is a Community member, typically a generalist, who plays an extraordinarily interesting role in our world.

There are of course major differences when comparing Agile to a competitive community like TopCoder. I did want to share a bit from our vantage point, focus on how self-selection is a huge aspect of what drives our individuals to compete and at least hint that our methodology is more collaborative than what might meet the eye. Appreciate the opportunity to share this with you and please feel free to get in touch if you’d care to continue the conversation.

Thank you again for highlighting the HBR article and TopCoder, we are appreciative.

Hi Clinton—
Thanks for sharing this additional information about TopCoder. It’s a very interesting model.

We are all specialists to some extent, but “hyperspecialization” implies that there is an extremely well-defined deliverable and plan, which is rarely the case. And even if there is an extremely well-defined deliverable and plan, you would need generalizations to define the deliverable and plan.