A Question About Software Product Lines and their Architectures, Transcript of a very interesting e-mail discussion Options

[Paul Clements]

Moderator's Note: This discussion thread is the transcript of a very interesting e-mail discussion, initiated by a message from Paul Clements to David Weiss, Rob van Ommering, and Charles Krueger.

Gentlemen,

Can I ask you guys a question based on your product line experience?

I have often seen it written that every software product line has a single architecture. This is often implicit (when reference is made to “the” product line architecture) but it is often explicit as well.

My question is this: Why can’t a software product line have two architectures? That is, why can there not be two architectures in the core asset base that are populated by (many of) the same software elements, just wired together in different ways to meet different behavioral and quality goals of different products? It seems like this would make a lot of sense.

I could imagine two architectures, for example -- one for very fault-tolerant products, one for the cheap products. You'd still have a lot of the same components, many common requirements, lots of test cases in common, etc., etc. Sounds like a product line to me, with a perfectly respectable core asset base. For another example, suppose I want to migrate to a service-oriented architecture. But I don’t want to throw out my current family of products, which do not use a service-oriented architecture. So it seems like I’d have two architectures in my core asset base.

An alternative is to say “two architectures means two product lines” but this seems like a lot of trouble just to preserve a rule, whose purpose I cannot see.

My question is this: Am I nuts? If not, can you give me an example from the real world that supports this idea?

OK, I guess that's two questions.

[Charles Krueger]

Hi Paul,

There may be a third option: You are right AND you are nuts. (:^)

I've never been a proponent of the architecture dogma in software product line practice, so it's easy for me to agree with you. I think architecture is a secondary issue in SPLs (blasphemy!), so having several architectures under the umbrella of one product line is fine.

But then we have to get into the arguments about "what constitutes one product line vs. multiple product lines" and "what constitutes one architecture vs multiple architectures"...

Engenio is a good example. They used to keep their low-end and high-end products on separate SCM branches since there was sufficient diversity among the requirements, architecture, and source code to initially justify that. Of course, (as you suggest) they ended up doing a poor job of capitalizing on the commonality between these two branches.

Now, under Gears, all of their products are managed as a single product line. If you do a pair wise comparison among any two products, some are very similar and some are quite different -- all through the requirements and architecture levels. However, there is absolutely no duplication or divergence in places where they is commonality among any two or more products.

Engenio could have chosen to (or still could) split the low-end and high-end products into two different product families within one van Ommering "product population". The same level of sharing occurs, but the diversity among any pair of products within one product family is less. Would this mean they have one or two product line architectures? I'm fuzzy on that answer.

Engenio actually has four choices (1) all products under one product line, (2) products partitioned into multiple product lines within one product population, (3) products partitioned into a combination of populations and independent (outside of any population) product lines, and (4) products partitioned into multiple independent product lines. Moving from 1 to 4, it becomes easier to manage wide diversity, but harder to take advantage of commonality. A perfect economic model could help predict the optimal tradeoffs. Whether or not there is one, two, or many different architectures seems to be a secondary issue.

I'm still not sure in the above scenarios that I could draw the line between one architecture with lots of diversity or several architecture with little diversity. Maybe the three of you have better insights into this than I do...

[David Weiss]

In answer to the first question, no. I do not think that the theory bars this. Recall that the definition of product line that I use is a family that takes advantage of its commonalities and predicted variabilities. Architecture is not mentioned in the definition. As I recall the definition that the SEI uses, it requires that there be a set of managed assets, but again says nothing about architecture. Recall also that I think one way to produce members of a product line is to generate them from a domain specific language. Once again, there need not be (although there often is) an underlying architecture, in the sense that we usually use architecture. (I haven't seen many papers about this approach in recent SPLCs, although I think there were a number in the early SPLCs.)

That was easy. Now for the hard one, i.e., an example. When I worked with the 5ESS organization, we created a product line of switch maintenance software (you may have seen me talk about this - it is one of the examples I use in public talks and we published a paper about it). Members of the product line were variations on software that was used to replace devices connected to the switch while it was still running. The switch maintenance software was generated from a set of managed assets, in fact fragments of C code. I think you could argue this either way, i.e., that those fragments constituted an architecture, or that they did not. In fact, they were fragments based on algorithms that described the steps necessary to remove a device from service and place a new one into service. The application engineer had a graphical language that he/she used to specify the member of the product line that he/she wanted. A generator was invoked that composed the fragments and instantiated them properly.

Hope all is well.

[Rob van Ommering]

Hi Paul,

every product line has one architecture

I must confess that I'm not too surprised by this statement; I have been observing this 'opinion' already for a long time. Whether the statement is true depends on your definition of a software product line and on your definition of software architecture. To explain this, I need more than just a few words :-)

Management Summary :-)
Products in a product line may have different architectures!

Product Lines, Families and Populations
My work on product families started around 1993, when we wanted to share software between TVs and video recorders, as produced by different organizations in Philips. I observed how hardware modules could be easily reused between such products, and therefore decided to develop a software component model (in 1996) that could do the same.

In 1998, when I first published on this, I was surprised to see so many people talk about the (single) architecture with variation points. In all fairness, there was a paper that described this to be just one of the possibilities (ref [88] in my PhD thesis), but I must confess I usually quoted this paper only for the phrase 'variant free architecture' :-)

This is why I coined the term populations, to describe sets of products with many commonalities but also many differences, as opposed to families, where products only have few differences. Only later I learned the term product line (from you guys:-).

To bring this into my mental framework, I've always seen families and populations as what you make, and product lines as how you make it. I like the word product line, because it suggests a very systematic (and partially automatic) way of deriving products. But since this has mostly been studied in the context of families with a (quite) narrow scope, where a variant-free architecture works, this probably explains why people now see this as the only way.

There are however many situations in which there is an economic reason to share between a set of products of much wider scope, and I'll give examples later. If you don't call that a product line, fine by me, but then you need to study how to share between product lines, and you need a term for that (product factory?).

So, my answer could be: okay, every product line has ONE architecture, but then we need another term and start studying that. A better answer, I think, is to use the term product line in a larger context, and accept that there is not necessarily a single architecture.

What is Software Architecture :-)
The second reason that the question is difficult lies in the meaning of the word 'software architecture'. If this is interpreted as 'the set of subsystems and their interfaces', then products will have different (concrete) architectures. If this is interpreted as 'a reference architecture', then products may have the same (abstract) architecture. But again, if the scope of products widens, there will me more and more differences in the architecture of different products.

By the way, I know of no architect that has specified all interfaces between all subsystems, so the first definition is impossible anyway... :-)

My own definition of architecture is: the first 100-1000 decisions that you have to take to efficiently build efficient systems. Among these could be the structure of the systems, but also patterns, rules, guidelines, mechanisms, etc.. My experience is: the wider the scope of the family, the more the overall architecture shifts to rules and patterns rather than 'the' structure of the system.

Terms for this are lightweight architecture or federated architecture. You only define what is absolutely necessary at the highest level, and leave the rest to the architects of subsystems.

Intra- versus inter-organization
A bit of warning is on its place here. Ordinary reuse usually fails in organizations for two reasons (I believe): (1) it's difficult (maybe even impossible) to install a proper internal structure that makes the economic benefit visible, and (2) the scale of application is usually too small anyway. The introduction of product lines solves these problems by making very explicit how reusable the software must be, namely only fitting in the scope of the family to build. This is good!

But if you want to widen the scope again, going from families to populations, you get the same problems back, and while this may be successful for a while, there are all kinds of forces that work against you. It may therefore be smarter not to strive for reuse between product lines (in the narrow sense), but rather buy shared software from outside (if available of course). Then you profit from a clear structure (explicit inter-organization instead of implicit intra-organization), and all the usual economic rules work.

But this may be over-pessimistic, and since not many people have studied it yet, I would highly recommend looking into inter-product-line (in the narrow sense) reuse.

Examples
Our TV architecture is a federated architecture, as I've explained elsewhere :-) No two products have precisely the same structure. They can differ in choice of subcomponents, glue code between components, et cetera. I must confess that I never bothered to try to model products in a single (concrete) architecture, so I do not know how well that would work. I do know that our technique works, and that we never suffered from having different concrete architectures.

Actually, the first mid-range product made with our architecture (originally intended for high-end products) had different implementations of most of the components, but many interfaces remained the same. Would you call that the same architecture? Some people claim that the interfaces are actually the skeleton of the architecture :-) In our case the concepts were the same, the interfaces, only the concrete implementations were different. I would call that at least part of the same family approach, and certainly not two independent product lines, since the interfaces (and concepts, and architectural rules) were shared between them.

In your case of the [fault-tolerant] architecture: if you could manage to implement fault tolerance as an aspect (as in AOP), then it would just be a variation point in the same product line, wouldn't it :-) ? In any case, I would say that you share enough between the two (concrete) architectures to warrant to call it a product line (or factory, or whatever).

Engenio
Charlie says what I always have been saying: share where needed, and diverge where needed. There is a commercial need to differentiate products, and there is an (internal) economic drive to share where possible. But there are many (hidden) costs in sharing, so one should not be afraid of having some copies of the code :-)

Conclusion
I would go for the wider definition of software product lines: sharing of certain assets between products, which not necessarily includes the architecture. In practice, there would be a shared part of the architecture (rules, guidelines) but there may very well be differences in architecture between products, ranging from small ones to fundamental ones. Koala may be a prime example how you can parameterize architecture (using switches, lazy evaluation, reachability analysis and code generation). But also note that life is much easier if you can use a single architecture for your product line :-)

[Rob van Ommering]

Product populations are one area where you have systematic reuse between products yet very different (concrete) architectures of the individual products. I also likes Paul fault-tolerance example. Yet I can see the reason for discussion, since architecture is sometimes defined as 'the fundamental organization of a system', implying that it is difficult to change the architecture. So if you manage to make the architecture a variation point, and can change it easily, are we still talking about the architecture, or is the architecture then something more deeply hidden?

[Charles Krueger]

I must say that I'm rather amazed that we are all 4 in such vigorous agreement. This must be a first for the SPL and PhD community. (:^)

[Paul Clements]

Indeed, and thanks to all of you for shedding light on this.

In fact, even though we agree, it wasn't all that clear-cut. I had to read the discussion very carefully to make sure that was, in fact, agreement I was seeing. To be honest, I expected three short answers of the form "Of *course* a product line can have more than one architecture. What a dumb question." So now I'm starting to think there's more to this issue than I thought.

[Rob van Ommering]

Okay, let's have some fun :-)

(1) I think there *is* an issue...
(2) I think, Paul, that you're wrong, and...
(3) I think that your colleagues are even more wrong!

In reverse order and a nutshell :-)

(1) Your colleagues are wrong because they are viewing product lines in a
(too) narrow sense, and are clearly missing out on sharing assets
between these narrow product lines, which may have a lot of economical
benefit, as David, Charlie and I are arguing (with different examples).

(2) You are wrong because you are viewing architecture in a (too) narrow
sense, as the specific structure of a product (I'm exaggerating a little here :-),
rather than as the common set of agreements between the products.

(3) There *is* an issue because many researchers in the product line field
share one or both of these views, and too little attention is paid to widen
the scope of product lines (hey, wasn't that the title of an interesting paper :-)

So, what are the interesting research questions?

(1) We know that reuse between organizations is successful (paid or unpaid,
i.e. buying (large) parts of the software from other companies, or taking it
from the OpenSource domain). We know that we can make reuse within an
organization successful by carefully defining the scope and then analyzing
commonalities and differences and setting up a (classical) product line. But
can we also make larger scale reuse within a company happen, or are the
organizational complexities too large and the economic benefits too small?

(2) Of course a product line (however wide) has a single architecture :-),
otherwise the assets wouldn't work together (architectural mismatch)... Of
course, each product has a specific architecture that differs from that of the
next product in the same product line, whether in small details or in structure...
The real question is: what do you have to agree on at the product line level to
make the assets (requirements, code, tests, ...) work together, while maintaining
a sufficient amount of freedom to implement products differing in functionality
and qualities and therefore architecture (like your fault-tolerant example, or
my TV - DVD example).

(3) Derived from (2): what kind of analysis can you do at the level of the
product line, so taking all possible values of variation points (which may include
changes in the structure) into account, and what kind of analysis must you do
at the level of individual products (or subsets of the product line).

I think we are strong in setting up and analyzing specific architectures for
single products and (very) small sets of products. I think much work still needs
to be done in widening the scope.

Note that more and more architectures are emerging that allow lots and lots
of variation, or put differently, that are not design for a single specific purpose
but rather for a whole class of systems, like Eclipse (everything you ever wanted
to do with inheritance, and then much more :-) but also Apache which I'm
studying now, which claims to implement any generic network server, rather
than just a specific http daemon...

Somehow, my nutshells always turn out to be quite large... :-)

[Paul Clements]

So here's a challenge for you. Show me two architectures that YOU would call different. You can use whatever criteria you choose, but whatever they are, you have to say "Yep, there is no way I would call those the same architecture." I bet you a meal at a decent restaurant of my choosing that I can construct a plausible scenario where those two architectures share a lot of core assets, including components, that could together form a product line. (I issue the caveat that I consider what you call a product population to be a product line, because while I think it is a wonderful concept, I see no hard line that differentiates between the two.)

My point is that I don't accept your claim that components must work under the same architecture, otherwise we'd have architectural mismatch, because we know lots of ways to repair mismatch. Consider a core asset base that includes (a) one architecture; (b) a set of components that work with that architecture; © a different architecture that passes the "van Ommering difference test"; and (d) wrappers and mediators and bridges and lots of other stuff to make the components work with the second architecture.

I also find this whole conversation unsettling because it is stuck squarely in the world of code, when there are a million things OTHER THAN code that go into a core asset base. Suppose you have two architectures, and two entirely disjoint set of components that populate them. I think that's unlikely, but let's pretend. If I have a common set of requirements, system acceptance tests, user manuals, production plans, budgets and schedules and work breakdown structures, processes, tools, and on and on, then I'm going to be really unhappy if you tell me "No, that can't be a single product line because ALL of the code isn't shared."

(But I did like that part about how my colleagues were even *more* wrong...)

[Charles Krueger]

I issue the caveat that I consider what you call a product population to be a product line, because while I think it is a wonderful concept, I see no hard line that differentiates between the two.

Hmm, I think this difference in perceptions must be reconciled before you can start a meaningful debate about what constitutes one versus multiple architectures, and one versus multiple product lines.

I can walk up to a Philips electronics exposition and my brain will easily partition the TV section from the DVD section. Something tells me in a blink that there are two product lines. When I look at little closer at their features, capabilities, and possible interconnects, then I recognize that they also have some things in common, thus they are part of a larger population.

The family versus population concept is very real in the world in which I work, not just intuitively as in the above example, but because it serves a valuable role in designing, establishing and running SPL practices.

If anyone attempts to argue that TV and DVD are part of the same family or have the same architecture, that effort will only convince me that you have firmly placed two feet on an academic slippery slope and will soon claim that everything in the universe is one. (:^)

[Paul Clements]

If you substitute "larger product line" where you wrote "larger population" would it be wrong?

Are you saying a population is NOT a product line?

[Charles Krueger]

If you substitute "larger product line" where you wrote "larger population" would it be wrong?

Yes.

Are you saying a population is NOT a product line?

Yes.

If I hand anyone a TV and ask them to place it in the proper product line, there's a 100% certainty they won't put it with the DVD units.

If I take Philips' top TV design guru and ask them to design next year's product line of DVD units, those DVD products will suck because that person doesn't understand the gestalt of the DVD product line.

If I take Philips' top RF component guru and ask them to create an RF component (e.g., chip and firmware) that will serve both the TV and DVD product lines, they will create an elegant solution for this one population of two product lines that need RF components.

I think there is sufficient "gravity" that clusters different product lines, and that trying to blur the boundaries around these clusters results in an over generalization that dramatically degrades reuse. I can be an expert in TV and I can be an expert in DVD, but probably not both.

Reuse across product lines (within a population) is like an aspect-oriented reuse. The RF component example is really a little product line that cross-cuts the TV and DVD product lines.

One of my favorite capabilities we added to Gears in the last year is hierarchical product lines, where one product line can be composed out of not only components, but also smaller nested product lines. Both components and nested product lines can be shared across multiple product lines, extending the notion of population to include the sharing of both components and nested product lines. The hierarchical product line and product population capability was driven by customer requests and needs, so that's why I believe pretty strongly that having these concepts in our field has real and practical value.

[Paul Clements]

So doesn't that mean the RF component is a product line, and the TV and DVD families are really populations, not product lines?

What's the difference between (a) a product line within a population, and (b) a "hierarchical product line"?

[Charles Krueger]

Yes, the RF component is definitely a product line since there are different instances with commonality and variation among them.

In terms of the distinction between family and population using our TV and DVD example, Rob argues eloquently that we have two choices -- it's either (1) one family with all of the TV and DVD members as part of that family, or (2) two different families that together represent one population. Thus a population is the union of multiple "similar" families.

Just to clarify on terminology, note that I'm using "product family" and "product line" to mean the same thing, which I think is what you (Paul) are using, too. Rob uses "product line" to mean the "how" -- I've been using the term "production line" to describe the "how" since I'm a big fan of the manufacturing analogy. In my terminology, a production line creates a product line (or equivalently, produces a product family). In Rob’s terminology, a product line creates a product family. Not sure if this is part of the confusion in this discussion.

Where does the RF component fit into the argument about the TV/DVD family/population issue? It has a "part-of" relationship in members of the TV/DVD population, but it does not have a "member-of" relationship with TV or DVD. It's too dissimilar to be considered in a family or population of TV/DVD. Thus families and populations are all about member-of relationships, while hierarchical product lines are all about the part-of relationships.

[Paul Clements]

Looking back over the message traffic makes me think that in fact we don't agree.

Rob, Charlie says a population is NOT a product line. Do you agree?