Staring at code from across the sea

Offshore programming seems to be a hot issue in the past few years. Well, I'm on my third placement as a software engineer reviewing the output of code written from India. I can't generalize the entire industry from the evidence I've seen, but in this short experience there are some depressing patterns.

You can't ask for a piece of technology without clear requirements, many of which are only flushed out as development starts to present a partly-complete product. "Oh, I want something like this, but more of..." is a common sentiment of the requester. So, iteration and involvement are key concepts in this endeavor. The problem is, if you do this too late in the process, you have to pay for the un-building of some parts to accommodate your changes. Ok, so this much is obvious. Already, one can see that putting a development team far away from the customer would present a hurdle to a smooth flow of communication.

The programming industry has two levels: "SELL/DELIVER!" - The personality who wants to focus on getting the requests met quickly and within budget. Then there's "ANALYZE/ADVISE!" - The personality seeks to focus on flushing out all the details so that the request is clear and potential issues are resolved before its too late.

Both of these are in balance the entire way. Too much on either side and problems arise: SELL/DELIVER will release a product that does everything initially asked, but it's not very dynamic. The product needs a lot of effort to adjust to the real-world, which is always in flux. "We want to add to this..." can be a death-knell for a product that hasn't thought about so-called "Flex Points." Flex Points are the software equivalent of "ways to accommodating change." If you built a swing set, you wouldn't weld the chains to the seat, because the chains may need to be lengthened or shortened at some point. Well, without a flex point, a change's effort is higher - this means expensive. This lack of flexibility can become apparent even before the project is released - which early on is a relatively better find, since 1 delay to install a flex point is better than years of expensive changes.

Conversely, too much focus on the ANALYZE/ADVISE can have other adverse effects: First, you may have to start defending longer and longer periods of design and discussion about "what are we going to do?" without yet delivering product parts (this criticism is due in part to the miss-classification of analysis as "non-work" by some of the other camp). One ends up with a large amount of descriptive information about a project. For the most part, many folks view this as a Good Thing. However, documentation is sometimes unnecessary: only what is read later or presented to the customer for discussion is useful. The rest is bloat. Also if the construction team is the same as the analysis team, documenting the entire system is usually unnecessary, since they will soon be iterating prototypes and early releases to the client. The formalization of the design documents is best used when handing the design to another team - like an offshore team. At that point, a whole new set of risks arise.

Similar to the value of Design Documents that only have value if used, the Flex Points built into software are only valuable once *used*. Before that, they are effort (investment) in anticipation of change. One must draw a line between how much change you will accommodate in an initial design, and how much work an enhancement may entail. There are advantages to each - one makes work analogous to "entering a record in the table" or "changing a value in the profile" versus "rework the logic and recompile." For example, if you anticipate a future of your swing set that may allow for a tree house on top of it, you invest a lot of effort in the construction of the base level. If your guess is wrong and no tree house is ever needed, that cost was needless. One shouldn't speculate on the market when building in Flex Points. Instead, you leave those changes up to another release, which requires not Flex Points, but Modular Design. Modular Design suggests that you can hang the swings from another, different frame in the future without a lot of effort, and save the design of a 2-story swing set with tree house for another project. In software, this is essentially building-in a low coupling between components, anticipating that entire components, while not flexible enough initially, can be rewritten and then replace the prior component without disturbing the entire system.

As an aside, this is why XML as a communication encoding gained popularity. Each component in the application can read when it needs from the information flowing around, and ignore other parts. XML alone doesn't make this happen, but the standard tools for using XML-encoded data facilitate this easily. Other components in the system can ignore the new data a future releases start adding to the data, until individual replacement components can take advantage of such data. Meanwhile, existing components need not be touched, saving a large part of the testing and code management costs.

In an offshore project, you typically have a team show up at your doorstep to flush out a design. If they are going to write it, one suspects you no longer need any software experts in-house to help. However, this is the mistake I've been hired to clean up time and again: The software Delivered but severely lacked on both the Flex Points and the Modular Design approaches. The team analyzed the problem to solve, but never seemed to ask questions about what the customer envisioned in their future, and never presented a menu of trade-offs of flexibility/cost to the customer.

Thus, at some point after the birthing process of the project (a big deal), the perceived "win" degrades. The market or business needs hit the software and it cannot change easily. Each programmer diving into the product tells the customer "this wasn't written to accommodate what you're asking" and the customer is left frustrated in paying more for changes than they expected. Changing the system not longer mimics a replacement of parts wired lightly together; it resembles delicate surgery.

Note, personally I tell customers 2 things when starting to design a system: (1) The balance of what any team, customer or consultant, can foresee is limited. One must focus on productive concepts and modular design that can grow easily. Change will always cost something, but minimizing it is in everyone's best interest. (2) The initial analysis has two parts: the business problem we're solving and the technical description of the Flex Points and Modules. This mean the analysis make take a bit longer than expected, but it reduces the risk of making a poor decision.

Thankfully, software engineering principles of the last five years have taken this into account. Multiple technology layers have arisen that allow for Flex Points and Modular Design to be easily incorporated into a platform. The Object-Oriented drive of software in the prior 15 years has matured into a distributed component-oriented design. Web Services, XML, object frameworks, and a multitude of standard toolsets from large vendors make this easy. The only drawback to a business is training their programmers to understanding and using these principles. Younger programmers may learn this in school, but older programmers usually know the business data better. Solution Space (modern design) versus Problem Space (business data relationships). Both must be merged and used together.

This again comes back to a drawback of offshore programming: The Business Data relationships have to be completely flushed out and communicated clearly to a team that more-often-than-not doesn't understand the client's market. Now the skills of the Business Analyst comes into play, and low skills here can spell disaster when the Offshore team assumes relationships about the data which later prove to be false. Instead, using in-house developers or at least market-aware consultants allow the "right questions to be asked" as managers are fond of saying.

Given my experience, a business should prioritize the acquisition of software in tiers:

1:
If the issue is market-wide and not a custom need and solved with a common tool – use one! One doesn't usually see companies writing their own Email clients, for example. The commodization of tools such as those is mature and sufficiently flexible. Some tools allow for significant customization, which can encompass the usage pattern of the customer ("we need extra fields in our email..." for example). For highly dynamic requirements, perhaps a large-vendor solution suffices (Lotus Notes communication platform), but one must start to examine why a customer needs such a level of dynamics in a common tool area. Continuing the example, this would be when a customer demands a custom email tool because their needs are beyond market tools - I begin to ask "Are you asking for features in the wrong part of the information flow?" Perhaps a combination of tools or another path accommodates the request. Technology knowledge can trump a savvy Business Analyst here, who sees an all-in-one tool when really a set of of-the-shelf tools integrated elegantly would suffice.

2:
If the business need is truly custom and dynamic, then writing a tool is probably necessary. Using any in-house developers is almost always a better investment, all other factors unchanged. Low morale, high turnover, and other issues of a workforce can make programmer training seem like a waste (indeed, training is doubly expensive, since more knowledgeable programmers can compete in a more expensive market for wages). But for a team that actually enjoys creating great solutions, an interesting project that provides real value to the business is always a win-win between programmer and business. For a business the ROI of highly skilled programmers can be realized over several projects. This implies a need for standardization on the solutions platforms. Managers that demand or suggest diverse platforms, and thus diverse skill sets, usually end up juggling too many workforce skills to know their team well. Consolidation on a robust platform is an easy bet.

3:
Bringing in help to augment a team can be beneficial. One note on that: Sequestering contractors away from the salaried folks (due to perceived secrecy or other politics) only ends up preventing the contractors needed skills from migrating to the people who will be maintaining the system long-term. I've seen businesses treat contractors like peasants, uninvited from most meetings and such - but then they are trusted to provide a much-needed skill and deal with some of the most delicate business data. This resulted in disaster when the contractor left, since a new one with the same skills had to immediately take their place. Although it benefits a contractor to be constantly in the "savior" position (it's quite lucrative!), a true consultant advises a business to take over the system entirely - training the team as the system is built. So, contractors are best used in pairs or teams.

4:
Lastly, as much as I've argued against remote development, it can be used constructively. I would advise businesses to allow their development team to break a solution into modules, allowing the remote team to deliver the modules piecemeal, and have the in-house team integrate them. This allows for the guidance on standards and the quality of the remote team to be reviewed in an iterative manner. Remember, this still means involvement of the local team to a large degree. They are designing the unit tests for the modules, which are written and run by the remote team. The local team is also reviewing the code of the modules. Obviously, they are performing the system testing.

Overall, there’s no such thing as a Done System. The closer software is to aiding in businesses processes, the more dynamic and maintainable it must be. The skills to manipulate it must be common in the market and familiar to the in-house developers. Training, paired with direct usage of the new skills on in-house projects, is a necessary part of keeping a team happy. This does not mean “adopting new technologies” so much as “going deep” on the decided platforms already in-house.

“Turning the ship” of a company can be frustrating and involving and lot of pontificating and preaching, but if the effects of poor decisions can be predicted, a business will notice. If you see your client or company making an obvious mistake in your eyes, speak up and be ready to listen. Papers such as this aren’t the only place wisdom can be imparted.