Return to the Moon postponed?

[Data Cable]

phantomwolf said:

It's amazing how often you get told to just cut corners because "no one will ever do that." Generally that means the first person to use the program will do exactly that.

Insert old engineering chestnut about idiot-proofing against ingenious idiots here.

[JayUtah]

Which is sort of my point.

Yes it was -- exactly.

If nothing had happened then management would have felt justified in cutting deeper into the safety margin and relying on luck to get them through, a rather bad attitude, but one I see often in the computer programming area.

The reason you see it in the programming area is because it's a universal human trait.

A prolonged string of successes leads one into complacency, from which position he questions the cost and effort needed to maintain a margin that doesn't seem to be necessary. Invading the margin in order to reduce costs or increase production leads inevitably to a failure. Analyzing the failure reveals one or more shortcomings in design or operation. With designs repaired and blame apportioned (not always fairly), one then enjoys a period of success helped along by renewed vigilance. Until the cycle begins again.

It is human nature to accept risk, even grave risk, for the sake of convenience. But the alignment of those priorities changes from time to time, typically based on how long it has been since our last accident. In the 1930s we wanted buildings that could be built cheaply. So we used a lot of concrete. In the 1960s we wanted buildings that could soar to the heavens. So we used a lot of steel trusses. In the 1990s we wanted buildings (again) that could be built cheaply. So we used a lot of pre-fab steel. In the 2000s we want buildings that are safe and protect us. So we're back to using a lot of concrete.

Unfortunately we're doomed as a race to push the envelope, even unwisely. NASA /Marshall and Morton Thiokol worked themselves into a state of complacency believing that a flawed design could still be operated safely. In fact they had edged right up to the brink of failure, until life and death hung from something as fragile as a chunk of slag.

I really hope that the 777 software wasn't developed by managers like mine.

Thankfully there are federal regulations mandating a process for avionics software development. It doesn't guarantee bug-free software, but it does mean that no one can come into your office and tell you what to do without submitting to the process.

Most engineers, including computer programmers, want to do a good job. They want to build systems that work well and safeguard the interests of those who use them. They want to have robust, resilient systems that defend against failure. However, the measurement of success along those lines is difficult. And so the cost of doing a good job is hard to justify.

This is where I love to quiz new business-school graduates who aspire to engineering management. In a cost-benefit analysis of additional, say, 3,000 lines of error-checking and mitigation code, how do you propose to measure the benefit? How can you count the number of defects not filed against your product because it was well-built from the start? That's how you separate the real budding business leaders from the jargon-spewing, fad-waving wannabes.

[reynoldbot]

I imagine you are a pretty intimidating force to egotistical managers.

[JayUtah]

Very. But to conscientious, well-qualified, honest managers I am a dream: I don't lie to them and I keep them well informed. Nearly all then respond in kind.

I have a friend on the faculty at the Wake Forest University business school. He is becoming increasing disillusioned with some business school students who intend to take a management track. He says more and more of them seem to expect that management graduates simply have to be good at talking the talk, and that they're following this career path under the delusion that it will be the easy road to fame and fortune without actually having to know or understand anything. He's noticing a lot of people with a get-rich-quick attitude and no interest in actually learning how a business works. And he helps me ask questions that weed that sort of candidate out.

The person to whom I currently report used to work at JSC. Believe me, we see eye to eye on almost everything.

[PeterB]

jayutah said:

This is where I love to quiz new business-school graduates who aspire to engineering management. In a cost-benefit analysis of additional, say, 3,000 lines of error-checking and mitigation code, how do you propose to measure the benefit? How can you count the number of defects not filed against your product because it was well-built from the start? That's how you separate the real budding business leaders from the jargon-spewing, fad-waving wannabes.

Out of interest, what's the answer?

[JayUtah]

There really isn't an answer. It's a trick question. Candidates only trying to please will try to answer the question. Good candidates will push for a question they can answer.

Good candidates will ask, for example, whether it's to be added to an emerging new design or to existing code. With new code, you never know the cost of the path not taken. You can only vaguely try to compare it to similar code elsewhere. And that becomes an assessment of the candidate's approach to risk.

Existing code has established defect rates, feature counts, lines of code, duration of deployment, customer base, and a whole host of numerical data from which some model can be derived. If by adding 3,000 lines of "hardening" code you can reduce the overall defect rate per month or year per lines of code, or some other such normalization, then you can have a basis for determining a cost benefit. It's then reasonable to set goals such as a 10% reduction in defects per lines of code over a certain period of time, and from that determine whether your gain from lack of defects offsets your development time.

Turning back to new code and designs, I expect good candidates to be familiar with some model of software quality. You can get them from various industries, or from IEEE, or from any number of private consulting companies. It doesn't largely matter which quality model the candidate understands, because they all contain factors that aren't determined by functional requirements, but instead by such things as modularity, transparency, and other dimensions that affect how well the resulting code can be modified, maintained, reused, or retasked from the software engineer's perspective.

Those models will not necessarily give you a cost-benefit basis right out of the box, because it's hard to put a dollar figure on, say, "transparency" in a program code base. (Transparency is a property of program code that allows a newcomer to abduct the correct intent of the code by looking at it.) But a good candidate will use the inward-facing factors of his favorite software quality model to pin project-specific estimates on an expected gain in terms of how easy it will make subsequent tasks for his team.

Any design has a total cost of ownership, both for those who buy it and those who produce it. Too many eager managers consider only up-front costs and neglect the lingering costs throughout a design's lifetime. And they err on the side of being willing to trade long-term advantages for short-term gains. The good candidate will not necessarily have all the answers up front. But he will know what things to keep his eyes on, and he will realize that the metrics that affect profitability aren't always customer-facing or revenue-dependent.

Put simply: the good candidate knows about qualitative gains and how to think about them.

[PeterB]

jayutah said:

Any design has a total cost of ownership, both for those who buy it and those who produce it. Too many eager managers consider only up-front costs and neglect the lingering costs throughout a design's lifetime. And they err on the side of being willing to trade long-term advantages for short-term gains. The good candidate will not necessarily have all the answers up front. But he will know what things to keep his eyes on, and he will realize that the metrics that affect profitability aren't always customer-facing or revenue-dependent.

Put simply: the good candidate knows about qualitative gains and how to think about them.

Ah, yes, this sounds familiar.

I come from the world of human resources, intially payroll and now industrial relations.

I'd been the team leader of my department's payroll team when I was asked to spend a couple of weeks doing the same for the payroll team of another agency we were taking over. That couple of weeks turned into several months, and gave me an interesting insight into how some other agencies run (or don't, in this case).

Anyway, the team I normally led had about 12 people in it, and looked after the pay of about 3500 employees. The team I temporarily looked after had about 9 people in it, but looked after only about 1200 employees.

At the time, their IT people were in the process of upgrading the payroll software (another issue in its own right, but not relevant here). The guy in charge of the IT team was confident the new software would be so efficient the pay team would only need about 2 or 3 staff (yet another issue), but when it was released, it was quickly apparent that it was nowhere near that good. Instead, the software development had been done on the cheap, to save up-front costs, and the result was that the entire payroll team needed to be kept on. The up-front savings had been in the order of a couple of hundred thousand dollars, IIRC, but they were swallowed up in a year by salary savings not made.

To give an example, the software assumed that all employees worked 9 to 5, Monday to Friday. If someone happened to be rostered for a shift at some other time of the week (or worked other than 8 hour shifts), and took leave for the shift, there were several work-arounds required to make the system deduct the right amount of leave from the employee's relevant leave balance. The problem was that >95% of the agency's employees are shift workers, and many of them worked 10 or 12 hour shifts.

Sigh...

[JayUtah]

An analogy outside the software world would be manufacturing tooling. Because it is
ephemeral and because you can't charge a customer (generally) for it, unwise managers want it done on the cheap in order to cut costs. What they don't realize is that tooling is itself intended to cut costs by making the manufacturing easier, safer, better, and cheaper. Good tooling involves a substantial up-front cost in development, and pays for itself many times over in saved manufacturing costs. Bad or non-existent tooling saves up-front costs, but incurs additional expense for every unit manufactured.

That's the quantitative aspect. What's the qualitative aspect?

Let's go back to your predicament. You work in human resources (I think human capital is now the preferred term) because you want to work with people and help them when they have problems with their employment. And you want to help your company maximize and account for its resources in a responsible way. Those are all excellent goals. And when you try to meet them, you think in terms of tasks to be accomplished and tools that you need.

Now when those tools break, or fail to do what they promise, your attention gets yanked away from what you really want to do and fastened involuntarily on the behavior of the tool. You spend your time trying to work the tool, not doing your job. You have to do things in silly or inefficient ways to get around the limits of the tool.

A printer wants to print books, not figure out why his typesetter is making grinding noises or dropping type. He just wants the typesetter to work. HR staff want to balance the books and get everyone paid, not figure out what rat-dance on the keyboard properly deducts vacation time taken.

When someone's attention goes from the task to the tool, the toolmaker has failed. And most often it fails because the toolbuilder has been too selfish in making it. Yes, it may be easier to make hammers with balsa-wood handles, but the hammer won't be worth anything. And it won't matter how much profit you make on each unit if word gets out that your hammers are crap and no one buys them. That's the qualitative aspect.

[Ginnie]

At my former job I used to try to troubleshoot before I had problems. I was pretty good at seeing what would happen down the road. It was especially important when working with organizational data in a database. For instance, sometimes it was easy to convert mass titles of books to a certain catagory for a special promotion, but not so easy to convert it back. Or keeping book imprints organized so that if it changed to another company, lets say from Random House to Penguin Books you could do it en masse, but still be able to keep track of all the imprints titles distinct from the head vendor. After all, the imprint could be sold again to someone else in the future and you don't want to have to extract one title at a time from the database.

I remember at work we had a three thousand dollar barcode machine that printed out barcodes on sticker labels. Problem was, most stores starting scanning their items, including books. And they were using 13 digit EAN barcodes that our barcode machine couldn't produce from certain UPC numbers.
We were at a loss - the books had to get to the stores and they had to be able to scan. Luckily, we had PC's and the web was in its infancy. I was able to download a shareware barcode program and using Works 2.0 printed them out on label sheets. Actually on a sheet of paper. Then I used our photocopier to print them on label sheets to attach the labels to the back of the book. That solved our problem. But what really p*ssed me off was that the programs author wanted $39 if you used the program and my boss was too cheap to send him a cheque. I tried to point out how the program bailed us out of a predicament and was indespensible to supplying our customers with some of our product, but to no avail. I almost felt like deleting the program from the computer and telling my boss I forget where I got it on the web so that he'd have to shell out another few thousand dollars for a new barcode machine!

[JayUtah]

One can make the same mistake in the other direction too, especially in software. If you're going to make a vacuum cleaner or a mining dump-truck, you're probably not inclined to depart too far from the core application. You won't be taking the kids to soccer practice in the dump truck. You won't be using the blower in the vacuum cleaner to dry your hair, so making the discharge outlet compatible with hair-styling attachments would be of dubious commercial value and therefore unworthy of engineering attention.

Some software, on the other hand, seems to gather extraneous features like writers gather metaphors. MP3 players need to embed web browsers. Flight-management software needs to connect to APOLLO nodes via wi-fi at the gate to download passenger manifests. Interoperability is a great thing, of course. But it comes at a price.

In my consulting I once saw a software system created by a small company. The software architect had unwisely expanded the customer's requirements into a litany of invented, extraneous requirements intended not only to satisfy the customer's needs but also the imagined needs of all similar customers. The developers couldn't build it, and couldn't make it work for the one customer who was the only one actually writing a check.

Because three-tiered architectures were all the rage, he had mandated using one even though the application didn't require it or benefit from it. As a result, even the simplest tasks were greatly complicated and delayed.

He had designed and written a software library to interpret a particular data format. He had convolved the format interpretation inextricably with both compression and encryption functionality (rarely used). He had provided both forward and reverse traversal functions, requiring elaborate buffering strategies and complicated data structures.

In short, it was the worst of what can happen when software designers try to reach for the holy grail on every project. An enormous amount of development expense was undertaken with little return. A good manager will know how to trim those unwanted features and reign in eager programmers so that they can focus on the actual needs.

Regarding the format-interpretation library, the first thing we did was analyze how the various client programs used the library. Only two clients used any of the backwards-traversing functions, and they could be easily rewritten either not to require them, or to implement their own buffering. Then we noticed that the functions that were most heavily used were grossly overcomplicated by the machinations required to support the never-used operations.

So the library was rewritten to provide only straightforward iteration. The compression and encryption support was removed, to be provided by third-party utilities. The number if API calls it exposed was reduced from 15 to 6. It required only 20% of the original lines of code and was essentially bug-free from that point on.

The imagined requirements were eliminated, including the synchronization by satellite uplink (no kidding!). The resulting system then had a scope that the small development team could tackle. The unwieldy multi-tiered architecture was refactored to two nodes, which immediately became scalable. Oddly enough -- and I didn't even anticipate this -- it then became attractive to other customers sporting its new, stripped-down feature set.

The moral is that every feature you add comes at a cost. It costs to develop it. It costs to support and maintain it. But it most heinously costs in how it complicates the entire system, including those features that need to work well and would be straightforward to build by themselves. So a good manager knows how to stop programmers from interfacing an MP3 player with satellites just because they can, and from embedding web browsers in flight-management software that needs to run correctly every time, and from all sorts of frivolous pursuits that kill otherwise good ideas.

The exercise I like to do is this: have the stakeholders list each of a software product's desired features and functions in priority of estimated value to the customer. Let them debate as long as they want, and use whatever criteria they want. Then you tear the list in half and throw out the bottom-ranked features, hand them the pruned list, and say, "Build that."

[JayUtah]

...my boss was too cheap to send him a cheque.

To counter that: consider that I witnessed a $14 million supercomputer pass its qualifying runs, hours before the customer's funding cycle expired, under threat of overheating (insufficient cooling in the test room). This was accomplished by a six-dollar can of "canned air" squirted into the air intake openings at warm moments. The system delivered on time, and the high-school student summer intern who thought of the solution got a $250 bonus in his next paycheck.

[Ginnie]

It amuses me when a new computer user feels they have to spend hundreds of dollars on bloated software like Microsoft Word and Excel when they could just download for free small efficient software like Abiword and Gnumeric - which are available for Windows now BTW. If they still want bloat, they could use OpenOffice for free.

[Ginnie]

jayutah said:

...my boss was too cheap to send him a cheque.

To counter that: consider that I witnessed a $14 million supercomputer pass its qualifying runs, hours before the customer's funding cycle expired, under threat of overheating (insufficient cooling in the test room). This was accomplished by a six-dollar can of "canned air" squirted into the air intake openings at warm moments. The system delivered on time, and the high-school student summer intern who thought of the solution got a $250 bonus in his next paycheck.

Wow. A cheap solution that was! I guess the company didn't want the student to get too big headed.

[JayUtah]

The bonus was the better part of a week's pay for him. Not bad for a company that doesn't routinely give bonuses.

[JayUtah]

It's even more amusing to watch someone launch a $1,000 spreadsheet program just to create a table of text entries.