Working the System

“Systems Biology Calls for New Way of Training Doctors” – sidebar headline in Chemical and Engineering News

The first feedback control device was probably the float valve, used in ancient water clocks. I’m discounting biological and other feedback systems, obviously. Those are usually called homeostasis.

Prior to the 20th century, there aren’t a lot of examples of feedback devices. Watt’s governor is the one most commonly cited, and its 18th century origin was close to concurrent with the steam release valve, which is also a feedback device. The governor is also of note because it is an example of proportional control. It didn’t just shut the steam on and off; it throttled the steam by varying the size of an aperture.

On/Off control is the sort that you get with a thermostat. When the temperature drops, the furnace kicks on at full force, then it stops when the temperature rises. That produces a limit cycle because the process is non-linear. If the furnace heating were proportional to the difference between the room temperature and the thermostat’s set point, then you’d have proportional control. That also produces a cycle, but the cycle is sinusoidal, and the process is termed linear, because of the type of equation that describes it.

The 19th century invention of a torpedo control system by Robert Whitehead was probably the first mechanical invention that addressed the oscillation problem. The first torpedo designs used a simple hydrostatic valve to adjust the control fins, but this caused “porpoising,” an up-and-down motion that sometimes put the torpedo above the surface of the water. Whitehead realized that something was needed to damp out the fluctuations, so he devised a pendulum that crudely measured the torpedo’s angle and modified the control in the direction to reduce that angle. This added a rate-of-change term (aka, a derivative) to the control equation, and reduced the depth fluctuations of the torpedo from 40 ft. to less than 6.

The problem with a proportional controller with damping is that the system often settles to a point of stable error, because the small error signal is damped out by the derivative signal. The solution to that is to add what is called the integral term, so a small error signal is integrated over time, and thus builds to a large enough signal to move the settling point.

The first example of a full PID (proportional-integral-derivative) controller comes in 1922, when N. Minorsky devised an automatic controller for the steering of ships. The mathematical characterization of control systems was also advanced enough by then to properly analyze such systems.

The “feedback loop” as it came to be called, seemed to offer some benefit to another, more general problem, of the sort that a wide variety of scientists and others were facing, that of the reductionist trap. When someone says, “We’re nothing but a bunch of atoms that think we’re alive,” that’s voicing the reductionist trap. A bunch of atoms we certainly are, but it doesn’t seem accurate to say that we’re nothing but a bunch of atoms. There are, after all, a lot of bunches of atoms around, but none of them behave just like me. I rather doubt that any of them think they are me, either.

Another way of addressing the problem is to use phrases like “emergent phenomena,” which is a fancy way of saying that the whole is more than the sum of its parts. Since a feedback loop is also more than the sum of its parts, and since homeostasis (feedback, remember) is a general characteristic of living organisms, there came a general belief that feedback analysis might offer some insights into biology, or psychology, or sociology.

Thus was born the Cybernetics Movement, which included some folks like A. H. Maslow, whom I mentioned in a recent essay, as well as Margaret Mead and Gregory Bateson, plus some heavy hitters like Claude Shannon, John von Neumann, and Norbert Weiner, whose 1950 book, The Human Use of Human Beings: Cybernetics and Society became a best-seller. I’ll mention in passing that Claude Shannon had just pretty much invented information theory, which, aside from revolutionizing electronic communications, also became part of the cybernetics movement.

Later, Cybernetics became General Systems Theory, which was not exactly a cult and not exactly a movement. But it did have some Believers, and I was probably one of them. The systems guys were of the belief that systems theory could be applied to, if not everything, an awfully big part of everything, and that it could and would revolutionize everything it was applied to.

In my own case, as I’ve previously written, I was attracted to the idea of simulation modeling of large scale biological, environmental, and social systems. I started off doing lake ecology, then slid over to atmospheric chemistry with barely a hiccup, because the methods of analysis were so similar. So that part of the program worked pretty well, at least from my viewpoint. However, I hit the downside of it all pretty quickly.

The downside was first, that while the tools of analysis were top notch, to use them in real world situations, you need a lot of data, and the methods of data collection weren’t really up to it. I hit that first in lake ecosystem modeling, where data from sunlight, nutrients, and plankton were pretty good, but the data we had for fish populations were horrible. And, oddly enough, the fish were important. After that experience, atmospheric science was wonderful; there was so much data available.

The second drawback was the real killer: analysis isn’t enough. In order to “change the world” you have to change the world. You can have the right answer, but if people aren’t willing to use it, what good is it? And, if your way of doing things is different in any way from what people are already doing, what they are, in fact, trained to do, you’re not going to make much headway.

It doesn’t help to blame the other guy, either. Everyone thinks their job is hard and everyone else’s is easy. No, what they are is different. Getting the correct engineering analysis isn’t the same as getting the right policy analysis, and neither of them make getting the policy adopted that much easier.

The worst of it was with the physicians. They go through hell getting their medical education. If you want medicine to change, you’re going to have to wait for an entirely new cohort. Worse, because medical education is also controlled by those same people, you’re actually talking about many generations. I watched more than one systems engineer bash his head into that brick wall, over and over again.

The quote at the beginning of this essay is from July, 2006. It could just as easily have been from 1976. Or 1956 Maybe it will happen, but I’m not holding my breath.

Without Delay

In reel-to-reel tape decks, there is a record head and a play head and they are separated by a small gap. The play head comes after the record head, and the record and playback circuitry are separate, so it's possible to monitor a tape recording more or less as its being recorded, albeit with a small delay.

The small delay was often used to produce an "echo effect" on recordings and in the studio. For the echo effect, the tape output was mixed with the line in and patched back into the tape input. Depending on the tape speed, the echo delay could be controlled, and the gain between output and input controlled the echo strength. A gain of greater than 1 produced the "infinite echo" that rapidly became a sound pulsation with its frequency centered at the maximum frequency response of the system.

One practical joke that was often played at radio stations was to hook up a tape deck to generate a delay, then feed the announcer's voice back to him with a fraction of a second delay. I was once trying to get an echo effect on my voice and I found that I'd practical joked myself; I had to remove my headphones in order to continue. The delay makes it almost impossible to speak. It's hard to explain why, but the experience is compelling.

In a course, Voice and Image Processing, that I took at RPI there was a similar demonstration with video. A ball was placed behind a small barrier, and a video camera showed the ball on a TV screen. Normally, you could just watch the monitor and reach behind the wall to pick up the ball. But with a half-second time delay, such a seemingly ordinary task became almost impossible. You soon found yourself reaching for the ball, overshooting, then overcorrecting, then overshooting, etc.

Such a thing is called a 'limit cycle' in systems control theory, but it's pretty eerie to be a part of a limit cycle and unable to break out of it. Eventually, you just stop moving entirely, then veeeeerrrrrrrryyyyyy slowly move your hand to get the ball. It could literally take 30 seconds or more to do that simple task.

There's a bunch of mathematics in systems theory that deals with time delay and "controllability." The upshot is that if you add enough time delay into a control system, it becomes uncontrollable. Your ability to affect events is slower than those events. Imagine trying to pick up the ball behind the wall if it is moving erratically.

One of my favorite jokes is about the economics professor walking through the Quad with his students. One of his students says, 'Look, there's a ten dollar bill on the ground.' The professor replies, 'Can't be. If it were, someone would have picked it up already.'

For a long time, economics was dominated by what are called "equilibrium calculations," models of an economy under steady state conditions, no shortages, prices in equilibrium, all the usual assumptions. Those are the simplest conditions to model and to easy calculate, so they were the first results. Evolutionary biology tended toward the same simplifications, for the same reasons. The advent of the computer, and the growing access to massive amounts of computing power changed the landscape, but it took a while for theoretical models to catch up to the improved tools. In fact, the catch-up is still going on.

I had lunch with a colleague a while ago, and he asked my opinion about global warming/climate change/greenhouse gases. I told him that it was pretty obvious that the signal was out of the noise, the whole process was clearly underway, and was he surprised at this answer? He noted my well-known contrarian streak. I observed that James Hansen hadn't made a wrong prediction since 1988, and I wasn't going to challenge that sort of success.

In truth, I was a little late to the global warming party, partly because of that contrarian streak, but also because I was focusing on the science and not the policy. I was also perhaps yielding too much to my own libertarian leanings. So let's review why I should have been convinced sooner than I was, at least on the policy issues.

From the standpoint of political philosophy, one fact should be paramount: if we do not have a right to the air we breathe, then human rights, including property rights, are meaningless. And that should include the right to have that air remain unaltered. You shouldn't have to prove that harm is being done to you, any more than you should have to prove that people are harming you in order to not want a stream of trespassers walking across your lawn.

Now any given individual has no real impact on the contents of the entire atmosphere, although it's certainly possible for an individual to affect your current breathable air, and you generally have recourse. If someone smokes in your house and you don't like it, you can throw them out. If the neighbor's barbecue is noxious, you can usually complain to some agency, and I, for one, do not consider that to be an infringement on your neighbor's rights, though your neighbor may disagree.

But group behavior can, and does, affect urban, regional, and global resources. The industrial world's propensity for fossil fuels has had an undeniable effect on the concentration of some important trace gases in the atmosphere. Regulating group behavior is not the same as regulating individual behavior. Regulating corporations or national economies is not the same as regulating individuals, and giving free license to groups and organizations reduces individual freedom.

In the case of global climate change, regulating group behavior is essential. Actually, of course, group behavior is regulated. It just happens that it is regulated by those who rule, manage, control, and lead those organizations, the corporate boards, the CEOs, the congresses, presidents, agency heads, judges, and lawyers whose fingers are entwined with the strings of authority.

But authority and control are meaningless if the system is uncontrollable. The global climate system takes decades, if not centuries to equilibrate to any given greenhouse gas level. Glaciers take even longer to melt or rebuild. And the human political process likewise has major delays built into it.

There is a thin straw to clutch at, called feedforward in control theory. Using feedforward, you attempt to compensate for feedback delays by anticipating the system response. But feedforward control is seriously limited by your understanding of the underlying system. Without that understanding, feedforward is useless.

In regulatory policy, science is the feedforward control signal. Science, however, is currently under political attack from numerous quarters. And big money is being spent to target climate research in one part of that attack.

We're going to lose south Florida, and, my colleague suggests, most of Louisiana and Mississippi. California will acquire a new inland sea. Much of Bangladesh will vanish, as will plenty of islands in the Pacific and Indian Oceans. The fact that these things are going to happen long after you and I are dead does not make the future more palatable. It makes it more inevitable.

Information Technology, Systems, and Simulation

Some years ago, a time sufficiently distant that I was still a member of the IEEE (Institute of Electrical and Electronics Engineers) I attended one of that organization’s Simulation Conferences. I there heard an anecdotal tale from two engineers, who had prepared a queuing model for a particular warehouse operation. The model had the unfortunate tendency to fail with some regularity; the simulation would progress to a particular point and then cease to function, responding with a series of error messages.

The two engineers decided to visit the site of the warehouse that they were simulating, in order to observe a day at the loading dock, which was the input to the queuing model. They observed the following sequence:

A truck arrived at the loading dock.

The driver and dock attendant began unloading the truck, whose cargo was a number of cardboard boxes, all of identical size.

As each box was unloaded, it was placed on a moving conveyor belt, which transported the boxes into an automated stacking system within the warehouse.

When the final cardboard box was placed on the conveyor belt, the dock attendant also mounted the conveyor belt, and when it reached the entrance to the warehouse, he proceeded to kick and pummel the box until it entered the warehouse.

The engineers were nonplussed. “Why did you just do that?” they asked the dock attendant. The dock attendant then explained that there had been a blueprint error in the construction of the warehouse and the storage area that fed the automated stacking system was about nine inches too short, so it was necessary to force the last box into the warehouse. The boxes in the storage area had just enough give for that task to be accomplished, provided some substantial force was applied to the last box.

In a sense, then, the simulation had been a success. The automated warehouse system would fail if used as constructed. However, human intervention has compensated for what was basically a defective system design.

In recent years, a substantial portion of the productivity gains in the U.S. economy have been ascribed to Information Technology (IT) as the causal agent. There is probably substantial truth to this conjecture, especially if IT is expanded to include telecommunications of all sorts, email, cellular telephone technology, broadband internet connections, etc. However, there are a number of implications to heavy IT involvement on business processes that should be borne in mind. I will speak primarily to two such implications. The first is that a high degree of IT involvement in a business tends to make that business “system centric” as opposed to “process centric.” There are several further implications of that observation, and I’ll get to those in a bit.

The second thing to bear in mind is that IT systems are generally simulations of something, rather than res ipsa “the thing itself.” This is easy to see when we speak of a computer simulation model, but this hardly scratches the surface. A telephone, for example, does not deliver the voice of the person at the other end; rather, electrical information is sent from one place to another that then allows a simulated human voice to appear at the receiving end. Similarly, financial and accounting systems are simulations, and economists speak of “the financial economy” as different from “the real economy.” When decisions made for financial accounting reasons are derided as “coming from the bean counters,” that derision carries with it the idea that the financial simulation is missing some important part of the real situation.

Nevertheless, accountants and financial officers are charged with protecting the financial systems of an organization, and with good reason; if the financial system fails, it likely mirrors a failure in the “real” organization, and even if it does not, financial system failure makes some part of the real organization either unobservable or uncontrollable, or both. As is often the case, financial systems are both simulation systems and control systems.

The same may be said for IT systems. IT systems are generally either communications or control systems (or both), and failure of an IT system can have adverse or even catastrophic consequences, whether it be losing an inventory database, sensitive customer information, or a set of emails. Without adequate information, management decisions become mere guesses, and without adequate communications, those decisions cannot be implemented.

Moreover, as the tale of the automated warehouse suggests, IT systems are not as robust as real processes, because human beings are inherently more adaptable than algorithms and hardware. If the box doesn’t fit, the dock foreman may try kicking it; if that doesn’t work, perhaps a box cutter will do. If that fails, then perhaps he stores the box temporarily, until the holding area clears. As long as a human being has a goal in mind, he or she will often find away around obstacles. There is even a common name for this: “work-arounds.”

So it is understandable, perhaps even inevitable, that IT organizations will tend to become “system-centric,” but this carries numerous drawbacks. The obvious drawback is the tendency to concentrate of system development at the expense of business process development. This has several follow-on failure modes.

First, concentrating on systems rather than process tends to starve the processes themselves of needed resources. This is particularly problematic in a cost-cutting environment, where IT systems are, in themselves, supposed to reduce the need for personnel. Too often, however, the personnel savings are assumed rather than demonstrated, with an outcome that is deleterious to the underlying process. Moreover, IT systems usually carry their own overhead burdens, so it is not uncommon to see the effect of merely trading one set of employees for another set, the latter often being more expensive than the former. This, incidentally, is a problem that is not at all confined to IT. It is often true that attempts to implement cost-cutting through personnel reduction results in an increase in managerial overhead (“off-shoring” projects are notorious in this regard) to no great benefit to the organization as a whole. Hospital management is another area where this phenomenon has been observed, with reductions in nurse-to-patient ratios often occurring simultaneous to substantial increases in administrative (and sometimes legal) overhead.

Another implication of “system-centricity” is that it allows, even encourages, “gaming the system.” A substantial portion of the enhanced productivity created by IT systems in large organizations is the deletion of several layers of middle management. The managerial review process is then replaced with a variety of system-derived “metrics.” However, employees then often find it easier to “work the metric,” than to perform the work that the metric is supposed to reflect. This is the equivalent of the use of standardized testing in the public school system. Given the need for a certain percentage of students to achieve particular scores on the tests, teachers find themselves under pressure to “teach to the test,” emphasizing material that is known to be on the tests and de-emphasizing material which either does not appear on the tests, or material for which it is inherently difficult to test. In some cases, school administrators and teachers have been found restricting the student population that takes the tests, sending those who will do poorly home on test days. In other cases, there has been instructor assisted cheating.

There are no easy answers or quick fixes to these problems. Certainly the older “thick middle management” model contained its own pitfalls and cannot be taken as a model of information clarity and inevitably sound judgment. If higher level managers wish to be kept ignorant of problems within the organization, they will accomplish this task, whether by killing the messengers bearing bad news, promoting those who tell flattering lies, or receiving “dashboards” that do not adequately convey a picture of the organization. However, those wishing to be dutiful officers and representatives of the organization as a whole, should listen to the whispered memento mori (“Remember Thou art Mortal”) and take heed.

This Years Model III

The massive (and historically cheap) increase in computing power that began in the 1950s and 60s changed the way that many previously difficult problems could be solved. In some cases, the analytical techniques had been around for a long time; cheap computing simply meant that they could be extended to larger problems, or applied to things that hadn’t been worth such analysis in times past. But there were also a number of methods of analysis that were invented during (or just immediately before) that fortuitous period: Shannon’s information theory, Danzig’s linear programming, von Neumann and Morgenstern’s game theory, and Kalman filtering, just off the top of my head.

One big chunk of real estate in this New World was labeled Operations Research. It came out of the post-war Department of Defense and more or less took the corporate world by storm, since it offered the hope (and often delivered) of untangling complicated production and logistics operations into an at least somewhat optimized operation. I learned my OR material from the RPI School of Management, along with a good stiff dose of advanced statistics.

But Systems Engineering, and its not-quite-a-cult sibling, General Systems Theory was where the really big game lived. Here was a tool kit of almost mind bogglingly powerful techniques, things like linear systems analysis, discrete time systems, transform theory, including frequency domain analysis, Laplace transforms, and on and on.

There was already a lot of history in linear systems theory, because linear systems could often be solved analytically. Moreover, any RCL circuit is a linear system, so circuit theory and electrical engineering had all these cool techniques already in the can. Mechanical engineering also had linear systems as a backbone, with every mass-spring system (and in ME, almost everything is a mass-spring system), getting the same treatment. So the science of simulations already had plenty of analog models being used, with tuned circuits often used to simulate mechanical devices, or chemical processes, or, well, other electrical devices.

All well and good, but I had my eye on non-linear systems.

Non-linear systems are where “Everything You Know is Wrong.” The phrase most often used is “counterintuitive.” What are now called chaotic systems are non-linear. In fluid mechanics, this was called “the turbulence problem,” and it had been known to be a bitch for a long, long time. What the computer revolution did was to give people a handle on just how bitchy it was.

Now linear systems can be very complicated, too complicated for a human mind to understand it, even. And linear systems can surprise you when they get very complicated as well. But in non-linear systems, even simple systems can be surprising. And it gets even worse when you connect them together.

The result is that there aren’t very many ways of analyzing non-linear systems. Most of them, in fact, consisted of making the system somehow appear to be linear, then using the linear techniques. It that didn’t work, you were stuck with the only other general method, which was modeling. Prior to electronic computers, the model was usually a real, physical model, like an aerodynamic model in a wind tunnel, or a small propeller in a tub of water.

With computers, however, you could make large scale numerical models, computer models as they were called, then just models, as the computer oracle took over the world.

I’ll talk a little about some of the math in my next essay, but I’ll end this one on a note about the sociology of it all. The thing is, computers became a magic wand. The Next Big Thing is often a magic wand in the popular imagination. That’s why Clarke got so much mileage out of his “Any sufficiently advanced technology…” aphorism. When people see something doing cool stuff, they expect it to do other cool stuff, without understanding any of it. So “computer model” became a powerful buzz phrase. It also fed into the tendency of people to project their own fantasies onto technology. The computer model became an oracle, its pronouncements inherently partaking of the mojo of science.

For myself, arrogant individual(ist) that I am, I never bought into the idea that “the model says…” “Models don’t say anything,” I would assert, “Modelers do.”

I was right, of course. As if that has ever done me any good.