Manufacturing Reflections

Last week NPR interviewed the author of Why Bad Things Happen.  In an automation world we diagnose causes to improve prevention, but ultimately without going into MPC religion, “stuff happens”.  My engineering brain focuses on remedies.

Part of the solution is good control design – spending the time to figure out what can happen.  Look at reams of past data (electronically) and test a control design against significant process history, interview experienced and not-so-experienced operators about what they worry about, watch a lot (and why) and compare a problem with similar solution designs in your or peers experience.  Make the design sustainable within the imperfect data and processing world of it’s environment.

The other step is to test the solution through each designed function and conceivable events.

1. What will the controller do if important controllers are unavailable, what will if do if a possible plant section is not operating
2. Is the controller designed to recover smoothly from loss and re-connectivity to your control system
3. Is the controller designed to recover smoothly and automatically from a computer reboot or loss/recovery of power.
4. How does the controller respond to projected bad data or flaws – can you test it with clearly bad data sent to it’s systems (like a zero to a controlled variable)

Your final catch-all is to confirm that the controller limits are appropriately set and at levels your controller can survive.  Manipulated Variable high and low, rate-of-change limits are to keep you out of the ditches and limits on the control system itself, required for plant, equipment and operator safety should be tested.  Confirm that if computer based control sends zeros out (which has happened on uninitialized controllers) that a lower-level control would, where appropriate reject or maintain it’s limits even with wrong values from a computer-based control system.

Operator training needs to include basic troubleshooting and required monitoring training.  An operator should know the basics of what your controller is designed to do and be able to reasonably question things that don’t seem right.  The operator should be able to turn off a suspicious control function (with on/off control swithes), but be required to write down the whats and whys.  This log is critical in your troubleshooting (along with uptime monitoring) and determining if a controller patch or operator training is warranted.  Then each event can rapidly be resolved and repeats avoided.

Mike T.

Many people use the term “Golden Batch” as a strategy to control batch processes.  The concept is to compare a set of batches and find the ‘good’ batches to identify the best possible recipe then set up a control system to follow this recipe to improve batch results.

As a concept – this makes simple sense.  It matches how your mother makes cake or meatloaf.  But there have been many questions and doubts about how much good it delivers.  First almost every batch runs on a recipe – there is no problem with trying to do the same thing every time and most batch control systems include a recipe manager.

But the variation from batch to batch that you are trying to ‘control’ by repeating the same recipe is infrequently caused by failing to follow your design recipe.  Most variability is caused by: feedstock quality variation, utility system upsets, and by natural bio-organism or bio-catalyst efficacy differences.  These variations cannot be impacted by an identification of a golden batch except for the case where your feedstock tends to vary in higher or lower quality than the design and your innate process variability finds a better way to run with this shifted quality.

There is another limitation with most ‘golden batch’ recipes – time is the primary driver of when to do what. Life/bio-processes grow, digest, and convert at varying rates.  This is partially because of natural bio-processing rates, but a sense-and-respond capable control system responds to these causes of variation.  A better recipe is based on batch progress not from the simpler time.

We have delivered batch-quality control on bio- and other processes, most commonly in yeast fermentation. The focus has been on taking advantage of modified Michaelis-Menten kinetic models, tuned to a batch performance history and deploying a sence and respond system looking at all available batch state information.  We found another limit of time in that for our model-based dynamic control perspective it is not realistically differentiable (we cannot make time move faster as we can with batch progression).  A key for us is that interim QC sampling is provided so that batch progression can be confirmed or corrected while still actionable. End-of-Batch sampling is informative and can support the ‘next’ batch, but we do much better with interim results measuring quality progression. A quality model, backed-up by measurement lets you run much closer to an optimal batch trajectory every batch even with biologic, feedstock and utility variability.  This provides today’s golden batch – adjusted for what is possible with the ingredients that are available.

Mike T.