Tops Read
How a problem got resolved when the the problem statement was changed.
How a problem got resolved when the the problem statement was changed.

100 milliseconds, 22 years

1 comment On the Job

In March 1994, the the commissioning of the first unit of Talcher Super Thermal Power Project got underway. Commissioning of power plants is rather a complex, and lengthy procedure. But this story belongs to the beginning of this complex process, where the water system of the plant gets ready leading to hydro test of the boiler. The technical name of the water system is "de-mineralization system" or DM system in short. DM plant is that building of a power plant where water from local source is purified so that it becomes suitable for use in the boiler.

Why we cannot use normal drinking water in a boiler is a good question, but from an entirely separate subject. I am not a water chemistry expert, but with 27 years in power plants, I will give a shot at it and write two lines. Boilers require water with very little (almost nil) dissolved minerals and chemical properties that help in avoiding corrosion, scaling and foaming. This helps in a longer life for boiler tubes, sustained and superior heat transfer, and high quality steam generation without dissolved impurities like silica. With that said, let's get back to the story.

Annunciator FasciaSo when the water treatment system was getting commissioned, the engineers faced a peculiar problem. There was a control room with an operating desk from where all drive motors were operated. The system belonged to the pre-PLC era. On the top of the control panel was a large number of annunciation facia describing problems that the operator should be aware of immediately. When something undesirable happens, a pre designated facia with the description of the problem will start flashing, and a hooter will sound so that the operator cannot miss the alert. This system was working fine, but there was some problem with the hooter and the flashing annunciation. Whenever some equipment was started or stopped, a couple of arbitrary annunciation facia will start flashing and the hooter will go off without any apparent reason. It also happened when there was a voltage fluctuation or a welding machine starting nearby. Sometimes for no apparent reason the system will jump up and dance a little. Every time this happens, the operator has to acknowledge the alarm and reset the fault. This was considerable pain for the operator. If such a problem persists, the operator will lose all faith on the annunciation system and might become insensitive to genuine alarms. After lot of R&D (read intense troubleshooting) the problem could not be resolved. What was done during this phase of troubleshooting was trying to find out what was causing this problem. Voltage/current activities in one circuit affecting another circuit is not uncommon when the right cabling practice is not followed. All possibilities for such interference was thoroughly investigated. Certain problems were found and rectified but the problem of spurious annunciation apparently reduced but did not get resolved. We could not find out what are the various events that was triggering this. Since the triggers were not known the problem remained unsolved.

Let's take a break and think for a moment, how we solve a problem. Before we start solving a problem (in fact any problem) we frame a question. This question sets the tone and tenure of the efforts that follow. And then we run behind the question looking for an answer. In trying to resolve the problem, consider the following questions.

What are the possible reasons that is triggering this problem?
This will lead us to identify the possible causes of electromagnetic induction and remove each of them. We may find many of them, may not find some of them. The answer to this question may or may not solve the problem.

What can we do to reduce the problem of the operator?
This will lead to develop some kind of automatic method of accepting and resetting the spurious alarms. This may solve the operator's pain but will generate more potential problems if genuine alarms get reset without enough diligence.

What can be done so that spurious annunciation will be eliminated (without jeopardizing the genuine alarms) even in the presence of the events that are triggering this?
We had not asked this question yet, but that was the way to go.

Now let's get back to the story.

What ever was triggering the false alarms was very short lived, and it was spurious - meaning, the equipment status was healthy but there was a trigger that should not have occurred. So we thought what will happen if we introduce a small delay - like fraction of a second - between the receipt of the trigger and the initiation of the flashing and the hooter? Two things will happen. First, all triggers (spurious and genuine) will get delayed by this duration. This is not a problem since the operator response will be same in both cases. Secondly, after the set delay, if the trigger still persists, it will be allowed to generate flashing and hooter. If during the delay, the false trigger dies down, it cannot generate the alarm. That way we are changing the very definition of a trigger. A trigger will be considered a trigger only if it lasts more than the delay duration. Otherwise the system will consider this as a short lived trigger that never happened or came in.

circuitry annunciatorWe opened up the annunciation circuit and identified the points on the circuit board where we can put a small capacitor and introduce a small delay of about a 100 milliseconds – that is one tenth of a second. A good quality polystyrene capacitor was soldered on the circuit board. Then we tested the response for genuine triggers and could not observe any perceptible delay. And then we waited for false triggers. We waited for a couple of hours and not one false trigger came in. We told the operators to observe for a couple of days. For next 48 hours, not one trigger came in. Then after a week we assumed the problem solved and forgot the episode.

Many years later when I visited the project again, I made it a point to visit the DM plant to see if the problem has reappeared ever. No one even remembered that there was any such problem. I imagined, the little capacitor sitting in its place must be doing its job exactly as we designed, year after year for 22 years.

Last modified onTuesday, 09 August 2016 18:41
(1 Vote)
Read 5794 times
  • What ever was triggering the false scott properties was very short lived, and it was spurious - meaning, the equipment status was healthy but there was a trigger that should not have occurred

Post comment as a guest