by Mark Noseworthy, Group Engineering Manager
I was recently working at a client and made some DeltaV code changes on the Development system to a composite1 used in a control module. One of the code changes involved adding an Alias2 reference out to a different control module to obtain some information running in that other module concurrently. Once it was all configured and tested, the code was downloaded successfully to the Production system, and I didn’t think any more on it.
Fast forward a few months, and a colleague is modifying the same code for a different project. He was experiencing some odd behavior in the code that he couldn’t explain. He pulled in the client and me to look at it, and none of us could figure out what was happening. The phase that was running on the unit module that contained the control module with the composite that we had both updated would not run successfully the first time through the code, but if manually manipulated, the phase would run as expected subsequent to that first pass through the logic.
We troubleshot the issue by first reinstalling the code without my colleague’s changes, assuming that would fix things. It did not. We were flummoxed. Since my colleague’s logic needed to be used in Production, the client indicated that it was acceptable to put on the Production system, attributing the odd behavior as a Development system issue. Upon doing this there were no issues running in Production. Although grateful, we were doubly flummoxed now. Clearly the issue lies in the Development system, but where?
Trying to Uncover the Mystery Flaw in the Control System
Further investigation discovered a yellow question mark (which generally means some sort of error) on a CALC (calculation) block that was definitely not there when the initial code was verified. This also happens to be the location of the code update that included the Alias reference out to that other module that I mentioned previously.
I proceeded to test a few things. First, I monitored when this CALC block gets accessed, and even when not executing this CALC block, the same symptoms described above kept happening. Second, I removed the logic by commenting out the code, and the yellow question mark went away, and the code executed without the odd behavior. This was helpful, as it pointed toward the Alias reference as the culprit. But remember, this code worked without issue in Production, and also worked in Development in a previous iteration, not to mention that it was qualified code that was required, so removing it long-term was not a viable solution.
At this point, the issue was escalated again to the client. After troubleshooting without success, the client remembered that the unit module had been downloaded from the Production system into the Development system as a means of ensuring that code updates on the Development system align with the actual running logic in Production. As it turns out, this download was made around the time that these issues started appearing. Now there were two potential culprits, the unit module download and the Alias reference in the composite’s CALC block.
Limited Controllers in Dev Leads to Limited Knowledge in Prod
Production system infrastructure vs Development system infrastructure. Ofttimes, a Development system for DeltaV is a scaled back version of the Production system, due to cost and licensing constraints of both the Input/Output (I/O) capacity, the virtual controllers, and the IT networking and infrastructure capabilities. This is a business decision and is quite normal for clients with a lot of equipment.
In the case above, there are maybe a quarter of the number of virtual controllers in the Development system as there are physical controllers in the Production system. This is relevant as not all physical controllers found in the Production system are represented as virtual controllers in the Development system.
Finding the Flaw in the Automation System
Upon further investigation, it was determined that the Production system physical controller for the unit module in question was different than the Development system virtual controller. This can happen, for example, when a virtual controller is not available with the same name as the Production controller due to all virtual controllers being used up in Development, so to expedite getting things done, a change to the unit module and subordinate modules are made that change the controller reference from what is in Production.
The download of the unit module by the client actually changed the controller reference in the Development system to match Production. At this point, the bulk of the control modules were now on a different virtual controller than the unit module.
Why is this important? Because Alias resolution in class-based modules occur in the controller, meaning they do not work across multiple controllers, which is what caused the issue. Once this was realized, changing the controller reference of the unit module to match the controller reference of control module that contains the Alias reference solved the issue.
Cases where this will most likely show up are:
- The aforementioned controller mismatches when migrating code between multiple environments (Production vs Development)
- Initial system design and build, where code sharing is common across many controllers.
These types of support calls are common and a natural consequence of handing a control system over to the client after project completion. It takes teamwork and cooperation, but any problem can be solved with the right automation engineers.
1A composite is a reusable piece of common code that can be linked or embedded in a higher-level module (control module, equipment module, unit module, phase)
2An Alias is a name substitution (typically for control or equipment modules) used most commonly by unit classes when executing common phase logic that resolve at the instantiation of the unit class (the unit module).