ctrl variables incorrectly reporting

[Daron Chabot]

This problem was originally reported here - https://github.com/pyepics/pyepics/issues/40
There are some test results at the above link as well.

DBR_CTRL variables (units, etc) are reported correctly only on the initial connection for a channel subscription when the channel is hosted by either a Gateway or via pcaspy. This hints at a problem in libcas or libgdd.

Test code may be found here - https://github.com/dchabot/ca_client_test

This is simply a version of the caClient example code genereated via makeBaseApp, and modified to print the units of a channel.

To test, point the caMonitor executable at channel hosted by a Gateway or hosted by pcaspy.

[mdavidsaver (mdavidsaver)]

What you describe is a long standing deficiency in CA. In an attempt to address this the DBE_PROPERTY flag was introduced for use when creating monitor subscriptions. This was introduced in Base 3.14.11, it's likely you IOCs support it. Support in cagateway is a different story. I think this is in progress, but not yet released.

To make use of this you'll want to create two subscriptions. One for DBR_TIME_* with DBE_VALUE|DBE_ALARM, and a second for DBR_CTRL_* with only DBE_PROPERTY. Be prepared that the second subscription request can fail if the IOC (or gateway) doesn't recognise DBE_PROPERTY. If this happens the best you can do is fall back to polling on some long interval.

[Ralph Lange (ralph-lange)]

I don't think this is the same issue.

The reason for DBE_PROPERTY was that changes in metadata did not cause an update.

This issue is about a change in the value creating an update that contains wrong metadata.

[Daron Chabot (daron-chabot)]

Yes, thank you Ralph. That is a very succinct (and accurate!) description of the problem.

[mdavidsaver (mdavidsaver)]

Ok, verified. It's a PCAS and/or gateway issue. Direct connection to a softioc works as expected. Through cagateway the value continues to change, but units doesn't. My test situation was epics base 3.14.12.3 (debian pkg -11) w/ cagateway 2.0.6.0. I have a softioc with a calcRecord counter and change .EGU.

[Ralph Lange (ralph-lange)]

The pyepics guys reported the same behavior with pcaspy - that's why I proposed adding the bug here.

It's a PCAS issue.

[Ralph Lange (ralph-lange)]

Another verification from Wang Xiaoqiang (PSI) <xiaoqiang.wang _AT_ psi.ch>:

TL;DR: When DBE_PROPERTY event is posted in a PCAS application for an Enum typed PV, the client will be noticed but with the stale state strings.

I have create an enum typed PV “MTEST:ENUM", initially with two states [ZERO, ONE], in a PCAS(py) application.
Then this is how I expand it to 3 states and send the DBE_PROPERTY event.
// pseudo code, only to show the sequence
dd = app_table.getDD(gddAppType_dbr_ctrl_enum)
dd[gddAppTypeIndex_dbr_ctrl_enum_value].put(0)
dd[gddAppTypeIndex_dbr_ctrl_enum_enums].put([‘ZERO’,’ONE’,’TWO'])
myPV.postEvent(DBE_PROPERTY, dd)

One the client, I put a monitor with type DBR_GR_ENUM and mask DBE_PROPERTY, and print the states string in the callback.
# a.py
def valueCB(epicsArgs, userArgs):
    print(epicsArgs['pv_statestrings’])

chan1 = CaChannel('MTEST:ENUM')
chan1.searchw()
chan1.add_masked_array_event(ca.DBR_GR_ENUM, None, ca.DBE_PROPERTY, valueCB)
chan1.flush_io()
while True:
    time.sleep(2)

The result is that the callback is called for the property change event, but the state string remains the same.
$ python a.py
('ZERO', 'ONE')
('ZERO', 'ONE’)
If I then run “caget -d DBR_GR_ENUM MTEST:ENUM”, it does print the changed state strings.

However if I load such a mbbo record into IOC,
record(mbbo, "MTEST:ENUM")
{
    field (ZRST, "ZERO")
    field (ONST, "ONE")
}
 and change the state string with "dbpf MTEST:ENUM.TWST TWO”, the client will receive the event with the changed state strings.
$ python a.py
('ZERO', 'ONE')
('ZERO', 'ONE', 'TWO’)

I have doubted the PCASpy implementation. So I tested the above mbbo record through the ca-gateway(2.1.0), the result is consistent with the PCASpy application. It receives the event but with the stale state strings.

The test is with Base 3.14.12.5.

[Ralph Lange (ralph-lange)]

A suggested patch from Wang Xiaoqiang (PSI) <xiaoqiang.wang _AT_ psi.ch>:

I seem to find some clue.

The bug that is reported on launchpad can be worked around by creating a gdd of gddAppType_dbr_ctrl_double type and filling in the meta properties. This has to been do even if there is only a value update.

The bug with the enum typed PV is because casPVI caches the enum string table and only refresh it when PV gets created. So I implemented a fix to update this table during the call of postEvent.
https://github.com/xiaoqiangwang/epics-base/commit/3ca62daa6045d12cc7e71284885309310e9a062e

After these two changes, the DBE_PROPERTY event is properly received by client with updated meta property.

The test I used is here https://github.com/paulscherrerinstitute/pcaspy/blob/master/example/dynamic_property.py
Changing “MTEST:CHANGE” will cause MTEST:ENUM states and MTEST:RAND high alarm limit to change. And this change can be monitored from a client.
$ caput MTEST:CHANGE 3

$ python a.py
('ONE', 'TWO')
('ZERO', 'ONE', 'TWO')

[Ralph Lange (ralph-lange)]

Testing with the CA Gateway (commit 4a1f2b5)
https://github.com/epics-extensions/ca-gateway/commit/4a1f2b508b01d42da0758bb6c03c30db0ed9016e
shows the patch working.

[Ralph Lange (ralph-lange)]

Note that the ENUM issue found by Wang Xiaoqiang (PSI) is only a part of the problem (or maybe even a separate one).

I added another test to the CA Gateway (commit c6f69c7)
https://github.com/epics-extensions/ca-gateway/commit/c6f69c7babe0337ff2cd0b14a290f5b674433fb0

that shows the originally reported bug, and is not fixed by the attached patch.

[Ralph Lange (ralph-lange)]

Other than my first statements, I have come to the conclusion that the problem should not be fixed in CAS or GDD, but in the server applications.

CAS itself does no caching of data that is posted by the server app. It keeps a prototype GDD of the structure that the client requested, and whatever GDD the server app posts is smart-copied into a clone of the prototype and pushed to the client.
read() operations give the prototype clone (e.g. a ctrl container) to the server app to fill in. That's why the first update on setting up the connection is correct: in that case CAS does a read(), and the server app fills the complete structure. If the server app then posts e.g. value/time/status GDDs, the smart-copy to the prototype clone leaves all other elements empty - the DBR_CTRL structure only gets the posted elements, the remaining are zero or empty.
The server app has no hint if a CAS issued read() is originating from a client-side read or a subscription request.

Adding a merging cache inside CAS would add a lot of complexity and threaten the stability of CAS.
It might be easier to have the server app do whatever is needed, and have it always post complete containers (super set of DBR types), so that CAS's smart-copy in any prototype clone GDD will always fill the requested structure completely.

[Ralph Lange (ralph-lange)]

Noted that Wang Xiaoqiang (PSI) already fixed the issue in PCASpy 0.6.1 (https://pcaspy.readthedocs.io/en/latest/news.html)

I will do something similar in the Gateway.

[Ralph Lange (ralph-lange)]

The issue with the enum string table cache not being updated, reported by Wang Xiaoqiang (PSI) who also provided a fix, is indeed a separate issue.

I created https://bugs.launchpad.net/epics-base/+bug/1612944 to track it.