when subscribing for zero elements, array element count should be dynamic

[Jeff Hill]

> You just stumbled over what I think is one of the most irritating
> mis(sing)-features of EPICS as it stands: support for array fields is very
> weak. Particularly it is limited to arrays which are of fixed size at
> run-time.

Our current situation:
The server does learn the element count form convert_db_addr when the
channel
connects, and it doesn’t ask for the current array length when sending
subscription updates. We also have been careful about changing the array
length to other than what the client has specified when issuing the IO
request
as this might break backwards compatibility (responsible API design etc).

In the short term:
Clients _are_ currently allowed to _not_ specify the element count when
they subscribe (they can specify an element count of zero). So perhaps
the CA code should be changed allowing the array length to be dynamic in
just
that situation. Currently, we are locking the element count for that
situation
to what was there when the channel connects, but perhaps this is wrong
(lazy)
etc. Any opinions from Andrew or others? We would need an efficient way to
query the current element count before each call to db_get_field. The way to

proceed is to first get general agreement on this being a defect, and next
to
install a Mantis entry against R3.14 or R3.15 so that we will remember to
attend to the matter.

For the future:
This is certainly one of the more complex issues when designing the
programming
interfaces. IMHO, the client _should_ have an option to rigidly specify how
many elements will be received, but should also have an option to _not_
specify
the element count and just receive however many elements happen to be
there. Clients should probably have flexibility to specify, or not, for each

bound on the hypercube (multidimensional array). And, of course, also a first element number for
each bound on the hypercube - so that one can define a
slice.

Am I off base on that as a
future direction for the API design? Open to suggestions.

Original Mantis Bug: mantis-368
    http://www.aps.anl.gov/epics/mantis/view_bug_page.php?f_id=368

[Jeff Hill (johill-lanl)]

From Andrew:

On Monday 05 October 2009 10:24:25 Jeff Hill wrote:
>
> In the short term:
> Clients _are_ currently allowed to _not_ specify the element count when
> they subscribe (they can specify an element count of zero). So perhaps
> the CA code should be changed allowing the array length to be dynamic in
> just that situation. Currently, we are locking the element count for that
> situation to what was there when the channel connects, but perhaps this is
> wrong (lazy) etc. Any opinions from Andrew or others?

I agree with Michael in favor of changing the behavior so that when a client
specifies COUNT=0 in its subscription, it is requesting a dynamic element
count such that successive events may have different numbers of elements.
However it should never be given more than ca_element_count() elements so it
can always use that number to allocate data storage at connection time. If at
some time in the future we want to make it possible for even that number to be
increased, this could be indicated to the client by a pair of disconnect/
reconnect events.

Looking at my Perl 5 API code, I don't currently allow the user to specify
COUNT=0 but if I change just that the above behavior should work perfectly
since I use peha->count in the callback routine to allocate the array storage.
I suspect the same would be true of a number of other clients.

It would be interesting to consider whether doing a ca_array_get_callback()
with COUNT=0 should copy the new subscription behavior and return all of the
current elements as well. I did wonder whether that should return the current
number of elements without fetching all the data, but I don't think it should,
the problem being that users might assume they can use that to allocate the
storage for the data, but the IOC might change the data length in between that
request and the following one that actually gets the data.

> We would need an efficient way to
> query the current element count before each call to db_get_field.

That might get slightly complicated. The dbGetField() routine that it uses
calls dbScanLock(precord) at the beginning and dbScanUnlock(precord) at the
end, but we can't allow the record to be processed in between fetching the
size and the actual data. I really don't like the idea of putting a wrapping
dbScanLock() call in some new db_get_nelements() routine and the corresponding
dbScanUnlock() in the db_get_field() routine.

I think the best solution would be to change the int no_elements parameter of
db_get_field() into an int* and change its value inside the routine, just like
dbGetField() does. This would still require the server to allocate the buffer
for the full array before entry (which it already has to do), but it could
reclaim the unused part immediately afterward. Note that the task of padding
the remaining buffer with zeros which db_get_field() currently does could even
be moved into the client library (if the client asked for a fixed length
subscription and not the dynamic one that we're discussing).

> Clients should probably have flexibility to specify, or not, for each
> bound on the hypercube (multidimensional array). Am I off base ...