Bulk endpoint for working with resources

[Jim Baker]

For both setting and getting resources, including their variables, it can be valuable to provide a bulk resources endpoint.

Presumably this is not an antipattern for REST APIs. Regardless, it still is very useful for real usage.

One possible schema starts with something like this payload for setting the values of resources in the Craton API server, which then constructs the corresponding object graph; and serializes it to the database.

[{resource_type: <type>, resource_id: <id>, [attribute: value]*, variables: <json>}, ...]

Perhaps this can be done via JSON PATCH? A good design that meets this bulk usage is what we need; design is flexible as long as it is reasonably easy for client usage to build the necessary payload. Other questions include making this a general update mechanism, so variables/labels/RBAC role assignments can be appropriately manipulated.

Lastly most bulk loads like this for databases are done in a sequence of batch updates. Similarly, this should be retrievable via pagination. Standard queries should apply for getting such resources as well; eg, via a region, label, var search, etc.

See discussion starting here:
http://eavesdrop.openstack.org/irclogs/%23craton/%23craton.2017-02-03.log.html#t2017-02-03T17:50:00

[Jim Baker (jimbaker)]

We can also use this to remove the existing ansible-inventory endpoint: we can push its functionality for building the Ansible formatting specifics into client code, while retaining its functionality for bulk reads out of the db.

[Thomas Maddox (thomas-maddox)]

I think it would make sense to convert this to a BP, especially since we're pursuing a spec for it. Thoughts?

[Thomas Maddox (thomas-maddox)]

Some initial thoughts...

POST /resources

{"resources": [
  {
    "resource_type": "region",
    "resource": {...},
  },
  {
    "resource_type": "cell",
    "resource": {...}
  }
]}

PATCH /resources

[
  {"op": "add", "path": "/cells/1/name", "value": "cellone"},
  {"op": "remove", "path": "/cells/2"},
  ...
]

[Thomas Maddox (thomas-maddox)]

Are we considering PATCH because it's the OpenStack API WG standard, or because we think it'll help? I ask because partial-PUTs are generally easier to work with, and the industry as a whole uses them since there's not much use for the HTTP standard use of the PUT verb.

[Thomas Maddox (thomas-maddox)]

Bulk changes could look like:

PUT /resources

{
  "resources": [
    {
        "resource_type": "region",
        "data": {
            "id": 1,
            "name": "regionone"
        }
    },
    {
        "resource_type": "cell",
        "data": {
            "id": 3,
            "region_id": 1,
            "name": "celltwo"
        }
    },
    ...
}

This would be updating the name for the first region, and updating the region reference and name for the cell.

[Thomas Maddox (thomas-maddox)]

Maybe move the "id" attribute up one level so you could conceivably place the same request to a PUT on the resource itself in the "data" field.

{
  "resources": [
    {
        "resource_type": "region",
        "resource_id": 1,
        "data": {
            "name": "regionone"
        }
    },
    {
        "resource_type": "cell",
        "resource_id": 3,
        "data": {
            "region_id": 1,
            "name": "celltwo"
        }
    },
    ...
}

[Jim Baker (jimbaker)]

We can move dev into a companion BP and corresponding spec, that makes the most sense to me.

I like the idea of keeping the PUT the same for a list of resources as it would be for a given resource as discussed in #6. It's also possible that PUT will be the preferred way to do this implementation in general. Note that the bulk update as I'm envisioning it is already closer to PUT anyway, since it's looking at resources as individual records, vs some other way of serializing the object graph.

Re JSON PATCH: given that we expect most usage will be through the Python client, following this standard should be fine in principle. But generating a suitable patch is likely to be nontrivial.

Lastly, one issue that was raised by Ian was whether this should be async or not. If we are doing it chunks from the client, this point seems to be moot; it certainly simplifies the server implementation. We could always expose an async proxy to the API server if that ever makes sense.

[git-harry (git-harry)]

Based on the description, the priority of this task seems high. What would this be doing that a bash for-loop can't do?

If looking to do a bulk import, I would have thought most of the work comes from converting the data from its existing format to one craton recognises. Once it's in that format, using it with the existing tools should be fairly simple. It would be helpful to see an example of what has been tried so far and found inadequate.

If there is a strong user story, that is solved by bulk imports, what about simply using the client to iterate over the data and make a series of requests? I would prefer us to look at simple solutions using the existing API and only extend it when there is a good justification.

[Thomas Maddox (thomas-maddox)]

I think that's a fair point. Also, thinking about it, there are some snags when it comes to communicating that the update or import is complete. Depending on the size of the update, it could take a fair amount of time to complete. How does the end-user know it's done and one can expect those variables to be set properly before doing something with those resources? We'd either have to provide this more as a Job concept, or the end-user would have to verify the data (looping over it anyway). Thoughts?

We can probably go to antonym for the story and details around shortcomings of current solutions?

[Thomas Maddox (thomas-maddox)]

To clarify, how can one expect the resources to be created and in the expected state without either us communicating via our API that the bulk operation is complete (so they'd watch a URL, I guess?) or they'd have to loop over said resources to confirm they're how they want them to be... by the time they're doing that, we might as well have looped over the resources creating them.

[Ian Cordasco (icordasc)]

> If there is a strong user story, that is solved by bulk imports, what about simply using the client to iterate over the data and make a series of requests? I would prefer us to look at simple solutions using the existing API and only extend it when there is a good justification.

Harry, the justification is as follows:

An operator with an existing inventory of several thousand items wants to adopt craton. They aggregate that information in a way and then they use cratonclient to import it... one request at a time.

Let's say, very opportunistically, that each request takes 100 milliseconds. That import will not be fast. Further, network performance between the operator and Craton will like mean that a single request might be closer to somewhere between 500 milliseconds and 1 second. If the operator has 10,000 devices (10,000 * .5s) = 5,000s ~= 83.33 minutes or nearly 1.5 hours. Compare that to a bulk import of the same data which would generate one (but probably more depending on proxies, while being probably less than 100) requests to do the same. Each one would reply with a 202 Accepted immediately and then allow the operator to poll for the status of the bulk import. Keep in mind that this is a very simple transformation and batch update of the database so that change will probably finish quickly. If we assume that it would take .5s (500 milliseconds) for the first request and two more (at most) requests to determine that the import is complete we have 1.5s for something on the order of hundreds or thousands of devices as compared to 1.5s for 3.

The performance benefit for our users is clear.

[git-harry (git-harry)]

If an operator has to do a one-time import that takes some time to run, I don't necessarily see that as a major blocker.

What would the rate limit look like for an API request that can add thousands of devices in one go, plus potentially tens of thousands of variables and labels?

Your example of using the existing API assumes sequential requests, what's to stop the client making concurrent connections?

Clearly users will always want something that can be done faster. What I'd like to be clear on is what is fast enough. So for example, if we:

- defined a format the client would accept for the bulk import of data
- modified the client to process that file and make concurrent requests to upload the data so that now an import of ten thousand devices took less than ten minutes.

How would that measure against operator requirements? What is the minimum acceptable speed of import? How would the development time for the client compare to modifying the API and client to make a bulk import endpoint?

Would that provide the ability for us to iterate on the solution in a simpler fashion?

I am not opposed to the idea of a bulk import API, I just question how important it is. I suspect it would be a lot simpler and quicker to do something in the client that would get most of the benefit. This would not prevent adding a bulk import API at a later date but simply defer it until it was relatively more important, on the assumption the client solution was found wanting.

[Ian Cordasco (icordasc)]

> What would the rate limit look like for an API request that can add thousands of devices in one go, plus potentially tens of thousands of variables and labels?

This is an inventory service. We have no concept of rate-limiting, nor would we ever grow one. The rate-limiting would only be able to apply to a single API service if we do that, not all deployed instances of the service (unless we used something external to Craton). So this question is moot. Plus the limit would be based on whatever sits between Craton and the User. Many tools (e.g., Apache, Nginx, etc.) have request body size limitations. That would effectively create the ratelimit you want.

> what's to stop the client making concurrent connections?

I would operate under the assumption that you know nothing of Python's concurrency issues and explain to you in detail why that would be unrealistic, but I won't. I know you're more intelligent than that. What I don't know is why you're wasting time asking this.

> concurrent requests to upload the data so that now an import of ten thousand devices took less than ten minutes.

Concurrent requests in Python would not take less than 10 minutes for something that size. Again, I won't go into the details of Python's concurrency or why it wouldn't work here.

> I just question how important it is.

That is actually *not* what you're doing in this thread. If the question is of importance, than keep in mind that this has already been de-prioritized despite specific requests for the functionality from one specifically large group of potential users.

> If an operator has to do a one-time import that takes some time to run, I don't necessarily see that as a major blocker.

This is not one-time use functionality. As operators set up new regions, cells, etc. they would likely prefer to do one bulk import rather than a several minute import. The bulk endpoint for import will not be used frequently, but the bulk export capability will absolutely be used regularly to provide inventory to configuration management tools and to replace the bit-rotting /ansible_inventory endpoint.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/432283