Storlets API v1¶
Swift Storlets extend Swift with the capability to run computation near the data in a secure and isolated manner. With Swift Storlets a user can write code, package and deploy it as a Swift object, and then explicitly invoke it on data objects as if the code was part of the Swift pipeline. We use the term Storlet to refer to the binary code deployed as a Swift object. Invoking a Storlet on a data object is done in an isolated manner so that the data accessible by the computation is only the object’s data and its user metadata. Moreover, the computation has no access to disks, network or to the Swift request environment.
Operations on Storlets¶
Storlets are uploaded to a special container called ‘storlet’. Therefore, ‘/storlet’ shows up in the url of various operations, such as to upload a storlet or to list all of the available storlets.
List All Storlets¶
[GET] /v1/{account}/storlet
The body of the response contains a list of the names of the uploaded storlets. The Content-type of the response is (text/plain).
storlet1.jar
storlet2.jar
storlet3.jar
Upload a New Storlet¶
You may write a storlet in Java according to the instructions in the Developer’s manual and upload it to the Object Store. The storlet is uploaded to the container named ‘storlet’, so ‘/storlet’ appears in the url. The storlet may depend on other existing libraries, which must be uploaded to the dependency container. When uploading a storlet, the X-Object-Meta-Storlet-Dependency header requires a value that is a comma separated list of dependencies. The main_class_name parameter for the X-Object-Meta-Storlet-Main header specifies the class in which the invoke method of the storlet is defined. The X-Object-Meta-Storlet-Language header specified the language in which the storlet is run. At present, only ‘Java’ is supported. The X-Object-Meta-Storlet-Interface-Version header should be provided and set to the value ‘1.0’. Although not currently used, the X-Object-Meta-Storlet-Object-Metadata header must be provided and set to ‘no’. See the Storlets Developer’s manual for details of the signature of the invoke method. The content-type of the request should be set to ‘application/octet-stream’. Additional details and examples can be found at <https://github.com/openstack/storlets/blob/master/doc/source/writing_and_deploying_storlets.rst>.
[PUT] /v1/{account}/storlet/{storlet_object_name}
'X-Object-Meta-Storlet-Language' : 'Java'
'X-Object-Meta-Storlet-Interface-Version' : '1.0'
'X-Object-Meta-Storlet-Dependency': dependencies
'X-Object-Meta-Storlet-Object-Metadata' : 'no'
'X-Object-Meta-Storlet-Main': {main_class_name}
'X-Auth-Token': {authorization_token}
Operations on dependencies¶
Upload a New Dependency¶
You may create and upload your own libraries to assist in running a storlet. These dependencies are uploaded to a special container named ‘dependency’. For an executable dependency (such as a compiled c program), you must specify the X-Object-Meta-Storlet-Dependency-Permissions header. Otherwise, that header may be omitted. The name of the library containing the dependency (i.e. dependency_name) goes into the URI. The content-type of the request should be set to ‘application/octet-stream’.
[PUT] /v1/{account}/dependency/{dependency_object_name}
'X-Object-Meta-Storlet-Dependency-Version': '1'
'X-Object-Meta-Storlet-Dependency-Permissions' : '0755'
'X-Auth-Token': {authorization_token}
Storlets Invocation¶
Once the storlet and its dependencies are deployed the storlet is ready for invocation. Storlets can be invoked in 2 ways:
- Invocation upon GET.
In this case the user gets a transformation of the object residing in the store (as opposed to the actual object). One use case for GET is anonymization, where the user might not have access to certain data unless it is being anonymized by some storlet.
- Invocation upon PUT.
In this case the data kept in the object store is a transformation of the object uploaded by the user (as opposed to the original data or metadata). A typical use case is metadata enrichment, where a Storlet extracts format specific metadata from the uploaded data and adds it as Swift metadata.
Invocation involves adding an extra header (‘X-Run-Storlet’) to the Swift original PUT/GET requests. Additional details and examples can be found in <https://github.com/openstack/storlets/blob/master/doc/source/invoking_storlets.rst>.
Invoke a storlet upon object download¶
[GET] /v1/{account}/{container}/{object}
An additional header (‘X-Run-Storlet’) must be provided to inform the system to run a storlet.
'X-Run-Storlet': {storlet_name}
'X-Auth-Token': {authorization_token}
Invoke a storlet upon object upload¶
[PUT] /v1/{account}/{container}/{object}
An additional header (‘X-Run-Storlet’) must be provided to inform the system to run a storlet.
'X-Run-Storlet': {storlet_name}
'X-Auth-Token': {authorization_token}