| Network Working Group | B. Gilbert |
| Internet-Draft | HJ. Cho |
| Intended status: Standards Track | A. Goode |
| Expires: February 08, 2013 | J. Harkes |
| L. Huston | |
| W. Richter | |
| M. Satyanarayanan | |
| R. Sukthankar | |
| August 9, 2012 |
Diamond Protocol
diamond-protocol
The Diamond system is a framework for interactive search of non-indexed content such as medical images. The Diamond system is centered around an open source implementation named OpenDiamond. The Diamond protocol defines the communication mechanisms between the Diamond server and the client. The Diamond protocol carries data serialized according to the RFC 4506 format, and uses a custom Remote Procedure Call (RPC) protocol.
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
This Internet-Draft will expire on February 08, 2013.
Copyright (c) 2012 IETF Trust and the persons identified as the document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (http://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.
The OpenDiamond system is a client-server search system that enables interactive search of Internet repositories that store vast amounts of complex and non-indexed data, such as webcam photographs and medical images. These data are referred to as objects, and a user can configure a search to return only the objects that meet some user-specified requirements. The user does this by requesting the OpenDiamond servers to run user-provided executables against the objects. For each executable the user also provides string parameters, a binary argument and various other configuration parameters (see Section 4.1.1 for a detailed list). The executable is called a filter, and the binary argument is called a blob argument; together, these are called blobs. At runtime, each filter produces a floating-point value called a score; for an object to pass the filter, its score must be within the range specified by the minimum and maximum threshold values in the configuration parameters.
Before initiating a search, a user must first define the scope: the set of objects that she desires to execute the search on. The user does this through the scopeserver, one of the main components of the OpenDiamond system. Using a web browser, the user connects to the scopeserver, authenticates, and selects a set of objects. Optionally, she can instruct the scopeserver to further restrict the set based on arbitrary object metadata. Once a scope has been selected, the user downloads a set of scope cookies, which are the entities that define the scope of objects to be searched.
Once the user obtains the scope cookies, she can invoke a client application to send the scope cookies to the servers, and the servers use the cookies to define the search scope. The client application also sends the filter configurations, which include a list of blob signatures that the servers use to determine whether they have all the blobs that are necessary to execute the search. After the client application has sent the blobs that are missing, the servers are ready for the search.
To initiate the search, the client application specifies the properties of objects that a user desires by sending the servers the names of a set of desired attributes. Attributes are named binary property values associated with an object, and the servers can use them to return properties of objects, such as thumbnail images, to the user. Attributes are typically generated by the filters while an object is being examined.
Upon the search request, the servers query the dataretrievers, which are essentially proxy servers that retrieve data from Internet repositories, to return the objects specified by the scope cookies. Once the servers obtain the objects, they carry out the selection process using the filters, and they return to the user the requested attributes for the objects that pass the filters. Typically in a search, a user will only request a subset of all the attributes, such as a small thumbnail image and the object name. Users can then request a more detailed set of information including more attributes. This process is referred to as reexecution.
The servers maintain double-precision floating-point values called session variables that are specific to a particular search. A filter can use these variables to keep track of typical parameter values for objects encountered during the search. This allows the filter to detect significant differences between the object it is currently examining and the objects that have already been examined. The OpenDiamond system is a distributed system, and by nature, the states across all servers can differ. To prevent the servers from becoming too far out of sync, the client can periodically retrieve the session variables, merge their values together, and send the combined values to all of the servers.
The remainder of this RFC is organized as follows. Section 1.2 briefly describes the OpenDiamond wire protocol. Section 1.3 describes the basics of requesting object attributes. Section 1.4 describes scope cookies. Section 2 describes the nonce exchange process that is required to begin a search. Section 3 describes the structure of an OpenDiamond RPC. Section 4 describes the protocol RPCs. Section 5 discusses system integrity and issues associated with security. Appendix Appendix A provides a reference guide of common Diamond terminology. Appendix Appendix B lists commonly-used Diamond attributes and attribute types. Appendix Appendix C lists server and filter-specific statistics values.
OpenDiamond uses a custom XDR-based Remote Procedure Call (RPC) protocol in a client-server model. Similar to a regular function or a procedure call, all operations are synchronous; when a client makes a request, it waits until the results of the remote procedure it requests are returned.
To perform a search, a client establishes two TCP connections with the server: the control connection and the blast connection. As the server can perform multiple simultaneous searches for multiple users, there must be a way to associate the control and blast connections of the same search. The two connections are therefore paired using an exchange of random nonces. The blast channel is used for transferring objects found during a search, and the control channel is used for all other client-server interactions.
Through the control channel, a client can use a set of RPCs to request searches, manipulate session variables or retrieve server statistics. The client typically begins a search by using the setup RPC (Section 4.1.1) to send the scope cookies and filter configurations to the server. The filter configuration does not include a copy of the filter code or blob arguments, as these are typically quite large; instead, a SHA-256 signature of this data is sent. The server replies with a list of filters and blob arguments it does not have. The client then uses the send_blobs RPC (Section 4.1.2) to send the missing filters and blob arguments. After these initial steps, the server is ready to perform a search. The client can then call the start RPC (Section 4.1.3) to initiate the search, and/or the reexecute RPC (Section 4.1.4) to obtain more information on a particular object. The client can also use the statistics RPC (Section 4.1.5) to obtain statistical information regarding the server, such as the average processing time of objects. If session variables are supported by the filters being used, the client can retrieve and update session variables with the get_session_variables (Section 4.1.6) and set_session_variables (Section 4.1.7) RPCs.
The blast channel is used for transferring search result objects. A client can call the get_object RPC (Section 4.2.1) to request an object. The get_object RPC does not have to be used synchronously, and the client will typically pipeline multiple get_object RPCs to minimize the effect of round-trip latency.
When the search has completed the server returns an object containing no attributes. The client MAY terminate the search at any time by closing the control and blast connections.
Below is a typical protocol interaction in a regex-like notation:
setup send_blobs? ((reexecute (send_blobs reexecute)?)+ |
(start (statistics | (get_session_variables set_session_variables))*))
A client may specify a list of attributes when starting a search or reexecution. This list is called the push attribute list and it serves to restrict the set of attribute values that will be returned: the server returns the keys of all object attributes, but only returns values for those attributes included in the push attributes list. If no push attribute list is specified, the server returns values for all attributes. As a special case, the server MUST always return a value for the _ObjectID attribute.
OpenDiamond servers use cookies for two purposes: for authorization and to specify the objects to search. A scope cookie includes a cryptographic signature, and the servers MUST verify that this signature is from a trusted public key before executing a search. Scope cookies also contain the URLs of scope lists, which enumerate the objects to be searched.
Scope cookies are Base64-encoded, and the beginning and the end of a scope cookie are delimited by the strings:
-----BEGIN OPENDIAMOND SCOPECOOKIE-----
-----END OPENDIAMOND SCOPECOOKIE-----
The Base64-decoded contents of a scope cookie can be represented as follows:
scope-cookie = signature LF data
signature = 1*(DIGIT / "a" / "b" / "c" / "d" / "e" / "f")
data = header LF body
headers = *(header LF)
(* Each header must appear once only *)
header = "Version:" version |
"Serial:" random-uuid |
"Expires:" expiration-time |
"Servers:" servers-list |
"Blaster:" URL
servers-list = server-address *((";" | ",") server-address)
body = scopelist-url *(LF scopelist-url) [LF]
Scope cookie headers are defined as follows:
A control connection and a blast connection are paired with a handshake using an exchange of random nonces. As soon as the socket connection is established for the control connection, the client MUST send a sequence of 16 null bytes as a nonce. The server MUST respond with a random nonce of 16 characters. The client uses this random nonce to establish the blast connection. As soon as the socket connection is established for the blast connection, the client MUST send this random nonce and the server MUST respond with the same random nonce.
A nonce is a sequence of 16 random bytes. Below is the XDR representation of a nonce, serialized according to the RFC 4506 format:
typedef opaque nonce[16];
Diamond client-server messages consist of requests from client to server and responses from server to client. The requests and responses consist of a header and an optional body depending on the RPC. Request and response messages are serialized according to the RFC 4506 format. Below is the generic protocol message in the XDR data description language.
struct message
{
int sequence_number;
int status; /* See Section 3.3 for status code definitions */
int command; /* See Section 3.4 for command code definitions */
opaque message_body<>;
};
Request messages are signified by a status code of MINIRPC_PENDING (Section 3.3).
A successful response message MUST have the status code of MINIRPC_OK (Section 3.3). A non-zero status code indicates an error. Negative codes indicate errors at the RPC protocol layer, and positive codes indicate errors in the application layer. The status code MINIRPC_PENDING MUST NOT be used in a response message. Error responses MUST have an empty message body.
Each request MUST have a unique sequence number. The sequence number of the server response MUST match the sequence number of the request.
The status element signifies the status of a particular RPC. Below is the list of status codes:
enum status_code
{
MINIRPC_OK = 0, /* request processed successfully */
MINIRPC_PENDING = -1, /* denotes that a message is a client request */
MINIRPC_ENCODING_ERR = -2, /* encoding error due to bad XDR structure */
MINIRPC_PROCEDURE_UNAVAIL = -3, /* requested remote procedure not available */
MINIRPC_INVALID_ARGUMENT = -4, /* request message body contains invalid value */
DIAMOND_FAILURE = 500, /* no search scope, filters configured; cookie invalid */
DIAMOND_FCACHEMISS = 501, /* requested object is not cached */
DIAMOND_COOKIE_EXPIRED = 504, /* scope cookies have expired */
DIAMOND_SCHEME_NOT_SUPPORTED = 505 /* server does not support blob uri scheme */
};
The command element indicates the remote procedure to execute. Below is the list of command codes:
enum control_command_code
{
setup = 25, /* used for search configuration (Section 4.1.1) */
send_blobs = 26, /* used to send filters and blob arguments (Section 4.1.2) */
start = 28, /* used to initiate search (Section 4.1.3) */
reexecute = 30, /* used to request reexecution (Section 4.1.4) */
statistics = 29, /* used to request server statistics (Section 4.1.5) */
get_session_variables = 18, /* used to request session variables (Section 4.1.6) */
set_session_variables = 19 /* used to update session variables (Section 4.1.7) */
};
enum blast_command_code
{
get_object = 2 /* used to request objects (Section 4.2.1) */
};
The setup RPC is used to configure the server for a search. In this RPC, a client sends the server the following entities to be used for search configuration:
The server uses the scope cookies to define the set of objects to be searched.
Each filter configuration contains the following:
The filter and blob argument URIs have the following format: sha256:<hexadecimal SHA-256 sum>.
The server MUST respond with a list of blob URIs that are not present in its cache.
The setup request body uses the following format:
typedef string cookie<>;
typedef string argument<>;
typedef string dependency<>;
struct filter_config
{
string name<>; /* filter name */
argument arguments<>; /* arguments to the filter */
dependency dependencies<>; /* list of other filters that the filter is dependent on */
double min_score; /* lower threshold to pass filter execution */
double max_score; /* upper threshold to pass filter execution */
string code<>; /* filter uri */
string blob<>; /* blob argument uri */
};
struct setup_request_body
{
cookie cookies<>;
filter_config filters<>;
};
The setup response body uses the following format:
typedef string uri<>;
struct setup_response_body
{
uri uris<>;
};
The send_blobs RPC is used to transmit filters and blob arguments to be added to the server's cache.
The response from the server MUST have an empty body.
The purpose of send_blobs is to send filters and blob arguments that are missing in the server. Therefore send_blobs RPC SHOULD only be used after the setup RPC, and only if the server indicates that there are missing filters and blobs.
The send_blobs request body uses the following format:
typedef opaque blob<>;
struct send_blobs_request_body
{
blob blobs<>;
};
The send_blobs response body MUST be empty.
The start RPC is used to request that the server initiate a search. In this RPC, a client sends to the server the following entities to be used for the search.
The search ID is used to correlate searches across multiple servers; each server used in a search MUST receive the same search ID. The search ID MUST be a UUID in canonical string representation, and SHOULD be unique to the requested search over the lifetime of the scope cookies used.
When the start RPC is invoked, the server begins evaluating the objects in the scope and returning passed objects via the get_object RPC. The start RPC MUST NOT be called until necessary configuration has been performed using the setup and send_blobs RPCs. The response from the server MUST have an empty body.
The start request body uses the following format:
typedef string attribute_name<>;
struct start_request_body
{
opaque search_id[36];
attribute_name *attribute_list;
};
The start response body MUST be empty.
A search invoked by the start RPC evaluates every object in the scope. The reexecute RPC, in contrast, allows a Diamond client to request that the filters be executed against one particular object, to obtain additional information about it. In this RPC, a client sends to the server the following:
The object ID identifies the object to be processed, and is obtained from the _ObjectID attribute from a previous search.
A client MAY specify an object ID consisting of the string "sha256:" followed by an SHA-256 hash value in hexadecimal. If an object matching this checksum is stored in the server's cache, the server MUST perform reexecution on this object. Otherwise, the server MUST respond with a DIAMOND_FCACHEMISS error. The client MAY then call send_blobs to send the object and then retry the reexecute RPC.
If the requested object is dropped by the filters, the server MUST return only the _ObjectID attribute.
The reexecute request body uses the following format:
struct reexecute_request_body
{
string object_id<>;
attribute_name *attributes_list; /* see Section 4.1.3.1 for attribute_name */
};
The reexecute response body uses the following format:
struct attribute
{
string name<>;
opaque data<>;
};
struct reexecute_response_body
{
attribute attributes<>;
};
The statistics RPC is used to request that the server return statistical information about the search.
The statistics request body MUST be empty.
The statistics response body uses the following format:
struct stat
{
string name<>; /* name of statistic */
hyper value;
};
struct filter_stat
{
string name<>; /* name of filter */
stat stats<>; /* list of statistics of this filter */
};
struct statistics_response_body
{
stat stats<>; /* list of statistics associated with search */
filter_stat filter_stats<>; /* filter-specific statistics */
};
A client can retrieve the server's session variables by calling the get_session_variables procedure.
The get_session_variables RPC MUST have an empty request body.
The client MUST NOT call get_session_variables twice without an intervening set_session_variables call.
The get_session_variables request body MUST be empty.
The get_session_variables response body uses the following format:
struct session_variable
{
string name<>;
double value;
};
struct get_session_variables_response_body
{
session_variable session_variables<>;
};
A client can use the set_session_variables RPC to modify the server's session variables.
The server MUST update each specified session variable to contain the specified value plus the difference between the current value of the variable and its value at the time of the most recent get_session_variables RPC.
The response to this RPC MUST have an empty message body.
The set_session_variables request body uses the following format:
struct set_session_variables_request_body
{
/* See Section 4.1.6.1 for session_variable definition */
session_variable session_variables<>;
};
The set_session_variables response body MUST be empty.
The get_object RPC is called on the blast connection to request an object that has passed the filters. This RPC will block until an object is ready. If no get_object RPCs are pending and the server produces a search result, the server MAY block, MAY queue the object and continue executing the search, but MUST NOT drop the search result. Multiple get_object requests MAY be pipelined so that the effect of round-trip latency between object requests can be minimized.
The client MAY perform the get_object RPC at any time after the blast connection is established. The server MUST indicate that the search has completed by returning an object containing no attributes on the blast channel. If the client performs further get_object RPCs after the completion of the search, the server's response is undefined.
The attributes included in the object struct MUST comply with the rules specified in Section 1.3. The list of push attributes is taken from the start RPC.
The get_object request body MUST be empty.
The XDR representation of a Diamond object is shown below:
struct object
{
attribute attributes<>; /* See Section 3.1.4.2 for attribute definition */
};
The OpenDiamond system allows users to run arbitrary x86 executable filter code on the servers. Although this feature makes the OpenDiamond platform a versatile search system, it also poses a major threat to server integrity. Server implementations must take care to execute filter code in a restricted sandbox to minimize the damage that can be caused by rogue filters.
Most attributes are generated by filters while an object is being examined, but some attributes are generated directly by the server. These include:
By convention, Diamond attribute names include a suffix that indicates the type of data stored in the value. This allows the client to implement generic display handlers for different types of data.
struct RGBImage
{
uint32_t type; /* RGBImageType */
uint32_t nbytes; /* size of this struct */
int32_t height;
int32_t width;
struct RGBPixel data[0];
};
struct RGBPixel
{
uint8_t r;
uint8_t g;
uint8_t b;
uint8_t a; /* mainly just padding */
};
enum RGBImageType
{
IMAGE_UNKNOWN = 0,
IMAGE_PBM = 1,
IMAGE_PGM = 2,
IMAGE_PPM = 3,
IMAGE_TIFF = 4,
IMAGE_JPEG = 5,
IMAGE_PNG = 6
};
struct __attribute__((packed)) patches
{
int32_t num_patches;
double distance;
struct patch patches[0];
};
struct patch
{
int32_t min_x;
int32_t min_y;
int32_t max_x;
int32_t max_y;
};