Records¶
A Record
is an concurrency-safe, array-friendly view of a directory. Records support four types of data transactions: reading, writing, appending, and deleting.
Records pointing to directories created by Command
s also provide access to command metadata.
Obtaining a record¶
record = cg.Record('some/directory/path/')
Since a record is just a view into a directory, constructing it does not perform any filesystem operations. Files and directories are created as needed.
Reading entries¶
Subscripting a record with a key corresponding to an HDF5 file (minus the extension) returns an array:
array = record['file/path']
Subscripting a record with a key corresponding to a non-HDF5 file returns the file’s path (the file is presumed to be encoded in an application-specific format):
image_path = record['image/path.jpg']
Subscripting a record with a key corresponding to a directory returns a subrecord:
subrecord = record['directory/path']
Records also have dict
-style iteration methods (keys
, values
, and items
). These methods iterate over all entries in the directory corresponding to the record, with the exception of those with names that start with “_”.
Writing entries¶
Subscript-assigning can be used to write an array to a file.
record['file/path'] = array
Subscript-assigning can also be used to copy the contents of one record into another, deleting its previous contents.
record['directory/path'] = another_record
A [nested] dict
of array-like objects can also be used to tersely write to multiple files.
record['beings/animals'] = {
'dogs': {'snoopy': snoopy_data},
'cats': {'garfield': garfield_data}}
Appending to entries¶
Appending works analogously to writing, and creates files and directories as necessary.
record.append('file/path', array)
record.append('directory/path', another_record)
record.append('directory/path', dict_of_arrays)
Deleting entries¶
Deleting an entry removes files/directories recursively, from the key downward, and deletes empty parent directories, up to record.path
. (In other words, deleting performs the inverse of the “create as necessary” operations writing performs.)
del record['some/path']
Accessing command metadata¶
Records also supports reading command metadata (stored in _cmd-spec.yaml and _cmd-status.yaml) via the cmd_spec
and cmd_status
properties.
Running a data server¶
Records can also be accessed via HTTP. Currently, only GET
operations are supported. Call serve
to start a data server allowing clients to access the contents of a directory via a REST API.
# The following routes are supported:
# - /<record-path>/_entry-names
# - /<record-path>/_cmd-info
# - /<record-path>/<entry-name>
cg.serve('my-data/', port=5555)
When running the data server on a publicly accessible machine, SSH tunneling combined with a firewall can be used to prevent public data access.