Storage Service API

From Archivematica
Jump to navigation Jump to search

Main Page > Development > Storage Service API

The Storage Service API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using TastyPie.

Improvement Note: TastyPie is less well supported than Django REST Framework, both in terms of docs & community. We should look at replacing TastyPie with DRF.

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg ?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2) or as a header (eg Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2)

A note about browsing

A detailed schema can be found for each of the resources by adding "schema" to the get all URL.

Example: 

$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/location/schema/?format=json
{
   "allowed_detail_http_methods": [
       "get",
       "post"
   ],
   "allowed_list_http_methods": [
       "get"
   ],
   "default_format": "application/json",
   "default_limit": 20,
   "fields": {
       "description": {
           "blank": false,
           "default": "No default provided.",
           "help_text": "Unicode string data. Ex: \"Hello World\"",
           "nullable": false,
           "primary_key": false,
           "readonly": true,
           "type": "string",
           "unique": false,
           "verbose_name": "description"
       },
       "enabled": {
           "blank": true,
           "default": true,
           "help_text": "True if space can be accessed.",
           "nullable": false,
           "primary_key": false,
           "readonly": false,
           "type": "boolean",
           "unique": false,
           "verbose_name": "Enabled"
       },
       "path": {
           "blank": false,
           "default": "No default provided.",
           "help_text": "Unicode string data. Ex: \"Hello World\"",
           "nullable": false,
           "primary_key": false,
           "readonly": true,
           "type": "string",
           "unique": false,
           "verbose_name": "path"
       },
       "pipeline": {
           "blank": false,
           "default": "No default provided.",
           "help_text": "Many related resources. Can be either a list of URIs or list of individually nested resource data.",
           "nullable": false,
           "primary_key": false,
           "readonly": false,
           "related_schema": "/api/v2/pipeline/schema/",
           "related_type": "to_many",
           "type": "related",
           "unique": false,
           "verbose_name": "pipeline"
       },
       "purpose": {
           "blank": false,
           "default": "No default provided.",
           "help_text": "Purpose of the space.  Eg. AIP storage, Transfer source",
           "nullable": false,
           "primary_key": false,
           "readonly": false,
           "type": "string",
           "unique": false,
           "verbose_name": "Purpose"
       },
       "quota": {
           "blank": false,
           "default": null,
           "help_text": "Size, in bytes (optional)",
           "nullable": true,
           "primary_key": false,
           "readonly": false,
           "type": "string",
           "unique": false,
           "verbose_name": "Quota"
       },
       "relative_path": {
           "blank": false,
           "default": "",
           "help_text": "Path to location, relative to the storage space's path.",
           "nullable": false,
           "primary_key": false,
           "readonly": false,
           "type": "string",
           "unique": false,
           "verbose_name": "Relative Path"
       },
       "resource_uri": {
           "blank": false,
           "default": "No default provided.",
           "help_text": "Unicode string data. Ex: \"Hello World\"",
           "nullable": false,
           "primary_key": false,
           "readonly": true,
           "type": "string",
           "unique": false,
           "verbose_name": "resource uri"
       },
       "space": {
           "blank": false,
           "default": "No default provided.",
           "help_text": "A single related resource. Can be either a URI or set of nested resource data.",
           "nullable": false,
           "primary_key": false,
           "readonly": false,
           "related_schema": "/api/v2/space/schema/",
           "related_type": "to_one",
           "type": "related",
           "unique": false,
           "verbose_name": "space"
       },
       "used": {
           "blank": false,
           "default": 0,
           "help_text": "Amount used, in bytes.",
           "nullable": false,
           "primary_key": false,
           "readonly": false,
           "type": "string",
           "unique": false,
           "verbose_name": "Used"
       },
       "uuid": {
           "blank": true,
           "default": "",
           "help_text": "Unique identifier",
           "nullable": false,
           "primary_key": false,
           "readonly": false,
           "type": "string",
           "unique": true,
           "verbose_name": "uuid"
       }
   },
   "filtering": {
       "pipeline": 2,
       "purpose": 1,
       "quota": 1,
       "relative_path": 1,
       "space": 2,
       "used": 1,
       "uuid": 1
   }
}

This schema, among other things, describes the fields in the resource (including the schema URI of related resource fields) and the fields that allow filtering. Valid filtering values are: Django ORM filters (e.g. startswith, exact, lte, etc.) or 1 or 2. If a filtering field is set to 2 it can be filtered over the related resource fields. For example, the locations could be filtered by their pipeline UUID setting it in a request parameter formatted with two underscore chars: /api/v2/location/?pipeline__uuid=<uuid>

For more info on how to interact with the API see:

http://django-tastypie.readthedocs.io/en/v0.13.1/interacting.html

Pipeline

Get all pipelines

  • URL: /api/v2/pipeline/
  • Verb: GET
  • Parameters: Query string parameters
    • description: Description of the pipeline
    • uuid: UUID of the pipeline
  • Response: JSON
    • meta: Metadata on the response: number of hits, pagination information
    • objects: List of pipelines. See #Get pipeline details for format

Returns information about all the pipelines in the system. Can be filtered by the description or uuid. Disabled pipelines are not returned.

Example: 

$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
    "meta": {
        "limit": 20,
        "next": null,
        "offset": 0,
        "previous": null,
        "total_count": 1
    },
    "objects": [
        {
            "description": "Archivematica on alouette",
            "remote_name": "127.0.0.1",
            "resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
            "uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
        }
    ]
}

Create new pipeline

  • URL: /api/v2/pipeline/
  • Verb: POST
  • Parameters: JSON body
    • Should contain fields for a new pipeline: uuid, description, api_key, api_username
    • create_default_locations: If True, will associated default Locations with the newly created pipeline
    • shared_path: If default locations are created, create the processing location at this path in the local filesystem
    • remote_name: URI of the pipeline.
      • Before v0.11.0: If create_default_locations is set, SS will try to guess the value using the REMOTE_ADDR header.
      • In v0.11.0 or newer: If not provided, SS will try to guess the value using the REMOTE_ADDR header.
  • Response: JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example: 

$ curl -X POST \
   -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" \
   -H"Content-Type: application/json" \
   -d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", \
       "description": "Test pipeline", "create_default_locations": true, \
       "api_username": "demo", \
       "api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' \
    'http://192.168.1.42:8000/api/v2/pipeline/'
{
   "create_default_locations": true,
   "description": "Test pipeline",
   "remote_name": "192.168.1.42",
   "resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
   "uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

Get pipeline details

  • URL: /api/v2/pipeline/<UUID>/
  • Verb: GET
  • Parameters: None
  • Response: JSON
    • description: Pipeline description
    • remote_name: IP or hostname of the pipeline. For use in API calls
    • resource_uri: URI for this pipeline in the API
    • uuid: UUID of the pipeline

Space

Get all spaces

  • URL: /api/v2/space/
  • Verb: GET
  • Parameters: Query string parameters
    • access_protocol: Protocol that the Space uses. Must be searched based on the database code.
    • path: Space's path
    • size: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django field lookups.
    • used: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django field lookups.
    • uuid: UUID of the Space
  • Response: JSON
    • meta: Metadata on the response: number of hits, pagination information
    • objects: List of spaces. See #Get space details for format

Returns information about all the spaces in the system. Can be filtered by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

Get space details

  • URL: /api/v2/space/<UUID>/
  • Verb: GET
  • Parameters: None
  • Response: JSON
    • access_protocol: Database code for the access protocol
    • last_verified: Date of last verification. This is a stub feature
    • path: Space's path
    • resource_uri: URI to the resource in the API
    • size: Maximum size of the space in bytes.
    • used: Bytes stored in this space.
    • uuid: UUID of the space
    • verified: If the space is verified. This is a stub feature
    • Other space-specific fields

Browse space path

  • URL: /api/v2/space/<UUID>/browse/
  • Verb: GET
  • Parameters: Query string parameters
    • path: Path inside the Space to look
  • Response: JSON
    • entries: List of entries at path, files or directories
    • directories: List of directories in path. Subset of `entries`.

Create space

See Issue 36.

  • URL: /api/v2/pipeline/space
  • Verb: POST
  • Parameters: JSON body
    • Should contain fields for a new space: See the Storage Service Documentation or Space for fields relevant to each type of space. Basic fields for a local file system space are listed below.
    • access_protocol: this defines the type of space
    • path: Absolute path to the Space on the local filesystem
    • size: (Optional) Maximum size allowed for this space. Set to 0 or leave blank for unlimited.

Example (to create an S3 space): 

$ curl \
    -X POST \
    -d @payload.json \
    -H "Content-Type: application/json" \
    -H "Authorization: ApiKey test:test" \
        http://127.0.0.1:62081/api/v2/space/

Where payload.json contains
{
    "access_protocol": "S3",
    "path": "",
    "staging_path": "/",
    "endpoint_url": "http://127.0.0.1:12345",
    "access_key_id": "_Cah4cae1_",
    "secret_access_key": "_Thu6Ahqu_",
    "region": "us-west-2"
}
Version 1: Returns paths as strings

Version 2: Returns all paths base64 encoded

Location

Get all locations

  • URL: /api/v2/location/
  • Verb: GET

Create new location

Added in v0.12 - see issue 367 and issue 37.

This endpoint creates a location in the storage service, but it doesn't actually create the directory that the location points to.

  • URL: /api/v2/location/
  • Verb: POST
  • Parameters: JSON body
    • description.
    • pipeline: URI of the pipeline.
    • space: URI of the space.
    • default: If 'true' this location will be the default for it's purpose.
    • purpose: (below is a list of possible values)
      • AR (AIP_RECOVERY)
      • AS (AIP_STORAGE)
      • CP (CURRENTLY_PROCESSING)
      • DS (DIP_STORAGE)
      • SD (SWORD_DEPOSIT)
      • SS (STORAGE_SERVICE_INTERNAL)
      • BL (BACKLOG)
      • TS (TRANSFER_SOURCE)
      • RP (REPLICATOR)
    • relative_path: Relative to the space's path.

Example: 

curl -s -d '{
    "pipeline": ["/api/v2/pipeline/90707555-244f-47af-8271-66496a6a965b/"],
    "purpose": "TS",
    "relative_path": "foo/bar",
    "description": "foobar",
    "space": "/api/v2/space/141593ff-2a27-44a1-9de1-917573fa0f4a/"
}' \
    -X POST \
    -H "Authorization: ApiKey test:test" \
    -H "Content-Type: application/json" \
        "http://127.0.0.1:62081/api/v2/location/"

Get location details

  • URL: /api/v2/location/<UUID>/
  • Verb: GET

Move files to this location

  • URL: /api/v2/location/<UUID>/
  • Verb: POST
  • Parameters: JSON body
    • origin_location: URI of the Location the files should be moved from
    • pipeline: URI of the pipeline. Both Locations must be associated with this pipeline.
    • files: List of dicts containing source and destination. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

Browse location path

  • URL: /api/v2/location/<UUID>/browse/
  • Verb: GET
  • Parameters: Query string parameters
    • path: Path inside the Location to look
  • Response: JSON
    • entries: List of entries in `path`, files or directories
    • directories: List of directories in `path`. Subset of `entries`.
Version 1: Returns paths as strings

Version 2: Returns all paths base64 encoded

SWORD collection

  • URL: /api/v2/location/<UUID>/sword/collection/
  • Verb: GET, POST

See Sword API for details


Package

Get all packages

  • URL: /api/v2/file/
  • Verb: GET

Create new package

  • URL: /api/v2/file/
  • Verb: POST
  • Parameters: JSON. Fields for a new package:
    • uuid: UUID of the new package
    • origin_location: URI of the Location where the package is currently
    • origin_path: Path to the package, relative to the origin_location
    • current_location: URI of the Location where the package should be stored
    • current_path: Path where the package should be stored, relative to the current_location
    • package_type: Type of package this is. One of: AIP, AIC, DIP, transfer, SIP, file, deposit
    • size: Size of the package
    • origin_pipeline: URI of the pipeline the package is from
    • related_package_uuid: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified obj_create function, which calls Package.store_aip or Package.backlog_transfer

Get package details

  • URL: /api/v2/file/<UUID>/
  • Verb: GET

Update package contents

  • URL: /api/v2/file/<UUID>/
  • Verb: PUT
  • Parameters: JSON body
    • reingest: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
    • uuid: UUID of the existing package
    • origin_location: URI of the Location where the package is currently
    • origin_path: Path to the package, relative to the origin_location
    • current_location: URI of the Location where the package should be stored
    • current_path: Path where the package should be stored, relative to the current_location
    • package_type: Type of package this is. One of: AIP, AIC
    • size: Size of the package
    • origin_pipeline: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through Package.misc_attributes.reingest_pipeline)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call Package.finish_reingest and merge the new AIP with the existing one.

This is implemented using a modified obj_update which calls obj_update_hook.

Update package metadata

  • URL: /api/v2/file/<UUID>/
  • Verb: PATCH
  • Parameters: JSON body
    • reingest: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.

This is implemented using a modified obj_update which calls obj_update_hook. update_in_place also helps.

Delete package request

  • URL: /api/v2/file/<UUID>/delete_aip/
  • Verb: POST
  • Parameters: JSON body
    • event_reason: Reason for deleting the AIP
    • pipeline: UUID of the pipeline the delete request is from
    • user_id: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
    • user_email: Email of the user requesting the deletion.

Recover AIP request

  • URL: /api/v2/file/<UUID>/recover_aip/
  • Verb: POST
  • Parameters: JSON body
    • event_reason: Reason for recovering the AIP
    • pipeline: URI of the pipeline the recovery request is from
    • user_id: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
    • user_email: Email of the user requesting the recovery.

Download single file

  • URL: /api/v2/file/<UUID>/extract_file/
  • Verb: GET, HEAD
  • Parameters: Query string parameters
    • relative_path_to_file: Path to the file to download, relative to the package path.
  • Response: Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP

If the package is in Arkivum, the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns 202 and emails the administrator about the attempted access.

Download package

  • URL: /api/v2/file/<UUID>/download/
  • URL: /api/v2/file/<UUID>/download/<chunk number>/ (for LOCKSS harvesting)
  • Verb: GET, HEAD
  • Parameters: None
  • Response: Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.

If the package is in Arkivum, the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns 202 and emails the administrator about the attempted access.

Get pointer file

  • URL: /api/v2/file/<UUID>/pointer_file/
  • Verb: GET
  • Parameters: None
  • Response: Stream of the pointer file.

Check fixity

  • URL: /api/v2/file/<UUID>/check_fixity/
  • Verb: GET
  • Parameters: Query string parameters
    • force_local: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
  • Response: JSON
    • success: True if the verification succeeded, False if the verification failed, None if the scan could not start
    • message: Human-readable string explaining the report; it will be empty for successful scans.
    • failures: List of 0 or more errors
    • timestamp: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

AIP storage callback request

  • URL: /api/v2/file/<UUID>/send_callback/post_store/
  • Verb: GET

Request to call any Callbacks configured to run post-storage for this AIP.

Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).

Get file information for package

  • URL: /api/v2/file/<UUID>/contents/
  • Verb: GET
  • Response: JSON
    • success: True
    • package: UUID of the package
    • files: List of dictionaries with file information. Each dictionary has:
      • source_id: UUID of the file to index
      • name: Relative path of the file inside the package
      • source_package: UUID of the SIP this file is from
      • checksum: Checksum of the file, or an empty string
      • accessionid: Accession number, or an empty string
      • origin: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

Update file information for package

  • URL: /api/v2/file/<UUID>/contents/
  • Verb: PUT
  • Parameters: JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
    • relative_path: Relative path of the file inside the package
    • fileuuid: UUID of the file to index
    • accessionid: Accession number, or an empty string
    • sipuuid: UUID of the SIP this file is from
    • origin: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

Delete file information for package

  • URL: /api/v2/file/<UUID>/contents/
  • Verb: DELETE

Removes all file records associated with this package.

Query file information on packages

  • URL: /api/v2/file/metadata/
  • Verb: GET, POST
  • Parameters: Query string parameters. Must have at least one, but not all are required
    • relative_path: Relative path of the file inside the package
    • fileuuid: UUID of the file
    • accessionid: Accession number
    • sipuuid: UUID of the SIP this file is from
  • Response: JSON. List of dicts with file information about the files that match the query.
    • accessionid: Accession number, or an empty string
    • file_extension: File extension
    • filename: Name of the file, sans path.
    • relative_path: Relative path of the file inside the package
    • fileuuid: UUID of the file to index
    • sipuuid: UUID of the SIP this file is from
    • origin: UUID of the Archivematica dashboard this is from

Reingest AIP

  • URL: /api/v2/file/<UUID>/reingest/
  • Verb: POST
  • Parameters: JSON body
    • pipeline: UUID of the pipeline to reingest on
    • reingest_type: Type of reingest to start. One of METADATA_ONLY (metadata-only reingest), OBJECTS (partial reingest), FULL (full reingest)
    • processing_config: Optional. Name of the processing configuration to use on full reingest


Move Package

This endpoint is introduced in v0.14.

  • URL: /api/v2/file/<UUID>/move/
  • Verb: POST
  • Parameters: JSON body
    • location_uuid: UUID of the location to send the package. The location must have the same Purpose as the package's current location (e.g. AIP Storage, DIP Storage).

Example: 

$ curl -d 'location_uuid=7c5c48d4-50db-4018-b4b1-7ed29d1ef9d3	' -H"Authorization: ApiKey test:4525fd5272275caeac04a28447698c51" 'http://mysite.archivematica.org:8000/api/v2/file/8b8aa1a2-79c9-490e-b630-28f90bb7e654/move/'

SWORD endpoints

  • URL: /api/v2/file/<UUID>/sword/
  • URL: /api/v2/file/<UUID>/sword/media/
  • URL: /api/v2/file/<UUID>/sword/state/

See Sword API for details.

Retrieved from "https://wiki.archivematica.org/index.php?title=Storage_Service_API&oldid=13078"