Dataverse - Revision history

Grant: /* Current Status */ Added ref to iPRES paper

2020-04-08T15:28:58Z

Current Status: Added ref to iPRES paper

Grant: /* Current Status */ Update to reflect newer documented issues

2020-04-08T15:24:06Z

Current Status: Update to reflect newer documented issues

Sallain at 22:16, 11 February 2020

2020-02-11T22:16:23Z

Grant at 17:38, 19 March 2019

2019-03-19T17:38:15Z

Grant at 19:09, 28 February 2019

2019-02-28T19:09:53Z

Grant at 20:03, 22 November 2018

2018-11-22T20:03:26Z

Grant: /* Current Status */

2018-11-20T21:03:13Z

Current Status

Grant: /* Current Status */

2018-11-20T21:02:50Z

Current Status

Grant at 21:02, 20 November 2018

2018-11-20T21:02:31Z

Grant at 20:46, 20 November 2018

2018-11-20T20:46:31Z

@@ Line 27: / Line 27: @@
 This [https://drive.google.com/open?id=1XlHZF2Sryg_79qzw7G-R4PeWmMcPgRug screencast] provides a demonstration of the current implementation.
-OCUL/Scholars Portal is currently hosting a demonstration sandbox for interested users to test the integration. Please visit the [https://spotdocs.scholarsportal.info/display/DAT/Archivematica+Demo+Sandbox sandbox page on the OCUL Confluence site] for information on how to access it
+OCUL/Scholars Portal is currently hosting a demonstration sandbox for interested users to test the integration. Please visit the [https://spotdocs.scholarsportal.info/display/DAT/Archivematica+Demo+Sandbox sandbox page on the OCUL Confluence site] for information on how to access it. You can read more about the project and its outcomes in Meghan Goodchild and Grant Hurley's 2019 iPres paper and presentation slides here: https://osf.io/wqbvy/.
 == Overview of Dataverse to Archivematica Integration ==

@@ Line 11: / Line 11: @@
 ==Current Status==
 ''March 19, 2019''
@@ Line 18: / Line 24: @@
 * Multiple authors are not captured in the Dataverse METS - only the first author listed is. [https://github.com/archivematica/Issues/issues/278 Issue 278]
 * It is not possible to delete packages after extraction using the Dataverse transfer type if the package contains derivatives. [https://github.com/archivematica/Issues/issues/269 Issue 269]
 This [https://drive.google.com/open?id=1XlHZF2Sryg_79qzw7G-R4PeWmMcPgRug screencast] provides a demonstration of the current implementation.
 OCUL/Scholars Portal is currently hosting a demonstration sandbox for interested users to test the integration. Please visit the [https://spotdocs.scholarsportal.info/display/DAT/Archivematica+Demo+Sandbox sandbox page on the OCUL Confluence site] for information on how to access it
 == Overview of Dataverse to Archivematica Integration ==

← Older revision		Revision as of 22:16, 11 February 2020
Line 1:		Line 1:
	[[Main Page]] > [[Documentation]] > [[Requirements]] > Dataverse		[[Main Page]] > [[Documentation]] > [[Requirements]] > Dataverse
		+
		+	<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

	This page sets out the requirements and designs for integration with [http://dataverse.org Dataverse]. As of Archivematica [https://wiki.archivematica.org/Archivematica_1.8_and_Storage_Service_0.13_release_notes v. 1.8], integration with the Dataverse repository as a transfer source type supports the selection and processing of Dataverse research datasets. Archivematica [https://wiki.archivematica.org/Archivematica_1.9_and_Storage_Service_0.14_release_notes v. 1.9] introduced two fixes to the workflow.		This page sets out the requirements and designs for integration with [http://dataverse.org Dataverse]. As of Archivematica [https://wiki.archivematica.org/Archivematica_1.8_and_Storage_Service_0.13_release_notes v. 1.8], integration with the Dataverse repository as a transfer source type supports the selection and processing of Dataverse research datasets. Archivematica [https://wiki.archivematica.org/Archivematica_1.9_and_Storage_Service_0.14_release_notes v. 1.9] introduced two fixes to the workflow.

@@ Line 1: / Line 1: @@
 [[Main Page]] > [[Documentation]] > [[Requirements]] > Dataverse
-This page sets out the requirements and designs for integration with [http://dataverse.org Dataverse]. As of Archivematica [https://wiki.archivematica.org/Archivematica_1.8_and_Storage_Service_0.13_release_notes (v1.8)], integration with the Dataverse repository as a transfer source type supports the selection and processing of Dataverse research datasets.
+This page sets out the requirements and designs for integration with [http://dataverse.org Dataverse]. As of Archivematica [https://wiki.archivematica.org/Archivematica_1.8_and_Storage_Service_0.13_release_notes v. 1.8], integration with the Dataverse repository as a transfer source type supports the selection and processing of Dataverse research datasets.  Archivematica [https://wiki.archivematica.org/Archivematica_1.9_and_Storage_Service_0.14_release_notes v. 1.9] introduced two fixes to the workflow.
-This page was originally created as part of an early proof-of-concept integration project in 2015, which was only made available in a development branch of Archivematica. Phase 2 of this project improves on that original integration work and merges it into a public release of Archivematica (v1.8).  This work was sponsored by [https://scholarsportal.info/ Scholars Portal], a service of the Ontario Council of University Libraries (OCUL).
+This page was originally created as part of an early proof-of-concept integration project in 2015, which was only made available in a development branch of Archivematica. Phase 2 of this project improved on that original integration work and merged it into a public release of Archivematica (v1.8). This work was sponsored by [https://scholarsportal.info/ Scholars Portal], a service of the Ontario Council of University Libraries (OCUL).
 [[Category:Feature requirements]]
@@ Line 10: / Line 10: @@
 ==Current Status==
-''November 20, 2018''
+''March 19, 2019''
-The integration has been released as part of Archivematica version 1.8
+The integration has been released as part of Archivematica version 1.8. Version 1.9 (released March 6, 2019) integrated the following fixes:
-The following additional fixes will be included in a future release:
+* Multiple authors are not captured in the Dataverse METS - only the first author listed is. [https://github.com/archivematica/Issues/issues/278 Issue 278]
-* Multiple authors are not captured in the Dataverse METS - only the first author listed is.
+* It is not possible to delete packages after extraction using the Dataverse transfer type if the package contains derivatives. [https://github.com/archivematica/Issues/issues/269 Issue 269]
-To see the current status of work, and any outstanding issue, please see the Waffle Board or Board's linked to [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse below]:
+To see the current status of work, and any outstanding issue, please see the Archivematica issues repository on GitHub with the tag [https://github.com/archivematica/Issues/labels/OCUL%3A%20AM-Dataverse "OCUL: AM-Dataverse"]:
-* [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse Waffle board for the Dataverse Feature]
+This [https://drive.google.com/open?id=1XlHZF2Sryg_79qzw7G-R4PeWmMcPgRug screencast] provides a demonstration of the current implementation.
-This [https://drive.google.com/open?id=1XlHZF2Sryg_79qzw7G-R4PeWmMcPgRug screencast] provides a demonstration of the current implementation.
+OCUL/Scholars Portal is currently hosting a demonstration sandbox for interested users to test the integration. Please visit the [https://spotdocs.scholarsportal.info/display/DAT/Archivematica+Demo+Sandbox sandbox page on the OCUL Confluence site] for information on how to access it
 == Overview of Dataverse to Archivematica Integration ==
 ===Setting up the Integration===
-In order to set up and use the integration, users should consult Archivematica’s documentation, particularly the [https://www.archivematica.org/en/docs/storage-service-0.13/administrators/#dataverse Dataverse section] of the Archivematica Storage Service documentation for setup and the section on [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/dataverse/#dataverse-transfers Dataverse transfers]. If users are unfamiliar with Archivematica, they should consult Archivematica’s [https://www.archivematica.org/en/docs/archivematica-1.8/#overview overview] and [https://www.archivematica.org/en/docs/archivematica-1.8/getting-started/quick-start/quick-start/#quick-start quick start documentation]. If users are unfamiliar with Dataverse, they should consult Dataverse's [http://guides.dataverse.org/en/latest/ guides]. Users will need access to both an appropriately provisioned Archivematica instance, and to an installation of Dataverse for which they have an account.
+In order to set up and use the integration, users should consult Archivematica’s documentation, particularly the [https://www.archivematica.org/en/docs/storage-service-0.14/administrators/#dataverse Dataverse section] of the Archivematica Storage Service documentation for setup and the section on [https://www.archivematica.org/en/docs/archivematica-1.9/user-manual/transfer/dataverse/#dataverse-transfers Dataverse transfers]. If users are unfamiliar with Archivematica, they should consult Archivematica’s [https://www.archivematica.org/en/docs/archivematica-1.9/#overview overview] and [https://www.archivematica.org/en/docs/archivematica-1.9/getting-started/quick-start/quick-start/#quick-start quick start documentation]. If users are unfamiliar with Dataverse, they should consult Dataverse's [http://guides.dataverse.org/en/latest/ guides]. Users will need access to both an appropriately provisioned Archivematica instance, and to an installation of Dataverse for which they have an account.
 ===Scope of the Integration===
@@ Line 33: / Line 32: @@
 * The current integration presumes a user who has an account with a Dataverse instance and has generated an associated [http://guides.dataverse.org/en/latest/user/account.html?highlight=api%20token API key], and the same (or a different, authorized) user who has access to an Archivematica instance and storage service that is connected to that Dataverse via the API key. You can read more in the [http://guides.dataverse.org/en/latest/installation/config.html#id40 Dataverse documentation under "Root Dataverse Permissions"] about users, admins and superuser categories that might impact access to Dataverse datasets via the API.
 * It is assumed the user has obtained the necessary rights to process and store dataset files in Dataverse for preservation and has appropriate access to the dataset and/or associated files based on the rights related to their Dataverse API key (see above).
-* It is assumed that the preserver is interested in selecting specific datasets in a Dataverse for preservation. SIPs and their resulting AIPs are created from current versions of Dataverse datasets with one or more associated files in that dataset. A dataset is therefore equivalent to a SIP. Individual files cannot be selected for preservation, nor can older versions of files. However, users may make use of Archivematica’s [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/appraisal/appraisal/#appraisal Appraisal] functions to select individual files in a particular dataset to create a final AIP.
+* It is assumed that the preserver is interested in selecting specific datasets in a Dataverse for preservation. SIPs and their resulting AIPs are created from current versions of Dataverse datasets with one or more associated files in that dataset. A dataset is therefore equivalent to a SIP. Individual files cannot be selected for preservation, nor can older versions of files. However, users may make use of Archivematica’s [https://www.archivematica.org/en/docs/archivematica-1.9/user-manual/appraisal/appraisal/#appraisal Appraisal] functions to select individual files in a particular dataset to create a final AIP.
 * At present, a function to automate the ingest of all datasets in a Dataverse has not been developed.
@@ Line 100: / Line 99: @@
 ''Dataverse-created derivative bundles''
-A second set of packages are created by Dataverse in the form of derivative bundles. Derivatives are copies of files in tabular format that Dataverse creates from user-submitted files. Dataverse delivers these packages to Archivematica as zip packages. See [[Bundles for tabular data files]] for more details below. See Dataverse’s [http://guides.dataverse.org/en/latest/user/tabulardataingest/index.html guide on tabular ingest] for additional documentation. These packages are always extracted by Archivematica by default. Setting the processing configuration to not extract packages will not function for this type of transfer. Furthermore. in Archivematica 1.8 it is not possible to set the processing configuration to delete packages for transfers with derivatives - users should select 'no' for this option in their processing configuration. This is a bug that will be fixed in an upcoming release. The issue is that Dataverse produces .RData files as part of tabular ingest which are themselves packages, which are then deleted and cause transfer failures due to missing files.
+A second set of packages are created by Dataverse in the form of derivative bundles. Derivatives are copies of files in tabular format that Dataverse creates from user-submitted files. Dataverse delivers these packages to Archivematica as zip packages. See [[Bundles for tabular data files]] for more details below. See Dataverse’s [http://guides.dataverse.org/en/latest/user/tabulardataingest/index.html guide on tabular ingest] for additional documentation. These packages are always extracted by Archivematica by default. Setting the processing configuration to not extract packages will not function for this type of transfer.
 ==Known Issues Impacting Transfers==
@@ Line 129: / Line 128: @@
 |-
 | Dataset has derivative packages and “Delete packages after extraction” is set in the processing configuration
-| If the transfer contains derivative files (i.e., files that have been uploaded through the tabular ingest process in Dataverse) and the option “delete packages after the extraction” is selected in the Archivematica processing configuration, the transfer will fail. This is because the .RData files contained as part of derivative files are themselves packages and will be deleted. This issue is slated for a fix in a future release. The current fix is to select ‘no’ as the option in the processing configuration.
+| If the user is running Archivematica version 1.8, and the transfer contains derivative files (i.e., files that have been uploaded through the tabular ingest process in Dataverse) and the option “delete packages after the extraction” is selected in the Archivematica processing configuration, the transfer will fail. This is because the .RData files contained as part of derivative files are themselves packages and will be deleted. This issue was fixed in Archivematica 1.9. The work-around if running Archivematica 1.8 is to select ‘no’ as the option in the processing configuration.
 | Parse External Files: Parse Dataverse METS XML
 | IntegrityError: (1048, "Column 'eventOutcomeDetailNote' cannot be null")

← Older revision		Revision as of 19:09, 28 February 2019
Line 219:		Line 219:

	More metadata from dataverse could be mapped into the METS files. Scholar's Portal would like to see more metadata in the AIP to enable better indexing & search / discovery of datasets. To show which fields could be used, we took a version of the Dataverse metadata crosswalk, and created our own version that includes Archivematica. The [https://docs.google.com/spreadsheets/d/18Xn4yR-nvbZV5lfrxVNQ8GHM18ilZ_IPocP9UeOtCY4/edit?usp=sharing Dataverse 4.0+ to Archivematica Metadata Crosswalk] provides the same details in the table below but also highlights additional fields that should ultimately be mapped into METS.		More metadata from dataverse could be mapped into the METS files. Scholar's Portal would like to see more metadata in the AIP to enable better indexing & search / discovery of datasets. To show which fields could be used, we took a version of the Dataverse metadata crosswalk, and created our own version that includes Archivematica. The [https://docs.google.com/spreadsheets/d/18Xn4yR-nvbZV5lfrxVNQ8GHM18ilZ_IPocP9UeOtCY4/edit?usp=sharing Dataverse 4.0+ to Archivematica Metadata Crosswalk] provides the same details in the table below but also highlights additional fields that should ultimately be mapped into METS.
		+
		+	Note that if a user enters descriptive metadata via the Archivematica interface during the transfer process (by going to the transfer report pane > Metadata > Add), the addition of this new metadata will overwrite any imported DDI metadata from Dataverse in the final Archivematica METS file.

	</br>		</br>

@@ Line 97: / Line 97: @@
 ===Packages-related Workflows===
 ''User-submitted packages''
-It is common for Dataverse users to “double-zip” files when uploading files to datasets: the practice of packaging files and then packaging them again a second time. Dataverse always unpacks packages, but if users double-zip, they can save the labour of uploading many files one-by-one. Archivematica users may choose whether they wish to have these packages extracted and/or deleted afterward by setting the corresponding processing configuration.
+It is common for Dataverse users to “double-zip” files when uploading files to datasets. This is the practice of packaging files and then packaging them again a second time. Dataverse always unpacks submitted packages, but if users double-zip, they can save the labour of uploading many files one-by-one. Archivematica users may choose whether they wish to have these packages extracted and/or deleted afterward by setting the appropriate corresponding processing configuration.
 ''Dataverse-created derivative bundles''
-A second set of packages are created by Dataverse in the form of derivative bundles. Derivatives are copies of files in tabular format that Dataverse creates from user-submitted files. Dataverse delivers these packages to Archivematica as zip packages. See Dataverse’s [http://guides.dataverse.org/en/latest/user/tabulardataingest/index.html guide on tabular ingest] for more details. These packages are always extracted by Archivematica. In Archivematica 1.8 it is not possible to delete packages after extraction using the Dataverse transfer type for packages with derivatives.
+A second set of packages are created by Dataverse in the form of derivative bundles. Derivatives are copies of files in tabular format that Dataverse creates from user-submitted files. Dataverse delivers these packages to Archivematica as zip packages. See [[Bundles for tabular data files]] for more details below. See Dataverse’s [http://guides.dataverse.org/en/latest/user/tabulardataingest/index.html guide on tabular ingest] for additional documentation. These packages are always extracted by Archivematica by default. Setting the processing configuration to not extract packages will not function for this type of transfer. Furthermore. in Archivematica 1.8 it is not possible to set the processing configuration to delete packages for transfers with derivatives - users should select 'no' for this option in their processing configuration. This is a bug that will be fixed in an upcoming release. The issue is that Dataverse produces .RData files as part of tabular ingest which are themselves packages, which are then deleted and cause transfer failures due to missing files.
 ==Known Issues Impacting Transfers==
@@ Line 131: / Line 131: @@
 | If the transfer contains derivative files (i.e., files that have been uploaded through the tabular ingest process in Dataverse) and the option “delete packages after the extraction” is selected in the Archivematica processing configuration, the transfer will fail. This is because the .RData files contained as part of derivative files are themselves packages and will be deleted. This issue is slated for a fix in a future release. The current fix is to select ‘no’ as the option in the processing configuration.
 | Parse External Files: Parse Dataverse METS XML
+| IntegrityError: (1048, "Column 'eventOutcomeDetailNote' cannot be null")
 |-
 | User attempts to process dataset with restricted files
@@ Line 163: / Line 163: @@
 When Dataverse [http://guides.dataverse.org/en/latest/user/tabulardataingest/index.html ingests some forms of tabular data], it creates derivatives of the original data file and additional metadata files. All of these files are provided in a [http://guides.dataverse.org/en/latest/user/dataset-management.html?highlight=bundle bundle] as a zipped package, containing:
 * The original file uploaded by the user;
@@ Line 169: / Line 168: @@
 * Variable Metadata (as a DDI Codebook XML file);
 * Data File Citation (currently in either RIS or EndNote XML format);
 '''TO DO''' - update notes on how bundles are retrieved. the original version of this documentation included these notes which need to be updated / corrected:

@@ Line 10: / Line 10: @@
 ==Current Status==
-'''September 6, 2018'''
+"November 20, 2018"
-Development work is almost complete. QA is in progress. Changes are scheduled to be included in version 1.8 of Archviematica. To see the current status of work, and any outstanding issue, please see the Waffle Board or Board's linked to [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse below]:
 * [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse Waffle board for the Dataverse Feature]
 This [https://drive.google.com/open?id=1XlHZF2Sryg_79qzw7G-R4PeWmMcPgRug screencast] provides a demonstration of the current implementation.
 == Overview of Dataverse to Archivematica Integration ==
@@ Line 88: / Line 94: @@
 The AIP is stored in whatever location has been configured. Scholar's Portal intend to store their AIPs in an S3 location (which is a standard configuration option as of Storage Service version 0.12).
 ==Known Issues Impacting Transfers==
@@ Line 133: / Line 146: @@
 == Dataverse Datasets ==
 === Dataset Metadata file - dataset.json ===
@@ Line 491: / Line 509: @@
 * Warnings to users that would prevent transfer failures, such as when a user selects a metadata-only dataset, a restricted transfer, or transfer that does not verify compliance due to user not selecting ‘dataverse’ transfer type [https://github.com/archivematica/Issues/issues/79#issuecomment-415114349 issue 79, number 9] and [https://github.com/archivematica/Issues/issues/57 issue 57].
 * Download performance improvement [https://github.com/archivematica/Issues/issues/61 issue 61]
 ===New Features===
-* A separate microservice for verifying a Dataverse transfer before the transfer begins would make it easier to identify issues and ensure compliance for Dataverse transfer types.
+* A separate micro-service for verifying a Dataverse transfer before the transfer begins would make it easier to identify issues and ensure compliance for Dataverse transfer types.
 * Use of AICs or other method to relate datasets that are part of a Dataverse collection.
 * Automatic transfer title naming from DOI or other useful name.

@@ Line 3: / Line 3: @@
 This page sets out the requirements and designs for integration with [http://dataverse.org Dataverse]. As of Archivematica [https://wiki.archivematica.org/Archivematica_1.8_and_Storage_Service_0.13_release_notes (v1.8)], integration with the Dataverse repository as a transfer source type supports the selection and processing of Dataverse research datasets.
-This page was originally created as part of an early proof-of-concept integration project in 2015, which was only made available in a development branch of Archivematica. Phase 2 of this project improves on that original integration work and merges it into a public release of Archivematica (v1.8).  This work is being sponsored by [https://scholarsportal.info/ Scholars Portal], a service of the Ontario Council of University Libraries (OCUL).
+This page was originally created as part of an early proof-of-concept integration project in 2015, which was only made available in a development branch of Archivematica. Phase 2 of this project improves on that original integration work and merges it into a public release of Archivematica (v1.8).  This work was sponsored by [https://scholarsportal.info/ Scholars Portal], a service of the Ontario Council of University Libraries (OCUL).
 [[Category:Feature requirements]]
@@ Line 19: / Line 19: @@
 == Overview of Dataverse to Archivematica Integration ==
 === Feature Files ===
@@ Line 79: / Line 89: @@
 The AIP is stored in whatever location has been configured. Scholar's Portal intend to store their AIPs in an S3 location (which is a standard configuration option as of Storage Service version 0.12).
 == Dataverse Datasets ==
@@ Line 432: / Line 484: @@
 == Future Requirements & Considerations ==
 This section includes working notes for future phases, as interesting opportunities or questions arise. At the end of the current phase we will be documenting the integration as well as future opportunities.
 === Notes from Feature File review meeting on May 1 2018 (2pm EST) ===
 '''Choice & Versioning of Dataverse API:'''
 The dataverse Search and Access APIs are not currently versioned.