Upload DIP - Revision history

Sallain at 22:05, 11 February 2020

2020-02-11T22:05:37Z

Hbecker: /* Send your DIPs using rsync */ Add how to do ssh-copy-id manually

2016-11-17T19:02:26Z

Send your DIPs using rsync: Add how to do ssh-copy-id manually

David: /* Configure AtoM to process your requests asynchronously */ Link to new docs for configuring atom gearman worker

2014-08-13T23:11:29Z

Configure AtoM to process your requests asynchronously: Link to new docs for configuring atom gearman worker

Sevein: /* Send your DIPs using rsync */

2013-06-25T00:08:26Z

Send your DIPs using rsync

Sevein: /* Send your DIPs using rsync */

2013-06-25T00:08:06Z

Send your DIPs using rsync

Sevein: /* Upload to AtoM */

2012-10-25T16:25:16Z

Upload to AtoM

Sevein: /* Upload to AtoM */

2012-10-25T16:24:40Z

Upload to AtoM

Sevein at 16:16, 25 October 2012

2012-10-25T16:16:19Z

Sevein at 13:14, 25 October 2012

2012-10-25T13:14:24Z

Sevein: /* Upload to AtoM */

2012-10-25T12:44:35Z

Upload to AtoM

← Older revision		Revision as of 22:05, 11 February 2020
Line 1:		Line 1:
		+	<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>
		+
	== Upload to AtoM ==		== Upload to AtoM ==

@@ Line 24: / Line 24: @@
   archivematica.server $ sudo -u archivematica ssh-keygen
-You will be asked to enter the location of the key, leave it empty by pressing enter as the default location is fine. That will generate the public/private keys under <code>/var/lib/archivematica/.ssh/id_dsa.pub</code> (public) and <code>/var/lib/archivematica/.ssh/id_dsa</code> (private).
+You will be asked to enter the location of the key, leave it empty by pressing enter as the default location is fine. That will generate the public/private keys under <code>/var/lib/archivematica/.ssh/id_rsa.pub</code> (public) and <code>/var/lib/archivematica/.ssh/id_rsa</code> (private).
 On the AtoM side, we have to authorize that key in any of your local users. My recommendation is to create an exclusive local user for this purpose:
@@ Line 31: / Line 31: @@
   atom.server $ sudo passwd -l archivematica
-Now, let's dump the contents of the public key into your AtoM server. The default location is <code>/home/archivematica/.ssh/authorized_keys</code>.
+Now, let's dump the contents of the public key into your AtoM server.
 <div class="note">Before you try all this from Archivematica, please make sure that the SSH connection can be stablished as expected. Furthermore, the first time you try the client will store the host key fingerprint and that requires the user intervention. If you don't do that, this service won't work.</div>

@@ Line 39: / Line 39: @@
 ===  Configure AtoM to process your requests asynchronously ===
-This section is described under the AtoM wiki: https://www.qubit-toolkit.org/wiki/Job_scheduling
+See: https://www.accesstomemory.org/en/docs/2.0/admin-manual/installation/asynchronous-jobs/
 == Upload to ContentDM ==

@@ Line 24: / Line 24: @@
   archivematica.server $ sudo -u archivematica ssh-keygen
-You will be asked to enter the location of the key, leave it empty as the default location is fine. That will generate the public/private keys under <code>/var/lib/archivematica/.ssh/id_dsa.pub</code> (public) and <code>/var/lib/archivematica/.ssh/id_dsa</code> (private).
+You will be asked to enter the location of the key, leave it empty by pressing enter as the default location is fine. That will generate the public/private keys under <code>/var/lib/archivematica/.ssh/id_dsa.pub</code> (public) and <code>/var/lib/archivematica/.ssh/id_dsa</code> (private).
 On the AtoM side, we have to authorize that key in any of your local users. My recommendation is to create an exclusive local user for this purpose:

@@ Line 24: / Line 24: @@
   archivematica.server $ sudo -u archivematica ssh-keygen
-That will generate the public/private keys under <code>/var/lib/archivematica/.ssh/id_dsa.pub</code> (public) and <code>/var/lib/archivematica/.ssh/id_dsa</code> (private).
+You will be asked to enter the location of the key, leave it empty as the default location is fine. That will generate the public/private keys under <code>/var/lib/archivematica/.ssh/id_dsa.pub</code> (public) and <code>/var/lib/archivematica/.ssh/id_dsa</code> (private).
 On the AtoM side, we have to authorize that key in any of your local users. My recommendation is to create an exclusive local user for this purpose:

@@ Line 11: / Line 11: @@
 The script can be found under <code>/usr/lib/archivematica/upload-qubit/upload-qubit.py</code> (check out the [https://github.com/artefactual/archivematica/blob/master/src/upload-qubit/lib/upload-qubit.py source code]).
 If you expect to work with large DIPs please refer to the next two sections to make the process more reliable:
-==== Send your DIPs using rsync ====
+=== Send your DIPs using rsync ===
 This is the most complicated part of the process. rsync is able to send files over a network connecting to a rsync server, that you can setup on your AtoM side, but most of the time people just prefer to use SSH as a secure and reliable transport. The trick is to setup both sides so the authentication step is passwordless and the good news is that SSH provides an easy way to do that using SSH keys.
@@ Line 39: / Line 37: @@
 <div class="note">rsync with SSH needs a shell in the remote server to work properly. If you are concerned about it try rssh, a restricted shell that can be configured to limit its usage to rsync. There are other methods like using the SSH authorized_keys file to limit the commands executed by the remote client.</div>
-====  Configure AtoM to process your requests asynchronously ====
+===  Configure AtoM to process your requests asynchronously ===
 This section is described under the AtoM wiki: https://www.qubit-toolkit.org/wiki/Job_scheduling

@@ Line 1: / Line 1: @@
 == Upload to AtoM ==
-This service is implemented as a Python script that will deposit a DIP into the AtoM service specified using the SWORD protocol. The script accepts a number of arguments and can be executed manually but it is normally called from Archivematica.
+This service is implemented as a Python script that given an UUID it will deposit the corresponding DIP into the AtoM service specified. The communication protocol between both ends is called [http://swordapp.org/ SWORD]. The script accepts a number of arguments and it can be executed manually although it is normally called from Archivematica.
-The Archivematica dashboard provides an interface so the parameters passed to <code>upload-qubit.py</code> can be personalized under Administration > AtoM DIP upload. In the same page you will find a description for every argument possible. During its configuration, we should consider four points:
+The Archivematica dashboard provides an interface to configure the parameters passed to the script. This interface can be found under Administration > AtoM DIP upload. In the same page you will find a description for every argument possible. Please read the following explanation about the different options:
-# The DIP UUID: use <code>--uuid</code>.
+# Pass the DIP UUID using <code>--uuid</code>.
-# Details of the AtoM service: use <code>--url</code>, <code>--email</code> and <code>--password</code>.
+# Pass the AtoM service details using <code>--url</code>, <code>--email</code> and <code>--password</code>.
-# (optional) By default, this service includes the DIP as part of the deposit. This is not feasible if the size is too big. Our recommendation is to avoid that if the DIP is over two megabytes. For that purpose, this service is able to send your DIP using [http://en.wikipedia.org/wiki/Rsync rsync], an advanced data transfer tool very flexible. Two options are available if you want to use rsync: <code>--rsync-command</code> and <code>--rsync-target</code>.
+# (optional) By default, this service includes the DIP within the deposit request. This is not feasible if the size of the package is too big. Our recommendation is to avoid that if the DIP size is over two megabytes. For that purpose, this service is able to send your DIP using [http://en.wikipedia.org/wiki/Rsync rsync], an advanced data transfer tool very flexible. Two options are available if you want to use rsync: <code>--rsync-command</code> and <code>--rsync-target</code>. You can find more details later.
 # (optional) Are you troubleshooting a problem? Use <code>--debug</code> to increase the script verbosity.
@@ Line 14: / Line 14: @@
 === Tips to deal with big DIPs ===
-# Send the DIPs to AtoM using rsync as explained later.
+If you expect to work with large DIPs please refer to the next two sections to make the process more reliable:
-==== How to use rsync? ====
+==== Send your DIPs using rsync ====
 This is the most complicated part of the process. rsync is able to send files over a network connecting to a rsync server, that you can setup on your AtoM side, but most of the time people just prefer to use SSH as a secure and reliable transport. The trick is to setup both sides so the authentication step is passwordless and the good news is that SSH provides an easy way to do that using SSH keys.
@@ Line 39: / Line 38: @@
 <div class="note">rsync with SSH needs a shell in the remote server to work properly. If you are concerned about it try rssh, a restricted shell that can be configured to limit its usage to rsync. There are other methods like using the SSH authorized_keys file to limit the commands executed by the remote client.</div>
 == Upload to ContentDM ==

← Older revision		Revision as of 12:44, 25 October 2012
Line 1:		Line 1:
	== Upload to AtoM ==		== Upload to AtoM ==
		+
		+	This service is implemented as a Python script that will deposit a DIP into the AtoM service specified. The script accepts a number of arguments and can be executed manually but it is normally called from Archivematica.
		+
		+	The Archivematica dashboard provides an interface so the parameters passed to <code>upload-qubit.py</code> can be personalized under Administration > AtoM DIP upload. In the same page you will find a description for every argument possible. During its configuration, we should consider four points:
		+
		+	# The DIP UUID: use <code>--uuid</code>.
		+	# Details of the AtoM service: use <code>--url</code>, <code>--email</code> and <code>--password</code>.
		+	# (optional) By default, this service includes the DIP as part of the deposit. This is not feasible if the size is too big. Our recommendation is to avoid that if the DIP is over two megabytes. For that purpose, this service is able to send your DIP using [http://en.wikipedia.org/wiki/Rsync rsync], an advanced data transfer tool very flexible. Two options are available if you want to use rsync: <code>--rsync-command</code> and <code>--rsync-target</code>.
		+	# (optional) Are you troubleshooting a problem? Use <code>--debug</code> to increase the script verbosity.
		+
		+	The script can be found under <code>/usr/lib/archivematica/upload-qubit/upload-qubit.py</code> (check out the [https://github.com/artefactual/archivematica/blob/master/src/upload-qubit/lib/upload-qubit.py source code]).
		+
		+	=== How to use rsync? ===
		+
		+	This is the most complicated part of the process. rsync is able to send files over a network connecting to a rsync server, that you can setup on your AtoM side, but most of the time people just prefer to use SSH as a secure and reliable transport. The trick is to setup both sides so the authentication step is passwordless and the good news is that SSH provides an easy way to do that using SSH keys.
		+
		+	You can just [https://www.google.com/search?q=ssh+passwordless google it], but I'll write it a brief explanation.
		+
		+	On the Archivematica end, we have to create a SSH key for the <code>archivematica</code> user running the command shown below. Please, leave the passphrase field blank.
		+
		+	archivematica.server $ sudo -u archivematica ssh-keygen
		+
		+	That will generate the public/private keys under <code>/var/lib/archivematica/.ssh/id_dsa.pub</code> (public) and <code>/var/lib/archivematica/.ssh/id_dsa</code> (private).
		+
		+	On the AtoM side, we have to authorize that key in any of your local users. My recommendation is to create an exclusive local user for this purpose:
		+
		+	atom.server $ sudo useradd -d /home/archivematica -m -N -r -s /bin/bash archivematica
		+	atom.server $ sudo passwd -l archivematica
		+
		+	Now, let's dump the contents of the public key into your AtoM server. The default location is <code>/home/archivematica/.ssh/authorized_keys</code>.
		+
		+	<div class="note">Before you try all this from Archivematica, please make sure that the SSH connection can be stablished as expected. Furthermore, the first time you try the client will store the host key fingerprint and that requires the user intervention. If you don't do that, this service won't work.</div>
		+
		+	<div class="note">rsync with SSH needs a shell in the remote server to work properly. If you are concerned about it try rssh, a restricted shell that can be configured to limit its usage to rsync. There are other methods like using the SSH authorized_keys file to limit the commands executed by the remote client.</div>

	== Upload to ContentDM ==		== Upload to ContentDM ==