Archivematica - User contributions [en]

User:Jhs

2024-07-19T06:29:09Z

Artefactual:

Justin Simpson is the Managing Director at Artefactual Systems.

User:Kpunia

2024-06-06T15:32:02Z

Artefactual: Creating user page for new user.

This is Katie's Artefactual account. Thank you - thank you!

User:Diab

2024-06-03T12:13:43Z

Artefactual: Creating user page for new user.

I'm 53 years old and a software architect. I'm interested in archivematica to learn more about integration of the tool as ingestion-tool.

Main Page

2023-08-14T15:49:25Z

Artefactual:

<div style="background: #fff; float: right; padding: 1em; width: 30%">
<div class="download">

== Free Demo and Download ==
* [http://sandbox.archivematica.org Demo] - see [[Sandbox|Sandbox info]] for login
* [[Release Notes|Release notes]]
* [https://www.archivematica.org/en/docs/latest/admin-manual/installation-setup/installation/installation/ Installation]
* [http://www.artefactual.com/services/ Installation services]

</div>

<div class="documentation">

== [https://www.archivematica.org/ How to use Archivematica] ==
* User and administrative manuals at [http://www.archivematica.org archivematica.org]
* [https://www.artefactual.com/services/training/ Training services]

</div>

<div class="development">

== [[Development|Developer resources]] ==
*[[Development roadmap: Archivematica|Development roadmap]]
* Development [[Documentation|documentation]]
*[[Improvements|Proposed Improvements]]
*[https://projects.artefactual.com/projects/archivematica Issues list]
*[https://www.artefactual.com/partners/ Become an innovation partner]

</div>

<div class="community">

== [[Community|Community]] ==
*[https://groups.google.com/group/archivematica Discussion list]
* [https://wiki.archivematica.org/Community/Regional_User_Groups Regional groups]
*[[https://github.com/archivematica/archivematica-case-studies Case Studies]]
*[[Community|Free community resources]]
*[http://www.artefactual.com/services/archivematica/maintenance/ Get support]
*[[Contributors|Contributors]]
*[[Community/Camps|Camps]]
*[https://twitter.com/archivematica Twitter]

</div>
</div>

== ''This is the Archivematica development wiki. For user and administrative manuals, as well as installation information, please visit [http://www.archivematica.org archivematica.org].'' ==

='''Archivematica gets its own issues repo'''=
Artefactual has moved towards a centralized [https://github.com/archivematica/Issues Issues repository on Github ] to track all bugs, enhancements and feature requests in the Archivematica environment.

From now on issues should be filed against the Issues repo instead of against a repo in the Artefactual organization.

='''What is Archivematica?'''=

Archivematica is a free and open-source [[wikipedia:Digital preservation|digital preservation]] system that is designed to maintain long-term access to digital memory. Archivematica is packaged with the web-based content management system [https://www.accesstomemory.org/ AtoM] for access to your digital objects.

The [[overview]] section provides a detailed description of Archivematica's functionality and technical architecture. This screencast gives a demo of the core features in the most current release of the system.

{| width="100%" border="0" style="margin: 20px 0;" class="youtube"
| align="center" | <youtube>EjStNDXK48U</youtube>
|}

==='''Standards-based'''===

Archivematica uses a [http://en.wikipedia.org/wiki/Microservices micro-services] design pattern to provide an integrated suite of software tools that allows users to process digital objects from ingest to access in compliance with the ISO-OAIS functional model. Users monitor and control the micro-services via a web-based dashboard. Archivematica uses METS, PREMIS (events, agents, rights and restrictions), Dublin Core, the Library of Congress BagIt specification and other best practice standards and practices to provide trustworthy, authentic, reliable, and interoperable archival packages (AIPs) for storage in your preferred repository.

==='''Open source'''===

All Archivematica code is released under a GNU Affero General Public License ([http://www.gnu.org/licenses/agpl.html AGPL 3.0]) – giving you the freedom to study, modify, improve, and distribute it. We believe that an important part of preservation is transparency, and that memory institutions should be able to demonstrate at every stage what happens when they process cultural heritage materials for preservation. Archivematica [https://github.com/artefactual code] is always freely available, and our documentation is also released under a Creative Commons Share-alike license.

==='''Flexible and customizable'''===

Archivematica provides several decision points that give the user control over choices about format identification tools, printing the original order of the directories ingested, examining contents for private and personal information, extracting contents of packages and forensic images, transcribing content, and more. Users may also preconfigure most of these options for seamless ingest to archival storage and access. Archivematica offers many ingest workflows: metadata and submission documentation import, zipped and unzipped Bag ingest, digital forensic image processing, SIP arrangement, manual normalization, and dataset management.

==='''Compatible with hundreds of formats'''===

In the Format Policy Registry (FPR), Archivematica implements its default [[media type preservation plans|format policies]] based on an analysis of the [[significant characteristics]] of file formats. The FPR also offers an editable, flexible framework for format identification, package extraction, transcription and normalization for preservation and access. Your institution can update tools, rules and commands in your local FPR from the Artefactual-managed FPR server. You can also add your own, local policies to your internal FPR. The FPR is integrated with PRONOM.

==='''Advanced search and storage management'''===

You can easily search your backlog and your archival storage from within your Archivematica web-based dashboard. This means you can download stored AIPs as complete packages, individual objects, or every package in an AIC. You can also manage your storage and processing locations using the Archivematica Storage Service, including a two-step deletion process that requires justification and approval to eliminate a stored AIP.

==='''Integration with third-parties'''===

Memory institutions have dedicated voluminous resources over the past couple of decades to implement various software platforms and tools to manage digital objects. For this reason, we believe in leveraging the strength of other tools and integrating with them wherever possible. Highlights include: [https://www.accesstomemory.org/en/ AtoM], [http://www.dspace.org/ DSpace], [http://www.contentdm.org/ CONTENTdm], [http://islandora.ca/ Islandora], [http://www.lockss.org/ LOCKSS], [http://www.duracloud.org/ DuraCloud], [http://www.openstack.org/ OpenStack] and [http://www.archiviststoolkit.org/ Archivists' Toolkit]

The software applications integrated into Archivematica are each released under their own open source license. These are checked for license compatibility before they are integrated into the project. A full list of applications with their respective license is available on the [[External tools|external software tools]] page.

==='''Lead developers and partners'''===
This project is managed by [http://artefactual.com/archivematica.html Artefactual Systems]. Key funding contrbutors have included UNESCO Memory of the World's [http://portal.unesco.org/ci/en/ev.php-URL_ID=1720&URL_DO=DO_TOPIC&URL_SECTION=201.html Subcommittee on Technology], the [http://vancouver.ca/ctyclerk/archives/ City of Vancouver Archives], [http://www.library.hbs.edu/ Harvard Business School Baker Library], the [http://www.moma.org/ Museum of Modern Art (MoMA)], the [http://www.library.ualberta.ca/ University of Alberta Libraries], the [http://diginit.library.ubc.ca/ University of British Columbia Library], the [http://rockarch.org/ Rockefeller Archive Center], [http://www.sfu.ca/archives/ Simon Fraser University Archives and Records Management], [http://www.library.yale.edu/ Yale University Library], [http://www.zib.de/en/home.html Zuse-Institute Berlin], [http://www.coppul.ca/ Council of Prairie and Pacific University Libraries (COPPUL)], [http://bentley.umich.edu/ Bentley Historical Library, University of Michigan], [http://duraspace.org Duraspace], [http://libraries.mit.edu/ MIT Libraries], [http://www.ocul.on.ca/ Ontario Council of University Libraries], [https://www.llgc.org.uk/ National Library of Wales], [http://www.archivescanada.ca/ Canadian Council of Archives] and a number of other collaborators.
</div>

[[File:Artefactual_Logo.png|240px|center|link=http://www.artefactual.com/]] [[File:Cva.gif|160px]] [[File:Ubcblue_full.png|80px]] [[File:RAC-logo.png|80px]] [[File:Logo_en.gif|140px]][[File:SFULogo.jpg|150px]] [[File:UofAlogo.png|180px]][[File:LOGO_ulib_4cp_under75_png_4f05de2698.png|200px]] [[File:MoMA.png|180px]] [[File:FinalULBlues_copy.png|300px]] [[File:ZIBlogo-mediumlines.png|75px]] [[File:HarvardBaker30_image.png|250px]] [[File:Coppullogo.jpg‎]] [[File:UM-BHL-logo.png|120px]] [[File:DuraSpace_logo_horiz_300.png|260px]][[File:Mit_logo.png]] [[File:ocul_logo.png|200 px]] [[File:york_logo.png|240px]] [[File:hull_logo.png|220px]] [[File:NLW_logo1.jpg|330px]] [[File:Blue_Eng_First.png|360 px]] [[File:logo-en.png|360 px]]

__NOTOC__

Archivematica 1.13.0 and Storage Service 0.18.0 release notes

2021-08-18T17:45:01Z

Artefactual: /* Other changes */ minor typo

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.13.0 and Storage Service 0.18.0

'''Release date: July 12, 2021'''

== Environments ==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.13.0 and Storage Service 0.18.0 have been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the short term.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Added ==

=== Disableable virus scanning ===

* Issue: https://github.com/archivematica/Issues/issues/869

Virus scanning can now be disabled from the processing configuration.

This change was '''contributed''' by Bodleian Library. Thank you!

=== Purge management command ===

* Issue: https://github.com/archivematica/Issues/issues/1239

A new [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/maintenance/maintenance/#management-commands management command] has been added to provide a mechanism to remove old processing data from the system filling up the application database and search indices.

=== Strong password validation ===

* Issue: https://github.com/archivematica/Issues/issues/1332

Archivematica and the Storage Service can now require strong passwords for user accounts. Prompts are given when creating a new user if the password doesn't meet the requirements.

This feature was '''sponsored''' by the City of Winnipeg Archives. Thank you!

=== Audit logging capability ===

* Issue: https://github.com/archivematica/Issues/issues/1341

This addition allows audit logs to be written to third party applications- Artefactual Systems' implementation of this is in a new application called [https://github.com/artefactual-labs/auditmatica Auditmatica]. The change in Archivematica and the Storage Service is only to facilitate the capture of audit logging information in other applications.

This addition was '''sponsored''' by the City of Winnipeg Archives. Thank you!

=== Customized workflow file ===

* Issue: https://github.com/archivematica/Issues/issues/1441

This feature allows the user to indicate the location/existence of a customized workflow document, in json. This would allow institutions to add to the Archivematica workflow (e.g. custom micro-services) in a way that is easier to maintain through upgrades. Developer documentation is pending.

== Changed ==

=== Python 3 ===

* Issues: https://github.com/archivematica/Issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.13.0+label%3A%22%3Asnake%3A%C2%B3+Python+3%22

All Archivematica components are now running on Python 3.6.

=== Archivematica Storage Service now uses MySQL by default ===

* Issue: https://github.com/archivematica/Issues/issues/952

MySQL is now the default database in Archivematica Storage Service. SQLite is still supported but we encourage users to migrate. We have documented the process: [https://www.archivematica.org/en/docs/storage-service-0.18/migration-sqlite-mysql/#migration-sqlite-mysql Migrating data from SQLite to MySQL].

=== Allow replicated AIPs to be packaged in a different format from the original ===

*Issue: https://github.com/archivematica/Issues/issues/1440

By using an offline replica storage space it is now possible to replicate AIPs in a different packaging format than the original AIP (helpful in use cases such as offline tape storage, etc).

This change was '''sponsored''' by Norwegian Health Archives. Thank you!

=== Tasks will open in one tab only ===

* Issue: https://github.com/archivematica/Issues/issues/85

Now instead of opening multiple tabs every time a user clicks on a task gear, they will all open in one new tab.

This change was '''contributed''' by Bodleian Library. Thank you!

=== Dev tools deprecated / Hiding active packages ===

Issue: https://github.com/archivematica/Issues/issues/68

Issue: https://github.com/archivematica/Issues/issues/1446

Due to maintainability issues, we have removed the devtools repo. The most commonly used dev tool was used for resolving hidden transfers, which should no longer be relevant since resolving [https://github.com/archivematica/Issues/issues/1446 Issue 1446]- active transfers or SIPs can no longer be hidden. However as a precaution, we have made that command available within Archivematica (https://github.com/artefactual/archivematica/blob/qa/1.x/src/dashboard/src/main/management/commands/resolve_pending_jobs.py).

=== DIP storage locations now allowed in S3 spaces ===

* Issue: https://github.com/archivematica/Issues/issues/1442

This is a '''contribution''' by Fashion Institute of Technology NYC. Thank you!

=== Other changes ===

* Archivematica now records granular time stamps for original files, Issue 1427: https://github.com/archivematica/Issues/issues/1427
* Archival storage csv file sortable by size column, Issue 1450: https://github.com/archivematica/Issues/issues/1450
* Deleting AIPs does not delete associated directories from AIP store, Issue 359: https://github.com/archivematica/Issues/issues/359
* Cannot add a processing configuration if the name has diacritics, Issue 1104: https://github.com/archivematica/Issues/issues/1104
* Cannot create a replicator location in Duracloud space, Issue 1350: https://github.com/archivematica/Issues/issues/1350
* Fixity status tab sorts alphabetically, now sorts by date, Issue 1196: https://github.com/archivematica/Issues/issues/1196
* Failure report not generated when email not configured, Issue 1033: https://github.com/archivematica/Issues/issues/1033
* Replicator does not replicate AIPs on re-ingest or delete replicated AIPs, Issue 985: https://github.com/archivematica/Issues/issues/985

== Fixed ==

Please see the 1.13 milestone in GitHub for all issues addressed in this release: https://github.com/archivematica/Issues/milestone/17.

User:JennR

2021-03-01T18:23:39Z

Artefactual: Creating user page for new user.

I am Artefactual's northernmost digipres enthusiast. Which is only 5 words long...

Update mapping and reindex Elasticsearch indices

2020-12-21T16:52:58Z

Artefactual: /* List indices to resize the Elasticsearch heap size when needed */

= Backup Elasticsearch data =

The easiest way to backup the Elasticsearch data is copying the data directory:

sudo service elasticsearch stop
tar cvfz var_lib_elasticsearch.tgz /var/lib/elasticsearch
sudo service elasticsearch start

= List indices to resize the Elasticsearch heap size when needed =

Use the following command to list indices:

curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'

The output should show something like this:

root@archivematica-test-server:~# curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 0 0 1.2kb 1.2kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 11987 0 2.9gb 2.9gb

Take the elasticsearch heap size from ''/etc/default/elasticsearch'' (Ubuntu) or ''/etc/sysconfig/elasticsearch'' (CentOS):

root@ny-gclibrary-test-release-1:~# grep ES_JAVA_OPTS= /etc/default/elasticsearch
#ES_JAVA_OPTS=
ES_JAVA_OPTS="-Xms2g -Xmx2g"

The heap size for the example is 2G.

Ensure your Elasticsearch heap size is greater than the max ''store.size'' in the indices list. For our example, it should be greater than 3GB.

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch''.
* Change ''ES_JAVA_OPTS'' to a bigger value, in our example:
ES_JAVA_OPTS="-Xms3g -Xmx3g".
* Restart Elasticsearch service for the changes to take effect (''sudo service elasticsearch restart'')

= Run script to reindex and use new mappings =

Use the following script:

<nowiki>
#!/bin/bash

es_url="http://localhost:9200"

index_list='aips aipfiles transfers transferfiles'

echo -e "\nIndex list before reindexing:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Clone indices with _reindex API call:
for index in $index_list;do
echo "Reindex ${index} in ${index}_new..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}"'"
},
"dest": {
"index": "'"${index}_new"'"
}
}' > /dev/null
done

echo -e "\n\n"

echo -e "Index list after tmp indices creation\n"
indices_output=$(curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc")
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Delete old indices
for index in $index_list;do
echo "Deleting ${index}..."
curl -s -X DELETE ${es_url}/${index} > /dev/null
done

#Restart archivematica-dashboard to create indices with new mappings
echo -e "\nRestarting archivematica-dashboard"
sudo service archivematica-dashboard restart

#Wait 30 seconds
echo "Wait 30 seconds to ensure dashboard has created the empty indices with new mapping"
sleep 30
echo -e "\n"

#When index has no docs the reindex doesn't create the new index (typically transferfiles index)
#There's a check to ensure the new index has been create before reindexing.
#Reindex fron *_new indices:
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Indexing ${index} using ${index}_new ..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}_new"'"
},
"dest": {
"index": "'"${index}"'"
}
}' > /dev/null
fi
done

echo -e "\n"

#Delete tmp indices
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Deleting ${index}_new..."
curl -s -X DELETE ${es_url}/${index}_new > /dev/null
fi
done

echo -e "\n\nReindexing done:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"
</nowiki>

For our example it takes 11 minutes, and this is the output:

<nowiki>
root@archivematica-test-server:~# time ./script_reindex_new_map.sh

Index list before reindexing:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Reindex aips in aips_new...
Reindex aipfiles in aipfiles_new...
Reindex transfers in transfers_new...
Reindex transferfiles in transferfiles_new...

Index list after tmp indices creation

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers_new gdFevH8yRdiNTdrPcfo8Lg 5 1 0 0 460b 460b
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips_new uJ-ehaYLTfe_1lOSErfu3Q 5 1 17 0 96.8mb 96.8mb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles_new 00Xxu7v2QvWsq92gM247xQ 5 1 12905 0 3.1gb 3.1gb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Deleting aips...
Deleting aipfiles...
Deleting transfers...
Deleting transferfiles...

Restarting archivematica-dashboard
Wait 30 seconds to ensure dashboard has created the empty indices with new mapping

Indexing aips using aips_new ...
Indexing aipfiles using aipfiles_new ...
Indexing transfers using transfers_new ...

Deleting aips_new...
Deleting aipfiles_new...
Deleting transfers_new...

Reindexing done:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers FC7aSVPmSmmCc_LTv1AQRA 5 1 3 0 1.2kb 1.2kb
yellow open transferfiles 5JMAft3FQwmosZQFi7eJNw 5 1 0 0 1.2kb 1.2kb
yellow open aips EtwXG3-4SO2Px-4QMRufXA 5 1 17 0 102.1mb 102.1mb
yellow open aipfiles -PFuzslgTeWJ4CWny8VZoA 5 1 12905 0 3gb 3gb

real 10m47.114s
user 0m0.068s
sys 0m0.032s
</nowiki>

The script uses the elasticsearch API and makes the following actions:

* Reindex the ''transfers'', ''transferfiles'', ''aips'' and ''aipfiles'' indices in new temporary indices
* Delete original indices
* Restart ''archivematica-dashboard'' service to create empty indices with new mappings
* Reindex from temporary indices
* Delete temporary indices

= Restore Elasticsearch heap size when needed=

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch'' when needed.
* Change ''ES_JAVA_OPTS'' when needed.
* Restart Elasticsearch service when needed (''sudo service elasticsearch restart'')

Update mapping and reindex Elasticsearch indices

2020-12-21T13:50:22Z

Artefactual: /* Run script to reindex and use new mappings */

= Backup Elasticsearch data =

The easiest way to backup the Elasticsearch data is copying the data directory:

sudo service elasticsearch stop
tar cvfz var_lib_elasticsearch.tgz /var/lib/elasticsearch
sudo service elasticsearch start

= List indices to resize the Elasticsearch heap size when needed =

Use the following command to list indices:

curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'

The output should show something like this:

root@archivematica-test-server:~# curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 0 0 1.2kb 1.2kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 11987 0 2.9gb 2.9gb

Take the elasticsearch heap size from ''/etc/default/elasticsearch'' (Ubuntu) or ''/etc/sysconfig/elasticsearch'' (CentOS):

root@ny-gclibrary-test-release-1:~# grep ES_JAVA_OPTS= /etc/default/elasticsearch
#ES_JAVA_OPTS=
ES_JAVA_OPTS="-Xms2g -Xmx2g"

The heap size for the example is 2G.

Ensure your Elasticsearch heap size is greater than the max ''store.size'' in the indices list. For our example, it should be greater than 3GB.

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch'' when needed.
* Change ''ES_JAVA_OPTS'' when needed.
* Restart Elasticsearch service when needed (''sudo service elasticsearch restart'')

= Run script to reindex and use new mappings =

Use the following script:

<nowiki>
#!/bin/bash

es_url="http://localhost:9200"

index_list='aips aipfiles transfers transferfiles'

echo -e "\nIndex list before reindexing:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Clone indices with _reindex API call:
for index in $index_list;do
echo "Reindex ${index} in ${index}_new..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}"'"
},
"dest": {
"index": "'"${index}_new"'"
}
}' > /dev/null
done

echo -e "\n\n"

echo -e "Index list after tmp indices creation\n"
indices_output=$(curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc")
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Delete old indices
for index in $index_list;do
echo "Deleting ${index}..."
curl -s -X DELETE ${es_url}/${index} > /dev/null
done

#Restart archivematica-dashboard to create indices with new mappings
echo -e "\nRestarting archivematica-dashboard"
sudo service archivematica-dashboard restart

#Wait 30 seconds
echo "Wait 30 seconds to ensure dashboard has created the empty indices with new mapping"
sleep 30
echo -e "\n"

#When index has no docs the reindex doesn't create the new index (typically transferfiles index)
#There's a check to ensure the new index has been create before reindexing.
#Reindex fron *_new indices:
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Indexing ${index} using ${index}_new ..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}_new"'"
},
"dest": {
"index": "'"${index}"'"
}
}' > /dev/null
fi
done

echo -e "\n"

#Delete tmp indices
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Deleting ${index}_new..."
curl -s -X DELETE ${es_url}/${index}_new > /dev/null
fi
done

echo -e "\n\nReindexing done:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"
</nowiki>

For our example it takes 11 minutes, and this is the output:

<nowiki>
root@archivematica-test-server:~# time ./script_reindex_new_map.sh

Index list before reindexing:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Reindex aips in aips_new...
Reindex aipfiles in aipfiles_new...
Reindex transfers in transfers_new...
Reindex transferfiles in transferfiles_new...

Index list after tmp indices creation

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers_new gdFevH8yRdiNTdrPcfo8Lg 5 1 0 0 460b 460b
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips_new uJ-ehaYLTfe_1lOSErfu3Q 5 1 17 0 96.8mb 96.8mb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles_new 00Xxu7v2QvWsq92gM247xQ 5 1 12905 0 3.1gb 3.1gb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Deleting aips...
Deleting aipfiles...
Deleting transfers...
Deleting transferfiles...

Restarting archivematica-dashboard
Wait 30 seconds to ensure dashboard has created the empty indices with new mapping

Indexing aips using aips_new ...
Indexing aipfiles using aipfiles_new ...
Indexing transfers using transfers_new ...

Deleting aips_new...
Deleting aipfiles_new...
Deleting transfers_new...

Reindexing done:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers FC7aSVPmSmmCc_LTv1AQRA 5 1 3 0 1.2kb 1.2kb
yellow open transferfiles 5JMAft3FQwmosZQFi7eJNw 5 1 0 0 1.2kb 1.2kb
yellow open aips EtwXG3-4SO2Px-4QMRufXA 5 1 17 0 102.1mb 102.1mb
yellow open aipfiles -PFuzslgTeWJ4CWny8VZoA 5 1 12905 0 3gb 3gb

real 10m47.114s
user 0m0.068s
sys 0m0.032s
</nowiki>

The script uses the elasticsearch API and make the following actions:

* Reindex the ''transfers'', ''transferfiles'', ''aips'' and ''aipfiles'' indices in new temporary indices
* Delete original indices
* Restart ''archivematica-dashboard'' service to create empty indices with new mappings
* Reindex from temporary indices
* Delete temporary indices

= Restore Elasticsearch heap size when needed=

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch'' when needed.
* Change ''ES_JAVA_OPTS'' when needed.
* Restart Elasticsearch service when needed (''sudo service elasticsearch restart'')

Update mapping and reindex Elasticsearch indices

2020-12-21T13:38:49Z

Artefactual: /* Run script to reindex and use new mappings */

= Backup Elasticsearch data =

The easiest way to backup the Elasticsearch data is copying the data directory:

sudo service elasticsearch stop
tar cvfz var_lib_elasticsearch.tgz /var/lib/elasticsearch
sudo service elasticsearch start

= List indices to resize the Elasticsearch heap size when needed =

Use the following command to list indices:

curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'

The output should show something like this:

root@archivematica-test-server:~# curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 0 0 1.2kb 1.2kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 11987 0 2.9gb 2.9gb

Take the elasticsearch heap size from ''/etc/default/elasticsearch'' (Ubuntu) or ''/etc/sysconfig/elasticsearch'' (CentOS):

root@ny-gclibrary-test-release-1:~# grep ES_JAVA_OPTS= /etc/default/elasticsearch
#ES_JAVA_OPTS=
ES_JAVA_OPTS="-Xms2g -Xmx2g"

The heap size for the example is 2G.

Ensure your Elasticsearch heap size is greater than the max ''store.size'' in the indices list. For our example, it should be greater than 3GB.

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch'' when needed.
* Change ''ES_JAVA_OPTS'' when needed.
* Restart Elasticsearch service when needed (''sudo service elasticsearch restart'')

= Run script to reindex and use new mappings =

Use the following script:

<nowiki>
#!/bin/bash

es_url="http://localhost:9200"

index_list='aips aipfiles transfers transferfiles'

echo -e "\nIndex list before reindexing:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Clone indices with _reindex API call:
for index in $index_list;do
echo "Reindex ${index} in ${index}_new..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}"'"
},
"dest": {
"index": "'"${index}_new"'"
}
}' > /dev/null
done

echo -e "\n\n"

echo -e "Index list after tmp indices creation\n"
indices_output=$(curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc")
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Delete old indices
for index in $index_list;do
echo "Deleting ${index}..."
curl -s -X DELETE ${es_url}/${index} > /dev/null
done

#Restart archivematica-dashboard to create indices with new mappings
echo -e "\nRestarting archivematica-dashboard"
sudo service archivematica-dashboard restart

#Wait 30 seconds
echo "Wait 30 seconds to ensure dashboard has created the empty indices with new mapping"
sleep 30
echo -e "\n"

#When index has no docs the reindex doesn't create the new index (typically transferfiles index)
#There's a check to ensure the new index has been create before reindexing.
#Reindex fron *_new indices:
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Indexing ${index} using ${index}_new ..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}_new"'"
},
"dest": {
"index": "'"${index}"'"
}
}' > /dev/null
fi
done

echo -e "\n"

#Delete tmp indices
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Deleting ${index}_new..."
curl -s -X DELETE ${es_url}/${index}_new > /dev/null
fi
done

echo -e "\n\nReindexing done:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"
</nowiki>

For our example it takes 11 minutes, and this is the output:

<nowiki>
root@archivematica-test-server:~# time ./script_reindex_new_map.sh

Index list before reindexing:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Reindex aips in aips_new...
Reindex aipfiles in aipfiles_new...
Reindex transfers in transfers_new...
Reindex transferfiles in transferfiles_new...

Index list after tmp indices creation

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers_new gdFevH8yRdiNTdrPcfo8Lg 5 1 0 0 460b 460b
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips_new uJ-ehaYLTfe_1lOSErfu3Q 5 1 17 0 96.8mb 96.8mb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles_new 00Xxu7v2QvWsq92gM247xQ 5 1 12905 0 3.1gb 3.1gb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Deleting aips...
Deleting aipfiles...
Deleting transfers...
Deleting transferfiles...

Restarting archivematica-dashboard
Wait 30 seconds to ensure dashboard has created the empty indices with new mapping

Indexing aips using aips_new ...
Indexing aipfiles using aipfiles_new ...
Indexing transfers using transfers_new ...

Deleting aips_new...
Deleting aipfiles_new...
Deleting transfers_new...

Reindexing done:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers FC7aSVPmSmmCc_LTv1AQRA 5 1 3 0 1.2kb 1.2kb
yellow open transferfiles 5JMAft3FQwmosZQFi7eJNw 5 1 0 0 1.2kb 1.2kb
yellow open aips EtwXG3-4SO2Px-4QMRufXA 5 1 17 0 102.1mb 102.1mb
yellow open aipfiles -PFuzslgTeWJ4CWny8VZoA 5 1 12905 0 3gb 3gb

real 10m47.114s
user 0m0.068s
sys 0m0.032s
</nowiki>

= Restore Elasticsearch heap size when needed=

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch'' when needed.
* Change ''ES_JAVA_OPTS'' when needed.
* Restart Elasticsearch service when needed (''sudo service elasticsearch restart'')

Update mapping and reindex Elasticsearch indices

2020-12-21T13:37:37Z

Artefactual: Created page with "= Backup Elasticsearch data = The easiest way to backup the Elasticsearch data is copying the data directory: sudo service elasticsearch stop tar cvfz var_lib_elasticsea..."

= Backup Elasticsearch data =

The easiest way to backup the Elasticsearch data is copying the data directory:

sudo service elasticsearch stop
tar cvfz var_lib_elasticsearch.tgz /var/lib/elasticsearch
sudo service elasticsearch start

= List indices to resize the Elasticsearch heap size when needed =

Use the following command to list indices:

curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'

The output should show something like this:

root@archivematica-test-server:~# curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 0 0 1.2kb 1.2kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 11987 0 2.9gb 2.9gb

Take the elasticsearch heap size from ''/etc/default/elasticsearch'' (Ubuntu) or ''/etc/sysconfig/elasticsearch'' (CentOS):

root@ny-gclibrary-test-release-1:~# grep ES_JAVA_OPTS= /etc/default/elasticsearch
#ES_JAVA_OPTS=
ES_JAVA_OPTS="-Xms2g -Xmx2g"

The heap size for the example is 2G.

Ensure your Elasticsearch heap size is greater than the max ''store.size'' in the indices list. For our example, it should be greater than 3GB.

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch'' when needed.
* Change ''ES_JAVA_OPTS'' when needed.
* Restart Elasticsearch service when needed (''sudo service elasticsearch restart'')

= Run script to reindex and use new mappings =

Use the following script:

<nowiki>
#!/bin/bash

es_url="http://localhost:9200"

index_list='aips aipfiles transfers transferfiles'

echo -e "\nIndex list before reindexing:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Clone indices with _reindex API call:
for index in $index_list;do
echo "Reindex ${index} in ${index}_new..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}"'"
},
"dest": {
"index": "'"${index}_new"'"
}
}' > /dev/null
done

echo -e "\n\n"

echo -e "Index list after tmp indices creation\n"
indices_output=$(curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc")
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Delete old indices
for index in $index_list;do
echo "Deleting ${index}..."
curl -s -X DELETE ${es_url}/${index} > /dev/null
done

#Restart archivematica-dashboard to create indices with new mappings
echo -e "\nRestarting archivematica-dashboard"
sudo service archivematica-dashboard restart

#Wait 30 seconds
echo "Wait 30 seconds to ensure dashboard has created the empty indices with new mapping"
sleep 30
echo -e "\n"

#When index has no docs the reindex doesn't create the new index (typically transferfiles index)
#There's a check to ensure the new index has been create before reindexing.
#Reindex fron *_new indices:
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Indexing ${index} using ${index}_new ..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}_new"'"
},
"dest": {
"index": "'"${index}"'"
}
}' > /dev/null
fi
done

echo -e "\n"

#Delete tmp indices
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Deleting ${index}_new..."
curl -s -X DELETE ${es_url}/${index}_new > /dev/null
fi
done

echo -e "\n\nReindexing done:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"
</nowiki>

For our example it takes 11 minutes, and this is the output:

<nowiki>
root@archivematica-test-server:~# time ./script_reindex_new_map.sh

Index list before reindexing:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Reindex aips in aips_new...
Reindex aipfiles in aipfiles_new...
Reindex transfers in transfers_new...
Reindex transferfiles in transferfiles_new...

Index list after tmp indices creation

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers_new gdFevH8yRdiNTdrPcfo8Lg 5 1 0 0 460b 460b
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips_new uJ-ehaYLTfe_1lOSErfu3Q 5 1 17 0 96.8mb 96.8mb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles_new 00Xxu7v2QvWsq92gM247xQ 5 1 12905 0 3.1gb 3.1gb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Deleting aips...
Deleting aipfiles...
Deleting transfers...
Deleting transferfiles...

Restarting archivematica-dashboard
Wait 30 seconds to ensure dashboard has created the empty indices with new mapping

Indexing aips using aips_new ...
Indexing aipfiles using aipfiles_new ...
Indexing transfers using transfers_new ...

Deleting aips_new...
Deleting aipfiles_new...
Deleting transfers_new...

Reindexing done:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers FC7aSVPmSmmCc_LTv1AQRA 5 1 3 0 1.2kb 1.2kb
yellow open transferfiles 5JMAft3FQwmosZQFi7eJNw 5 1 0 0 1.2kb 1.2kb
yellow open aips EtwXG3-4SO2Px-4QMRufXA 5 1 17 0 102.1mb 102.1mb
yellow open aipfiles -PFuzslgTeWJ4CWny8VZoA 5 1 12905 0 3gb 3gb

real 10m47.114s
user 0m0.068s
sys 0m0.032s
<nowiki>

= Restore Elasticsearch heap size when needed=

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch'' when needed.
* Change ''ES_JAVA_OPTS'' when needed.
* Restart Elasticsearch service when needed (''sudo service elasticsearch restart'')

Archivematica 1.8 and Storage Service 0.13 release notes

2018-10-12T18:34:44Z

Artefactual: /* Indexing can be enabled/disabled for Transfers and/or Archival Storage */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Major release notes template

'''Work in progress'''

==Supported environments==

Link to installation instructions.

Specify supported environments.

Make special note of any changes to supported environment.

==Added==

Describe new features.

===New feature template===

This is a description of this amazing feature! Here's why it's a net benefit to the project and the community. Also included are any special notes, like if it's a beta feature.

This work was sponsored by some amazing institution. Thank you!

* Documentation: link
* Pull requests: link

===Processing configuration reset and download buttons===

A new installation of Archivematica comes with a pre-set processing configuration called "default", and a second one (used only in Jisc workflows) called "automated". In testing, users are encouraged to change the configurations to suit their workflows, but may need to reset the configuration to the installation pre-sets. A reset button has been added so that users can easily change the default and automated processing configurations back to their installation pre-sets.

The second part of this feature is the addition of a download button for the processing configuration files. If you create a custom processing configuration, you can download the resulting processingMCP.xml file using the button and then include it at the top level of your transfer. Archivematica will then use this to automate your transfer selections, rather than the default configuration.

This work was sponsored by Jisc. Thank you!

* Documentation: [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration documentation]
* Issue: [https://github.com/artefactual/archivematica/issues/1138 #1138]

==Changed==

Describe enhancements or major fixes.

===Streamline checksum verification===

This enhancement de-duplicates checksum verification in Archivematica, which helps to improve the performance of Archivematica in processing large transfers (many files and/or large files). This enhancement includes three changes:

* Remove the "Verify checksums generated on ingest" micro-service
* Enhance the "Verify AIP" micro-service to bulk query the database for transfer-generated checksums and then verify that they match what is documented in the bag-generated manifest-<ALGORITHM>.txt.
* Have "Verify AIP" create an AIP-level "fixity check" PREMIS:EVENT that it can pass to the Storage Service, which will document this verification in the pointer file.

This should not impact regular workflows, but it is worth noting that there is no AIP-level fixity check PREMIS event for uncompressed AIPs, which don't have pointer files. For uncompressed AIPs, there are still object-level fixity events in place. Note that there is an issue in the Archivematica Issues repository regarding this note - [https://github.com/artefactual/archivematica-storage-service/issues/324 Problem: uncompressed AIPs need pointer files #32]

This work was sponsored by Columbia University Library. Thank you!

* Issue: [https://github.com/artefactual/archivematica/issues/918 #918]
* Pull requests: [https://github.com/artefactual/archivematica/pull/1012 PR 1012]

===File format identification updates===

Archivematica 1.8 is now up to date with PRONOM v.94! For more information on new data added to PRONOM, check the [http://www.nationalarchives.gov.uk/aboutapps/pronom/release-notes.xml PRONOM release notes].

This work was sponsored by the Denver Art Museum. Thank you!

===Indexing can be enabled/disabled for Transfers and/or Archival Storage===
Previously, the ElasticSearch index feature could be disabled globally as a scalability measure since indexing consumes a lot of resources. However, this also disabled Backlog and Appraisal features (which also uses indexing) and which some users still wanted to access. As of release 1.8, Archivematica can be deployed to run with indexing enabled just for Transfers (Backlog and Appraisal enabled), just for Archival Storage (Backlog and Appraisal disabled), for both indexes, or for none.

* Issue: [https://github.com/artefactual/archivematica/issues/1172 1172]

==Fixed==

List bugfixes with a link to the Github issue.

* Bugfix 1: ASCII codes can't decode when the filename contains a backtick https://github.com/archivematica/Issues/issues/16
* Bugfix 2: AIP re-ingest fails. https://github.com/archivematica/Issues/issues/42
* Bugfix 3: PREMIS events from previous transfers are re-appearing https://github.com/archivematica/Issues/issues/43
* Bugfix 4: Metadata reingest fails when dc:type is null https://github.com/artefactual/archivematica/issues/1132
* Bugfix 5: Use 7-zip without compression (Copy) mode https://github.com/archivematica/Issues/issues/46
* Bugfix 6: Metadata added before "Approve Transfer" disappears https://github.com/archivematica/Issues/issues/140
* Bugfix 7: Generate AIP METS fails for bag SIPs if bag-info.txt has multiple instances of the same label https://github.com/archivematica/Issues/issues/173
* Bugfix 8: Zip files with diacritic characters are failing to extract https://github.com/artefactual/archivematica/issues/1104

==Upgraded tools and dependencies==

* Tool has been updated to version X.
* Fido has been upgraded to version 1.3.12
* Siegfried has been upgraded to version 1.7.10

==End of life dependencies==

===Archivists' Toolkit integration===

Archivists' Toolkit has been deprecated since 2013. The Archivists' Toolkit DIP upload feature has not had active development or testing since then. There are no plans to start testing or to fix any problems with the feature. As a result, there is a [https://github.com/archivematica/Issues/issues/174 proposal deprecate this feature in Archivematica 1.9]. Community response is welcome via a comment on the issue in GitHub.

Archivematica 1.8 and Storage Service 0.13 release notes

2018-10-12T16:28:55Z

Artefactual: /* Indexing can be enabled/disabled for Transfers and/or Archival Storage */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Major release notes template

'''Work in progress'''

==Supported environments==

Link to installation instructions.

Specify supported environments.

Make special note of any changes to supported environment.

==Added==

Describe new features.

===New feature template===

This is a description of this amazing feature! Here's why it's a net benefit to the project and the community. Also included are any special notes, like if it's a beta feature.

This work was sponsored by some amazing institution. Thank you!

* Documentation: link
* Pull requests: link

===Processing configuration reset and download buttons===

A new installation of Archivematica comes with a pre-set processing configuration called "default", and a second one (used only in Jisc workflows) called "automated". In testing, users are encouraged to change the configurations to suit their workflows, but may need to reset the configuration to the installation pre-sets. A reset button has been added so that users can easily change the default and automated processing configurations back to their installation pre-sets.

The second part of this feature is the addition of a download button for the processing configuration files. If you create a custom processing configuration, you can download the resulting processingMCP.xml file using the button and then include it at the top level of your transfer. Archivematica will then use this to automate your transfer selections, rather than the default configuration.

This work was sponsored by Jisc. Thank you!

* Documentation: [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration documentation]
* Issue: [https://github.com/artefactual/archivematica/issues/1138 #1138]

==Changed==

Describe enhancements or major fixes.

===Streamline checksum verification===

This enhancement de-duplicates checksum verification in Archivematica, which helps to improve the performance of Archivematica in processing large transfers (many files and/or large files). This enhancement includes three changes:

* Remove the "Verify checksums generated on ingest" micro-service
* Enhance the "Verify AIP" micro-service to bulk query the database for transfer-generated checksums and then verify that they match what is documented in the bag-generated manifest-<ALGORITHM>.txt.
* Have "Verify AIP" create an AIP-level "fixity check" PREMIS:EVENT that it can pass to the Storage Service, which will document this verification in the pointer file.

This should not impact regular workflows, but it is worth noting that there is no AIP-level fixity check PREMIS event for uncompressed AIPs, which don't have pointer files. For uncompressed AIPs, there are still object-level fixity events in place. Note that there is an issue in the Archivematica Issues repository regarding this note - [https://github.com/artefactual/archivematica-storage-service/issues/324 Problem: uncompressed AIPs need pointer files #32]

This work was sponsored by Columbia University Library. Thank you!

* Issue: [https://github.com/artefactual/archivematica/issues/918 #918]
* Pull requests: [https://github.com/artefactual/archivematica/pull/1012 PR 1012]

===File format identification updates===

Archivematica 1.8 is now up to date with PRONOM v.94! For more information on new data added to PRONOM, check the [http://www.nationalarchives.gov.uk/aboutapps/pronom/release-notes.xml PRONOM release notes].

This work was sponsored by the Denver Art Museum. Thank you!

===Indexing can be enabled/disabled for Transfers and/or Archival Storage===
Previously, the ElasticSearch index feature could be disabled globally as a scalability measure since indexing consumes a lot of resources. However, this also disabled Backlog and Appraisal features (which also uses indexing) and which some users still wanted to access. As of release 1.8, Archivematica can be deployed to run with indexing enabled just for Transfers (Backlog and Appraisal enabled), just for Archival Storage (Backlog and Appraisal disabled), for both indexes, or for none.

* Issue: [https://github.com/artefactual/archivematica/issues/1172 #1172]

==Fixed==

List bugfixes with a link to the Github issue.

* Bugfix 1: ASCII codes can't decode when the filename contains a backtick https://github.com/archivematica/Issues/issues/16
* Bugfix 2: AIP re-ingest fails. https://github.com/archivematica/Issues/issues/42
* Bugfix 3: PREMIS events from previous transfers are re-appearing https://github.com/archivematica/Issues/issues/43
* Bugfix 4: Metadata reingest fails when dc:type is null https://github.com/artefactual/archivematica/issues/1132
* Bugfix 5: Use 7-zip without compression (Copy) mode https://github.com/archivematica/Issues/issues/46
* Bugfix 6: Metadata added before "Approve Transfer" disappears https://github.com/archivematica/Issues/issues/140
* Bugfix 7: Generate AIP METS fails for bag SIPs if bag-info.txt has multiple instances of the same label https://github.com/archivematica/Issues/issues/173
* Bugfix 8: Zip files with diacritic characters are failing to extract https://github.com/artefactual/archivematica/issues/1104

==Upgraded tools and dependencies==

* Tool has been updated to version X.
* Fido has been upgraded to version 1.3.12
* Siegfried has been upgraded to version 1.7.10

==End of life dependencies==

===Archivists' Toolkit integration===

Archivists' Toolkit has been deprecated since 2013. The Archivists' Toolkit DIP upload feature has not had active development or testing since then. There are no plans to start testing or to fix any problems with the feature. As a result, there is a [https://github.com/archivematica/Issues/issues/174 proposal deprecate this feature in Archivematica 1.9]. Community response is welcome via a comment on the issue in GitHub.

Archivematica 1.8 and Storage Service 0.13 release notes

2018-10-12T16:28:23Z

Artefactual: /* Changed */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Major release notes template

'''Work in progress'''

==Supported environments==

Link to installation instructions.

Specify supported environments.

Make special note of any changes to supported environment.

==Added==

Describe new features.

===New feature template===

This is a description of this amazing feature! Here's why it's a net benefit to the project and the community. Also included are any special notes, like if it's a beta feature.

This work was sponsored by some amazing institution. Thank you!

* Documentation: link
* Pull requests: link

===Processing configuration reset and download buttons===

A new installation of Archivematica comes with a pre-set processing configuration called "default", and a second one (used only in Jisc workflows) called "automated". In testing, users are encouraged to change the configurations to suit their workflows, but may need to reset the configuration to the installation pre-sets. A reset button has been added so that users can easily change the default and automated processing configurations back to their installation pre-sets.

The second part of this feature is the addition of a download button for the processing configuration files. If you create a custom processing configuration, you can download the resulting processingMCP.xml file using the button and then include it at the top level of your transfer. Archivematica will then use this to automate your transfer selections, rather than the default configuration.

This work was sponsored by Jisc. Thank you!

* Documentation: [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration documentation]
* Issue: [https://github.com/artefactual/archivematica/issues/1138 #1138]

==Changed==

Describe enhancements or major fixes.

===Streamline checksum verification===

This enhancement de-duplicates checksum verification in Archivematica, which helps to improve the performance of Archivematica in processing large transfers (many files and/or large files). This enhancement includes three changes:

* Remove the "Verify checksums generated on ingest" micro-service
* Enhance the "Verify AIP" micro-service to bulk query the database for transfer-generated checksums and then verify that they match what is documented in the bag-generated manifest-<ALGORITHM>.txt.
* Have "Verify AIP" create an AIP-level "fixity check" PREMIS:EVENT that it can pass to the Storage Service, which will document this verification in the pointer file.

This should not impact regular workflows, but it is worth noting that there is no AIP-level fixity check PREMIS event for uncompressed AIPs, which don't have pointer files. For uncompressed AIPs, there are still object-level fixity events in place. Note that there is an issue in the Archivematica Issues repository regarding this note - [https://github.com/artefactual/archivematica-storage-service/issues/324 Problem: uncompressed AIPs need pointer files #32]

This work was sponsored by Columbia University Library. Thank you!

* Issue: [https://github.com/artefactual/archivematica/issues/918 #918]
* Pull requests: [https://github.com/artefactual/archivematica/pull/1012 PR 1012]

===File format identification updates===

Archivematica 1.8 is now up to date with PRONOM v.94! For more information on new data added to PRONOM, check the [http://www.nationalarchives.gov.uk/aboutapps/pronom/release-notes.xml PRONOM release notes].

This work was sponsored by the Denver Art Museum. Thank you!

===Indexing can be enabled/disabled for Transfers and/or Archival Storage===
Previously, the ElasticSearch index feature could be disabled globally as a scalability measure since indexing consumes a lot of resources. However, this also disabled Backlog and Appraisal features (which also uses indexing) and which some users still wanted to access. As of release 1.8, Archivematica can be deployed to run with indexing enabled just for Transfers (Backlog and Appraisal enabled), just for Archival Storage (Backlog and Appraisal disabled), for both indexes or for none.

* Issue: [https://github.com/artefactual/archivematica/issues/1172 #1172]

==Fixed==

List bugfixes with a link to the Github issue.

* Bugfix 1: ASCII codes can't decode when the filename contains a backtick https://github.com/archivematica/Issues/issues/16
* Bugfix 2: AIP re-ingest fails. https://github.com/archivematica/Issues/issues/42
* Bugfix 3: PREMIS events from previous transfers are re-appearing https://github.com/archivematica/Issues/issues/43
* Bugfix 4: Metadata reingest fails when dc:type is null https://github.com/artefactual/archivematica/issues/1132
* Bugfix 5: Use 7-zip without compression (Copy) mode https://github.com/archivematica/Issues/issues/46
* Bugfix 6: Metadata added before "Approve Transfer" disappears https://github.com/archivematica/Issues/issues/140
* Bugfix 7: Generate AIP METS fails for bag SIPs if bag-info.txt has multiple instances of the same label https://github.com/archivematica/Issues/issues/173
* Bugfix 8: Zip files with diacritic characters are failing to extract https://github.com/artefactual/archivematica/issues/1104

==Upgraded tools and dependencies==

* Tool has been updated to version X.
* Fido has been upgraded to version 1.3.12
* Siegfried has been upgraded to version 1.7.10

==End of life dependencies==

===Archivists' Toolkit integration===

Archivists' Toolkit has been deprecated since 2013. The Archivists' Toolkit DIP upload feature has not had active development or testing since then. There are no plans to start testing or to fix any problems with the feature. As a result, there is a [https://github.com/archivematica/Issues/issues/174 proposal deprecate this feature in Archivematica 1.9]. Community response is welcome via a comment on the issue in GitHub.

Archivematica 1.8 and Storage Service 0.13 release notes

2018-10-12T16:18:33Z

Artefactual: /* Fixed */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Major release notes template

'''Work in progress'''

==Supported environments==

Link to installation instructions.

Specify supported environments.

Make special note of any changes to supported environment.

==Added==

Describe new features.

===New feature template===

This is a description of this amazing feature! Here's why it's a net benefit to the project and the community. Also included are any special notes, like if it's a beta feature.

This work was sponsored by some amazing institution. Thank you!

* Documentation: link
* Pull requests: link

===Processing configuration reset and download buttons===

A new installation of Archivematica comes with a pre-set processing configuration called "default", and a second one (used only in Jisc workflows) called "automated". In testing, users are encouraged to change the configurations to suit their workflows, but may need to reset the configuration to the installation pre-sets. A reset button has been added so that users can easily change the default and automated processing configurations back to their installation pre-sets.

The second part of this feature is the addition of a download button for the processing configuration files. If you create a custom processing configuration, you can download the resulting processingMCP.xml file using the button and then include it at the top level of your transfer. Archivematica will then use this to automate your transfer selections, rather than the default configuration.

This work was sponsored by Jisc. Thank you!

* Documentation: [https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration documentation]
* Issue: [https://github.com/artefactual/archivematica/issues/1138 #1138]

==Changed==

Describe enhancements or major fixes.

===Streamline checksum verification===

This enhancement de-duplicates checksum verification in Archivematica, which helps to improve the performance of Archivematica in processing large transfers (many files and/or large files). This enhancement includes three changes:

* Remove the "Verify checksums generated on ingest" micro-service
* Enhance the "Verify AIP" micro-service to bulk query the database for transfer-generated checksums and then verify that they match what is documented in the bag-generated manifest-<ALGORITHM>.txt.
* Have "Verify AIP" create an AIP-level "fixity check" PREMIS:EVENT that it can pass to the Storage Service, which will document this verification in the pointer file.

This should not impact regular workflows, but it is worth noting that there is no AIP-level fixity check PREMIS event for uncompressed AIPs, which don't have pointer files. For uncompressed AIPs, there are still object-level fixity events in place. Note that there is an issue in the Archivematica Issues repository regarding this note - [https://github.com/artefactual/archivematica-storage-service/issues/324 Problem: uncompressed AIPs need pointer files #32]

This work was sponsored by Columbia University Library. Thank you!

* Issue: [https://github.com/artefactual/archivematica/issues/918 #918]
* Pull requests: [https://github.com/artefactual/archivematica/pull/1012 PR 1012]

===File format identification updates===

Archivematica 1.8 is now up to date with PRONOM v.94! For more information on new data added to PRONOM, check the [http://www.nationalarchives.gov.uk/aboutapps/pronom/release-notes.xml PRONOM release notes].

This work was sponsored by the Denver Art Museum. Thank you!

==Fixed==

List bugfixes with a link to the Github issue.

* Bugfix 1: ASCII codes can't decode when the filename contains a backtick https://github.com/archivematica/Issues/issues/16
* Bugfix 2: AIP re-ingest fails. https://github.com/archivematica/Issues/issues/42
* Bugfix 3: PREMIS events from previous transfers are re-appearing https://github.com/archivematica/Issues/issues/43
* Bugfix 4: Metadata reingest fails when dc:type is null https://github.com/artefactual/archivematica/issues/1132
* Bugfix 5: Use 7-zip without compression (Copy) mode https://github.com/archivematica/Issues/issues/46
* Bugfix 6: Metadata added before "Approve Transfer" disappears https://github.com/archivematica/Issues/issues/140
* Bugfix 7: Generate AIP METS fails for bag SIPs if bag-info.txt has multiple instances of the same label https://github.com/archivematica/Issues/issues/173
* Bugfix 8: Zip files with diacritic characters are failing to extract https://github.com/artefactual/archivematica/issues/1104

==Upgraded tools and dependencies==

* Tool has been updated to version X.
* Fido has been upgraded to version 1.3.12
* Siegfried has been upgraded to version 1.7.10

==End of life dependencies==

===Archivists' Toolkit integration===

Archivists' Toolkit has been deprecated since 2013. The Archivists' Toolkit DIP upload feature has not had active development or testing since then. There are no plans to start testing or to fix any problems with the feature. As a result, there is a [https://github.com/archivematica/Issues/issues/174 proposal deprecate this feature in Archivematica 1.9]. Community response is welcome via a comment on the issue in GitHub.

Archivematica 1.8 and Storage Service 0.13 release notes

2018-10-01T21:57:56Z

Artefactual: /* Changed */

Getting started

2018-08-10T18:44:18Z

Artefactual:

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Getting Started

This wiki page describes getting started with Archivematica as a developer. For user and administrative manuals, please see http://www.archivematica.org.

== Vital Stats ==

* Language: Python (primarily)
* License: [https://en.wikipedia.org/wiki/Affero_General_Public_License AGPL]
* VCS: git
* Major libraries: [https://www.djangoproject.com/ Django], [http://gearman.org/ gearman] ([https://pythonhosted.org/gearman/ Python API])
* [[Contribute_code|Contribution guidelines]]
** [[Contribute_code#Code_Style_Guide_For_Archivematica|Coding style]]

== Projects ==

Archivematica consists of several projects working together.

* [https://github.com/artefactual/archivematica Archivematica]: Main repository containing the user-facing dashboard, task manager MCPServer and clients scripts for the MCPClient
* [https://github.com/artefactual/archivematica-storage-service Storage Service]: Responsible for moving files to Archivematica for processing, and from Archivematica into storage
* [https://github.com/artefactual/archivematica-fpr-admin Format Policy Registry]: Submodule shared between Archivematica and the Format Policy Registry (FPR) server that displays and updates FPR rules and commands

There are also several smaller repositories that support Archivematica in various ways. In general, you will not need these to develop on Archivematica.

* [https://github.com/artefactual/archivematica-devtools Development tools]: Scripts to help with development. E.g. restarting services, workflow analysis
* [https://github.com/artefactual/archivematica-fpr-tools FPR tools]: All the tools, commands and rules used to populate the FPR database. Changes to the FPR should be submitted here.
* [https://github.com/artefactual/archivematica-docs Archivematica Documentation]: Documentation found at https://www.archivematica.org/en/docs/ for Archivematica
* [https://github.com/artefactual/archivematica-storage-service-docs Storage Service Documentation]: Documentation found at https://www.archivematica.org/en/docs/ for the Storage Service
* [https://github.com/artefactual/automation-tools Automation Tools]: Scripts used to automate processing material through Archivematica
* [https://github.com/artefactual/deploy-pub Deployment]: Ansible scripts for deploying and configuring Archivematica
* [https://github.com/artefactual-labs/ansible-archivematica Deployment-Archivematica]: Ansible playbook for Archivematica package install.
* [https://github.com/artefactual-labs/ansible-role-archivematica-src Deployment-Archivematica-dev]: Ansible playbook for Archivematica github install.
* [https://github.com/artefactual/fixity Fixity checker]: Commandline tool that assists in checking fixity for AIPs stored in Archivematica Storage Service instances.
* [https://github.com/artefactual-labs/mets-reader-writer METS reader/writer]: Library to create and parse METS files.
* [https://github.com/artefactual-labs/agentarchives agentarchives]: Clients to retrieve, add, and modify records from archival management systems.
* [https://github.com/artefactual/archivematica-sampledata Sample data]: Data to test and show off Archivematica's processing
* [https://github.com/artefactual/archivematica-history History]: Contains the pre-git history of Archivematica. Useful for checking the origins of code.

== Installation ==

The recommended way to install Archivematica for development is with Ansible and Vagrant.

Alternatively, you can try our environment based on Docker Compose - see https://github.com/artefactual-labs/am/tree/master/compose.

=== Ansible & Vagrant ===

The following instructions detail how to install and run Archivematica from source on a virtual machine.
# Install VirtualBox, Vagrant, and Ansible with the following commands:
#* <code>sudo apt-get install virtualbox vagrant</code> (this is the command for Ubuntu; if you use Mac or a different Linux distribution, it may be slightly different).
#** Note: Vagrant must be at least 1.5 (it can also be downloaded from [https://www.vagrantup.com/downloads.html vagrantup.com]). Check your version with <code>vagrant --version</code>.
#* <code>sudo pip install -U ansible</code>
# Checkout the deployment repo:
#* <code>git clone https://github.com/artefactual/deploy-pub.git</code>
# Download the Ansible roles:
#* <code>cd deploy-pub/playbooks/archivematica</code>
#* <code>ansible-galaxy install -f -p roles/ -r requirements.yml</code>
# (Optional) Change the branch by opening the file <code>vars-singlenode.yml</code> and modifying the following:
#* <code>archivematica_src_am_version: "branch-name"</code>
#* <code>archivematica_src_ss_version: "branch-name"</code>
# Create the virtual machine and provision it:
#* <code>vagrant up</code> (it takes a while!)
# You can now log in to your virtual machine:
#* <code>vagrant ssh</code>
# You can now access the following services in a web browser:
#* Archivematica - http://192.168.168.192
#* Archivematica Storage Service: http://192.168.168.192:8000

You may also wish to do the following.
# Provisioning (via ansible) can be re-run with vagrant to update the code on the server (for example, if new features are added):
#* <code>vagrant provision</code>
# To re-deploy a new branch to the same VM, update the branch variables in <code>vars-singlenode.yml</code>:
#* <code>archivematica_src_am_version: "branch-name"</code>
#* <code>archivematica_src_ss_version: "branch-name"</code>
#** This will probably require resetting the Archivematica installation as well. This can be done by adding variables to <code>vars-singlenode.yml</code>
#* <code>archivematica_src_reset_am_all: "true"</code> This will reset the Archivematica database, clear ElasticSearch and clear shared directories
#* <code>archivematica_src_reset_ss_db: "true"</code> This will reset the Storage Service database
#* For more variables to control deployment, see the [https://github.com/artefactual-labs/ansible-role-archivematica-src/blob/master/README.md README]
# See also the FAQ below

=== Alternative Vagrant projects ===

Community-provided alternatives have also been developed.

*https://github.com/emltech/eml-archivematica-vagrant
*https://github.com/statsbiblioteket/archivematica-vagrant

== Tests ==

Archivematica and the related projects have a small but growing test suite. We use [http://pytest.org/ py.test] to run our tests, which should be listed as a requirement in the development/local requirements file.

To run the tests, go to the repository root and run <code>py.test</code>

See below for project-specific setup or changes to running the tests.

=== Archivematica ===

Before running Archivematica tests, set the following environment variable. Archivematica does not currently have a virtualenv that needs to be activated.

<pre>
#!/usr/bin/fish
set -xg PYTHONPATH /usr/share/archivematica/dashboard/:/usr/lib/archivematica/archivematicaCommon/
</pre>

<pre>
#!/usr/bin/bash
export PYTHONPATH=$PYTHONPATH:/usr/share/archivematica/dashboard/:/usr/lib/archivematica/archivematicaCommon/
</pre>

=== Storage Service ===

Before running Storage Service tests, activate the virtualenv and set the following environment variables. The tests should be run from the <code>storage_service</code> directory. This is the same directory that contains <code>manage.py</code>. You may need to install <code>requirements/test.txt</code> to install testing dependencies.

<pre>
#!/usr/bin/fish
set -xg PYTHONPATH (pwd)/storage_service # This directory contains manage.py
set -xg DJANGO_SETTINGS_MODULE storage_service.settings.test
set -xg DJANGO_SECRET_KEY 'ADDKEY'
</pre>
<pre>
#!/usr/bin/bash
export PYTHONPATH=$(pwd)/storage_service # This directory contains manage.py
export DJANGO_SETTINGS_MODULE=storage_service.settings.test
export DJANGO_SECRET_KEY='ADDKEY'
</pre>

== FAQ ==

=== How do I restart everything in Archivematica? ===

A default install using Ansible also installs the devtools. Run <code>am restart-services</code> to restart all services related to Archivematica and the storage service.

=== How do I restart just the dashboard? ===

<code>sudo /etc/init.d/apache2 start</code> or <code>sudo service apache2 restart</code>

=== How do I restart the storage service? ===

<code>sudo service uwsgi restart</code>

=== How do I restart nginx? ===

<code>sudo service uwsgi restart</code>
<code>sudo service nginx restart</code>

=== How do I update or reset an ansible install? ===

To update an install, re-run <code>vagrant provision</code>. If you only want to run part of the ansible tasks, you can use ansible's tags, for example: <code>env ANSIBLE_ARGS="--tags=amsrc-ss-code" vagrant provision</code> More tags are documented in the [https://github.com/artefactual-labs/ansible-role-archivematica-src ansible repo]

To reset an install (delete all existing data like a fresh install) you can use ansible's role variables. For example: <code>env ANSIBLE_ARGS="--extra-vars=archivematica_src_reset_ss_db=true" vagrant provision</code> will reset the storage service database. More role variables are documented in the [https://github.com/artefactual-labs/ansible-role-archivematica-src ansible repo]

The other way to control a deployment is to modify the Vagrantfile and vars-singlenote.yml files directly. Tags can be provided in the Vagrantfile. For example:
<pre>
# ... more above
# Ansible provisioning
config.vm.provision :ansible do |ansible|
ansible.playbook = "./singlenode.yml"
ansible.host_key_checking = false
# ansible.verbose = "v"
ansible.extra_vars = {
"archivematica_src_dir" => "/srv",
"archivematica_src_environment_type" => "development",
}
ansible.raw_arguments = ENV['ANSIBLE_ARGS']
ansible.tags = ['amsrc-pipeline']
end
# ... more below
</pre>

Role variables can be modified in vars-singlenote.yml. Default values are found in the Archivematica role (in <code>roles/archivematica-src/defaults/main.yml</code>). For example:

<pre>
---

# archivematica-src role

# What to install
archivematica_src_install_devtools: "yes"
archivematica_src_install_automationtools: "yes"
# archivematica_src_install_appraisaltab: "yes"

# SS django environment variables
archivematica_src_ss_env_django_setings_module: "storage_service.settings.local"

# Branches,
archivematica_src_am_version: "qa/1.x"
archivematica_src_ss_version: "qa/0.x"
# archivematica_src_devtools_version: "master"
# archivematica_src_automationtools_version: "master"

# Reset
# archivematica_src_reset_mcpdb: "true"
# archivematica_src_reset_shareddir: "true"
# archivematica_src_reset_es: "true"
archivematica_src_reset_am_all: "true"
archivematica_src_reset_ss_db: "true"

</pre>

=== How do I activate the Storage Service virtualenv? ===

<code>source /usr/share/python/archivematica-storage-service/bin/activate</code>

The location of the virtualenv is configured as part of the ansible install, which by default is <code>/usr/share/python/archivematica-storage-service</code> Sourcing the activate script should modify the prompt to display <code>(archivematica-storage-service)vagrant@am-local</code>. Note that if you're running fish, the activate script will not work; you may want to look in to [http://virtualfish.readthedocs.io/en/latest/ virtualfish].

You will also want to set environment variables as described in the storage service testing section.

== Documentation ==

Developer facing documentation can be found in the [[:Category:Development documentation|Development documentation]] category. Notable pages include:

* [[MCPServer]]
* [[MCPClient]]
* [[Storage Service]]
* [[Storage Service API]]
* [[Archivematica API]]

[[Category:Development documentation]]