Workflow dashboard

Main Page > [[Projects] > Vancouver Digital Archives > Software Integration/Development

Workflow dashboard tool

Digital Archives Workflow Event Tracker (WETboard!): (should come up with a better name so that it has a catchier acronym)

There should be a tool that tracks each SIP through the ingest process (this has expanded to all events in the system, not just Ingest events). I see this as a database that contains administrative metadata about the ingest process all events, and an interface that allows the archivist to query the database to view the status of SIPs/AIPs, and potentially an interface that allows the archivist to select SIPs/AIPs and initiate processes on them. We may also want some code that restricts certain actions if it is likely to cause the system to choke-up

• When SIPs are received by the archives, SIPs are registered on the WETboard
• As SIP progresses through the ingest process all event stages, the SIP record on the WETboard is updated to reflect what stage in the process the SIP is at, did specific processes run properly without crashing, summary results of inspection events (integrity checks, malware checks, etc.), the actual location of the SIP, the AIP or their components, type, name and location of reports produced, messages that have been sent to RIM or other creators (or users?) about SIP progress, etc
• Some of the information will be explicitly present in the metadata that is produced during ingest, some of it will be implicit in the metadata, and some will be implicit by the fact that some metadata exists. I think there should be a way of pulling all of this together so that the archivist can see what is going on, rather than having to look into several logs
• I put together a table with some of the WETboard metadata elements I would expect to see. This should be taken as an illustrative example of the types of things this tool should be tracking, rather than a legit attempt to sketch out the metadata.
• These will likely correspond to PREMIS event entities that are yet to be defined (attempts are being made here:

see also Metadata entity analysis

EXAMPLE

Etc.

Element	Description/Purpose	Data Type
SIP_ID	Unique ID of the SIP	string
Transfer_ID	Unique ID of transfer SIP was attached to	string
TransferDate	Time SIP was placed in network share	date/time
SIPtransfer_location	Location the received SIP was transferred to	Network path (e.g., /home/demo/sharedfolder/[SIPname])
SIPbackup_location	Location that the back-up copy of the received SIP was transferred to	Network path (/othermachine/[SIPname]
SIP_metadata	Network path of the .xml metadata in the SIP	Network path (e.g., /home/demo/sharedfolder/[SIPname]/[SIPname].xml)
SIP_file_count	Count of the total number of files included in the received SIP	Integer
SIP_contentObject_count	Count of the total number of content objects included in the received SIP	Integer
Current_State	Name of the process currently running on the SIP	{IngestQueue, Audit, AppraisalQueue, NormalizationQueue, Normalization, StorageTransferQueue, etc.}
Receipt_acknowledged	Has message been sent to RIM acknowledging receipt of SIP in transfer folder	{yes, no}
Audit_Start	Time audit process was initiated	Date/time
Audit_End	Time audit ended	Date/time
Manifest_Check_success	Did the manifest check run properly? (i.e., "did it run without crashing?", not "was the result of the check positive?")	{yes, no}
Manifest_Check	Result of manifest check	{pass, fail}
Metadata_Check_success
Metadata_Check	Result of metadata check	{pass, fail}
Integrity_Check_success
Integrity_Check	Result of md5 integrity check	{pass, fail}
Begin_Quarantine	Time SIP entered quarantine	Date/time
End_Quarantine	Time SIP exited quarantine	Date/time
Malware_Check	Result of Malware check	{pass, fail}
Files_inspected	Count of all files inspected by malware check	Integer
Clean_files	Count of all files with no malware present on inspection	Integer
Repaired_files	Count of infected files that were successfully repaired	Integer
Infected_files	Count of infected files that were not repaired	Integer
Isolation_location	Network location infected files were moved to	Network path (e.g., /home/demo/malwareIsolation/[SIPname]/)
Audit_Result	Result of the audit	{Pass, Fail, conditionalPass(?)}
Audit_Log	Network location of the audit log
RIM_auditNotification	Has message been sent to RIM notifying of audit results
RIM_resubmissionRequest	Has RIM been sent a resubmission request
Accession_ID	Accession ID assigned to SIP
Normalized	Has SIP content been normalized
Normalization_report	Location of normalization report