Dashboard

Main Page > Development > Development documentation > Dashboard

This page is no longer being maintained and may contain inaccurate information. Please see the Archivematica documentation for up-to-date information.

Design

This page proposes a new feature and reviews design options

Development

This page describes a feature that's in development

Documentation

This page documents an implemented feature

The dashboard manages a pipeline’s behaviour. It provides a view into the status of units, allows workflow decisions to be made, handles configuration, allows arrangement & description, and customization of the FPR. Code for the dashboard is found in src/dashboard/src/

Previously the dashboard also handled starting transfers from disk & storing files, but much of that functionality has been moved to the storage service.

Sections

• components/access: Internal API to talk to access systems (ArchivesSpace, Archivist’s Toolkit). Primarily supports the appraisal tab, and uses [agentarchives]
• components/accounts: Views related to account management (create, edit, delete, list)
• components/administration: Views & models related to the Administration tab. Contains the processing config form, DIP upload settings, etc
• components/api: External API endpoints.
• components/appraisal: Appraisal tab view.
• components/archival_storage: Views related to the Archival Storage tab. Contains ElasticSearch queries, creating AICs, deleting AIPs and starting reingest.
• components/backlog: Views related to the Backlog tab. Contains ElasticSearch queries, deleting & downloading backlogged transfers.
• components/file: Internal API to get file information from ElasticSearch & Storage Service. Primarily supports the appraisal tab.
• components/filesystem_ajax: Internal API to support SIP Arrangement and moving files to and from the pipeline. Much of its previous functionality has been superseded by the storage service
• components/ingest: Views and internal API related to the Ingest tab.
• components/mcp: Internal API for status updates from MCPServer
• components/rights: Views and forms for editing unit rights.
• components/transfer: Views and internal API related to the Transfer tab.
• components/unit: Views and internal API common between Transfers and SIPs

Improvement Note: This is newer than components/transfer and components/ingest. Functionality that is the same between the two should be moved here as appropriate

• external: External dependencies, often git submodules. Note: as of Archivematica 1.7, there is no external/ directory here. The base64-helpers subdirectory is now required by the transfer-browser and appraisal-tab and is downloaded during the NPM build phase.
• fpr: Views related to the Preservation Planning tab. This in the FPR-admin submodule from externals, and contains all FPR modification views. Note: as of Archivematica 1.7, there is no fpr/ directory here. The archivematica-fpr-admin module is now a dependency of the dashboard; see dashboard/src/requirements/base.txt.
• installer: Views related to installation and setting up Archivematica
• main: Models, views and internal APIs related to core dashboard functionality. Contains all model definitions & migrations, as well as app-wide configuration. Also includes Task & Job display, and views related to the Access tab.
• media: All CSS, JS & images for the dashboard
• middleware: Django middleware definitions.
• requirements: Python requirements files
• settings: Django settings modules for development or production configuration
• templates: All templates for the dashboard. Structure broadly mirrors that of the components directory.

Old documentation

Technical Requirements

The Dashboard is a web-based tool that is developed using Python-based Django MVC framework.

Functional Requirements

• provide a web-based, multi-user interface that will report on the status of system events and make it simpler to control and trigger specific micro-services.
• provide a user-friendly interface to add/edit metadata
• coordinate the read and write operations of the AIP to file storage and the syncing of metadata updates between the AIPs and the access system.
• process Consumer AIP requests
• provide statistical information about Archivematica operation
• provide preservation planning information

User interface

Release 0.7-alpha (Feb 18, 2011)

Django interface found in Archivematica 0.6.2 (dev tree, 29 Nov 2010)

Early mockup (March 2010)

Real-time interaction

Our preliminary design will be based in periodic refresh, trying to minimize the risks of more sophisticated solutions before Release 0.7-alpha is launched. In future releases, we will do more research on this topic trying to achieve the best user experience while we keep an eye on performance.

The Ajax web application model came to made the Web UI experience dynamic and asynchronous, as a replacement of the classic page-by-page web application model (see graph). However, Ajax applications don't offer a duplex communication where both client and server can send messages at any time. A new model of web applications frequently called Comet appeared providing bi-directional communications using persistent long-lasting HTTP connections between the server and the client (see graph). Comet is similar to Ajax in that it's asynchronous, but applications following the Comet model can communicate state changes on the server with almost negligible latency, which makes it suitable for monitoring or multi-user collaboration applications.

There exists different methods of implementing a Comet streaming transport (browser transport), but all of them are based in existing browser features: iframe HTML element, XMLHttpRequest or script tags. Between these methods, I think that we have two candidates:

• XMLHttpRequest long polling: firstly, the browser creates an asynchronous XMLHttpRequest with a long time-out. When we receive a response, the server closes the connection and we launch another XHR request immediately afterward, waiting for a new event.
• Script tag long polling: the browser creates script HTML elements dinamically and setting their source ("src" attribute) to the location of the server, which then send back JavaScript code. Each time the script requests is completed, the browser opens a new one, just like XHR long polling design does. This method bypass the same-origin policy security mechanism implemented in modern browsers.

Alternatives

There exists other alternatives that we should consider:

• WebSockets: this technology is part of HTML5 and provides full-duplex communications channels over a single TCP socket between the browser and the server. The WebSocket API is being standardized by the W3C and the WebSocket protocol is being standardized by the IETF (HyBi working group). Chrome 4, Safari 5, Firefox 4 (not yet in FF3) and Opera 11 support WebSockets. However, the last two ones have disabled this protocol by default. HTML5 Labs at Microsoft interoperability group recently launched a prototype compatible with IE8 and IE9 based in Silverlight. WebSockets is a promissing technology but unfortunately in the development phase yet.
  • There are some solutions which provide an API that looks like WebSocket API, and fallback to other techniques if WebSocket is not available. A good example is Socket.IO, which supports different transports: WebSocket, Adobe® Flash® Socket, Ajax long polling, etc... however, the server module was designed for Node.JS. Several implementations have been started for other languages / frameworks that are compatible with the Socket.IO client.
  • Other products like CometD, Lightstreamer and others provide a higher-level API using pubsub (see mod_pubsub) or some other messaging protocol, and use WebSocket or whatever other transport is available that is the fastest and safest option.
• Server-Sent Events: another draft API included in HTML5 designed for scenarios where data does not need to be sent from the client, just need updates from the server (server push only). This technology, only supported by some browsers like Chrome or Opera, could be considered as a formal and efficient alternative to Comet, but based in the same method: HTTP long-held requests. The big difference with WebSockets is, therefore, that it does not to implement a new protocol (it is based in HTTP) and it is not really full-duplex, although it could be simulated with parallel XHR requests.
• Periodic refresh (simple polling): to keep users informated about changes occurring on the server we can make the browser generates requests periodically, at fixed intervals, to gain new information: for example, one call every five seconds. This is a valid approximation where the server push data if data latency is not a critical for users. A callback function would be responsible for updating the DOM according to the server's latest report and the browser script can do some monitoring and dynamically adjust the period of refreshes to minimize the workload (e.g.: to cease it when the system detects the user is no longer active, see this article).
  • Lace: Old open source chat based in simple polling (periodic refresh).

Server design and scalability

When a web application creates bi-directional connections between the browser and the server, new server software is often required in order to scale well. Take into account that traditional web-based solutions would break down very quickly due to memory consumption and the excess overhead of framework for each HTTP (and possibly long-held) request made.

More research on this must be done if we decide to take advantage of Comet or WebSockets technologies. These are some initial notes:

• Apache MPM event: this experimental module included in Apache 2.2 has the potential to bring Twisted-esque funcionality within the Apache pipeline. It can save significant overhead in creating TCP connections, however, Apache traditionally keeps an entire child process/thread waiting for data from the client, which brings its own disadvantages. To solve this problem, this MPM uses a dedicated thread to handle both the listening sockets, and all sockets that are in a Keep Alive state.
• Tornado: an open source version of the scalable, non-blocking web server and tools that power FriendFeed. It is ideal for real-time web services. It is not just a web server, it could be considered a real-time web framework. It can serves Django applications.
• Twisted, eventlet, gevent, Tornado, Node.JS, greenlet, celery

Some recipes:

• Django + Orbited + Twisted: and an example, a realtime webapp: Hotdot.
• Using Django in Non-Standard Ways
• Simple long polling with Django and gevent
• Django and Comet using Orbited
• Comparing gevent to eventlet
• A clean, lightweight alternative to Python's twisted?
• Making moves w/ websockets and python/django(/twisted?)
• Quick notes on trying the Twisted websocket branch example

Debug mode

By default, the dashboard runs in "production" mode. To diagnose application errors it is usually useful to run in debug mode. Debug mode will display error messages. If you want to enable it, please follow these instructions:

1. Go to dashboard sources directory
2. Open settings.py file with your preferred text editor
3. Find the following line

   Debug = False

4. Update the False flag to True

   Debug = True

5. Save the file
6. Restart Apache

   sudo /etc/init.d/apache2 restart