Meeting 20130911

Archivematica

Development

• Holly did more updates in the dashboard to make it work with Django 1.5 #5207
• Holly updated the AIP pointer file to represent different compression algorithms (.7z vs .tar.bz2) with Evelyn1's METS-y help. #5159
• Holly added a GUI for editing IDRules #5073
• Holly is making slow progress integrating the FPR v2 API into the MCP Server so all our pretty new data models actually do something. Will probably take at least the rest of the week. #5575
• Mike C worked on making the FPR server's admin interface a pluggable Django app so we can reuse it on the Archivematica dashboard.
• Mike C worked on the FPR admon interface.
• Evelyn: We need an FPR usability review next week to see how much more should be done for 1.0 release
• Misty has a couple of FITS optimizations ready for testing in a realworld environment: #5512 (disabling checksum generation), and #5509 (using nailgun to remove JVM startup time). Evelyn: it would be good to test these on both archmat4 (Yale) and at MoMA.
• Holly and Misty were discussing the longterm possibilities for file identification and characterization. We're hoping to get some of the groundwork laid that we need to get that done as soon as possible, so we don't have to make too many dramatic chances to database structure in the future
• Misty has preliminary microservices for DROID identification and exiftool and mediainfo characterization which can be slotted in once the infrastructure is in
  • We can't change the database structure too much after 1.0, so a lot of this work has to be done before 1.0
  • Holly: Since I'm making changes to all that stuff anyway, I'll be leaving it open for the improvements Misty and I (and Justin and Mike) have been discussing.

Testing

• Evelyn has started QA for the 1.0 release, courtney will take over some of that when she gets back next week

Documentation

• Holly put some of her notes on the wiki (https://www.archivematica.org/wiki/MCP) and updated what was there. Started working on more notes to explain core MCP stuff. #5092

Chat log

(10:35:24 AM) evelyn1: I can take notes
(10:35:28 AM) JessicaB left the room (quit: Quit: Leaving.).
(10:35:51 AM) evelyn1: development?
(10:36:05 AM) mcantelon: Thx evelyn1
(10:36:09 AM) evelyn1: np
(10:36:42 AM) evelyn1: kerry: sorry, I answered phone, forgot you were here!
(10:36:55 AM) Holly: Holly did more updates in the dashboard to make it work with Django 1.5 #5207
(10:36:55 AM) qubot: Links: https://projects.artefactual.com/issues/5207
(10:36:56 AM) kerry: evelyn1: no problem!
(10:37:17 AM) Holly: Holly updated the AIP pointer file to represent different compression algorithms (.7z vs .tar.bz2) with Evelyn1's METS-y help.  #5159
(10:37:17 AM) qubot: Links: https://projects.artefactual.com/issues/5159
(10:37:32 AM) Holly: Holly added a GUI for editing IDRules #5073
(10:37:32 AM) qubot: Links: https://projects.artefactual.com/issues/5073
(10:38:01 AM) Holly: Holly is making slow progress integrating the FPR v2 API into the MCP Server so all our pretty new data models actually do something.  Will probably take at least the rest of the week.  #5575
(10:38:02 AM) qubot: Links: https://projects.artefactual.com/issues/5575
(10:38:09 AM) evelyn1: Holly: how many more hours on FPR dev for 1.0?
(10:38:14 AM) evelyn1: just looking for guesstimate
(10:38:43 AM) evelyn1: Is #5575 the last issue for 1.0?
(10:38:43 AM) qubot: Links: https://projects.artefactual.com/issues/5575
(10:39:12 AM) Holly: evelyn1: Not sure if there are more feature/bugs - Justin might know?
(10:39:15 AM) mcantelon: Mike C worked on making the FPR server's admin interface a pluggable Django app so we can reuse it on the Archivematica dashboard.
(10:39:29 AM) evelyn1: Holly: we'll check with Justin when he gets back
(10:40:29 AM) Holly: evelyn1: For time estimates, ummmm, at least the rest of this week, so 20-30 hours?  I'm really guessing.  (Skill I need to work on)
(10:40:46 AM) evelyn1: Holly: it's very hard to estimate those hours, I know
(10:40:58 AM) evelyn1: It looks like a lot of our 1.0 dev roadmap features are done
(10:41:06 AM) evelyn1: https://www.archivematica.org/wiki/Development_roadmap:_Archivematica_1.0
(10:41:25 AM) evelyn1: next week we'll probably need to start focusing on bug fixes and minor tweaks
(10:41:36 AM) kerry left the room.
(10:41:52 AM) kerry [~kcampbell@184.69.130.182] entered the room.
(10:42:03 AM) evelyn1: Holly and mcantelon unless there's some big monster feature I'm not aware of?
(10:42:07 AM) mcantelon: Mike C worked on the FPR admon interface.
(10:42:25 AM) mcantelon: Admin I mean.
(10:42:34 AM) evelyn1: I know, mcantelon :)
(10:42:40 AM) Holly: evelyn1:  I feel like the FPR qualifies, but it's getting worked on. :)
(10:42:42 AM) evelyn1: I didn't think we were going to admonish the users
(10:42:56 AM) evelyn1: Holly: yeah, we all know FPR is a monster!
(10:43:08 AM) evelyn1: I guess we need an FPR usability review next week
(10:43:13 AM) mcantelon is now known as m2
(10:43:17 AM) evelyn1: to see how much more should be done before 1.0
(10:43:33 AM) mcantelon [~mike@184.69.130.182] entered the room.
(10:44:08 AM) evelyn1: mcantelon: the dashboard fail reports are very nice
(10:44:36 AM) evelyn1: I just proposed one small enhancement, to add delete buttons to each row in the reports table
(10:44:40 AM) mcantelon: evelyn1: Thanks! :D
(10:44:43 AM) Misty: I have a couple of FITS optimizations ready for testing in a realworld environment: #5512 (disabling checksum generation), and #5509 (using nailgun to remove JVM startup time).
(10:44:43 AM) qubot: Links: https://projects.artefactual.com/issues/5512 | https://projects.artefactual.com/issues/5509
(10:45:01 AM) mcantelon: evelyn1: Makes sense... I'll make an issue for that.
(10:45:06 AM) Misty: I've submitted pull requests for these (and a third fix that doesn't directly impact our usecase) to Harvard.
(10:45:09 AM) evelyn1: mcantelon:  did already
(10:45:20 AM) Misty: In the meanwhile ARTi1 and I are working on packaging these so we can give the ma good test run.
(10:45:24 AM) mcantelon: evelyn1: ah cool
(10:45:31 AM) evelyn1: mcantelon: just added it to #1929
(10:45:31 AM) qubot: Links: https://projects.artefactual.com/issues/1929
(10:46:10 AM) Misty: Essentially we need a real world-ish test with a large dataset, so we can directly compare what benefit each brings compared to a vanilla run.
(10:46:25 AM) evelyn1: how large a dataset?
(10:46:56 AM) Misty: In the thousands of files for sure. I've tested with ~1500 files; something in that region, or larger if possible, would be preferable
(10:47:20 AM) evelyn1: can we just multiply some of our sample data?
(10:47:25 AM) evelyn1: eg OPF format corpus?
(10:49:25 AM) Misty: There are a lot of variables in terms of what affects FITS performance and we haven't tweaked all of them.
(10:49:27 AM) Misty: Sounds good.
(10:49:59 AM) evelyn1: oh boy, fast FITS!
(10:50:13 AM) mcantelon: Will be the end of a era
(10:50:25 AM) evelyn1: for sure
(10:50:47 AM) Misty: Don't get your hopes *too* high. This is fast*er* FITS but not fast by a long shot ;)
(10:50:59 AM) evelyn1: oh, I know, I know
(10:51:23 AM) Misty: Holly and I were discussing the longterm possibilities for file identification and characterization. We're hoping to get some of the groundwork laid that we need to get that done as soon as possible, so we don't have to make too many dramatic chances to database structure in the future
(10:51:45 AM) evelyn1: does this groundwork need to be laid before 1.0 release?
(10:51:50 AM) Misty: I do have preliminary microservices for DROID identification and exiftool and mediainfo characterization which can be slotted in once the infrastructure is in
(10:52:05 AM) Misty: My understanding is we can't change the database structure too much after 1.0, so probably as much as possible, yes
(10:52:15 AM) Holly: Since I'm making changes to all that stuff anyway, I'll be leaving it open for the improvements Misty and I (and Justin and Mike) have been discussing.
(10:52:31 AM) evelyn1: ah, sounds good
(10:56:27 AM) evelyn1: k, we can move on to documentation, testing etc
(10:56:56 AM) evelyn1: I've started QA for the 1.0 release, courtney will take over some of that when she gets back next week
(10:57:12 AM) evelyn1: I think that's about it
(10:57:43 AM) evelyn1: shall we call this a wrap?
(10:57:43 AM) Holly: Holly put some of her notes on the wiki (https://www.archivematica.org/wiki/MCP) and updated what was there.  Started working on more notes to explain core MCP stuff.  #5092
(10:57:43 AM) qubot: Links: https://projects.artefactual.com/issues/5092
(10:57:51 AM) evelyn1: oh, thanks Holly
(10:58:19 AM) Holly: Since Misty couldn't find them when I was gone, I figured the public wiki was a better place than my user page on the internal wiki. :)
(10:58:27 AM) evelyn1: heh. really?
(10:58:56 AM) Misty: Thanks Holly!
(10:59:29 AM) mcantelon: Holly: Nice!
(11:01:16 AM) evelyn1: ok, calling this a wrap