Meeting 20120328

Artefactual Systems, Internal Archivematica Dev Mtg, 2012-03-21

Development

• Joseph worked on METS struct map, waiting on QA review - Issue 939
• Joseph added a microservice to remove hidden directories and files in standard transfer processing Issue 910
• Joseph enclosed file pointers in Divs and worked on loading the label from the CSV file - Issue 939
• Mike did work on Issue 934 (incorrect icon displaying)
• Mike fixed Issue 575
• Mike did a bunch of refactoring and cleanup of the microservice grouping

Deployment

• CVA - Joseph gave commands to create a right click script to remove hidden files and directories
• SFU - Austin's fixes for email went well, added remote control scripts to their workstation so they could restart the MCP/Client

Testing

• Joseph has been working on queries to run on our distributed process testing
• Austin set up testing network on amazon ec2, running multinode with problems - Austin and Joseph testing
• Joseph set up testing network on Caius
• METS struct map fix needs QA review - Issue 939

Documentation

• Evelyn's requirements for METS changes: [1]
• Issue 593 requirements for BagIt ingest: [2]
• Testing distributed processing documented here: [3]

chat log

(10:31:28 AM) courtney: Archivematica time!
(10:31:45 AM) evelyn_mclellan: hi there
(10:31:46 AM) courtney: I think I can take notes 
(10:31:57 AM) evelyn_mclellan: thanks courtney
(10:32:18 AM) djjuhasz: JessicaB: thanks for taking notes JessicaB! :)
(10:32:18 AM) peterVG: dev?
(10:32:39 AM) berwin22: I did some work on sorting METS struct map
(10:33:03 AM) berwin22: it's waiting on qa review I believe.
(10:33:31 AM) evelyn_mclellan: the output you attached to the issue looked good
(10:33:54 AM) evelyn_mclellan: I've documented the requirements for METS changes at http://www.archivematica.org/wiki/index.php?title=METS#Changes_for_0.9
(10:34:02 AM) berwin22: added a microservice to remove hidden directories and files in standard transfer processing.
(10:34:48 AM) berwin22: Enclosed file ptr's in Div's and worked on loading the label from the CSV file.
(10:35:11 AM) mcantelon: Mike C did some work on issue 934 (incorrect icon displaying). http://code.google.com/p/archivematica/issues/detail?id=934
(10:35:45 AM) berwin22: epmclellan: Shouldn't it be file_labels.csv, not file_labels.txt?
(10:36:16 AM) evelyn_mclellan: yes, I changed the requirement to reflect that
(10:36:21 AM) berwin22: ah k
(10:36:28 AM) berwin22: I was just checking the issue
(10:37:07 AM) evelyn_mclellan: re issue 593, Add ability to ingest bagit SIPs
(10:37:12 AM) berwin22: I've been working on queires to run on our distributed process testing, but that falls into testing.
(10:37:22 AM) evelyn_mclellan: I've added requirements to the wiki at http://www.archivematica.org/wiki/index.php?title=Bag_ingest
(10:37:31 AM) berwin22: I saw those
(10:37:49 AM) berwin22: are we having a standar directory structure on ingest?
(10:37:58 AM) berwin22: are we renaming the bag with the UUID?
(10:38:08 AM) evelyn_mclellan: we should probably have a separate meeting to discuss the requirements
(10:38:13 AM) berwin22: are we only accepting zipped bags?
(10:38:46 AM) djjuhasz: somehow that ^ strikes me as funny
(10:39:08 AM) evelyn_mclellan: well, I suppose it is a little funny
(10:39:12 AM) ***djjuhasz waits for it
(10:39:31 AM) djjuhasz: peterVG: must be biting his digital tongue over there ;)
(10:39:39 AM) courtney: shall we talk about it after this meeting?
(10:39:51 AM) evelyn_mclellan: better to do it in person, I think
(10:39:53 AM) courtney: that is: general discussion about what we ingest
(10:39:55 AM) courtney: ok
(10:39:57 AM) evelyn_mclellan: next week ok?
(10:39:59 AM) peterVG: evelyn_mclellan: berwin22 we can review and update the bagit reqs in office soon? e.g Fri
(10:40:08 AM) evelyn_mclellan: sure
(10:40:13 AM) berwin22: sure
(10:40:18 AM) peterVG: i've zipped my bag before, it's no laughing matter
(10:40:24 AM) evelyn_mclellan: ow
(10:40:27 AM) courtney: will have to be wed - i'm out of the office tuesday (at border obtaining new work permit)
(10:40:41 AM) courtney: See Also "Something About Mary"
(10:40:51 AM) djjuhasz: lol, I waited for a long time, but I knew peterVG wouldn't disappoint :P
(10:40:54 AM) mcantelon: Hehe
(10:41:14 AM) evelyn_mclellan: I felt like I was the wrong gender to make a joke about it
(10:41:38 AM) evelyn_mclellan: er, moving on?
(10:41:43 AM) djjuhasz: yes please
(10:41:45 AM) mcantelon: Mike C fixed issue 575 (dashboard date localization). http://code.google.com/p/archivematica/issues/detail?id=575
(10:42:11 AM) courtney: awesome
(10:42:22 AM) berwin22: sweet
(10:42:37 AM) mcantelon: Mike C also did a bunch of refactoring and cleanup of the microservice grouping (which, I hadn't noticed, would kind of break after job refresh).
(10:43:01 AM) courtney: berwin22: is your div/csv work for Issue 939 or 937?
(10:43:15 AM) mcantelon: For issue 575 I just use JS to display dates so it uses the operating system's timezone rather than having the user set it specifically. Hopefully that's adequate.
(10:43:45 AM) peterVG: mcantelon: sounds great
(10:43:51 AM) mcantelon: Cool cool.
(10:44:04 AM) berwin22: courtney: both
(10:44:11 AM) courtney: thanks
(10:44:20 AM) berwin22: sorry, div is for 939
(10:44:39 AM) courtney: ok
(10:45:06 AM) courtney: testing?
(10:45:16 AM) courtney: oops, i mean deployment : )
(10:45:27 AM) courtney: i'm new
(10:45:46 AM) courtney: ok, then - testing?
(10:46:15 AM) ARTi: I setup a testing network has been setup on amazon ec2
(10:46:25 AM) berwin22: deployment - courtneyCVA - gave the commands to create a right click script to remove hidden files and directories
(10:46:34 AM) ARTi: its running multinode, but there may be something amiss atm  me and berwin22 have to take a look
(10:47:09 AM) berwin22: ARTi let me know if you need my help
(10:47:24 AM) djjuhasz: not testing multinode on caius?
(10:47:36 AM) berwin22: I've set up testing network on caius
(10:47:45 AM) ARTi: SFU, fix up went well, added some remote control scripts to their workstation so they could restart the MCP/Client 
(10:47:52 AM) berwin22: getting the first real resutls from it now.
(10:47:58 AM) djjuhasz: berwin22: nice
(10:48:36 AM) berwin22: getting: _mysql_exceptions.OperationalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')
(10:48:48 AM) berwin22: need to investigate
(10:48:52 AM) ARTi: berwin22: this afternoon, lets check it out
(10:49:08 AM) berwin22: ARTi: k
(10:49:48 AM) peterVG: want to follow progress boys, where's the wiki page on this testing? 
(10:50:48 AM) peterVG: should we add multi-node testing to http://archivematica.org/wiki/index.php?title=Scalability_testing?
(10:50:52 AM) peterVG: or keep seperate/
(10:50:52 AM) ARTi: peterVG: berwin22 just brought up standardization yesturday, maybe we should do it in the same spot we had been?
(10:50:58 AM) ARTi: I think its the same
(10:51:06 AM) peterVG: http://archivematica.org/wiki/index.php?title=Scalability_testing
(10:51:07 AM) peterVG: ?
(10:51:09 AM) ARTi: yeah
(10:51:10 AM) peterVG: yes,
(10:51:10 AM) berwin22: agree: it's the same
(10:51:13 AM) peterVG: okay ,cool
(10:51:25 AM) peterVG: please be diligent about documenting each step
(10:51:44 AM) peterVG: i know its a pain and not necessarily part of your usual flow
(10:52:03 AM) peterVG: but this is critical for Archivematica 1.0 success
(10:52:16 AM) ARTi: ok
(10:52:23 AM) peterVG: its easy to try 3 or 4 things and forget a week later where you started
(10:52:33 AM) peterVG: we need to be scientific about this
(10:52:42 AM) ARTi: YAY science!
(10:52:49 AM) JessicaB: scientific in a qualitative manner
(10:52:57 AM) berwin22: ARTi: what do you want to standardize on for testing data?
(10:53:00 AM) peterVG: ^ what Prof Bushey said
(10:53:31 AM) ARTi: berwin22: I saw that email about the 63GB folder
(10:53:36 AM) djjuhasz: that's an contradiction if I've ever heard one
(10:53:58 AM) ARTi: is there something offline we can grab?  might be better then transfering that folder to the web
(10:54:04 AM) peterVG: djjuhasz: then we have a textbook for you
(10:54:44 AM) djjuhasz: qualitative science is usually called BAD science
(10:54:46 AM) djjuhasz: :P
(10:54:59 AM) berwin22: ARTi, suggestions?
(10:55:40 AM) JessicaB: djjuhasz: bang on.
(10:55:41 AM) ARTi: there are those document collections
(10:55:51 AM) ARTi: 001.zip-999.zip
(10:56:59 AM) ARTi: anyways.. lets get multinode working on ec2
(10:57:05 AM) ARTi: and then we can decide on some heavy data
(10:57:32 AM) berwin22: what type of scale are we looking to test?
(10:57:32 AM) berwin22: Should we try to repeat? http://archivematica.org/wiki/index.php?title=Scalability_testing
(10:57:36 AM) berwin22: ok
(10:58:13 AM) courtney: ARTi: berwin22: you guys know about this? http://digitalcorpora.org/
(10:58:18 AM) courtney: lots of stuff for testing
(10:58:39 AM) evelyn_mclellan: yes, we have used that in the past
(10:58:42 AM) berwin22: Well I already know there is an issue from my testing. Need to do the investigation on that.
(10:59:15 AM) courtney: 1 more minute - anything else?
(10:59:27 AM) courtney: mcantelon: i'll have some mockups for the file browser to you by the end of the day today - and some for transfer
(10:59:47 AM) mcantelon: courtney: Ah, cool!
(10:59:47 AM) evelyn_mclellan: nothing from me, have been working mainly on SFU project
(10:59:57 AM) ARTi: courtney: yeah, those were the docs I was suggesting
(11:00:03 AM) courtney: cool
(11:00:28 AM) peterVG: courtney: we're going to need you to take your newly acquired mockup talents to DIP upload after that
(11:00:51 AM) peterVG: the contentDM work is raising issues about how to make this more generic, flexible and useable
(11:00:56 AM) courtney: will do