Difference between revisions of "Meeting 20110427"

From Archivematica
Jump to navigation Jump to search
(Created page with '== Development == * == Documentation == * == Testing == * == Chat log == <pre> </pre>')
 
 
(9 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
== Development ==
 
== Development ==
−
*  
+
* Sevein has been working on the dashboard, just solved son minor js issues with latest backbone/underscore libraries
 +
* Sevien will work on [http://code.google.com/p/archivematica/issues/detail?id=464 issue 464] as per evelyns request
 +
* Joseph found the underlying protocol the mcp was using had a max buffer size that needed to be increased for flv normalization output to be sent back
 +
* Joseph is working on getting accession documentation into workflow. Waiting for evelyn's feedback ([http://code.google.com/p/archivematica/issues/detail?id=458 issue 458]).
 +
* Joseph is working on getting a means to add metadata for manually normalized files.
 +
* Evelyn edited the database file to add the normalization path to Archivematica
  
 
== Documentation ==
 
== Documentation ==
  
−
*  
+
* Joseph created a transcoder db [http://archivematica.org/wiki/index.php?title=Transcoder diagram and documentation]
 +
* Evelyn  added [http://www.archivematica.org/wiki/index.php?title=Macintosh_PICT_Image preservation plan and documentation] for pct files
  
 
== Testing ==
 
== Testing ==
  
−
*  
+
* Austin's working on [[Scalability testing]].
  
 
== Chat log ==
 
== Chat log ==
 
<pre>
 
<pre>
 +
(10:34:10 AM) peterVG: cool
 +
(10:34:14 AM) berwin22: let me check, I know I have something
 +
(10:34:18 AM) Sevein: and will continue with some issues that Evelyn suggested me
 +
(10:35:20 AM) peterVG: What is next for Dashboard? I believe Joseph and Evelyn have been working on normalization error reporting and/or manual normalization support. This is ready for a Dashboard interface?
 +
(10:36:41 AM) Sevein: I am adapting our code to show multiple windows
 +
(10:36:41 AM) Austin: Sevein: can I give this to you - http://code.google.com/p/archivematica/issues/detail?id=467&q=owner%3Aaustin%40artefactual.com&colspec=ID%20Milestone%20Component%20Type%20Status%20Priority%20Owner%20Summary
 +
(10:37:01 AM) epmclellan: yes, we would like to trigger a PREMIS script from the dashboard, it will create PREMIS metadata for files manually normalized
 +
(10:37:10 AM) epmclellan: sec, I'll find the issue...
 +
(10:37:20 AM) Austin: Sevein: it will help trouble shoot multinode installs
 +
(10:37:36 AM) epmclellan: http://code.google.com/p/archivematica/issues/detail?id=464
 +
(10:37:45 AM) berwin22: found the underlying protocol the mcp was using had a max buffer size that needed to be increased for flv normalization output to be sent back
 +
Working on getting accession documentation into workflow. Waiting for evelyn's feedback (issue 458).
 +
Working on getting a means to add metadata for manually normalized files.
 +
 +
 +
(10:38:34 AM) berwin22: There is now an approval step after normalization
 +
(10:38:38 AM) Sevein: not sure what you exactly want to do in issue
 +
(10:38:50 AM) epmclellan: Sevein: I think Joseph can explain
 +
(10:39:02 AM) epmclellan: he has written a script
 +
(10:39:14 AM) epmclellan: and we'd like the user to be able to trigger it from the dashboard
 +
(10:39:24 AM) epmclellan: is that right, berwin22?
 +
(10:39:48 AM) Sevein: oh I see
 +
(10:40:26 AM) berwin22: Sevein, it's a python function to call with certain parameters
 +
(10:40:45 AM) berwin22: I'm hoping you can import it and use it from it's current place in the code.
 +
(10:41:19 AM) berwin22: it can be a script if it needs to be, but I thought a function might be easier for you to work with
 +
(10:41:37 AM) Sevein: ok, I'll see what can I do
 +
(10:41:51 AM) Sevein: give me some time to read more about and learn how it works
 +
(10:41:58 AM) Sevein: time = minutes :D
 +
(10:42:09 AM) Sevein: well, ASAP :D
 +
(10:42:32 AM) berwin22: http://code.google.com/p/archivematica/source/browse/trunk/src/transcoder/lib/premisXMLlinker.py
 +
def xmlCreateFileAssociationBetween(originalFileFullPath, outputFromNormalizationFileFullPath, SIPFullPath, eventDetailText, objects="objects/", outputFileUUID="", eventUUID="", edate=""):
 +
(10:42:50 AM) Sevein: nice
 +
(10:43:33 AM) peterVG: Austin what dev will you be working on this week?
 +
(10:44:00 AM) Austin: working on scalability testing
 +
(10:44:11 AM) peterVG: awesome
 +
(10:44:12 AM) Austin: have to get a 1TB hard drive
 +
(10:44:21 AM) Austin: or setup a bigger raid
 +
(10:44:24 AM) epmclellan: Austin: are we going to City of Vancouver Archives?
 +
(10:44:38 AM) epmclellan: to do testing there?
 +
(10:44:38 AM) Austin: right now largest I have to work with is 500
 +
(10:44:44 AM) peterVG: I've got a spare 1TB external hard disk at the office
 +
(10:45:11 AM) peterVG: epmclellan: first Austin will setup his own test environment. then we'll run the same tests on CVA's setup
 +
(10:45:16 AM) epmclellan: ah
 +
(10:45:21 AM) Austin: epmclellan: yes this ^
 +
(10:45:33 AM) peterVG: Austin: I guess we've ruled out using VMs on Ciaus due to the lag of the VM layer?
 +
(10:45:48 AM) Austin: peterVG: ok, I think Ill just pick up a internal
 +
(10:46:27 AM) Austin: yeah.. well and it was running into some space issues already
 +
(10:46:46 AM) peterVG: MJ Suhonos: anything to add about using ciaus for scalability testing?
 +
(10:47:01 AM) Austin: Im pretty sure backups are heading that way now.. and there isnt much room to play around
 +
(10:47:33 AM) peterVG: k
 +
(10:47:33 AM) Austin: much room like.. more then a few 100 gbs
 +
(10:47:39 AM) peterVG: k
 +
(10:48:27 AM) Austin: at the same time Ill be polishing up the install scripts
 +
(10:48:36 AM) MJ Suhonos: peterVG:  I think the VM on caius was fine for testing, but i wouldn't use it to deploy to production
 +
(10:48:53 AM) peterVG: MJ Suhonos: this is for more testing
 +
(10:48:54 AM) MJ Suhonos: a better real-world testing platform would be caius on bare metal, and then scrap/rebuild after each test project
 +
(10:49:07 AM) Austin: making them a bit more versatile.. and remove some of the excess errors during the environment build ( that folks have complained about)
 +
(10:49:39 AM) MJ Suhonos: peterVG:  i guess the question wrt. a VM is whether your testing is relative like it was in LAC or absolute, ie. to propose deployment scanerios
 +
(10:49:44 AM) MJ Suhonos: erm scenarios
 +
(10:49:54 AM) MJ Suhonos: Scanerios! part of a delicious breakfast!
 +
(10:50:02 AM) peterVG: yum
 +
(10:50:08 AM) peterVG: http://archivematica.org/wiki/index.php?title=Scalability_testing
 +
(10:50:15 AM) peterVG: MJ Suhonos: ^ it's similar to LAC testing
 +
(10:50:25 AM) MJ Suhonos: yeah i think an archivematica VM on caius would be fine
 +
(10:50:27 AM) Sevein: oh, we have news from LAC?
 +
(10:50:45 AM) MJ Suhonos: Sevein:  no news, just referring to our VM setup for that project
 +
(10:50:50 AM) Sevein: ok
 +
(10:51:22 AM) MJ Suhonos: peterVG:  i would just make sure the fact that the VM imposes significant overhead (esp. in disk i/o) is noted in the testing documentation
 +
(10:51:49 AM) Austin: peterVG: do you have the link to those free file sets we were looking at last week
 +
(10:52:29 AM) Austin: yeah, I was running a multinode install with 4 VMs on the same host.. and was getting very very poor performance
 +
(10:52:58 AM) ***peterVG looking for link
 +
(10:53:09 AM) MJ Suhonos: Austin:  this was using VMDKs?
 +
(10:53:10 AM) Austin: much worse then when we are doing bare metal
 +
(10:53:28 AM) Austin: nah, kvm qcow images
 +
(10:53:43 AM) MJ Suhonos: ah.  still consolidated disk images though
 +
(10:53:59 AM) Austin: mm, ahh.. as opposed to raw images?
 +
(10:54:04 AM) berwin22: I'm curious how that would differ from one vm running 8 "threads" instead
 +
(10:54:10 AM) MJ Suhonos: as opposed to native FS on bare metal :)
 +
(10:54:32 AM) Austin: mmm
 +
(10:54:41 AM) Austin: yeah berwin22, Im not sure it would be much different
 +
(10:54:46 AM) Austin: it was all on one host
 +
(10:54:48 AM) epmclellan: peterVG: http://digitalcorpora.org/corpora/files
 +
(10:54:56 AM) Austin: if I had raid.. I think it would work better
 +
(10:54:59 AM) Austin: epmclellan: thanks!
 +
(10:55:04 AM) berwin22: the vm software may handle it better
 +
(10:55:32 AM) berwin22: yeah... I actually lost a disk again this week :P turned my fans too low and went out.
 +
(10:57:43 AM) peterVG: okay any documentation?
 +
(10:58:49 AM) berwin22: from last week: transcoder db http://archivematica.org/wiki/index.php?title=Transcoder
 +
(10:58:50 AM) Austin: notes so far http://archivematica.org/wiki/index.php?title=Meeting_20110427
 +
(10:59:06 AM) Austin: berwin22: awesome
 +
(11:00:21 AM) epmclellan: added preservation plan and documentation for pct files
 +
(11:00:39 AM) epmclellan: http://www.archivematica.org/wiki/index.php?title=Macintosh_PICT_Image
 +
(11:01:05 AM) epmclellan: I did this because SFU has pct files, it was on the list of formats that Ian Forsyth gave me
 +
(11:01:35 AM) epmclellan: I think that's it for me
 +
(11:02:04 AM) epmclellan: I also edited the database file to add the normalization path to Archivematica
 +
(11:02:14 AM) epmclellan: sorry, that's dev, I'm not on top of this meeting
 +
(11:02:21 AM) epmclellan: due to impending workshop deadline
 +
(11:02:22 AM) peterVG: nice work on the Transcoder docs Joseph!
 +
(11:04:12 AM) peterVG: okay 11 o'clock. i think that's it
 
</pre>
 
</pre>
 +
[[Category:meetings]]

Latest revision as of 15:24, 21 September 2011

Development[edit]

  • Sevein has been working on the dashboard, just solved son minor js issues with latest backbone/underscore libraries
  • Sevien will work on issue 464 as per evelyns request
  • Joseph found the underlying protocol the mcp was using had a max buffer size that needed to be increased for flv normalization output to be sent back
  • Joseph is working on getting accession documentation into workflow. Waiting for evelyn's feedback (issue 458).
  • Joseph is working on getting a means to add metadata for manually normalized files.
  • Evelyn edited the database file to add the normalization path to Archivematica

Documentation[edit]

Testing[edit]

Chat log[edit]

(10:34:10 AM) peterVG: cool
(10:34:14 AM) berwin22: let me check, I know I have something
(10:34:18 AM) Sevein: and will continue with some issues that Evelyn suggested me
(10:35:20 AM) peterVG: What is next for Dashboard? I believe Joseph and Evelyn have been working on normalization error reporting and/or manual normalization support. This is ready for a Dashboard interface?
(10:36:41 AM) Sevein: I am adapting our code to show multiple windows
(10:36:41 AM) Austin: Sevein: can I give this to you - http://code.google.com/p/archivematica/issues/detail?id=467&q=owner%3Aaustin%40artefactual.com&colspec=ID%20Milestone%20Component%20Type%20Status%20Priority%20Owner%20Summary
(10:37:01 AM) epmclellan: yes, we would like to trigger a PREMIS script from the dashboard, it will create PREMIS metadata for files manually normalized
(10:37:10 AM) epmclellan: sec, I'll find the issue...
(10:37:20 AM) Austin: Sevein: it will help trouble shoot multinode installs
(10:37:36 AM) epmclellan: http://code.google.com/p/archivematica/issues/detail?id=464
(10:37:45 AM) berwin22: found the underlying protocol the mcp was using had a max buffer size that needed to be increased for flv normalization output to be sent back
Working on getting accession documentation into workflow. Waiting for evelyn's feedback (issue 458).
Working on getting a means to add metadata for manually normalized files.


(10:38:34 AM) berwin22: There is now an approval step after normalization
(10:38:38 AM) Sevein: not sure what you exactly want to do in issue 
(10:38:50 AM) epmclellan: Sevein: I think Joseph can explain
(10:39:02 AM) epmclellan: he has written a script
(10:39:14 AM) epmclellan: and we'd like the user to be able to trigger it from the dashboard
(10:39:24 AM) epmclellan: is that right, berwin22?
(10:39:48 AM) Sevein: oh I see
(10:40:26 AM) berwin22: Sevein, it's a python function to call with certain parameters
(10:40:45 AM) berwin22: I'm hoping you can import it and use it from it's current place in the code.
(10:41:19 AM) berwin22: it can be a script if it needs to be, but I thought a function might be easier for you to work with
(10:41:37 AM) Sevein: ok, I'll see what can I do
(10:41:51 AM) Sevein: give me some time to read more about and learn how it works
(10:41:58 AM) Sevein: time = minutes :D
(10:42:09 AM) Sevein: well, ASAP :D
(10:42:32 AM) berwin22: http://code.google.com/p/archivematica/source/browse/trunk/src/transcoder/lib/premisXMLlinker.py
def xmlCreateFileAssociationBetween(originalFileFullPath, outputFromNormalizationFileFullPath, SIPFullPath, eventDetailText, objects="objects/", outputFileUUID="", eventUUID="", edate=""):
(10:42:50 AM) Sevein: nice
(10:43:33 AM) peterVG: Austin what dev will you be working on this week?
(10:44:00 AM) Austin: working on scalability testing
(10:44:11 AM) peterVG: awesome
(10:44:12 AM) Austin: have to get a 1TB hard drive
(10:44:21 AM) Austin: or setup a bigger raid
(10:44:24 AM) epmclellan: Austin: are we going to City of Vancouver Archives?
(10:44:38 AM) epmclellan: to do testing there?
(10:44:38 AM) Austin: right now largest I have to work with is 500
(10:44:44 AM) peterVG: I've got a spare 1TB external hard disk at the office
(10:45:11 AM) peterVG: epmclellan: first Austin will setup his own test environment. then we'll run the same tests on CVA's setup
(10:45:16 AM) epmclellan: ah
(10:45:21 AM) Austin: epmclellan: yes this ^
(10:45:33 AM) peterVG: Austin: I guess we've ruled out using VMs on Ciaus due to the lag of the VM layer?
(10:45:48 AM) Austin: peterVG: ok, I think Ill just pick up a internal 
(10:46:27 AM) Austin: yeah.. well and it was running into some space issues already
(10:46:46 AM) peterVG: MJ Suhonos: anything to add about using ciaus for scalability testing?
(10:47:01 AM) Austin: Im pretty sure backups are heading that way now.. and there isnt much room to play around
(10:47:33 AM) peterVG: k
(10:47:33 AM) Austin: much room like.. more then a few 100 gbs
(10:47:39 AM) peterVG: k
(10:48:27 AM) Austin: at the same time Ill be polishing up the install scripts
(10:48:36 AM) MJ Suhonos: peterVG:  I think the VM on caius was fine for testing, but i wouldn't use it to deploy to production
(10:48:53 AM) peterVG: MJ Suhonos: this is for more testing
(10:48:54 AM) MJ Suhonos: a better real-world testing platform would be caius on bare metal, and then scrap/rebuild after each test project
(10:49:07 AM) Austin: making them a bit more versatile.. and remove some of the excess errors during the environment build ( that folks have complained about)
(10:49:39 AM) MJ Suhonos: peterVG:  i guess the question wrt. a VM is whether your testing is relative like it was in LAC or absolute, ie. to propose deployment scanerios
(10:49:44 AM) MJ Suhonos: erm scenarios 
(10:49:54 AM) MJ Suhonos: Scanerios! part of a delicious breakfast!
(10:50:02 AM) peterVG: yum
(10:50:08 AM) peterVG: http://archivematica.org/wiki/index.php?title=Scalability_testing
(10:50:15 AM) peterVG: MJ Suhonos: ^ it's similar to LAC testing
(10:50:25 AM) MJ Suhonos: yeah i think an archivematica VM on caius would be fine
(10:50:27 AM) Sevein: oh, we have news from LAC?
(10:50:45 AM) MJ Suhonos: Sevein:  no news, just referring to our VM setup for that project
(10:50:50 AM) Sevein: ok
(10:51:22 AM) MJ Suhonos: peterVG:  i would just make sure the fact that the VM imposes significant overhead (esp. in disk i/o) is noted in the testing documentation
(10:51:49 AM) Austin: peterVG: do you have the link to those free file sets we were looking at last week
(10:52:29 AM) Austin: yeah, I was running a multinode install with 4 VMs on the same host.. and was getting very very poor performance
(10:52:58 AM) ***peterVG looking for link
(10:53:09 AM) MJ Suhonos: Austin:  this was using VMDKs?
(10:53:10 AM) Austin: much worse then when we are doing bare metal
(10:53:28 AM) Austin: nah, kvm qcow images
(10:53:43 AM) MJ Suhonos: ah.  still consolidated disk images though
(10:53:59 AM) Austin: mm, ahh.. as opposed to raw images?
(10:54:04 AM) berwin22: I'm curious how that would differ from one vm running 8 "threads" instead
(10:54:10 AM) MJ Suhonos: as opposed to native FS on bare metal :)
(10:54:32 AM) Austin: mmm
(10:54:41 AM) Austin: yeah berwin22, Im not sure it would be much different
(10:54:46 AM) Austin: it was all on one host
(10:54:48 AM) epmclellan: peterVG: http://digitalcorpora.org/corpora/files
(10:54:56 AM) Austin: if I had raid.. I think it would work better
(10:54:59 AM) Austin: epmclellan: thanks!
(10:55:04 AM) berwin22: the vm software may handle it better
(10:55:32 AM) berwin22: yeah... I actually lost a disk again this week :P turned my fans too low and went out.
(10:57:43 AM) peterVG: okay any documentation?
(10:58:49 AM) berwin22: from last week: transcoder db http://archivematica.org/wiki/index.php?title=Transcoder 
(10:58:50 AM) Austin: notes so far http://archivematica.org/wiki/index.php?title=Meeting_20110427
(10:59:06 AM) Austin: berwin22: awesome
(11:00:21 AM) epmclellan: added preservation plan and documentation for pct files
(11:00:39 AM) epmclellan: http://www.archivematica.org/wiki/index.php?title=Macintosh_PICT_Image
(11:01:05 AM) epmclellan: I did this because SFU has pct files, it was on the list of formats that Ian Forsyth gave me
(11:01:35 AM) epmclellan: I think that's it for me
(11:02:04 AM) epmclellan: I also edited the database file to add the normalization path to Archivematica
(11:02:14 AM) epmclellan: sorry, that's dev, I'm not on top of this meeting
(11:02:21 AM) epmclellan: due to impending workshop deadline
(11:02:22 AM) peterVG: nice work on the Transcoder docs Joseph!
(11:04:12 AM) peterVG: okay 11 o'clock. i think that's it
Retrieved from "https://wiki.archivematica.org/index.php?title=Meeting_20110427&oldid=4579"