Meeting 20110721

Development

• Joseph/Austin
  • Look at ceph; possible problems with watched directories (inotify)
• Joseph
  • Work on MCP with gearman.

Deployment

• Austin is at RAC NY with Archivematica 0.7.1
  • Going well, 0.7.1 is fast.
  • got 5.25 floppy md5summing on win95 working

Chat log

(10:36:17 AM) berwin22: Dev:
(10:36:17 AM) berwin22: I've been working with gearman. It's looking good, I'll need to discuss some stuff about getting task information about what's processing with jesus.
(10:36:17 AM) berwin22: Specifically, gearman has it's own api for getting information on queued/processing events. I think we may want to get information directly from that AIP, rather than pulling it through the MCP server, to lighten the load on the server.
(10:36:19 AM) JessicaB: thanks david and jesus - see you in the office tomorrow.
(10:36:40 AM) djjuhasz: See you tommorrow JB
(10:36:44 AM) djjuhasz: :)
(10:37:01 AM) Sevein: djjuhasz: qubit notes - http://www.qubit-toolkit.org/wiki/index.php?title=Meeting_20110721, please paste the chat log, it is difficult with a shell client
(10:37:09 AM) berwin22: Had a more in-depth look at ceph yesterday. 
(10:37:09 AM) berwin22: I started doing so, because I wanted to know if we could easily identify which files were located on which machine in a distributed enviroment, and have the mcp assign jobs prioriting queue based on where files are located. (assign tasks to workers that already have the file, so they don't have to fetch it over the network). Gearman won't support this, but I might like to contact both gearman and ceph, to see if we might be able to collaborate on this feature.
(10:37:26 AM) Sevein: who is taking notes?
(10:38:40 AM) Sevein: ping broadcast
(10:38:59 AM) berwin22: I'm not sure how much there is today, I was just going to dump the chat log in the wiki.
(10:39:12 AM) djjuhasz: nice
(10:39:14 AM) Sevein: ok, sounds good
(10:40:11 AM) berwin22: I was also talking with Austin (ARTi) about the risk of ceph not working with inotify.
(10:40:37 AM) djjuhasz: hmm, why ceph no like inotify?
(10:40:58 AM) berwin22: inotify relies on hooks at the kernel level
(10:40:59 AM) ARTi: same reason nfs dosnt
(10:41:20 AM) ARTi: sshfs does though
(10:41:32 AM) ARTi: so maybe the ceph-fuse mount would be a option
(10:42:32 AM) berwin22: inotify may not be critical for the MCP watched directories, but would be for the arrangement section in thunar/archivematica
(10:44:07 AM) berwin22: While looking at that stuff, I also tried to document what were some of the bottlenecks in archivematica.
(10:44:08 AM) berwin22: http://archivematica.org/wiki/index.php?title=Bottlenecks
(10:44:42 AM) Sevein: nice docs berwin22 :)
(10:45:23 AM) berwin22: any other dev?
(10:45:31 AM) djjuhasz: yes, great notes
(10:46:41 AM) berwin22: deployment - ARTi
(10:47:54 AM) djjuhasz: Sounds like ARTi is in the thick of deployment :)
(10:47:59 AM) ARTi left the room (quit: Quit: Leaving.).
(10:48:02 AM) berwin22: agreed
(10:48:32 AM) berwin22: @RAC NY with Archivematica 0.7.1
(10:48:49 AM) djjuhasz: sweet :)
(10:49:25 AM) ARTi [~austin@static-71-167-174-3.nycmny.fios.verizon.net] entered the room.
(10:50:17 AM) ***ARTi thinks VM / package building is destroying his HD
(10:50:35 AM) berwin22: ssd?
(10:50:50 AM) ARTi: its one of those combo drives
(10:51:02 AM) berwin22: deployment - ARTi ?
(10:51:09 AM) ARTi: umm. RAC is going rad
(10:51:14 AM) ARTi: they are running 7.1
(10:51:40 AM) ARTi: got 5.25 floppy md5summing on win95 working
(10:51:53 AM) ARTi: dont let me near any sharp objects ;]
(10:52:12 AM) ARTi: 7.1 has been super fast
(10:52:24 AM) ARTi: have 5 workers and one masterServer
(10:52:42 AM) djjuhasz: wow, nice
(10:53:00 AM) berwin22: yeah... that should be a bit of a power house
(10:53:06 AM) epmclellan [~epmclella@static-71-167-174-3.nycmny.fios.verizon.net] entered the room.
(10:53:08 AM) ARTi: yeah, its rocking out
(10:53:41 AM) berwin22: Fits and normalization are probably the two slowest microservices, and they are distributed well
(10:53:41 AM) ARTi: but Ive got to get back to work..
(10:53:44 AM) ARTi: yeah
(10:53:56 AM) berwin22: cheers
(10:53:58 AM) ARTi: fits crashed once.. on one of the workers.. but we couldnt get it to repeat
(10:54:17 AM) ARTi: then the submission Docs normalization also crashed once
(10:54:22 AM) ARTi: but couldnt duplicated
(10:54:31 AM) ARTi: atleast nothing has brought the system down
(10:54:38 AM) ARTi: way to go berwin22 :]
(10:54:44 AM) berwin22: did submission docs crash on it trying to normalize a file that didn't exist?
(10:54:54 AM) ARTi: no, it was a pdf..
(10:55:01 AM) ARTi: actually I remember.. was a ghostscript error
(10:55:08 AM) ARTi: thought it might be the MS issue..
(10:55:14 AM) ARTi: but it worked the next few times around
(10:55:31 AM) berwin22: mind grabbing the crash info you can for me?
(10:55:31 AM) berwin22: logs and screen shots, idea
(10:55:38 AM) ARTi: made a 100 copies of the same pdf and ran it through.. just to make sure all the workers were tested
(10:56:04 AM) ARTi: berwin22.. Ill see if I can find.. sec
(10:56:38 AM) berwin22: Fail or crash... did it hang?
(10:57:04 AM) berwin22: I hoped we'd got rid of things hanging in 0.7.1 (for the most part)
(10:57:11 AM) ARTi: nope didnt hang
(10:57:12 AM) ARTi: fail
(10:57:18 AM) berwin22: Thanks ARTi, I know you're busy busy busy
(10:57:21 AM) ARTi: yeah.. ++ :]
(10:57:36 AM) ARTi: http://i.imgur.com/aMqkx.png
(10:57:39 AM) berwin22: Alright... any other business?
(10:58:02 AM) ARTi: berwin22^ err
(10:58:02 AM) epmclellan left the room (quit: Read error: Connection reset by peer).
(10:59:00 AM) ARTi: not from me.. evelyn is pretty busy atm
(10:59:40 AM) berwin22: k