Thursday, 16 May 2013

CMM's Chemistry X-ray Facility is on Mirage

As of a couple of days ago, the "B2" and "B5" instruments in the CMM's Lab in Chemistry are now connected to Mirage.

So now all 4 CMM Labs are "on Mirage" to some extent.

Technical stuff

The Chem Lab instruments don't save their data directly to a Samba server (like the other Labs).  Instead, they save their data locally, and the data is "pulled" by a periodic rsync from CMMData.  (This used to happen, once a day, but now it happens every 20 minutes.)

This presented a couple of "interesting challenges":
  • The data grabber software needed significant changes to cope with files delivered by rsync.  The files get touched repeatedly by the rsync, so we need to take special steps to prevent them being ingested multiple times.
  • The B5 machine is one of a few machines that are operated by CMM staff exclusively doing runs for CMM users. So we needed to implement a special mechanism for automatically assigning files to the right CMM user.

Friday, 26 April 2013

Recent changes to the Mirage User Interface

There have been a number of changes to the Mirage user interface in the past few days.

Source metadata

Mirage now displays "administrative" metadata about the data capture process; e.g. who was logged in, what instrument was used, and the "save" file and folder names for each captured Datafile.

Downloaded ZIP / TAR files

When you download an Experiment or selected Datasets or Datafiles the TAR / ZIP archive now has file and folder names that match the names used to save the files on the instrument.   If you are downloading an entire Experiment, you can still get the "classic" MyTardis organization.

Update - 2013-05-06:  Mac users are no longer offered the option of downloading in ZIP format because of Mac's problems with reading the "streamed" ZIP file format that MyTardis uses.  You can use TAR format instead.

Sharing data with other Mirage users

It is now possible to share Experiments with other Mirage users, or groups using the "sharing" tab in the Experiment view.  You can grant read-only access, read-write access, or full ownership rights.

If you wish to create a Mirage group for sharing data within your research group, please contact me.


The "data sharing" changes and other ones that you may notice are the work of Steve Androulakis and Grischa Meyer at Monash Uni.  For the MyTardis audience, this the result of the "master-3.0 merge".  Kudos to Steve A!

Sunday, 7 April 2013

QBP Lab is now on Mirage

The campaign to integrate the instruments in QBP Labs with the rest of CMM is now (nearly) finished.

On Friday I finalized the deployment of the data grabber and ACLS proxy to the QBPDATA server (the replacement for DejaVu) and hooked up the first instrument.  This morning I hooked up the atom feed to the production MIRAGE system, and ... the data has started to flow.

The last step is to hook up as many of the QBP instruments as possible.

Tuesday, 12 March 2013

Data Migration in production on Mirage

The long awaited MyTardis "datafile migration" extensions are now deployed to CMM's Mirage production system.  The data is being mirrored as I write this.

What does this mean?  The short answer is that I can now add the rest of CMM's instruments to Mirage without filling our local file servers with a few months of data.  So expect to see the remaining AIBN and Hawken instruments added to Mirage in the next few days and weeks.

For people interested in developing / deploying MyTardis, there is a "pull request" waiting for merging into mytardis:master on Github.  Read the pull request comments for details on what is involved.

Update: The pull request has been merged, so the "migration" code is not in the "master" MyTardis codebase.

Sunday, 17 February 2013

The Mirage project web pages

If you are interested in how Mirage is implemented, and/or are considering implementing your own data management system, take a look at the Mirage Project web pages.  They include an architectural overview, a description of the components we developed, and a section on how we interface with ACLS.