Read our blog

Development Goals: APIs

Earlier this year the Avalon Media System received a generous grant from The Andrew W. Mellon Foundation. This grant allows for the continued development of Avalon over the next two years, through January 2017. But what are the development plans and goals during this time?

 

The Avalon team is developing Application Programming Interfaces (APIs) that will provide greater flexibility. For one, we will develop an API allowing institutions to automate ingestion of large collections of digitized media using their own customized workflows.

 

Currently, we are developing an API allowing institutions to use the media transcoding engine of their choice. Now, Avalon uses Opencast Matterhorn for the media processing pipeline. But we realize that this is a barrier to adoption for some institutions since, unlike the rest of Avalon, it is a series of OSGI modules which runs under Apache Felix (a Java OSGI server), increasing the complexity of the dependencies. So we are developing an API enabling replacement of Matterhorn with other solutions, such as, but not limited to, Ruby Resque, Amazon Elastic Transcoder, and Zencoder.

 

In the coming weeks we will look at other development goals for the Avalon Media System that will be implemented over the the next two years.

Blog Categories:

Sprint 72 Update

During this sprint we continued our work on structural metadata support by implementing a means of uploading structure information in an XML file, whether via the interactive interface or via batch upload. We fixed a timeout problem where the interaction was confusing when the playback session timed out. And we did the metadata and design work necessary for adding new fields to hold imported notes and system IDs from MARC records. We made progress on integrating oEmbed for the player page, which will eventually allow us to support Facebook embeddability.

Issues completed during this sprint:

VOV-484 - I want to upload a multi-part item where the parts represent multiple CDs, each of which have hierarchical structure for within-file navigation.

VOV-3544 - I want to have the player work reliably and not get stuck.

VOV-3549 - I want to know what is needed for notes field(s) and other system IDs.

VOV-3551 - Ensure parameters are passed through with permalinks

Blog Categories:

Sprint 71 Update

During sprint 71, we made a number of improvements, fixed several bugs, updated Wiki pages, and upgraded some production servers to Avalon 3.3. We made our first step towards structural metadata, implementing a means of navigating the media player by clicking on structure labels (could be tracks in an audio recording, or scenes in a play or opera, for example). We updated the Avalon demo server with the recent 3.3 release, and Avalon is now upgraded to Hydra 8, which was released on March 31st. The embed code and sharing links are now collapsed under a Share button. Other visual cues have been added to improve the user experience.

 

The specific stories we accomplished during these sprints were:

 

VOV-519 - I want to navigate an audio or video media item by clicking on the scene titles, which are hierarchically displayed

VOV-3401 - I want to right-click on a section label and send the resultant permalink-based URL to someone, and have it work

VOV-3406 - I want to know what is needed for the Transcoding API

VOV-3452 - I want to try out release 3.3 on the demo server

VOV-3454 - I want to see 3.3 on IU prod systems

VOV-3459 - I want to create and upgrade to Hydra 8

VOV-3505 - I want to have some simple examples to test/demo structural metadata navigation with

Blog Categories:

Sprints 68-70 update

During sprints 68 and 69 we put the final touches on release 3.3. That work included testing and fixing bugs, writing release documentation and upgrading our local systems. We also had an in-person team meeting during sprint 69 where we started on some major system work including updating our Blacklight and Hydra versions.
 
In sprint 70 we continued our work on Hydra 8 and upgrading to the newest Blacklight. The team also started investigating what it would take to make Avalon work with Fedora 4. We also started work on two new features, structural metadata and a transcoding API. During this sprint, we made progress on designing these features as well as implementing the ability to navigate to a specific time offset via URL. 
 
Release 4 will include the ability to add structural metadata to media items via the batch process. This will allow viewers to more easily navigate to specific segments within media, for example a specific track on a CD. Another major feature in Release 4 will be a transcoding API. This API will allow Avalon to use additional transcoding services in addition to the current Matterhorn option. 
 
The specific stories we accomplished during these sprints were:
 
I want to configure MARC import for my institution
 
I want 3.3 to work robustly
 
I want to be able to manage my disk space and not have Matterhorn fill it up mysteriously
 
As a new developer, I want to come up to speed on Avalon
 
I want to upgrade to 3.3
 
I want to know how to use the new 3.3 functionality
 
I want to have the item page work well even when extensive structural metadata is displayed
 
I want to configure MARC import for my institution
 
I want 3.3 to work robustly
 
I want to be on the latest version of backlight
 
I want to try out release 3.3 on the demo server
 
I want to see 3.3 on IU prod systems
 
I want to create and upgrade to Hydra 8
 
I want to understand the impact of upgrading Avalon to Fedora 4
 
I want to understand the options for functional test tools
 
I want to have a url move the scrubber to a specific time offset within a file
 
I know further details of the structural metadata UI design

Blog Categories:

Andrew W. Mellon Foundation Grant Received

We're happy to announce that the Indiana University Libraries and Northwestern University Library have received a new $750,000 grant from the Andrew W. Mellon Foundation to support work on Avalon development and community building through January 2017. See the press release for more information.

Blog Categories:

Avalon 3.3 Now Available

Indiana University and Northwestern University are pleased to announce Avalon Media System 3.3. Release 3.3 adds the following capabilities:

  • MARC Metadata Import
  • Ingestion of pre-transcoded derivatives with multiple quality levels 
  • Script for recovering disk space taken up by temporary Matterhorn files
  • UI Improvements and Bug fixes

Users of Avalon 3.2 can take advantage of these new features by Upgrading Avalon 3.2 to Avalon 3.3.

 

For a more comprehensive list of changes, see the 3.3 release notes.

 

For more details on each of these new features, visit the What’s New in Avalon 3.3 wiki page: https://wiki.dlib.indiana.edu/display/VarVideo/What%27s+New+in+Avalon+3.3

 

Please feel free to try out Avalon 3.3 on our public test server (http://pawpaw.dlib.indiana.edu) before installation. Installation options include virtual machine image, manual installation, and source code installation. More information on all available options can be found on the Avalon web site’s Download page: http://www.avalonmediasystem.org/download 

 

We welcome your feedback on Avalon 3.3 via the avalon-discuss-l discussion list. Join the discussion list at http://www.avalonmediasystem.org/connect

 

Sprint 57 – February 6, 2015

With this sprint, the Avalon team completed work on one of the main new features of Release 3.3: importing descriptive metadata from an online public access library catalog (OPAC) into Avalon. This may be accomplish via batch ingest by using a bibliographic record identification number. The metadata import functionality will be configurable to meet local cataloging practices and currently includes catalog key, Online Computer Library Catalog (OCLC), and Library of Congress Control Numbers (LCCN) identifier types as options.

 

We are continuing to improve disk space management for system administrators and messages about the status of bulk actions on media items for collection managers. Next, we will be testing the 3.3 release candidate in preparation for the upcoming release.

 

Learn the details of each story and view our Sprint Demo recording below:

 

Sprint 67 – February 6, 2015 Demo Recording

 

  • VOV-1640 – When adding items via batch, I want to pull in the bibliographic info from the OPAC
  • VOV-3353 – I want to be confident for bulk updates that something is happening and know when it will finish
  • VOV-3358 – I want to apply 3.2.2
  • VOV-3318 – I want to configure MARC import for my institution
  • VOV-3309 – I want 3.3 to work robustly
  • VOV-3246 – I want to be able to manage my disk space and not have Matterhorn fill it up mysteriously

Blog Categories:

HydraDAM2

The National Endowment for the Humanities recently awarded the Indiana University Libraries and WGBH Boston a grant to support the development of HydraDAM2. This preservation-oriented digital asset management system for time-based media will improve upon WGBH’s existing HydraDAM system and work seamlessly with the Avalon Media System for user access, among other features. 

 

Both HydraDAM and the Avalon Media System grew from the Hydra community. Hydra is an open source technology framework that supports the creation of preservation and access applications for digital assets based on the Fedora repository system. A community of institutions known as the Hydra Partners works together to maintain the framework and create applications for local or shared use by libraries, archives, and cultural institutions. Both Indiana University and WGBH Boston are among the 25 Hydra Partner institutions. Indiana University is collaborating with Northwestern University on the development of the Avalon Media System and WGBH developed the original HydraDAM system with help from the Data Curation Experts group.

 

HydraDAM

The current version of HydraDAM is designed to help libraries, archives, and cultural institutions preserve audio and video files. HydraDAM accepts batches of files via FTP or a browser upload, extracts technical metadata, and moves the files and metadata into a Fedora 3 based preservation repository and a hierarchical storage management system. HydraDAM accepts multiple file formats. Via the user interface, users may also edit metadata and set access controls.

 

Watch a demo of HydraDAM during an Open Source Digital Preservation and Access session at the Association of Moving Image Archivists Conference 2014 (start at 35 minutes):

 

The Avalon Media System

The Avalon Media System is focused on the needs of these same types of institutions in providing access to digital audio and video. The software is designed with particular metadata and authentication capabilities, along with many other features that support its usefulness for students, researchers, and teachers. Avalon is currently in production or in production planning at several institutions.

 

HydraDAM2

In the HydraDAM2 project, Indiana University and WGBH will:

 

1) Extend HydraDAM to work on Fedora 4

 

2) Develop Fedora 4 content models for audio and video preservation objects based on current standards and best practices and utilizing new features in Fedora 4 for storage and indexing of RDF

 

3) Implement support in HydraDAM for two different storage models, appropriate to different types of institutions:

a. direct management of media files stored on spinning disk or on tape in a hierarchical storage management (HSM) system; and

b. indirect management and tracking of media files stored offline on LTO tapes

 

4) Integrate HydraDAM into preservation workflows that feed access systems including Avalon and WGBH’s Open Vault and conduct testing of large files and high-throughput workflows

 

5) Document and disseminate information about implementation and experience to the library, archive, digital repository, and audiovisual preservation communities

 

HydraDAM2 builds upon the philosophy, purpose, and experience of the HydraDAM, Avalon Media System, Open Vault, and Fedora projects. It will support digital preservation of time-based media files in a way that can easily be tied in with access. The Avalon team is very excited to see this project develop and be shared with the community.

 

 

Blog Categories:

2014 in Review

2014 was a year filled with development milestones and community engagement for the Avalon Media System team. We achieved implementation at both development partner institutions, Northwestern University and Indiana University this year. We also focused on spreading the word about the Avalon Media System across the library and archives community and connected with several institutions planning to implement the Avalon Media System or in the evaluation process. Some of the adopters even started work on new features that will be included in the Avalon core code in 2015.

 

The Avalon development team also continued their progress throughout the year, embarking upon our first series of minor releases. Continue reading for development, implementation, and presentation highlights from each month of 2014.

 

Fleming, Declan. 2014. Hydra Connect, La Jolla, CA. Flickr. Web. 20 Jan. 2015. <https://www.flickr.com/photos/bigdpix/12097356495/>.

 

In January, Jon, Mark, Julie R., Michael, Chris, and Claire attended the first Hydra Connect meeting on the University of California, San Diego campus. Over the course of four days, the developers and managers of our team joined with other Hydra project participants to synchronize development plans involving the Hydra framework and share their experiences.

 

In February, the Avalon Media System became compatible with different Learning Management Systems (LMS) that use the Learning Tools Interoperability (LTI) standard, such as Blackboard, Canvas and Sakai (Sprint 45 – February 21, 2014). Students and faculty members associated with classes gained the ability to view Avalon media items through their institution’s LMS. This feature was an important goal for Avalon since it supports the use of media collections for teaching and learning.

 

In March, Michael and Julie R. presented “Avalon Media System: Building for Ourselves and Others” at Code4Lib in Raleigh, North Carolina. Their presentation discussed how community engagement has shaped the requirements of the Avalon Media System and emphasized its importance in developing future goals and Avalon’s continued success.  Their 16 foot long graph of Avalon’s dependencies made quite a splash!

 

In April, the team released Avalon 3.0.

 

In May, the team began development of Avalon’s first minor release, 3.1. We also began ingesting the Indiana University Libraries Moving Image Archive’s digital collections into Indiana University’s test instance of the Avalon Media System.

 

Photo of Open Repositories conference via Flickr user todd.visionShared without alterations under CC BY-SA 2.0<https://www.flickr.com/photos/16894927@N08/14228456869/>

 

In June, Jon, Claire, and Julie H. attended the Open Repositories conference in Helsinki, Finland. Jon and Claire had the opportunity to discuss and demonstrate Avalon 3.0 at the poster reception.

 

In July, we celebrated the release of Avalon 3.1 and Northwestern University’s new production instance of the Avalon Media System.

 

In August, we conducted our first community survey and made connections with six new institutions working on production versions of the Avalon Media System. Around a dozen institutions reported work on pilot implementations and about two dozen institutions reported current or future investigations of Avalon.

 

Photo via Flickr user MarkNotess. Shared without alterations under CC BY-SA 2.0. <https://www.flickr.com/photos/61919416@N00/15254158978/

 

In September, Jon, Claire, Mark, Chris, Julie H., Julie R., Brian, Leah, and Michael attended Hydra Connect 2 on the Case Western University campus in Cleveland, OH. Michael demonstrated Avalon during a panel of production systems and Jon participated in the Opening Plenary (see photo above.)

 

In October, we achieved production implementation at Indiana University. At the Association of Association of Moving Image Archivists Annual Conference in Savannah, GA, Stefan discussed the way the Hydra framework is used at Northwestern University and Jon discussed the Avalon Media System in further depth. At the Digital Library Federation Forum in Atlanta, GA, Julie H. presented a poster on metadata in the Avalon Media System and Jon and Julie R. led a panel discussing experiences with Avalon at five different universities. The Avalon team also began coordinating with the University of Virginia to develop several new features.

   

In November, Jon, Mark, and Julie R. visited the Yale University Library to demonstrate Avalon and explain its technical architecture and future development plans.

 

In December, The National Endowment for the Humanities announced the award of a two-year Preservation and Access Research and Development grant to Indiana University and Hydra partner WGBH/Boston to createa Hydra-based digital preservation repository environment for audio and video media that builds on WGBH’s previous work on the HydraDAM system and will integrate with Avalon for access. We ended the year with the release of Avalon 3.2.

 

Over the course of 2014, we presented at 5 different conferences, completed 22 sprints, and shared one major and two minor releases of the Avalon Media System. We’re looking forward to another productive year, filled with new partnerships and exciting new features. Avalon 4.0 is slated for this spring and will include media processing queue management and more robust metadata features such as media section navigation based on structural metadata and bulk metadata updates for media items. Migration to Fedora 4 is also planned, along with accessibility improvements, playlist and clip creation features, and much more. 

 

Blog Categories:

Sprint 66 - January 23, 2015

Back from the holiday break, we began work on our next release, 3.3. We added the capability of importing descriptive metadata from an OPAC by specifying a catalog ID on the Resource Description page. We also added the capability of ingesting multiple quality levels of pre-transcoded derivatives via the batch ingest process. Previously, batch ingest could only add one quality level of pre-transcoded derivative. We also fixed two 3.2 release bugs we discovered. A patch for one is available and the second will shortly be available.
 

Sprint 66 - January 23, 2015 - Demo recording
 

Patched Bugs:

  • VOV-3344 - When you edit a collection name, the items disappear. (Patched in release 3.2.1.)
  • VOV-3347 - Bulk actions only operate on 10 items at a time. (Will be patched in release 3.2.2.)
     

Completed Stories:

  • VOV-1639 - When adding one item, I want to pull in bibliographic info for the new item from our OPAC
  • VOV-2949 - I want to ingest pre-transcoded derivatives with multiple quality levels via batch
  • VOV-3317 - I want automated MARC import records to look good in my system

 

Blog Categories:

Pages