res·tor·a·tion [noun]
– the return of something to a former or original state. (Canadian Oxford Dictionary, Second Edition)
– the process of restoring a building, work of art, etc. to its original condition.
(Oxford Dictionary of English, Third Edition)

The Legacy Project

A few years ago, I was hired as a VR consultant, (initially) to restore the audio for Char Davies’ two pioneering immersive artworks, Osmose (1995) and Ephémère (1998). These works used the latest immersive technology available at the time, including a Silicon Graphics (SGI) Onyx computer for the main program and visuals, Division head-mounted display and Polhemus 3D trackers, along with a bespoke navigation system based on breath and balance. The audio alone for the works depended on an entire rack of hardware, including a PC and a Mac, two Kurzweil K2000 samplers, a mixer, effects units and a Crystal River Engineering (later AuSIM) Acoustetron for binaural spatialization.

The works had already evolved several times in their past lives. Some of the audio gear had been upgraded during the first half-decade. The last major porting effort was in 2002, when John Harrison updated the graphics code to run on a high-end Linux machine (instead of SGI) — but otherwise using the same hardware and software. The last time this most recent configuration was shown was here in Spain, in 2007. The hefty and expensive Onyx was gone, but installing the works in a gallery still required shipping several crates, with two large racks of gear.

Immersence is Char’s research company, which was founded in 1998 to continue the work she’d started at Softimage. Shortly after its founding, I believe Immersence purchased the last four dVisor headsets produced, just before Division went out of the HMD business entirely. This turned out to be a smart move; a couple of those headsets continue to function — sort of — though neither of them does so gladly. And until the Vive and Rift headsets came along this year, the pickings were slim for a good replacement headset.

The Immersence team did a good job preserving Char’s works in (more or less) their original state; a number of replacement parts such as those HMDs, samplers, memory chips and disk drives had been acquired over the years. Failing pieces were swapped out with components scavenged from these reserves. Public exhibitions, and the passage of time, had taken their toll…and it was becoming infeasible to present the works in their original forms. This degradation process doesn’t just occur with hardware devices and physical media, though, it also happens with software: programming languages, libraries, operating systems and other dependencies evolve, diverge and eventually disappear altogether. Old computers fail, and over time it also becomes complicated (eventually impossible) to build or even run an old program on a modern-day computer. Sometimes you can’t teach a new dog old tricks…

Audio Work

The plan in 2013 was for me to replace the refrigerator-sized rack of physical gear with a single recent-model computer, to perform all audio processing in software, removing the dependency on failing, two-decades-old hardware. I had access to some of the original hardware, some of the time I was working on this conversion, but not all of it — that complicated my task. Therefore, it entailed a kind of archaeology, digging deep into murky layers and shining a (virtual) light into black boxes — studying old user manuals, doing some reverse engineering, a lot of A/B listening tests, and often taking a best guess at how some of those boxes made their sounds.

I started working on Osmose, knowing I’d be able to reuse much of the work when I got to Ephémère (a more complex work, but using the same audio setup). In the SuperCollider audio programming language, I coded software synths, recreated hundreds of preset programs, virtual busses and mixers, effects and all the control code to glue it together, along with binaural spatialization using the Ambisonic Toolkit (ATK) library. Osmose had used Opcode (now Cycling ’74) Max for most of the audio “intelligence” in the work, and fortunately the original patches loaded and worked (with minimal changes) in the latest version of Max. It’s something of a miracle when anything in the domain of electronic art can load and run on contemporary hardware without changes, twenty years later!

I was very fortunate to be in contact with Dorota Blaszczak (in Warsaw), who originally developed the works’ sonic architecture (along with the composer Rick Bidlack). Her help was invaluable, because Dorota knows these works better than anyone, and could listen to my work critically and give guidance on whether something was “good enough”. We were able to meet and work together in person on several occasions during this process, so she could answer detailed questions, help identify problems and validate my ongoing work.

As the Osmose audio port wrapped up, I began to work on Ephémère, which was a more complex task. Although lots of the groundwork was done — it shares the basic sound framework with Osmose — the synth instruments were considerably more varied and complex. Also, it used a custom Windows program (Windows 95, no less!), written by Rick Bidlack, as the controlling sonic “brain”. This program had to be ported to run on a Mac (and other platforms, for future flexibility). As with the Max patches for Osmose, these audio programs now communicate with the main “graphics” programs via OSC messages — previously they used a mix of physical MIDI and serial cables to send messages back and forth between programs and devices.

For Ephémère, I obtained the most recent copy of the source code from Rick (in Seattle; he dug it up on an old floppy disk), and set to work. Since all the audio gear has now been transformed into source code, future conservation/restoration should be much easier, as everything is now contained in a Git repository that includes all required code and data. By using a version control system, not only is everything consistent, organized and backed up, but it also provides a historical record of future work over time.

For both legacy projects, much of the audio work was meticulous “detail work”, going through hundreds of synth programs, parameter by parameter, seeing what functionality was needed for each layer and patch, what range of MIDI notes and controllers it needs to respond to, and trying to reproduce the correct sound for all possible inputs. Adding to the difficulty is that these are interactive works, not simple “playbacks”. On one run-through, certain events and parameters may be encountered that won’t appear in a subsequent run; everything depends on the user’s specific interactions, as well as some degree of predefined behaviour and randomness. I obviously did not try to recreate a K2000 sampler with its vast (pun intended) range of functionality — only the specifics needed for these works. That alone was a big enough job. However, having everything explicitly written in code now means that it should be easier to port the audio to any other software implementation, in future. The work was not just about making things run in a specific hardware and software environment today, but also preparing things to be more flexible and portable for the future.

Graphics Work

I was also in close contact with John Harrison (in Montreal) during this time. He was the programmer of the original VR software for both Osmose and Ephémère. The plan had been that he would port the graphical parts of the two legacy works, in parallel with my audio restoration work. However, John was busy developing software for a new work with Char, so when my work eventually became blocked (the graphics and audio programs need to interoperate), I offered to do the graphics port as well. This offer was enthusiastically received.

The original works ran on the IRIX operating system, and were reliant on SGI’s Performer high-performance visual simulation library. Performer is no longer available and, although there is an open source library that offers similar functionality (OpenSceneGraph), there was a strong desire to minimize dependencies on external software (within reason). At the time I began working on the project, the goal was to run the two works on a Mac Pro computer, using OpenGL. So, in a similar way to the audio work, I began studying and dissecting the ways these programs used Performer. I concluded it was not necessary to rely on a 3rd-party toolkit, and set about creating a layer of pure OpenGL functionality that reproduced the required parts previously provided by Performer (no, I didn’t write a full Performer implementation!).

There was a large amount of cleanup and refactoring to do in the code, especially to bring it from 1990s C to modern C++11, with an eye towards future cross-platform compatibility. GLFW was used for windowing and event management, the Oculus (Mac beta) SDK and eventually Valve’s OpenVR were used to interface with HMDs, but otherwise external dependencies were minimized. Inter-process communication and device driver code was ported and made cross-platform using Boost.

The complete porting/restoration of Osmose and Ephémère took place (on-and-off, i.e. part time) from 2013 to 2016, and included a number of in-person sessions (trips to Canada) to work with Char, John, Dorota, Daniel Chudak — who managed the restoration work — as well as a session with Georges Mauro (the animator who had worked under Char’s direction to create the original textures and models). The vast majority of the work, however, was done remotely (here, in Barcelona).

One factor that complicated the port was the desire to keep the works “exactly as they were” — as much as possible, at least. I’d understood that to be my original mandate: the graphics and sound should be the same as the original “legacy” works. Later in the project, when we began meeting with Char to present the new versions, this mandate was relaxed somewhat, because in some cases it was sufficient (or better) to aim instead for the same “sensibility” as the original works. Modern HMDs (such as the Vive and Rift) provided a somewhat different immersive experience, and not just because the displays and optics were different. The old works tended to run at around 20 fps — even on that SGI graphics supercomputer! — whereas with some optimization, the new works were able to run at 90 fps on modern hardware (while rendering over seven times the pixels).

The look in these new headsets is quite different, not just in terms of resolution, but also colour, distortions and “softness” (or rather, a distinct lack thereof, when compared to the old Division headsets). The low resolution of the original HMD (and its lack of screen-door effect) contributed to Char’s desired soft aesthetic; however, with the new high-resolution headsets, we decided that the low polygon counts of a few of the original models looked too harsh and hard-edged. In fact, we chose to deliberately add back in blur in some places, to get closer to the original softness. In general, though, because the works largely rely on semi-transparency, softness, and layering, they stand up quite well, even by “today’s standards” (whatever those may be).

To a large degree, this restoration was an objective, methodical, almost scientific process: exploration, experimentation, implementation…and repetition. However, these are artworks, and therefore “success” is more subjective than the meticulous process might suggest; a pixel-for-pixel match may not be the best criterion after all, some aesthetic decisions needed to be re-evaluated, negotiated and decided case-by-case.

Ultimately, Char’s vision for these works — then and now — is what guided the work. We were not out to remake them, creating a v2.0 or a director’s cut “with a different ending”. The goal was to bring the works back to life, preserving their place in the history of interactive, immersive artworks. As part of this process, it was necessary to be true to the works, but also flexible, allowing ourselves some small “improvements”, in the same way that an old tape recording might be remastered, or the colour subtly restored to a fading painting. The restored/remastered versions of Osmose and Ephémère are not identical to the originals, but they are very close, and hopefully are the same in spirit. And…now the graphics and audio all run on a single PC!