OSCulator - remaining close to our medium

Intro

OSCulator is an OSC + MIDI plugin for Unreal Engine, designed to simplify bi-directional interaction between multiple tools.

Unreal Engine has unique rendering capabilities that you might want to use. But maybe you don't want to make a game with it. This project aims at making it easy to remote control Unreal Engine with another tool or sensor of your choice or export data from Unreal Engine to control something else.

OSCulator parses and routes incoming data directly to the BP Actors you want to interact with. Within each BP, the data will be received through existing functions or custom events. This greatly reduces data plumbing since you don't need to build code dedicated to interfacing with your virtual world. From the user's perspective, the OSC/MIDI messages will directly interface with Unreal's existing structure. An upside of this approach is that everything you build to be remote controlled will also be callable from within the engine. There is no redundancy.

https://github.com/baronlanteigne/OSCulator

Why am I building this?

As a new media or digital artist, (so I'm neither a traditional media artist nor a programmer!) I was not satisfied with my current toolset and how they were connected. I’m looking to get the best out of each tool I use and the more I add, the more fragile the system gets. Each of these connections introduce a technical pirouette that distracts me and introduces a greater distance between me and my medium. I'm looking for more proximity with my medium during my creative process.

My first contact with the digital medium was music, which at the time was already light enough to perform in real time within the software. You could tweak a parameter and instantly hear its effect. The process was intuitive, exploratory. Later, when I moved into video editing, datamoshing and effects that required rendering, I immediately felt the distance from the result.

I think artists are too good at adapting and overcoming those frictions… and I think we deserve better because how we feel during the creative process has a lot of influence on its outcome. In this text, I intend to zoom in on the dynamics between the artist, the interfaces, the experience and the artwork by going over my process of getting into Unreal Engine.

In 2022, I knew I wanted to explore the virtual world building capabilities of Unreal Engine but I also knew I wasn't trying to make a video game. I had to adapt something that exists before I can use it.

(... maybe this is what best describes my practice: I hijack tools designed for an industry and repurpose them to create something else. In this case, I'm turning a game engine into a real-time improvisational instrument able to create various aesthetic propositions by adjusting the fragile balance between a camera, light sources and virtual objects held together by a physics constraints system. This virtual studio is where I document various explorations until something peculiar emerges.)

Learning a new tool initially means a lower output. A necessary evil for artists. Over the years I've learned that society discourages that but I can't help it; I'm addicted to learning.

Luckily, the learning curve becomes much smoother when existing skills can be reused in a new context. I've been building my own control interfaces in TouchDesigner for a long time now and while they aren't perfect, my tools offer a couple of tricks that help me keep things procedural and avoid laying down my animation on a timeline for just a bit longer. It's what helps me remain coherent across mediums and technological innovations. So my intention here was to benefit from the rendering capabilities of Unreal Engine while retaining my established workflow.

Right off the bat, when I started learning a tool as daunting as Unreal Engine, and before I even got to have fun with it, I had to customize the tool itself. I needed a way to remote control it with my custom controllers filled with pages of buttons and encoders. I already knew the ergonomics and the experience I wanted to feed my creative process. This was not negotiable and if you think I'm wrong here and that maybe I should see where Unreal as a medium would lead me by itself, well.. You might be right… but I have my reasons and even if I was wrong to impose my ways on the engine, the dynamic I'm presenting here remains at the core of creation: How we interface does matter and we can't neglect that.

I also have a typical programmer's mindset about this: I would rather spend five hours building a tool that saves me five minutes of repetitive work. That approach naturally led me to alter the software I use and eventually design my own virtual studio.

Back then, there were a few youtube tutorials demonstrating basic methods for exchanging OSC messages with Unreal but those methods didn't scale well and required me to map everything by hand. Creatively speaking, it's not the most inspiring first step to go through and it makes UE even less approachable for new media artists. If you're reading this, maybe you've been interested in using Unreal Engine as part of a broader workflow as well and this is exactly why I'm sharing this part of my project with the community. This is for you. It is intended for artists who want to bridge existing workflows from other softwares into an Unreal scene, with very minimal setup and maximum flexibility before committing to a rigid structure. In fact, it's meant to work even if you never commit to a rigid structure 🙂.

I'm releasing it openly for another reason too. I'm self-taught, and my practice exists because other people published their tools and their methods. It would be strange to benefit from that and then keep this closed. I also want other performers to play with my instruments, to break them, stretch them, and push them into places I wouldn't reach alone.

What I mean by proximity

I keep using this word so I should be more precise about it, because it sounds like a mood and it isn't.

To put it simply, proximity is how long the medium takes to answer you. In music, immediately. In a render, minutes or days. It’s also what happens during this latency. If you’re answering an email while you wait for your render, the flow has been broken. If your viewport isn’t real-time or you have to recompile the code for the project to update, even if it’s just for a second, the connection with your digital realm has been broken. That's the whole thing, and it's measurable.

When the answer comes back fast enough, you think inside the material. You try something, it responds, and the response gives you the next idea. You find out what you wanted by feeling around for it. When the answer is slow, you have to decide in advance what you want and then go through the process of getting it. Exploration becomes a specification. And happy accidents stop happening, because an accident only turns into a discovery if you're still there when it occurs and if you’re allowed to react to it.

The distance is made of specific things. Generic interfaces designed to be versatile rather than to feel like anything: the screen, the keyboard, the mouse. I deem them good enough for data entry. Then format conversions, mapping tables, plugins and APIs sitting between two applications that should already be talking to each other. Each one is a thin membrane. Stacked up, they get thick.

One thing I want to be careful about here, because "reducing friction" is the language of productivity software and that's not what I'm after. Friction in the material is where the work comes from. The physics simulation refusing to do what I want, a light behaving strangely, the tool imposing behaviors I never designed for myself: I protect all of that. It's the plumbing I want gone. The half hour spent getting two programs to acknowledge each other's existence gives me nothing back.

The most extreme case I've worked in is the Satosphere at the SAT, an 11.5 metre dome that spatializes sound and image at once. The place is in demand that almost all of the creation has to happen outside the dome. You build blind, then render, then wait, and only then do you see the work in the state the audience will see it. If something's off, you go back out, adjust without being able to check, and render again. Days go by inside that loop. What I wanted there wasn't speed. I wanted to sit inside the work while making it, at full scale, in its final form. That's when the artist and the medium are connected, and it's where the happy accidents come back. So when I’m in the dome and experience how fast or slow something moves and want to adjust it. This control needs to be right at my fingertips. Definitely not written as a variable in a function that requires it to be recompiled.

Why does it matter?

First of all, with digital arts, most of the tools were designed to fill the needs of a specific industry. This design is entirely focused on the output required by said industry. There is very little consideration for the actual process or the experience it provided. A simple example of this is how "Lightroom", a software meant to "develop" digital photos, has very little to do with the practical steps applied in a "dark room". What we care about is the finished product. So in the process of modernizing the tool, we got rid of an experience and created a new one, where the user is sitting in front of a computer and clicking around with a mouse. By standardizing the process, we also inadvertently removed key customizable aspects of the experience that were unique to its user.

It gets worse as the tools mature. Developer time is costly, so companies find it more appealing to add flashy new features than to optimize existing processes or refine subtle parts of the interface. Marketing departments reinforce this, because it is easier to promote a new buzzword-driven feature than a meaningful improvement in usability. The result is a widening gap between developers and users, with ergonomics pushed aside. This mindset then trickles down into schools, where the tools shape students into workers rather than encouraging them to think like artists.

Throughout the years, I believe we've been neglecting how much the user's experience influences the output we somehow care so much about. (Note that I'm not referring to UX design here, I'm specifically referring to artists and how they feel while practicing their craft.) Of course, what is a good or bad experience during the creative process is completely subjective. To me the only path forward in such a case would be that we maintain a greater diversity of approaches and techniques through as many tools as possible. Each leading to more ways to translate our ideas into art. To each their own.

The interoperable nature of the digital medium

Luckily the digital medium is, in theory, fully interoperable… it's all 0s and 1s after all. If it's within a computer, everything can control anything else. A sound can be viewed as an image. An image can be read as a timeline animating a robotic arm. The arm's motion can then be tracked by a moving spotlight… and so on.

There's no limit to these new connections we can draw across the mediums. So why don't we see a greater diversity of tools or mediums? Is it because we like familiar things? maybe… but it's definitely not because our toolset is complete. We are barely getting started trying to define how we build and approach this digital realm. I think it's because of how much work there is between every single simple idea and their realisation.

The cost of interoperability

The promise of everything remaining compatible with everything else is wonderful, but in its current state, there is quite a big cost to this interoperability. It's available through the use of an API, a plugin or a conversion tool that increases the distance between the artist and the creative process. I believe when the tools aren't seamlessly flowing one into the other, the artist who attempts to combine various mediums into a richer experience is quickly turned into a file processing agent having to sort and convert assets just to try something new. It takes time, and worse, it's distracting. Our focus shifts from creating to just making things work together and I believe too many artists, me included, are drowning in such frictionous processes.

The approach behind the process

The deeper I dive into the Engine, the more I appreciate its inner structure. I have started building more autonomous systems that no longer require bi-directional communication with TD and Ableton. As described above, the plugin calls functions and events similarly to how you'd call them from within the Engine. This gets me the best of both worlds. Early in the process, it's helpful to have very granular control over parameters to find the sweet spots using a midi controller exposing all of my parameters at once. Over time I might want to group those sweet spots into Structs and Material Instances and load them from within Unreal. This means, in some extreme cases, I could delete the plugin later on and still use all the work done within UE the exact same way without having to change my code. An example where I might want to do this is if I adapt a scene to be an executable mini game to share. In this case, I would simply remove the plugin from my packaged build at the very end.

Being able to keep my options open until the very end is vital here. This is what annoys me most in more traditional digital tools; they force you to commit to destructive manipulations of your material. I believe the biggest challenge in new media art occurs when we are forced to commit to a more conventional medium to present our work. Like turning an interactive virtual world into a short video clip for social media. By remaining non-destructive for as long as possible in my process, I am able to care longer for my virtual environment and sculpt it using various levels of control throughout. I only commit to a presentation medium when the opportunity and desire present themselves, so the same world can be documented and presented in multiple ways as I see fit.

I don't only make the work. I make the system that makes the work.

Brian Eno once compared the architect, who plans everything ahead, with the gardener, who must remain sensitive to how nature grows and adapt accordingly. I oscillate between the two. During creation I work like a gardener, staying attentive to my creative ecosystem and the surprises it brings. Then, like an architect, I step back to rework the tools themselves, designing new ways of interacting with the system. This plugin is what the architect leaves behind for the gardener.

AI

It's impossible to write those thoughts without considering the impact of AI. It has the power to lower the barrier of entry to the medium while reducing the workload associated with many tasks relative to creating digital art. It also has the power to simulate the act of creation along with a sloppy written artist statement with a single prompt. Everything we do now documents how AI has influenced what we define as a creative process or an artwork. That goes well beyond the scope of this text. What doesn't change is that the concept of proximity between the artist and their creation still applies, so that's how I'll position AI here.

There's one effect I keep coming back to. The tools I've been criticizing exist to produce a specific finish: clean and controlled. AI has made that finish nearly free, and now that this kind of perfection costs almost nothing, it has stopped signalling anything at all. What's become scarce is the evidence that somebody was there. Hesitation, tremor, a decision made in the moment because the material did something unexpected. Suddenly we value imperfection, spontaneity, glitches. Our whole relationship with that aesthetic has shifted.

So proximity stops being a preference and becomes the thing worth protecting.

It also gives me a usable test, and it's the same one I apply to every other tool: does this bring me closer to the material, or push me further from it? This is what we have to be careful of. If AI works on your behalf and your process is automated, the result arrives finished and you weren't there for any of it. The outcome is the opposite of what's intended. Using AI to build the instrument that puts your hands back on the material does the reverse.

Not everyone wants to automate this process using AI and not everyone who uses it wants to let AI handle everything. I love that AI is democratizing the digital medium for artists who have no interest in becoming programmers. And to those like me who are "in between" and do some programming, I love seeing how AI enables you to create that small tool, website or side project that would otherwise remain on the to-do list. Personally, I'm keeping AI on a tight leash with how it codes my stuff. Right off the bat when I started using it, it was to fix specific challenges I had in existing tools so the solution had to fit my existing architecture. Since then I've kept this approach but I've been having it rebuild code from scratch in contexts in which I already know the format of both my input and desired output. I remain the architect of the project while AI executes, so I'm not sure that qualifies as "vibe coding".

That said, whether we're assisted by AI or not, I believe the cost of creating new tools and new methods still remains. AI simply readjusted a workload that was disproportionate to begin with… allowing more artists to realistically get involved with the medium.

If I allow myself a single recommendation regarding the role of AI in your practice, it's this: make sure AI brings you closer to your medium.