Let's Discuss Visual Storytelling + Video Synthesis: Patching Event Based Narratives
Category: Forum · Tags: — · Posts: 42
#1 — creatorlars · 2022-01-07
When LZX first started, I was approaching video synthesis not as an electronics engineer, but as a would be filmmaker and game designer in the turn of the millennium: the era of web based interactive animation and MiniDV short films. It was an era before universities had “New Media” departments (in the modern sense.) The interactive fiction computer games of the 1980’s and 1990’s were waning in favor of larger budget titles and rendering-heavy 3D graphics, and YouTube / web video were still in their infancy.
I was attracted to video art tools, not just in an aesthetic sense but also in the potential of a modular instrument to produce emergent narratives.
Video is a medium which has been highly constrained to the concepts of linear vs nonlinear editing tools. Whether we’re talking about edit decision lists, splicing 16mm film, or dragging around ingested clips in Adobe Premiere or Final Cut Pro, these techniques are all based on assembling an intentional sequence of frames.
So while it may seem like a dichotomy such as non-linear editing vs linear editing covers all territory, I feel like it doesn’t – not even close, not even still. Any attempt at uncovering an emergent, live narrative in cinema usually ends up reduced to choose-your-own-adventure-like experiments such as Netflix’s Black Mirror: Bandersnatch or with VJ tools focused on sequencing the playback of prepared clips.
If you wonder why we have iterated through so many generations of modules and products, it’s largely in pursuit of this concept, of a video machine capable of not just beautiful imagery, but also engaging narratives full of rising and falling tension and structured, event based acts. A machine the video synthesist uses in a manner more like navigating one’s own subconscious while dreaming – an environment full of the unexpected, yet personal and familiar. A place only the individual artist could have found, yet a place impossible to plan or curate in advance. I feel like, as human beings, the hardware control surface – the place you put your hands – the steering wheel – is essential for the cybernetic relationship required to accomplish this.
Approaching the designs of Chromagnon and Gen3 modular, I’m attempting to design a toolkit that encapsulates our work to date, and gets us closer to a holistic creative environment than ever before. But there are some missing pieces we haven’t touched yet – mostly related to motion, nested animations, and sequencing. I want to do that in a way more integrated and intuitive than just clip sequencing and interesting LFO waveshapes. I think there’s something more there; and that to find it we need to think outside the box of both the synthesizer world and the videography/editing world.
I miss making these discussions part of the community. It’s why, I hope, you’re here on this forum – to be a component of the process of exploration and development we’ve dedicated ourselves to.
So let’s get some talking done, let’s discuss. What are your thoughts on this topic? How are you already using your video art tools this way? What works? What reads for you, in the work of others? What experiences do you have personally that could provide further background for this discussion? What’s still out there? What should “narrative” modular tools look like, and how should we be interacting with them?
Some of you are highly proficient animators and video editors and digital designers – what is your relationship with a tool more immediate and instantaneous, even, than pen and paper or rapidly assembling ingested camera footage?
Some of you have decades of experience in live television, stage direction and improv – what is your relationship towards a more introspective tool like a video art machine, where you are both improvising director and conjurer of the world on screen?
Some of you are genius polyrhythm programming modular synth musicians that craft melodies from the intersection of maths and probability – how do you make videos the same way, but without just directly applying the same techniques used in composing songs or ambient soundscapes?
Maybe you have no experience at all with video, or even audio, beyond video art hardware? What do you see in this discussion?
Thanks for reading, I am excited to dig in.
#2 — a_digital_index · 2022-01-07
Lot of thoughts on this but immediately I would say that my interest in all of this is to unhinge the dominance of singular ‘narratives’ and open up new avenues of understanding (itself another term for narrative).
I also think that the most dominant narrative in our lives is how we are presented with the idea of ‘our-selves’ via television and film, which is the most dominant world-building engine of the 20th century.
I do think there is something interesting about tinkering with that engine and these tools (LZX and other) let us disrupt the trend towards singularity.
It’s also just a lot of fun!
#3 — a_digital_index · 2022-01-07
Oops. I think I missed your question. Please continue to make cool modules and instruments that help do the above!
#4 — esnho · 2022-01-07
I try to answer quietly, bringing my own experience when (feel like a century ago) I was trying to survive mostly with live gigs using my generatIve visuals made with VVVV.
At that time I was creating visual processes like particles, moving geometries, weird shaders feedbacks, audio reactivity, geometry deformation; everything controlled with a lot of knobs (BCR2000) giving me the opportunity of change a parameter slowly, switch a scene/effect using a toggle, swipe a value, everything with custom ranges on knobs (I used a custom patch in max/map and midi nrpn).
This to say that I always worked preparing “scenes” that blended into each other, like multiple render targets textured on surfaces that worked both as screen and that got deconstructed in other shapes, I’m realizing that the most valuable thing for me is to control the signal path.
I don’t have a big lzx system and I’m actively experimenting with analog synth since less that 2 years, in the last weeks I played a lot re-routing the cvbs signal between the mixer (with keyer), the Vidiot, the structure, and some glitch fxs.
I feel that I probably will invest more in an RGB workflow with routing capabilities, and probably with more than one video source.
Probably routing capabilities should be controllable not only with switches and pots.
I’d also like to see something to process the video buffer, I don’t have a MP but I think it’s essential in such flow (unluckily structure doesn’t output 1V).
In both studio recordings and live I feel that signal routing is essential to explore and create variety.
This is my stream of consciousness.
Bye
#5 — dr_how · 2022-01-07
Thank you for introducing this topic Lars! Something I’ve been exploring the last couple years is how to make my live visuals relate and react to the music in the surrounding environment (either a band or a DJ) as much as possible, but by my own hands rather than automating too much with LFOs or envelope followers. While those tools do a great job of syncing synthesized visuals on a granular beat-by-beat basis or maybe across a couple measures in the music, I’ve been trying to develop patches on my LZX system that can more broadly follow the narrative of the music on a higher level, that I can react to in a hands-on interactive way in realtime. I want to follow the dynamics and moments of tension and release as much as possible in the moment as a performer reacting to the music, rather than just the technology having all the fun of doing the reacting.
Admittedly I still feel I have a long way to go with realizing this on my current LZX setup, and in parallel to that I’ve been also developing a setup based on the liquid light shows from the 60s, which provide a very hands-on organic, spontaneous, and precise method of relating visuals to music in the moment. When I’m working the liquids plates it also feels like I’m playing something akin to an actual visual instrument in the performing sense.
On my LZX setup I can create synthesized visuals with plenty of modulation, but in way still feel somewhat static in that you can tell there isn’t a human behind all of it reacting as dynamically as I would like. I’d ultimately like to interface with my modular in the same way that I interface with my liquid light plates. Right now I’m sending a camera feed from my liquids plates into the modular acting as a video processor which is working really well but with the right kind of control and precision over control voltages without the camera feed (maybe something like a giant Escher Sketch), I think the video modular could take things way further on its own.
#6 — creatorlars · 2022-01-07
a_digital_index wrote:
Oops. I think I missed your question.
Not at all! This is a very open ended prompt.
#8 — dryodryo · 2022-01-08
Fascinating topic.
Approaching video synthesis from a purely tool-based point of view, then there are no narrative limits. I.e., we can process any imagery and edit it in post to create whatever narrative we want. All kinds of happy accidents can happen along the way. That is one of the great strengths of video synthesis, its ability to surprise us with new imagery.
But if you want to talk about video sequencing, things get more complicated. It’s harder for an algorithm to generate interesting, unexpected patterns in time than in space. That added dimension of time means the system needs to work much harder/smarter. The problem of generating coherent narrative is non-trivial, and really can only be done by a sentient being that has subjective experience. Only then can the being actually model that experience to such a high degree of fidelity that it is accepted as genuine by another sentient being.
And so, moving generative art is pretty solidly in non-narrative territory. Machines have still not learned how to compose a story. Even the tech giants have not cracked this problem, despite the efforts of some very smart people and some very powerful hardware. So it’s probably a lost cause to hope that video synths can create narratives unaided by humans. The best we could do these days is program a complex graph of logic events that generates a pseudo-narrative structure. But seed that with some human intuition and you might have something. I.e, the human’s choice of clips/images to be fed into that logic graph would have a huge impact on its viability as a narrative.
When you start asking what makes something an interesting narrative in the first place, it turns out that it’s not the same for everyone. Sure, there are common threads… but people are wonderfully unique. Subject matter plays a massive role. Characterization is huge. A video synth can’t be expected to venture in this area. But from a purely structural point of view I think there are things that can be done.
If you’re looking at ideas for new modules, I would love to see something inspired by Buchla’s Source of Uncertainty. I am going to pick up one of the Tiptop clones when it’s available this spring. But that is just the jumping off point. Fractal algorithms are perfect for this. Multiparameter feedback with multiple recursion frequencies and amplitudes. Feedback channels going into one another. All of this to generate not just visual patterns but TIME patterns. When I have free time myself I am going to see what I can come up with on the computer using some visual programming environment like Resolume Wire or TouchDesigner.
Approaching video synthesis from the perspective of music, things get really interesting, I think. Instrumental composition is the original abstract pseudo-narrative. It’s all about patterns of tension and resolution, and yet it’s generally non-representational. There are exceptions, such as some Romantic composers, or musique concrete that incorporates sound effects. But by and large, instrumental music not only evokes a mood but also shifts that mood over time – all without words, pictures, or really representing anything but itself. This is my inspiration, an art that exists in its own right, makes reference to nothing outside itself, and yet somehow manages to tap into human perception and cognition to generate meaning.
#9 — Yatsek · 2022-01-08
I agree about the liquid light shows - playing it like an instrument. I have also done that as part of a live gig with a band. Perhaps in time I can learn to play an LZX modular synth like an instrument, creating the visuals in response to music and with the music rather than in mechanical sync.
#10 — Rik_bS · 2022-01-08
I dream of a “macro controller” and event sequencer, that many signals can be routed through and manipulated, with set “scenes” that can be programmed then either switched through or played.
The closest example I can draw from is lighting controllers - where the scenes are programmed ahead of time and played live with the music or sequenced as part of show automation.
As a step towards this idea of macro control, once the video rack is built I’ll be working on a 6U skiff that has all my modulation on hand, 9 signals at at a time to be piped through a Doepfer A-138E then via a Worng Vector Space out to the rack.
One turn of each crossfader will sweep through 3 different CV signals, and when it hits the V.S it gets combined/manipukated to be a dense source of 17 related signals.
The dream would be this level of control with added matrix routing and additional automation to be programmed as a scene, and each scene to be cut or blended with button, crossfader or an event sequencer
#11 — wednesdayayay · 2022-01-08
Audio influencesI’m not going to talk too much about audio but from a young age I’ve considered my audio output to be my journal. I create snapshots in time of my feelings to come back to at a later date. This has acted as a means of time travel for me throughout the years.
I also used to play lots of bass and that practice got me thinking about music in a geometrical way from an early age. I used to have bass tabs hung up on my wall where the geometry was outlined for songs that I found particularly pleasing/interesting (I’m looking at you primus). At this point I play piano and apply this same geometrical thought process to that practice.
Nichole and I have been together for almost my entire audio journey. However they haven’t always been a part of that process. We did play in a fully improvisatory 4 piece band for several years when we lived in indianapolis. This ended up being a huge boost of confidence for me. MFT :: Japatucky
“wonderful dreams of you” is a good example of the kind of things we did at the time.
These concepts of saving snapshots of time for later, seeing through the medium to its internal joke/structure and fully body improvisation have deeply influence the path we’ve taken toward our current workflows.
This would be a quick way of explaining these concepts that grew in my mind over the last 19 years of doing this without getting too far into the weeds.
Theatre influencesone of the biggest influences on our output/process this past decade has been our work together in/on our community theatre. We started this theatre together in 2013 with $500 from our bank account when we were dirt poor and we do every single aspect of production. When you do it all in a theatre it looks like costumes, set building/design/dressing, sound design, painting, lighting design, props, poster design, etc… all coming together at a specific time. Having big ideas at the start of a project and then slowly coming to terms with the reality of your situation/time frame as you get closer to opening night.
We do 6 shows a year where they are all two weekend performances.
This process of building up a larger output through careful work on individual parts and then piecing them together and letting them speak/breathe on a stage is absolutely the way we are approaching things at the moment.
Information influencesOne of our big pushes over the last couple years for video synthesis has been information spreading. This came about from not being able to find a place where someone could find all the pertinent information about what we do in one place. If for nothing else it allows us to understand old workflows/ideas but it also lets other people come up more easily and hopefully contribute their own “ah-ha’s”. All glory became because of the wonderous humming CRT god.
Clip show introductionin fact…
we just recently started sorting all the “clip show” videos that we’ve been saving as we found datamoshing videos. This will end up being a full length release.
These clips are things that needed more attention than a simple music video length datamosh. It could be that I saw narrative potential or that they illicit a certain feeling/time/space.
We are in the process of getting our very first new home and have had video things boxed up in the process of moving for a while. That has given us time to explore non hardware related video synthesis mostly through ffmpeg/touchdesigner.
Experiments with video interpolation/slit scanning/shape generation/projection mapping/physical object&light manipulation/lumia/datamoshing/feedback video looping etc have been piling up. Some short form some long but all about the process and evolution of technique over time.
this picture is the main folder overview for the release as it stands.
Naming convention is very specific and just as improvisatory as the artistic creation itself. In parentheses are the themes for the pieces. Lots don’t have final names because big work hasn’t been started on them yet, they are still amorphous.Some combinatory themes some single mostly conceptual (water) rather than workflow based (projection mapping). Lots of audio has already been pulled and more will be made during/after.
one thing to note is that it is ok to be a bit cheeky if only for your own enjoyment. For instance the water theme I’ve got plenty of regular water clips/imagery but I also have an unreleased shbobo shnth tutorial that talks specifically about the “water” opcode. I never released because I didn’t close my mail app and a sex toy email came through. You can hear my train of thought coming to an abrupt halt while realizing this.
Trusting your carefully crafted instinct the narrative becomes emergent as you gather these seemingly disperate pieces together. If it doesn’t you don’t have the right pieces/order/theme throw it away and start again. This might mean putting things in the trash or it might mean putting the original videos back into the “pick me” pool to be sorted again at a later date. Don’t be afraid to lose things in the wash. The act of letting go can become a catlyst to understand how to move forward.
workflow TLDRThis is where we are at currently with our workflow.
Collecting pieces over time that represent snapshots of video synthesis growth & reprocessing in current workflows.
Collecting pieces that represent the “current” in your life which can mean many things to many people.
Writing about those workflows and sharing the “how”.
Sitting on those pieces and letting them roil/boil in the back your mind.
Organizing them into a loose theme.
final composite/reprocessing.
it becomes a collage of yourself over a given period of time.
here is a very short example from one of the pieces as it stands.
How does this work in modular?We would need a module that could save clips/frames to nameable folders. Then being able to CV address the where/who/how of at least 2 (3 or 4 would be better) of those clips at once. Since I’m assuming each “track” wouldn’t necessarily have its own output (that would be rad though) I would love to see an assignable (assignable enough to say R channel from clip 1 G channel from clip 2 B channel from clip 3) “aux” lzx RGB output in addition to a regular main RGB out.
I’d like to be able to USB from that module to the computer and see it as at the very least a drive of those videos that could be taken and used however in a traditional editing environment.
being able to freeze frame per track with scrubbing (course, fine) would be a huge win
being able to resample the output as a new track easily would be huge as well
#12 — wednesdayayay · 2022-01-08
in order to ask a machine to tell a story
we need the machine to be able to describe video
Color
- R%, G%, B% White% black% modulators
Frame
- frame difference% modulator (fixed or variable delay time for differencing)
- motion vector modulators (even just 4 quadrants with phase modulated ramps “showing” the movement direction per quadrant)
- scene change detection modulator (could be based off frame difference but with a threshold for saying the scene has changed)
having modulations based on patchable video would be unbelievably fun.
you could probably just super slew a G channel to get an approximation of this effect but it certainly wouldn’t be the same.
having your machine speak back these details about your video so that they can end up controlling themselves/be controlled by us would go a long way toward making a generative narrator “sequencer”
#13 — creatorlars · 2022-01-08
7pip wrote:
To allow the subconscious to speak, unencumbered, one must be able to touch the tool without arousing any hesitation or doubt.
Yes! “Eyes on the screen, hands on the knobs.”
It’s a big ergonomic difference, in using a video vs audio instrument – with audio you can stare at what your hands are doing while listening – with video, it’s more important you can manipulate the patch blind, while staring at the screen to see the results. With Gen3 and Visionary series, the knobs are all evenly spaced across modules – this is so that you can spread your fingers out and count them with your fingertips without looking, like piano keys, regardless of what order you are putting them in.
#14 — creatorlars · 2022-01-08
These replies bring me great joy and I really appreciate the way all of you are engaging this subject! I’m going to be replying with my thoughts as I can, and I’ll likely be re-reading this thread for a while.
To further some of my own thoughts – they echo this sentiment:
dryodryo wrote:
Subject matter plays a massive role. Characterization is huge. A video synth can’t be expected to venture in this area. But from a purely structural point of view I think there are things that can be done.
Namely, a structural point of view – the dispersion of dramatic energy across a narrative arc, rather than signifiers such as language or characterization. I’m interested in the structure of dramatic tension – how do we turn that into a patchable signal? What does that signal interact with exactly? Can pure “narrative signaling” trigger a reaction in the viewer, even if what we see on screen is not representative of a real object or person? Can that become compelling enough to engage a viewer’s inherent understanding of visual metaphor, such that concepts like characters and confrontations are not only imagined, but communicated by the artist with clarity? How do we give the user total control, while also introducing enough complexity for emergent outcome?
If anything modular synthesis has taught me, it is that we don’t need to make this complicated. The complication comes from the interaction between simple signal sources and modifiers.
So, I theorize there are some universal human visual cues. To give an example, almost any 1 year old child from any culture or time or place, will react with surprise and laughter at the game of peek-a-boo.
And peek-a-boo is something we can dissect – both from a narrative perspective (sequence of events) and a visual perspective (what we see on screen).
- There is intentional concealment of a hidden object of interest. It is obvious to the viewer there is something exciting being hidden from them. In fact, the viewer knows what is there already!
- There is the rising, sustained tension of not knowing exactly when the hidden object will be revealed.
- The reveal comes suddenly, with great visual impact. It’s full of detail and motion, interaction. To a baby, this is comedy gold! Laughter ensues.
- Before the viewer’s eyes have time to fully process the reveal, it’s concealed once again, and we repeat from step #1. Only this time; the dramatic impact is amplified. The indeterminable waiting time between concealment and reveal becomes even more tense. The performer can push the waiting time further and further – a choice which only further excites the viewer.
Peek-a-boo is a simple example, but an easy one to apply. We have a video synth pattern on screen, but are only seeing one component of it. A slight visual tease of an overall shape or texture. After a random length of time, we open the Keyer or whatever is concealing it, showing something with more color, or more geometric complexity, or both. Then we hide it again. You don’t even need a special modulator for this – just your fingers.
And from here, there are countless places to go.
What if you have two peek-a-boo games going on at the same time in overlapping intervals?
What if you slightly alter the revealed “object” each time you reveal it?
What if one time you reveal it and it’s drastically different? Woah, big narrative moment!
Peek-a-boo is just a start, though – what other “narrative structure” games can we play like this?
#16 — Gavin · 2022-01-08
Like others here, I have enjoyed using modular video to push away from narrative. I had to struggle with editing my video synth stuff without turning it into too much of a narrative, and have tried to increasingly resist that.
What you describe feels like a parallel problem to modular audio synths. You can write techno on them, and meandering ambient, very easily. Music that has a strict narrative, that suddenly changes its global structure at agreed points (chord sequence shifts, time signatures) like a rock song does, are much harder to do in audio modular without some high-end sequencer action.
I would like to think about a module capable of global shifts while smaller patterns of a patch continue the same - which gives you song sections in the same ongoing key, or book chapters with the same progressing characters, etc. Switching the solarise or negative on my Visual Cortex would be basic examples of this, but it’d be nice to have those kind of overarching aesthetic aspects move in a less on/off way transition, or to imagine some new global parameters which could sit to one side of a patch.
In terms of narrative aspects, the other thing, aside from being ‘global’ (or at least multiple) parameters, is that they are slow parameters. I see a lot of folks, myself also, who are fans of very slow oscillators in video synths. I think this is because they have a more narrative purpose, for those slow changes that give a less immediately perceived change. And those oscillators function as one of those global parameters. I find these more useful than a sequencer for this sense of narrative change, but maybe a very slow sequencer would be the answer!
This is different to the way I also sometimes think about narrative progress, which is approaching this in terms of ‘scenes’ or switching between saved patches. That’s more what you get with VJ software I think, or the kind of narrative block created by a mixer wipe. The video synth is less amenable to this really, and it would be more about an individual operators’ patching strategy than anything actually embedded in the modules.
#17 — Agawell · 2022-01-08
in terms of tools for this:
I think that utility modules are always useful for this sort of thing, both in audio and in video
passage was a great module for this, hopefully there will be a future module in this vein - maybe with cv control as well as knobs
switching modules also are a great addition - the doepfer a-151 works at video rates
macro controllers - more advanced than available at the moment - possibly something that could work on both a timer and triggers and cv and possibly a dial - possibly an advanced version of mutable frames - add a switch for range - and it could easily be sold to both audio and video users
this would feed brilliantly into the proposed video player - maybe something like a more advanced music thing modular radio music - lzx tv video? where you can switch channels (clips), scrub forwards and backwards, play forwards and backwards and position within the clip - all under cv control - maybe one that can do both component and lzx standard video output
#18 — dryodryo · 2022-01-09
OK, when I was talking about structure I should have said temporal structure. I think we can definitely create patches that perform chaotic transitions/events/timings that will keep the viewer more engaged. I.e. fractal/recursive variation in event timing. Unpredictable, surprising, and yet not unfamiliar. It should not be that difficult to sort this out, given the tools we have today.
But when you start talking about programming the peekaboo game – again, things get more complicated. The machine doesn’t have a sense of what sights are interesting. I guess you could build face recognition into a module but that doesn’t seem like a good idea to me. We’re back at needing human experience and intuition, at least in the choosing of clips/images, to generate a meaningful narrative. If the peekaboo program works I guess you could also program editing/compositing patterns for other scenarios, e.g. suspense.
However, I would be quite happy with a high level event generator with multiple LFO frequencies, variable feedback, cross-parameter feedback, and an easy-to-use system to store presets.
Thanks
#19 — creatorlars · 2022-01-09
dryodryo wrote:
But when you start talking about programming the peekaboo game – again, things get more complicated. The machine doesn’t have a sense of what sights are interesting.
Agreed – and I am talking more about the human performance element than the idea of programming intelligence into the patch – anything from guiding the frequencies of the modulators by hand, manually controlling the knobs with a steady hand, or transitioning between higher level parameter groups and scene programs.
To put it another way – can you perform an animated film on your video instrument in the manner a musician performs songs on a musical instrument?
I feel like there’s something in that space that isn’t directly analogous to anything already existing in the world of audio synthesis or in digital media – I feel like the Scanimate took a big step in that direction – frame counters and low frequency ramp transformations, and the idea of a trained animation team at the console. It was kind of like a band, rehearsing then performing an animation. (At least that’s how I imagine it in my head.)
#20 — banyek · 2022-01-09
creatorlars wrote:
Some of you are genius polyrhythm programming modular synth musicians that craft melodies from the intersection of maths and probability – how do you make videos the same way, but without just directly applying the same techniques used in composing songs or ambient soundscapes?
As many of you have touched on certain directions which I like, I try to keep my addition brief.
I come from a modular synth environment, making electronic minimal dub and techno music of many kinds, plus sound plays with human voices which I also record and direct.
I started video synthesis with minimal knowledge of what is generated in what way, and had an almost 10 year background with audio synths and modular synthesis.
Video synthesis is quite tricky though. I have two approaches 1) the ‘painting’ approach where no animation is present, no transitions just final results (and versions of them).
2) or I have movement on the screen - sometimes I animate other peoples’ works (paintings, analog photography, etc).
I relied on generative patterns in modular synthesis in two ways:
a) audio-based events (envelope followers)
b) generating events with gesture (knobs, theremin, etc)
As these are the sources, the results would usually be aa) tonal or amplitude changes bb) rhythmic variations (clock divisions & multiplications, resets & combinations) & nothing else.
I think I never really generated ‘melodies’ for example.
A generative approach usually comes in handy in a third case, that is, interactive installations, where people with no background knowledge can control the instrument via theremin and their own voice. So generativity for me means either “spicing something up” or creating an environment for “interactivity”.
Bottom line: generativity for me is a means of achieving a result I had wanted as a person - the narrative vision had always been there somehow beforehand.
One more thing: a generative result can be “happy accidents” too (e.g. having an originally unintended color result in patching a multi-output video module into a mixer) - but this is mostly “getting lost in the maths of things” - I am a great Serge-enthusiast - you are left alone with functions and invited to do whatever you want with them instead of other synth modules which put you in a narrative tonal and functional context. Yet, I also like LZX’s naming convention in some of the series - they brought a lot of “Aha! I get it now!” moments in many of my workshops & performances.
I believe, “event-based” narratives also do not have to be very complex either. I like starving “the kitchen of the mind” & make it cook something wonderful with limited “resources” (i.e. visual inputs). Some of the users who saw my pictures came up with interesting reactions defining the “genre” of the “movie” they were looking at (let it be static video art or animated):
“Western” - (Arch & Topogram)
“Video Game”, “Sci-fi” - (Castle series)
“Retro” - (more “basic”, Cadet-based patches)
“Horror” “Suspense” (patches with feedback & darker colors)
You see, these are already “narratives” in themselves too!
#21 — dryodryo · 2022-01-09
creatorlars wrote:
I feel like there’s something in that space that isn’t directly analogous to anything already existing in the world of audio synthesis or in digital media – I feel like the Scanimate took a big step in that direction – frame counters and low frequency ramp transformations, and the idea of a trained animation team at the console.
Yes, I’m with you. This is what I was talking about regarding a high-level event generator. The user specifies base timing frequencies for LFOs, gaussian noise functions, and random events. Each oscillator/operator has controls for frequency and amplitude of feedback, as well as routing and amplitude of cross-parameter feedback. The system outputs gates/triggers and continuous waveforms to effect transitions. This would be the brain for other modules that actually perform the transitions, basically you need a multichannel sequencer / switcher.
Regarding continuous transitions such as dissolves and wipes, sine waves are a good start. Slow out / slow in, one of the Disney 12 principles of animation, is built into a sine wave. But any animator will tell you that the default B-spline curve is boring, artificial, mechanistic, and barely better than a linear transition. So an event generator should have some mechanism for waveshaping in order to de-couple the slow-out / slow-in from the overall transition time. In animation terms, extend or contract the Bezier tangent handles.
And not just that, but for quality animation, the slow-out / slow-in needs to be asymmetrical. Sometimes that asymmetry needs to be VERY EXTREME. The canonical example is a camera zoom. Screen space is exponentially more “sensitive” than a linear parameter such as focal length. To get a zoom that looks linear and natural to the audience, the function curve needs to have a very short slow-out of the initial value, and an extremely long slow-in to the final resting value.
This is definitely an area that is ripe for development. Video hardware has a problem, the automatic transitions all look artificial. This is why I’ve always preferred manual transitions with a T-bar, rather than that awful “TAKE” button. I presume some current high-end switchers and DVEs have more options for transition waveforms, but I’d wager that these are still pretty lackluster compared to the human touch. An event generator module could improve on this by modulating the base transition wave.
#22 — 337is · 2022-01-09
Great reading through this, thanks for the stimulating discussion all. Still letting this soak in and don’t have a ton to say yet. But…
creatorlars wrote:
Peek-a-boo is just a start, though – what other “narrative structure” games can we play like this?
Shifts of scale and zoom seem to be similarly related to peek-a-boo in terms processes which can reveal things that were hidden prior to a shift in perspective. If one is presented with a field of dots seen from a distance, the movement of approaching the field for a closer look could have a sense of drama, perhaps even more so when pushing past or through the field to view things from the other side. Maybe on the other side they’re not dots at all but triangles. Human vision can really only see one direction at a time yet we “know” that there are things that are happening outside of our current gaze. We can see something different by shifting our gaze. Part of an artistic process can be playing with those known expectations (often binaries, either/or, front/back, top/bottom) and using the expectation to emphasize an unexpected result that isn’t constrained by perceived binaries…not this/that but something else. I think for me, confounding dualities has lots of application in modular synthesis where often, things seem, at a very basic level, off/on. This is where things get interesting to me with more chaotic circuits that are are not so predictable in that way. Another layer here that I appreciate with sources of uncertainty is setting up systems where the duality of me/instrument is undermined. Nothing I like more than not being able to see/feel my hand at work.
#25 — phosphenes · 2022-01-10
This thread has been a joy to read, thank you everyone, such a diverse range of ideas.
My route into video synthesis was during my second year of university in 2015 (electronic engineering with music technology), Paracosm’s Lumen provided a more affordable path into the field. I used Lumen to do visuals for a regular techno party I was a part of, sequencing additional parameters and audio reactivity in Pure Data. In terms of narrative I felt the XY pad in Lumen being able to interpolate between 4 different parameter presets was really powerful, it allowed me to navigate between known points, islands of stability, into more chaotic behaviour and back out to familiar scenes.
At some (most!) points of the night I’d want to dance rather than knob twiddle so I also programmed some rudimentary random control patches in PD. I started to explore the idea of using stochastic matrices to create a framework for probability based control but ironically this got side lined by building my Cadet synth. I’d used them in my first year to create a C program that could ‘improvise’ jazz, which given my ropey understanding of both programming and jazz at the time, went surprisingly well. I figured if it worked for music it would work for video. I love the idea of a tool that could be seeded with preset scenes/control parameters and then move between them using an algorithm which could also be influenced in real time by the user.
In terms of what a hardware tool would look like, @Rik_bS is pretty much on the money
I dream of a “macro controller” and event sequencer, that many signals can be routed through and manipulated, with set “scenes” that can be programmed then either switched through or played.
The closest example I can draw from is lighting controllers - where the scenes are programmed ahead of time and played live with the music or sequenced as part of show automation.
Maybe something that can add in some of functionality of Sensory Translator, Escher Sketch, a VCA matrix with some additional computational controls.
#26 — wednesdayayay · 2022-01-10
old school lighting boards were a blast to operate/perform
nichole and I spent many years up the spiral staircase of an old art building running shows from a tiny black box
this kind of vibeI remember getting a new set of young techies and playing the spanish version of troll 2 over the headset at one point during a rehearsal
#27 — creatorlars · 2022-01-11
“Scene level control” is a must. But also tricky in a patchable context. We can either treat each individual connection as a node in the scene (in which case you end up needing a crossfader with a number of channels equal to the number of connections to morph between settings). But this quickly leads us back to the superiority of a digital system in this regard, where the cost of adding another parameter group is negligible.
So that’s a puzzle. I love giant modular systems, but I don’t think you should need a giant modular system to introduce higher order sequencing and scene control. Whatever we come up with for animation needs to be at least as accessible as the other components of the system. A powerful scene transition should be possible with only a couple of modules.
Pushing all of this into a purely digital control environment could potentially diminish the attractiveness of using a real time analog computing environment with a universal voltage standard for creating images, and get us thinking more about the results (frames), than about the individual components of shape and color which compose them (that is, to me, video synthesis!) There’s a place for it all, of course.
But there are two ways we’re trying to combat this design challenge. One is the concept of “embedding a transformation into the image itself.” If we can embed the spatial and scaling context of a transformation into XY vector images (aka HV ramps), and use those as the source to generate a pattern (using a separate module!), then we are able to transfer an entire group of parameter settings (x, y, width, height, lens distortion, etc) using only two cables! Now it is much easier to do a scene transformation – since to change scenes we can just crossfade between HV source ramps representing two separate compositions of spatial coordinates.
The second way is to make every generator also a processor for other generators. For example, if one module is creating a “space” then that space itself can be patched into another module and further modified, in a hierarchy of vector modifiers. So now if you can crossfade between the frontend parameters for the “parent module” you end up creating a cascading effect through every derivative branch of a modulation tree.
So this is exciting stuff for me; but describes more the structure of a patched transformation space, and describes what some of our new modules (ART3 especially) are designed for, and explains the lineage of thinking behind modules like Navigator and Shapechanger. I think there is room for a “spatial generator/processor” that is digital in nature, allowing a large variety of user saved presets, and fluid interpolation across their settings. TBC2’s internal generators will be an interesting place to start experimenting with this.
To add movement to that, we haven’t gone very far with it yet. Navigator is a ramps rotator and Diver is a ramps positioner – but there’s nothing yet for “the scene”. Coordinated motion across many objects on a singular timeline.
But I think we approach it as a hierarchy transformation as well. We need a master transport control of some kind, to start – a “low frequency ramp source” with the voltage of that ramp representing percentage of timeline length. And then that single low frequency ramp/envelope is the timeline for any number of subramps or envelopes created through waveshaping (easing curves! ideally an analog bezier implementation) and subdivisions through wavefolding. These modifications of an overall “timeline ramp” cascade to any subdivisions. We can stretch or modify the time scale of the entire parented animation simply by changing the frequency of our “low frequency transport ramp.” If we want to change the easing curve of the entire animation, we just need to bend the waveshape of that envelope.
To state it a bit simpler: we start with one envelope generator. You can Play/Pause/Stop that envelope with buttons. You can scrub it. You can change the overall speed of it and FM that speed. It shows you a frame count. It feels like a sequencer. It’s output though, is just an analogue linear ramp. It is a voltage constant for the timeline position itself. Next, perhaps we patch this ramp into a “3-act waveshaper.” This is like a very slow colorizer. Instead of color bands though, you have time segments as subdivisions of the transport ramp.
Perhaps “Act 1” is patched to slide a shape across the screen. “Act 2” is then patched to a module that makes the shape jump in scale, and “Act 3” fades the whole thing to black. Now we have a 3 part animation. The master transport ramp can start/stop/pause it, speed it up, slow it down, warp its timing, etc without touching the “3-act” controls at all.
How does it go from there? Another 3-act waveshaper. But instead of the transport ramp as it’s source, we use the Act 1 out from our first module. Now Act 1 has 3 more acts, and those subdivisions go to details of the animation. Maybe as the shape slides from left to right, it now wiggles up and down, then slows slightly, then bounces.
You get the idea.
So, I guess my current thoughts for an “animator” module lie in this area. “Clockless” timeline and subsegments through waveshaping techniques, inspired by the Scanimate. Perhaps an “event trigger” module designed more for singular events/envelopes that happen at time points (comparator thresholds) along the timeline. Needless to say, I think we’ll be doing at least 2 modules that fit together in this kind of workflow for Gen3 at some point. But this thread wasn’t really started to workshop product designs so much as just sit down and hear all your thoughts on the subject. It’ll be a minute before we can concentrate on these designs, so I want to listen and get the pulse of the community on the subject, so I have some time to think on it from different perspectives.
#28 — Rik_bS · 2022-01-11
creatorlars wrote:
But this thread wasn’t really started to workshop product designs
ahh damn, I’ll save my pitch for later…
(parks the idea of a digitally controlled 16 in, 16 out VCA/mixing matrix with a hands-on interface, internal modulation sources and per-event control of each parameter)
#29 — creatorlars · 2022-01-11
ahh damn, I’ll save my pitch for later…
Well I mean, I want to hear all of your ideas! Whether that’s theory concepts or concrete user interface imaginings. I just meant I don’t want our own R&D projects to dominate the discussion, there’s plenty of that in other threads.
I don’t have all this figured out yet. But I’ve been ruminating on all this for years.I think there’s huge potential for the concept of the “Matrix VCA router” in general. The EAB Videolab had one:
#30 — giantmecha · 2022-01-16
So cool to see people thinking about lots of the stuff that occupies much of my brain, and in such thoughtful ways. Been savoring everyone’s posts.
I’m super interested in the intersection of music, video, and (for lack of a better term) videogames.
First, music stuff: The modular and outboard tools available in the audio world (or a good software environment with the right MIDI controllers) allows for a certain kind of abstract storytelling that I really enjoy. Being able to play with “character” via melody/harmony/percussion/etc and “space” via reverb/delay/etc and “movement” (modulation) — and then find emotional resonance in the way these things are tied together — feels like it somehow charts a path for what could be done with “narrative” in video synthesis. Creating and then destroying walls of sound + light. Playing with things like tension and release (peek-a-boo really hit home as I’m a nü-dad), chaos and order, perspective, speed, and scale in a timeline being creating is realtime is what I strive toward in my AV setups. Being able to go from slowly traversing seas of feedback to percussive physical drama (ie realtime Autechre Ganz Graf) and anywhere in between over the course of a set.
Game stuff: I feel like there’s so much to talk about here, but I dream of being able to synthesize and interact with characters and environments (both terms used verrrry loosely), using hands-on controls. Ming Mecca had some really cool ideas (or, more accessible, this software sequencer called New Path), and games (and game controllers) in general are of course really good at letting players explore and interact in physically/expressively responsive ways. Ways to generate simple AI-like attractors and repellers and build them into a system that feels like a garden/starfield/turbulent ocean sea/kids’ playground of cross-pollination and procedural interactions. Systems of objects and spaces that relate to each other in highly controllable ways.
This is going down yet another path, but I’ve been messing with some neural net text generators and they are getting crazy impressive. I only bring this up because it allows places and people and dialogue to be conjured from a very limited input, very quickly.
Anyway, I guess I’ll summarize by saying, WTB: synth but also Earthbound but also a delay pedal but also F-Zero but also a Mood Organ but also a Euclidean sequencer but also a Memory Palace? IDK.
#31 — creatorlars · 2022-01-16
giantmecha wrote:
Game stuff:
Definitely. I’ve been obsessed since writing the Fortress firmware, with the idea of “self playing game AI” as a visualizer. Nothing complex, just very simple parameters; one agent draws a maze, one agent solves it – voltage control over the maze drawing algorithm’s complexity and the solving agent’s pathfinding algorithm. “Ant farm simulators”, is maybe a good way to put this. Just the idea of the actor/agent as a visual entity is something we can borrow from games directly when composing patches.
More event based states will help with this. For example, you can patch the collision between two shapes (like the pong paddle and ball) with AND logic and keyers on a video synthesizer, but to change the ball’s direction requires a flip flop and a counter. Ming Mecca definitely had some amazing ideas on how to make all of this patchable.
#32 — creatorlars · 2022-01-16
giantmecha wrote:
This is going down yet another path, but I’ve been messing with some neural net text generators and they are getting crazy impressive. I only bring this up because it allows places and people and dialogue to be conjured from a very limited input, very quickly.
Random Poetry Generator would be nice.
#33 — jwsmithwick1 · 2022-01-16
@creatorlars , thank you for starting this conversation. Most of my video synthesis work involves silent, long captures of animated images generated from my video synth. Here is my current creative conundrum with regards to my desired audio-visual workflow.
I discovered video synthesis though music. As is such, my creative process is built around songwriting. Typically, when I record audio, I will start with what is essentially a loop of drums and sequenced audio synths and record a take. From there, I’ll edit out the drums to map out a structure. After that, I layer on some overdubs. Then the recording is ready for mixdown. This is a pretty standard process.
I would love to be able to mirror this process with my video synthesizer whereas, I would be able to make a video capture for the foundation, and then capture a different patch that is reactive to each successive overdub. Once I have recorded and captured all of the audio-visual elements of the song, I would like to mix those captures together. I feel that with a TBC2, or more, I might be able to partially achieve. The piece of hardware that I feel is missing is a sync-able video pool or recorder that can capture and play back images in sync with my Audio DAW (with a basic cv or midi sync signal).
I’m not sure how this could be implemented. What are your thoughts?
#34 — creatorlars · 2022-01-17
jwsmithwick1 wrote:
I’m not sure how this could be implemented. What are your thoughts?
Yes! A sampler module/instrument is an inevitability – you’re right, we’re trying to build that off the TBC2 architecture and then implement something I imagine similar to a “video 4 track recorder.” Something with a 2X2 interface capable of inserts and overdubs through analog key input. I imagine stop-frame animation style transport control could be useful here as well as looping / drum machine style overdubs.
The next step for us in this particular direction will be with integrating hardware media decoding/encoding using the existing hardware platforms (TBC2/Memory Palace) as development environments.
But I’m curious about what any of you have put together with existing video gear. There are some video samplers out there, like the P10, and video software environments capable of this kind of workflow (most are not oriented to live streaming though.)
#35 — dryodryo · 2022-01-17
Love hearing about all of your amazing ideas, Lars.
I understand your motive for keeping things in the analog domain. You have pushed and continue to push the envelopes of the possible in the analog domain. However, I hope you will continue your development of digital tools. I think you know I want an HD version of Memory Palace. But I also want a miniature DVE.
#36 — creatorlars · 2022-01-17
Oh, I want to do it all, of course. If things pick up it would be nice to hire a full time software developer so we can make more progress. My main concern is with supporting new workflows or techniques that aren’t already possible with digital environments. But… I feel like most any hardware tool outside the industry boxes of “video mixers and converters” falls in that category, whether analogue or digital.
In the context of this discussion, a modular analogue animation system is something quite different from a digital event programmer / integrated digital processing / parameterization environment. They’re powerful in different ways. The digital animation stuff (parameter grids, modulation programs, etc) I see mostly as a modulation/animation subsystem for existing digital processors (like Memory Palace) rather than a separate module. For separate modules I naturally want to think in modular blocks – meaning not just a giant animation module/workstation, but building block modules that can support a small system, but also expand in a modular way to a massive system (level complexity / number of channels is user controllable, in other words.)
#37 — dryodryo · 2022-01-18
Yes, I’m with you completely. No need re-inventing the wheel. If DVEs already exist, there’s not much pressure to create one for Eurorack. However, I/O is still a problem. Without a DAC/ADC module that supports DVI/SDI, our options for integrating outboard gear are pretty limited. There’s no way to insert outboard gear into the LXZ signal path. I mean, I guess one could have a whole battery of TBC2s and ESG3s, but that seems like overkill, and an expensive solution.
But what would really open things up is a very simple DVE channel in the existing LZX standard. 1V RGB I/O, six degrees of 3D Cartesian freedom, CV control over those six parameters. A joystick and/or a few buttons/knobs, but no sequencing. That can be handled by external animation modules. MIDI would be nice, but not essential. Keep the menu-diving to an absolute minimum.
It may be entirely possible for you to do this all in the analog domain. Maybe you’ve already done it with Chromagnon, or some other module. Information is scarce, so I don’t really know its full capabilities. But the key for me is the six degrees of freedom in a Cartesian volume – or at least the illusion of such. Similarly to the HSL dilemma, I want to use my existing, deeply ingrained conceptualization of 3D space. I’m not at all opposed to re-wiring my brain to do things differently, but that would take time away from actually making art. And I’ve only got a few decades left before I’m too old to do any of this anymore.
#38 — Boneoh · 2022-02-22
I am so glad that I found this thread! It’s very interesting to read the many thoughts posted. So many different perspectives and ideas.
I’m planning on starting an LZX Gen3 system asap, I don’t currently have any video synth modules, but have a 12U eurorack. I want to be able to merge the music with video, and the demo videos have really drawn me in.
My main goal is to have fun being creative! The initial plan is to use Chromagnon and Keyer to merge recorded videos with interactive ‘knob-wiggling’. The possibilities seem endless. I’m interested in blending abstract video with recorded fun stuff to see where it leads me.
I enjoy computer graphics, have worked on-and-off in this area since 1980, when we had memory mapped frame buffers and we wrote one pixel at a time. It took forever to record frame at a time to film and eventually a large simple VCR. Today I’m experimenting with a NVIDIA Jetson Nano to see what kind of abstract stuff I can create. The hardware can model and animate at crazy rates in real time! I see using these kinds of videos with Keyer both as foreground and key.
I’m super excited to be starting this phase of creative expression!
#40 — reverselandfill · 2022-02-23
for some reason I’m picturing an old Umatic editing desk when I think of a timeline function.
with big jogwheels to set the begin and end positions and a start button, with segmentated LED displays for the voltage positions.
I loved working with those editors. the sounds they made and the frame to frame directness.
So if I translate this to a timeline editor, it could be a multisegmented envelope (such as the envelopes used in the DX7) with a way to jog through the points set in the editor.
Bezier curves would be great , so maybe OLED displays would be better to see these variables.
If you would have several channels of this, then I could see so much cool possibilities!
#41 — giantmecha · 2022-03-28
Do you have a Doepfer a-151? I read somewhere that this guy can’t really handle video rate signals, ie that if softens the video that passes though it. Thanks for any experience you can share!
#42 — Agawell · 2022-03-28
@giantmecha - it’s not that bad - I put an example on instagram - check out the #tag#lzxveurorack
#43 — Zackweb · 2022-07-12
AMAZING ideas in this thread WOW I’m just blown away !!! This topic is EXACTLY up my alley so I’m obligated to make a post and revive the thread haha !
To introduce myself: im a longtime lurker / lzx wishlister with a background in CG animation and touchdesigner.
Ive been trying to wrap my head around how to realize this sort of setup for the last four or five years. In my eyes this sort of storytelling is what this scene is missing out on when compared to other visual art disciplines. But this is also a result of being abstract art !!!
To TRY and answer the question though (from a more traditional animation and video mixing stand-point), I feel it comes down to:
-multiple stock video / png viewers with alpha
-blending modes
-deep branching trees of upstream / downstream keyers
-more ways to alter buffers ie: scrubbing, stretching time, seamless looping of some sort
-oscillographics / rutt etra / scanimate style animation to move in faux 3d space
-preferably an “FX send” per layer to effect pieces of the composition
-master controller for positioning “precomped” shape or source animations around the final frame
-preset saving and precomping to some degree
But as I’ve read, MANY people have pitched these ideas with better context to an lzx system ! These ideas are just to my personal taste.
I guess in theory this setup would consist of a TON of keyers and media players, layers of memory palace / fairlight cvi style processors, scan processors and oscillographic sources, and a master composition control of some sort to position the pieces in a frame for a final shot / scene switch / save presets etc.
I could imagine an artist designing an improvised scene like this live and possibly even having an improv actor acting alongside them captured with a camera and composited into the shot (or even tracked into mocap live and control a 3d character to composite in the shot)…hmmmm this is definitely a touchdesigner project…
I also felt the need to have this dream setup revolve around stock images / footage because this last year I REALLY started appreciating the possibilities in editing and mangling them into what u might need in the moment ! That being said, this setup is really more of a video collage editor than an analog video synth lol but thats just what I would personally want
#44 — killumina · 2022-07-14
creatorlars wrote:
But there are some missing pieces we haven’t touched yet – mostly related to motion, nested animations, and sequencing
Musing on this point-
Scale of industry-wise I can use Elektron MIDI p-locks / CV out, a Cirklon, or any other existing eurorack solutions like the Flux temporal sequencer.
Are there any interesting sequencing options with a video or visual design focus?
I’d be curious if there are any elements unique to video synthesis (and signal sync) that would necessitate a different mindset, especially if these sequencers would be less stuck in a quantized time domain or on “the grid”.
#45 — creatorlars · 2022-07-15
killumina wrote:
I’d be curious if there are any elements unique to video synthesis (and signal sync) that would necessitate a different mindset, especially if these “sequencers” would be less grid-y and in a quantized time domain.
That really nails the open question for me, too. I think we should experiment with every possible workflow to gain more data about that. On the visual side of things, we can look at differences between realtime music production / sequencing vs edit decision lists & animator’s worksheets, as a thought exercise. What’s missing in one but present in the other? What elements do they share? Can the ideas be combined in ways that make new kinds of sense?
#46 — Gavin · 2022-10-31
Some fragmented thoughts:
Animation of multiple elements in a scene: I have found the BSO bajaoscillator key here - multiple offset outputs means multiple things can move in different, but related ways and give the impression of a whole. Even when the scene is abstract it makes some visual sense that connects things as a scene, often suggesting ‘false’ depth (ie-it often looks a bit like layers of parallax scrolling, even when its only hue or brightness changes).
Multiple scenes: I think there’s a big conceptual jump here too. At what historical point does video art jump to using scenes (it’s the 90s, with the digital technology), and that means the viewer adjusting their expectations - its not a narrative, scene by scene, form. A bit like expecting a painting to move, or a sculpture to make sound. Maybe its not necessarily desirable. I do edit my patches into a series of scenes in premiere pro, but I treat that as a separate process and separate creative choice.
Games: The closest predecessor here is the Jeff Minter visualisers - colourspace or the console visualisers and ‘video-art games’ he made. These were inspired by video art, but intended to be playable like a game. An intermediary step might be porting the basic ‘playable’ algorithms in some of these old 8-bit visualisers to Diver or MP as animated ramps? I guess it starts to be a bit like a very primitive GLSL shader with a joystick input at that point, too.
#47 — dryodryo · 2022-11-06
Here to reiterate my desire for a no-frills patchable DVE.
A GLSL shader module that could handle 1080i60 would be great.
Processing requirements are high for both of those, so price points will be commensurate.
Not interested in hardware timeline sequencers, IMHO that’s much better done in software.
Stochastic / chaotic / dynamic pattern generation is the low-hanging fruit because relatively simple inputs & processing gives complex results. The hardware / software design challenge there is constraining the chaos into usable results. I can see this involving multi-phase oscillators, frequency shifters, mixers/compositors, and feedback.