♪ There may ♪ ♪ Be music or a star ♪ ♪ Into your mind ♪ ♪ All through this magic rules ♪ – Shimon has been reborn
as a singer-songwriter and now we collaborate
between humans and robots to make songs together. I can give him the theme, he can write the lyrics. He can suggest melodies for me. I can compose the melodies into a song. And then together with
a bunch of other humans, we can perform it while he’s singing. (Shimon singing) We are training the robot
using machine landing off of datasets, off lyrics. And it happens to be that these datasets are based on genre. And then whatever the
lyrics are from this genre, it learns the correlation and the features and the connections between
the lyrics and come up, based on prime words. I give it the prime words,
for example, space and time. It will go into the dataset
and generate new lyrics. Sometimes just by itself,
sometimes it fits to my melodies. All kind of different combinations. (Shimon singing) (bass guitar playing) – So we use deep lining
to generate lyrics, but it’s also combined
with semantic knowledge. So in a way, Shimon will
know the word “storm” is also related to “rain,” and then
so it’ll generate more based on “rain.” And then he’ll also generate
things related to rain, like maybe sunshine, call it to come back. So that was part of the
whole lyric system is making this interconnection,
ideas between all that. So I play bass with him. Actually, mainly play saxophone,
but a few of these songs I’ve been playing bass with him. It’s really cool. It’s really great just interacting. It really just feels like
another musician on stage. It’s like another person up there. It all, kind of, lots of
dialogue between me and him. Also, when you work with
the robot every day, it becomes like working with a person. And there’s ups and downs
in the relationship too. Some days it’s so annoying, that guy, and other days it’s like this is the best
thing ever to work with. It’s super great collaboration. – This new playing striking
system that can make Shimon to play much faster, about
25 to 30 hertz per second at the maximum and also a
much more expressive playing from a soft dynamic range
to a strong dynamic range. And also it can do visual gestures, send visual cues to the other
players as interactions. – Shimon has eyebrows now. Right now they have one degree of freedom, so they either both move
up or they both move down. And that has been a lot of
just manually placing it where we think it would be cute for him to give a little upwards movement. I started out with trying to
synchronize Shimon’s mouth to the lyrics, which was
a bit of a challenge. We have different poses of the mouth for different sounds of a word. So for example, for an “ah”
sound, his mouth will be wide. For a “T” it’ll be more close together, and we have different
ones assigned to different vowel and consonant sounds. And then we linearly
interpolate that over time, so that it kind of gets
a smooth transition. So it’s not just kind of
robotically moving up and down. – A big part of what we
trying to do is to bring it on the road, to take the research and bring it to the people. And we are going to release
a record and some videos, and we hope to start to schedule shows. So for many years, I felt as
people can come and enjoy it, partly because it’s kind of a gimmick. A robot that plays, that’s cool. And it looks at you, and
it bobs it’s head. So, definitely, I still
want all of this experience, hey it’s just like a human
and it’s interesting. But I think we reached a
level where I actually expect the audience to just
be blown by the music. I like the music. This is music that humans
like this have wouldn’t have written, and there’s
something unique about this song. And I want to go back and listen to it even if I don’t look at the robot. We may not be there yet, but if you ask me what is my dream, that would be it. (Shimon singing)

Author Since: Mar 11, 2019

Related Post