Speech separation involves extracting an individual speaker's voice from a multi-speaker audio signal.
The increasing complexity of real-world environments, where multiple speakers might converse simultaneously,
underscores the importance of effective speech separation techniques.
This work presents a single-microphone speaker separation network with TF attention aiming at noisy and reverberant
environments.
We dub this new architecture as Sep-TFAnet. In addition, we present a variant of the separation network,
dubbed Sep-TFAnetVAD VAD, which incorporates a VAD into the separation network.
The separation module is based on a TCN backbone inspired by the Conv-Tasnet architecture with multiple modifications.
Rather than a learned encoder and decoder, we use STFT and iSTFT for the analysis and synthesis, respectively.
Our system is specially developed for human-robotic interactions and should support online mode.
The separation capabilities of Sep-TFAnetVAD and Sep-TFAnet were evaluated and extensively
analyzed under several acoustic conditions, demonstrating their advantages over competing methods.
Since separation networks trained on simulated data tend to perform poorly on real recordings,
we also demonstrate the ability of the proposed scheme to better generalize to realistic examples recorded in our acoustic lab by a humanoid robot.
Samples from our robot dataset
Legend:
Red Words - Wrong words
Asterisk (*) - Missing words
Mix
Clean
Sep-TFAnetVAD Output
Conv-Tasnet Output
overlap ratio = 36%:
1st as a paris stt broker later as a celebrated author and yachtsman he went on frequent voyages
construction and were installed by the construction company.
that had been organized by edison to build and equipped central stations.
speaker 1:
1st as a paris stock broker later as a celebrated author and yachtsman he went on frequent voyages to britain america the mediterranean
speaker 2:
the street conductors were of the overhead pole line construction and were installed by the construction company
that had been organized by edison to build and equipped central stations
speaker 1:
1st as a paris stock broker later as A celebrated author and yachtsman he went on
frequent voyages to britain america the mediterranean
speaker 2:
the street conductors were ** the overhead **** line construction and were installed by the construction company that had been organized by edison to b
uild and equipped central stations
speaker 1:
1st as a paris stock broker later as a celebrated author and yachtsman he went on
frequent voyages to britain america the mediterranean ITS ALL ABOUT IT
speaker 2:
the AS A HE WENT ON street conductors **** OR
the OVERHANG THING line construction *** were ins
talled by the construction company that had been organized by edison to build and equipped central stations
overlap ratio = 45%:
it he has no traditions to bind him or guide him and his impulse to break away from the occupation Himself
but he gives us different sorts of Work according as he fits us for it and calls us to it
speaker 1:
he has no traditions to bind him or guide him and his impulse is to
break away from the occupation his father has followed and make a new way for himself.
speaker 2:
we can all be servants of god wherever our lot is cast but he gives us
different sorts of work according as he fits us for it and calls us to it.
speaker 1:
he has no traditions to bind him or guide him and his impulse IS to break away from the occupation his father has followed and make
a new way for himself
speaker 2:
we can ONLY be servants of god wherever our LIFE is TEST but he gives us different
sorts of work RECORDING as
he fits us for it and calls us to it
speaker 1:
he has no traditions to bind him or guide him and his impulse to break away from the occupation his father has followed and ****
* *** *** for himself
speaker 2:
HE HAS NO AND THIS IS we can all THE servants of god wherever *** *** *
* **** but he gives us different sorts of work RECORDING as he fits us for it and calls us to it
overlap ratio = 32%:
It is a gleaner Bringing down her one sheaf of corn to an Old Water Mill itself Mossy in rent Scarcely able to get its
stones to and think i showed him by my mode of address that i did not bear any grudge as to my individual self
speaker 1:
it is a gleaner bringing down her one sheaf of corn to an old watermill itself mossy and rent scarcely able to get its stones to turn
speaker 2:
as i spoke i made him a gracious bow and i think i showed him by my mode of address that i did not bear any grudge as to my individual self.
speaker 1:
it is a gleaner bringing down her one sheaf of corn to an old water mill itself mossy
in rent scarcely able to get its stones to TIME
ANY GRUDGE
speaker 2:
RIGHT as YOU spoke I made him a gracious bow and I think i showed him by my mode of address that i did not bea
r any PLEDGE as to my individual self
speaker 1:
it is a gleaner bringing down her one sheaf of corn to an old water mill
IT OFF mossy AND rent scarcely able to get
its STONE I THINK I SHOWED HIM BY MY MOTIVE AS to MY
speaker 2:
ITS SO I made him a gracious bow and ***** * ****** *** ** ** ****
of address that i did not bear any grudge as to my individual self
Samples from LibriCSS dataset
Legend:
Red Words - Wrong words
Asterisk (*) - Missing words
Mix
Clean
Sep-TFAnet Output
we sometimes have images that are by no means peculiarly vague which yet we do not
trust e g under the influence of fatigue we may see a friend's face vividly and clearly horribly
distorted cried the tree rejoicing.
speaker 1:
we sometimes have images that are by no means peculiarly vague which yet we do not
trust for example under the influence of fatigue we may see a friend's face vividly and clearly but horribly distorted
speaker 2:
i would fain know if i am destined for so glorious a career cried the tree rejoicing
speaker 1:
we sometimes have images that *** by no means peculiarly vague which yet we do not trust
for example under the influence of fatigue we may see a friend's face vividly and clearly but horribly distorted
speaker 2:
i would **** THINK if i am destined for SOME glorious a career cried the tree rejoicing
doubts now arose and some discussion followed tone so noble in its spirit
speaker 1:
doubts now arose and some discussion followed whether or not it was desirable for ben zoof to accompany his master.
speaker 2:
i had not ventured to hope for such a reply so considerate in its tone so noble in its spirit.
speaker 1:
doubts now arose and some discussion followed whether or not it was desirable for BE ZOO to accompany his master
speaker 2:
i HAVE not ventured to hope for such a reply so considerate in its tone so noble in its spirit
cried alice again for this time the mouse was bristling
all over and she felt trap in the far corner of the hut and stuteley had
tripped over the edge of the reversed flap mouth of this pit
speaker 1:
they were upon the verge of an open trap in the far corner
of the hut and stuteley had tripped over the edge of the reversed flap mouth of this pit
speaker 2:
cried alice again for this time the mouse was bristling all over and she felt certain it must be really offended.
speaker 1:
OVER they were upon the verge of an open trap in the
far corner of the hut and stuteley had tripped over the edge of the reversed flap mouth of this pit
speaker 2:
cried alice again for this time the mouse was bristling all
over and she felt certain THAT it must be really OFFENSIVE
