jiti-meet/react/features
Gabriel Borlea 61684b1071
feat(facial-expressions): add the facial expression feature and display them in speakerstats (#10006)
* Initial implementation; Happy flow

* Maybe revert this

* Functional prototype

* feat(facial-expressions): get stream when changing background effect and use presenter effect with camera

* add(facial-expressions): array that stores the expressions durin the meeting

* refactor(facial-expressions): capture imagebitmap from stream with imagecapture api

* add(speaker-stats): expression label

* fix(facial-expression): expression store

* revert: expression leabel on speaker stats

* add(facial-expressions): broadcast of expression when it changes

* feat: facial expression handling on prosody

* fix(facial-expressions): get the right track when opening and closing camera

* add(speaker-stats): facial expression column

* fix(facial-expressions): allow to start facial recognition only after joining conference

* fix(mod_speakerstats_component): storing last emotion in speaker stats component and sending it

* chore(facial-expressions): change detection from 2000ms to 1000ms

* add(facial-expressions): send expression to server when there is only one participant

* feat(facial-expressions): store expresions as a timeline

* feat(mod_speakerstats_component): store facial expresions as a timeline

* fix(facial-expressions): stop facial recognition only when muting video track

* fix(facial-expressions): presenter mode get right track to detect face

* add: polyfils for image capture for firefox and safari

* refactor(facial-expressions): store expressions by counting them in a map

* chore(facial-expressions): remove manually assigning the backend for tenserflowjs

* feat(facial-expressions): move face-api from main thread to web worker

* fix(facial-expressions): make feature work on firefox and safari

* feat(facial-expressions): camera time tracker

* feat(facial-expressions): camera time tracker in prosody

* add(facial-expressions): expressions time as TimeElapsed object in speaker stats

* fix(facial-expresions): lower the frequency of detection when tf uses cpu backend

* add(facial-expressions): duration to the expression and send it with durantion when it is done

* fix(facial-expressions): prosody speaker stats covert fro string to number and bool values set by xmpp

* refactor(facial-expressions): change expressions labels from text to emoji

* refactor(facial-expressions): remove camera time tracker

* add(facial-expressions): detection time interval

* chore(facial-expressions): add docs and minor refactor of the code

* refactor(facial-expressions): put timeout in worker and remove set interval in main thread

* feat(facial-expressions): disable feature in the config

* add(facial-expressions): tooltips of labels in speaker stats

* refactor(facial-expressions): send facial expressions function and remove some unused functions and console logs

* refactor(facial-expressions): rename action type when a change is done to the track by the virtual backgrounds to be used in facial expressions middleware

* chore(facial-expressions): order imports and format some code

* fix(facial-expressions): rebase issues with newer master

* fix(facial-expressions): package-lock.json

* fix(facial-expression): add commented default value of disableFacialRecognition flag and short description

* fix(facial-expressions): change disableFacialRecognition to enableFacialRecognition flag in config

* fix: resources load-test package-lock.json

* fix(facial-expressions): set and get facial expressions only if facial recognition enabled

* add: facial recognition resources folder in .eslintignore

* chore: package-lock update

* fix: package-lock.json

* fix(facial-expressions): gpu memory leak in the web worker

* fix(facial-expressions): set cpu time interval for detection to 6000ms

* chore(speaker-stats): fix indentation

* chore(facial-expressions): remove empty lines between comments and type declarations

* fix(facial-expressions): remove camera timetracker

* fix(facial-expressions): remove facialRecognitionAllowed flag

* fix(facial-expressions): remove sending interval time to worker

* refactor(facial-expression): middleware

* fix(facial-expression): end tensor scope after setting backend

* fix(facial-expressions): sending info back to worker only on facial expression message

* fix: lint errors

* refactor(facial-expressions): bundle web worker using webpack

* fix: deploy-facial-expressions command in makefile

* chore: fix load test package-lock.json and package.json

* chore: sync package-lock.json

Co-authored-by: Mihai-Andrei Uscat <mihai.uscat@8x8.com>
2021-11-17 16:33:03 +02:00
..
always-on-top chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
analytics fix(analytics) Fix analytics event names (#10332) 2021-11-09 12:20:40 +02:00
app feat(facial-expressions): add the facial expression feature and display them in speakerstats (#10006) 2021-11-17 16:33:03 +02:00
audio-level-indicator chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
authentication chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
av-moderation fix(av-moderation) Fix ask to unmute after allow video 2021-10-26 12:00:04 -07:00
base feat(welcome) added WelcomePage React Nav bottom tabs 2021-11-17 15:42:06 +02:00
calendar-sync feat(welcome) added WelcomePage React Nav bottom tabs 2021-11-17 15:42:06 +02:00
chat feat(chat/native) removed keyboard dismiss from JitsiKeybAvView 2021-11-16 17:50:09 +02:00
chrome-extension-banner chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
conference feat(welcome) added WelcomePage React Nav bottom tabs 2021-11-17 15:42:06 +02:00
connection-indicator chore(lint) tame the linter 2021-11-09 11:05:09 +01:00
connection-stats chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
deep-linking feat(billing-counter) Removed iframe billing-counter callbacks (#9537) 2021-07-20 11:58:42 +03:00
desktop-picker chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
device-selection chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
display-name chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
dropbox fix(rn) disable pip while authorising dropbox 2021-09-14 12:37:08 +03:00
dynamic-branding fix(dynamic-branding) Fix bogus mUI dependency on mobile (#10375) 2021-11-16 22:16:18 +01:00
e2ee feat(e2ee) add externally managed key mode 2021-11-16 12:12:10 +01:00
embed-meeting chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
etherpad feat(rn,welcome) React Navigation drawer 2021-11-11 15:32:56 +01:00
external-api feat(external-api) add data-channel-open event 2021-09-30 12:31:41 +02:00
facial-recognition feat(facial-expressions): add the facial expression feature and display them in speakerstats (#10006) 2021-11-17 16:33:03 +02:00
feedback fix(dynamic-branding) Fix bogus mUI dependency on mobile (#10375) 2021-11-16 22:16:18 +01:00
filmstrip chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
follow-me chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
google-api chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
invite fix(dynamic-branding) Fix bogus mUI dependency on mobile (#10375) 2021-11-16 22:16:18 +01:00
jaas chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
keyboard-shortcuts chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
large-video chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
lobby feat(rn,welcome) React Navigation drawer 2021-11-11 15:32:56 +01:00
local-recording chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
mobile chore(deps) adapt to logger package rename 2021-11-10 14:40:17 +01:00
no-audio-signal chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
noise-detection chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
notifications chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
old-client-notification chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
overlay bugfix(rn) sends CONFERENCE_TERMINATED to native after dismissing the ReloadOverlay 2021-11-10 13:34:16 +02:00
participants-pane feat(rn,welcome) React Navigation drawer 2021-11-11 15:32:56 +01:00
polls feat(rn,welcome) React Navigation drawer 2021-11-11 15:32:56 +01:00
power-monitor redux: refactor loading of middlewares and reducers 2020-06-16 11:24:15 +02:00
prejoin fix(Prejoin): Make prejoin name noneditable only when taken from jwt 2021-11-15 15:40:18 +02:00
presence-status chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
reactions fix(dynamic-branding) Fix bogus mUI dependency on mobile (#10375) 2021-11-16 22:16:18 +01:00
recent-list feat(welcome) added WelcomePage React Nav bottom tabs 2021-11-17 15:42:06 +02:00
recording fix(recorder): "already started" notification 2021-11-09 13:25:19 -06:00
rejoin redux: refactor loading of middlewares and reducers 2020-06-16 11:24:15 +02:00
remote-control chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
room-lock feat(rn,security) add security dialog 2021-04-09 14:30:25 +02:00
rtcstats chore(deps) update rtcstats, change package name 2021-11-11 08:04:28 +01:00
screen-share chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
screenshot-capture fix(dynamic-branding) Fix bogus mUI dependency on mobile (#10375) 2021-11-16 22:16:18 +01:00
security chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
settings feat(rn,welcome) React Navigation drawer 2021-11-11 15:32:56 +01:00
share-room redux: refactor loading of middlewares and reducers 2020-06-16 11:24:15 +02:00
shared-video chore(deps) adapt to logger package rename 2021-11-10 14:40:17 +01:00
speaker-stats feat(facial-expressions): add the facial expression feature and display them in speakerstats (#10006) 2021-11-17 16:33:03 +02:00
stream-effects chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
subtitles chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
talk-while-muted feat(av-moderation) Updated Advanced moderation (#9875) 2021-09-10 14:05:16 +03:00
toolbox fix(toolbox): enable fullscreen button on Android mobile browsers 2021-11-17 12:55:10 +01:00
transcribing chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
unsupported-browser fix(jaas) do not show overriden unsupported browser page for jaas users (#9962) 2021-09-20 11:38:46 +03:00
video-layout chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
video-menu fix(av-moderation) Don't stop local screensharing on mute all 2021-11-16 12:47:39 -06:00
video-quality chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
videosipgw chore(lint) tame the (uppdated) linter 2021-11-09 09:43:55 +01:00
virtual-background feat(facial-expressions): add the facial expression feature and display them in speakerstats (#10006) 2021-11-17 16:33:03 +02:00
welcome feat(welcome) added WelcomePage React Nav bottom tabs 2021-11-17 15:42:06 +02:00