Audio-based performance evaluation of squash players
CREDIT:journals.plos...read complete Article HereFigures
Abstract
In competitive sports it is often very hard to quantify the performance. A player to score or overtake may depend on only millesimal of seconds or millimeters. In racquet sports like tennis, table tennis and squash many events will occur in a short time duration, whose recording and analysis can help reveal the differences in performance. In this paper we show that it is possible to architect a framework that utilizes the characteristic sound patterns to precisely classify the types of and localize the positions of these events. From these basic information the shot types and the ball speed along the trajectories can be estimated. Comparing these estimates with the optimal speed and target the precision of the shot can be defined. The detailed shot statistics and precision information significantly enriches and improves data available today. Feeding them back to the players and the coaches facilitates to describe playing performance objectively and to improve strategy skills. The framework is implemented, its hardware and software components are installed and tested in a squash court.
IntroductionAt
present in competitive sports there are a lot of talented sportsmen and
the differences between individual performance are often very small to
spot. It catalyses a race condition to be present already in the
practising period, thus more and more coaches and players seek finding
different means and aids to elaborate and make the preparation for the
tournaments always more effective. There are a lot of new technological
achievements available in the market. Small electronic devices are
capable of measuring various metrics including those that are relevant
for the sports, like heart rate and blood temperature and pressure
registers, pedometers, speedometers and accelerometers to name a few.
Using such devices is more than necessary since the results in a
competition and then the final scores may depend on millesimal of
millimeters. Another reason why to use measurement devices yielding
objective performance metrics is because when sportsmen are overloaded
in a performance, with adrenalin in their vein, it is hard if possible
for them to spot and fix their failures. In certain types of sports a
continuous or prompt feedback is definitely helpful, squash is one of
them.
Squash is a very rapid ball and racquet game with typically 40-60 hit events per minute. Depending on the various surfaces the ball interacts during its flight defines the different shot classes. Some shot classes are very rare due to being tricky to deliver or may occur only in circumstances where the rally may seem already lost. So knowing the detailed statistics of various hits and shot patterns talks about the quality of the sportsmen and are very important information for both the coaches and the squash players. However, these data and their statistical analysis are not available at present because of the paste of squash. Given its fast speed the human processing of events enables the score registration in real-time only, but the recording of shot types and the detailed sequences of the shots are rendered definitely impossible. One possible solution might be to analyse videos of the matches using image processing as it has been shown to work for the tennis [1]. Though for the squash it turns out that this approach remains difficult even with the use of high speed and high resolution cameras, due to the small size of the ball and the view provided by the cameras. Traditionally cameras are placed behind the court, therefore the players will most often cover the sight of the ball during the match making the reconstruction of ball trajectories an inauspicious problem. To provide reliable statistics by this approach will require human processing and validation so in the end a thorough analysis of the tournament will cost many times of the duration of this sport events in man-hours.
In this study we introduce a framework to unhide these information based on the analysis of acoustic data. Playing squash produces characteristic sound patterns. The sound footprint of each rally is a projection of all the details about the strength and the position of the ball hitting various surfaces in the court. Naturally, this pattern, which maintains the natural order of the events, is contaminated by some additional noise. Recording the sound in more directions allows for inverting the problem and for giving statistical statements about where and what type of an event took place in the play. We are focusing on events generated by the ball hits, which serves as a basis for further analysis and the reconstruction of shot patterns or the ball trajectories. Note, the framework to be detailed can be applied to various other types of ball games.
One test that was developed by squash coach Geoffry Hunt is the “Hunt Squash Accuracy Test” (HSAT) [3], that is a reliable method used by coaches to assess shot hitting accuracy. The test is composed of 375 shots across 13 different types of squash strokes and it is evaluated based on a total score expressed as the number of successful shots.
Recent technological advances have facilitated the development of sport analytical software such as Dartfish video based motion analysis system [4, 5]. However, these systems still require a considerable amount of professional assistance.
To the best of our knowledge there is no previous research investigating the applicability of sound analysis techniques for squash performance analysis, therefore it is not possible to directly compare our system to existing solutions. In other application environments a wide literature can be found on real-time sound source localization that is the most closely related topic to our work. The emerging application of camera pointing in video conferencing environment motivated many research papers on the field of visual speaker localization [6–10]. A linear-correction least-squares estimation procedure is proposed in [6–8]. The simulation results in [7] show that the bias level of this technique is around 30 cm. In the work of Tobias Gehrig et al. [10] a method is presented to speaker tracking using audio-visual features, namely time delay of arrival estimation on microphone array signals and face detection on multiple camera images. The sound source localization is based on a maximum likelihood approach. In the experimental results the authors measure 57.2 cm root mean square error for the audio-only solution and 49.9 cm for the audio-video approach. One conceptually simple solution for source localization is beamforming [11], where the source location is estimated by calculating the steered output power of a beamformer over a set of candidate locations. Although this concept has advantages in speech localization and enhancement, it is computationally expensive and its resolution is too low for our purposes.
Squash is a very rapid ball and racquet game with typically 40-60 hit events per minute. Depending on the various surfaces the ball interacts during its flight defines the different shot classes. Some shot classes are very rare due to being tricky to deliver or may occur only in circumstances where the rally may seem already lost. So knowing the detailed statistics of various hits and shot patterns talks about the quality of the sportsmen and are very important information for both the coaches and the squash players. However, these data and their statistical analysis are not available at present because of the paste of squash. Given its fast speed the human processing of events enables the score registration in real-time only, but the recording of shot types and the detailed sequences of the shots are rendered definitely impossible. One possible solution might be to analyse videos of the matches using image processing as it has been shown to work for the tennis [1]. Though for the squash it turns out that this approach remains difficult even with the use of high speed and high resolution cameras, due to the small size of the ball and the view provided by the cameras. Traditionally cameras are placed behind the court, therefore the players will most often cover the sight of the ball during the match making the reconstruction of ball trajectories an inauspicious problem. To provide reliable statistics by this approach will require human processing and validation so in the end a thorough analysis of the tournament will cost many times of the duration of this sport events in man-hours.
In this study we introduce a framework to unhide these information based on the analysis of acoustic data. Playing squash produces characteristic sound patterns. The sound footprint of each rally is a projection of all the details about the strength and the position of the ball hitting various surfaces in the court. Naturally, this pattern, which maintains the natural order of the events, is contaminated by some additional noise. Recording the sound in more directions allows for inverting the problem and for giving statistical statements about where and what type of an event took place in the play. We are focusing on events generated by the ball hits, which serves as a basis for further analysis and the reconstruction of shot patterns or the ball trajectories. Note, the framework to be detailed can be applied to various other types of ball games.
Related work
Squash and soccer were the first sports to be analysed by ways of analysis systems. Formal scientific support for squash emerged at the late 1960s. The current applications of performance analysis techniques in squash are deeply investigated in the book of Stafford et al. [2].One test that was developed by squash coach Geoffry Hunt is the “Hunt Squash Accuracy Test” (HSAT) [3], that is a reliable method used by coaches to assess shot hitting accuracy. The test is composed of 375 shots across 13 different types of squash strokes and it is evaluated based on a total score expressed as the number of successful shots.
Recent technological advances have facilitated the development of sport analytical software such as Dartfish video based motion analysis system [4, 5]. However, these systems still require a considerable amount of professional assistance.
To the best of our knowledge there is no previous research investigating the applicability of sound analysis techniques for squash performance analysis, therefore it is not possible to directly compare our system to existing solutions. In other application environments a wide literature can be found on real-time sound source localization that is the most closely related topic to our work. The emerging application of camera pointing in video conferencing environment motivated many research papers on the field of visual speaker localization [6–10]. A linear-correction least-squares estimation procedure is proposed in [6–8]. The simulation results in [7] show that the bias level of this technique is around 30 cm. In the work of Tobias Gehrig et al. [10] a method is presented to speaker tracking using audio-visual features, namely time delay of arrival estimation on microphone array signals and face detection on multiple camera images. The sound source localization is based on a maximum likelihood approach. In the experimental results the authors measure 57.2 cm root mean square error for the audio-only solution and 49.9 cm for the audio-video approach. One conceptually simple solution for source localization is beamforming [11], where the source location is estimated by calculating the steered output power of a beamformer over a set of candidate locations. Although this concept has advantages in speech localization and enhancement, it is computationally expensive and its resolution is too low for our purposes.
The measurement equipment
This study is based on the analysis of sound waves generated during the squash play. Among many other, squash is a game where various different sources of sound are present, including the players themselves (their sighing or their shoes squeaking on the floor), the ball hitting surfaces (like the walls, the floor or the racquet) and also external sources (including the ovation of the spectators or sound generated in an adjacent court). Here we focus on audio events related to the ball.When planning the experiments the following constraints had to be investigated and satisfied. The framework should be fast in signal processing point of view, because the target information can be most valuable when in a competitive situation it helps fine tune tactical decisions made by the coach and/or the player. The cost of the equipment should be kept low and the installation of the sensors requires a careful design to prevent them from disrupting the play. As the spatial localization of the ball is one of the fundamental goals a lower bound to the sampling rate is enforced to remain able to differentiate between displaced sound sources.
In Fig 1 the hardware and software components are sketched. Hardware components include 6 audio sensors, three of which are omnidirectional microphones (Audio Technica ES945) sinking in the floor and the rest of them are cardioid microphones (Audio Technica PRO 45) hanging from the top. Amplification and sampling of the microphone signals are done by a single dedicated sound card (Presonus AudioBox 1818VSL) so that all channels in a sample frame are in synchrony. The highest sampling rate of the sound card is used (96 kHz), so by each new sample the front of a sound wave travels approximately 6 mm.
Fig 1. A schematic view of the components.
To process audio events in the squash court a three component architecture was designed.The ball impact detection
The localization and the classification of ball hits both require the precise identification of the beginning of the corresponding events in the audio streams. The detection of ball impact events is carried out for each audio channels independently and in a parallel fashion, which speeds up the overall performance of the framework significantly. Different detection algorithms of various complexities were investigated two extreme cases are sketched here. The first model assumes that the background noise follows the normal distribution. An event is detected if new input samples deviate from the Gaussian distribution to a certain predefined threshold value. Next for each channels the mean and the variance estimates of a finite subset of the samples are continually updated according to the Welford’s algorithm [12].The second method is an extension of the windowed Gaussian surprise detection by Schauerte and Stiefelhagen [13]. The algorithm tackles the problem evaluating the relative entropy [14]. It is first applied in the frequency domain and if there is a detection then a finer scale search is carried out in the time domain. The power spectrum of w-sized chunks of windowed data samples is calculated. Between detection regime the series of the power spectra is modelled by a w-dimensional Gaussian. The a priori parameters of the distribution are calculated for n elements in the past, and the posteriori parameters are approximated including the new power spectrum. The Kullback Leibler divergence between the a priori and the posteriori distributions exceeds a predefined threshold when a new detection takes place
where primed parameters correspond to the posteriori distribution. The time resolution at this stage is w
and to increase precision a new search is carried out in the time
domain evaluating the Kullback Leibler divergence for 1-d data. In order
to bootstrap a priori distribution parameters n samples from the former windows are used.The localization of sound events
In this section we lay down a probabilistic model to determine the time and location of an audio event. For a unique event we denote these unknowns t and rev respectively. The location vector rev is a 3 dimensional array of Descartes coordinates (x, y, z), however, the calculation presented here also applies for lower dimension setups. The inputs required to find the audio event are the locations of the N + 1 detectors
and the timestamps τi when these synchronized detectors sense the event (0 ≤ i ≤ N).The probability that microphone i detects an event at (r, t) is
where c is the speed of sound, ti = τi − t is the propagation delay and 
is the distance between the sound source and the microphone. The uncertainty σi depends on the characteristics of the microphone, which we will consider constant in the first approximation.By introducing relative delays
the joint probability of relative delays detected is
The formula can be rearranged
where 
is a quadratic function and in the expression for p the Gaussian integral follows
The first order derivative f′ vanishes in 
, where 
is introduced for convenience.After substitution of
we arrive at
This formula can be interpreted as a variance formula, which can be rewritten
A good approximation of the audio event maximizes the likelihood p, which at the same time minimizes
, thus we seek the solution of 
equations.In practice f behaves well and its minimum can be found by gradient descent method. Fig 2 shows a situation, where the ball hit the front wall and 6 microphones detect this event error free. To show the functions behaviour f is evaluated in the floor, in the front wall and in the right side wall. Finding the minimum of f takes less than ten gradient steps.
Fig 2. The visualization of the likelihood function.
The ball hit the front wall. 
can be evaluated in space given the positions of the sensors (marked by
white disks) to find its minimum, which indicates where the event took
place. (0.5 m from the right corner and 3 m above the floor, marked by a
blue disk).
Fig 3. The cumulative distribution of the localization error.
For a noiseless case most often localization will have an error comparable to the size of the ball. With a bad detector (σ = 50 samples) still the localization is exact in the order of 10 cm.Classification
It is the task of the classification module to distinguish between the different sound events according to their origin. Sound events are classified based on the type of the surface that suffered from the impact of the ball. This surface can be the wall, the racquet, the floor or the glass. When the sound does not fit any of these classes, like the squeaking shoes, then it is classified as a false event. The classification enhances the overall performance of the system by two means. First, skipping to localize the false events speeds up the processing. And second, in doubtful situations when the calculated location of the event falls near to multiple possible surfaces, by knowing the type of the surface that suffered from the impact can reinforce the localization. For example a sound event localized a few centimetres above the floor could be generated by a racquet hit close to the floor or by the floor itself.Classification utilizes feed-forward neural networks that had been trained with backpropagation [15–18]. The training sets are composed of vectors belonging to 5461 audio events, which have been manually labelled. Based on these audio events two types of input were constructed for teaching.
In the first case temporal data is used directly. A vector element of the training set T1 is the sequence of the samples around the detections for each channels.
where the channel index is dropped and d is a unique detection and w sets the length of the vector. Given the sampling rate 96 kHz and setting w = 300 the neural network is taught by 6.25 millisecond long data.The second feature set T2 is built up of the power spectra.
where 
denotes the discrete Fourier transform.A single neural network model where all event classes are handled together performed poorly in our case. Therefore, separate discriminative neural network models were built for all four classes (racquet, wall, floor and glass impact) and for both of the training sets. It has also been investigated if any of the input channels introduce discrepancy. In order to discover this effect models were built and trained for each unique channels and another one handling the six channels together. Note, that not all possible combinations of the models were trained due to the fact that some channels poorly detected certain events, for example microphones near the front wall detected glass events very rarely.
In the training sets the class of interest was always under-represented. To balance the classifier the SMOTE [19] algorithm was used, which is a synthetic minority over-sampling technique. A new element is synthesized as follows. The difference between a feature vector from the positive class and one of its k nearest neighbours is computed. The difference is blown by a random number between 0 and 1, to be added to the original feature vector. This technique forces the minority class to become more general, and as a result, the class of interest becomes equally represented like the majority set in the training data.
Different network configurations were realized to find that for the direct temporal input a 20 hidden layer network (with 10 neurons in each layer) performed the best, while for the spectra input a 10 hidden layer (each layer with 10 neurons) is the best choice.
Analysis
In this section the performance of each modules of the framework and the datasets are presented.Datasets
In order to analyse the components of the framework implementing the proposed methods two audio and video record sets were used. Datasets are available at https://figshare.com. Audio 1 was recorded on the 18th of May 2016 when a squash player was asked to target specific areas of the wall. This measurement was necessary to increase the cardinality of the different hits significantly in the training datasets T1 and T2, and it was also manually processed to be able to validate the operations of the detector and the localization components. Audio 2 resembles data in a real situation as it contains a seven minutes squash match recorded on the 8th of March 2016. Table 1 summarizes the details of these audio recordings.Training the neural network models require properly labelled datasets. After applying the ball impact detection algorithm to the audio records the timestamps of the detected events were manually categorized as front wall event, racquet event, floor event or glass event. In the categorization procedure video files helped in doubtful situations. Every sudden sound effect that was detected by the algorithm but does not belong to these relevant classes was labeled as false event. In Audio 1 prescribed audio events were generated and recorded and it does not contain any false events. In contrast, Audio 2 was captured in a real situation and it presents several false events by nature.
Detection results
The performance of the detector is analysed by comparing the timestamp reported by the detector ddetector and the human readings dhuman. For Audio 1 in Fig 4 the cumulative probability distribution of the time difference is shown for each channel and in Table 2 the average error and its variance are shown grouped by the event types present in the dataset. One can observe that the detectors in channels ch4 and ch5 perform poorly for front wall and racquet events. When estimating the position discarding one of or both of these channels will enhance the precision of the localization. However, for floor events, these two channels performed the best along with channel ch2. For glass events the smallest deviations were measured on channels ch2, ch3 and ch5.
Fig 4. The error of the detector.
The detection error is defined as the difference between the timestamps generated by the module and read by a human.
Table 2. The class and channelwise error of the detector.
The false discovery and the false negative rate of the detector were examined on Audio 2. False positives are counted if detector signals for a false event, and false negatives are the missing detections. The results are summarised in Table 4.
Classification results
Approaching the problem at first and to use as much information as possible to teach the neural networks a large training set was constructed of the union of the detections of all the six channels. However, this technique gave poorer results than treating all the channels separately. The different settings of the microphones and the distinct acoustic properties of the squash court at the microphone positions are found to be the reasons of that phenomenon.Eight-fold cross-validation [20] was used on the datasets to evaluate the performance of the classifiers. Three measures are investigated closer: the accuracy, the precision and the recall. Accuracy (in Fig 5) is the ratio of correct classifications and the total number of cases examined (
). Precision (in Fig 6) is the fraction constrained to the relevant cases (
). Recall (in Fig 7) is the fraction of relevant instances that are retrieved (
).
Fig 5. The classifiers’ accuracy.
The classwise accuracy of each channel is presented in T1 (blue) and T2
(red) input sets. Front wall classification gives high accuracy on all
channels in both sets. It is interesting to observe that floor
classification is more accurate in input T2. Racquet classification performs best on channel 2 in both sets.
Fig 6. The classifiers’ precision.
The classwise precision of each channel is presented in T1 (blue) and T2 (red) input sets. Front wall classification gives high precision in input T1.
The precision of floor classification is low. Racquet classification
still performs best on channel 2. The precision of glass classification
is only acceptable on channel 4.
Fig 7. The classifiers’ recall.
The classwise recall of each channel is presented in T1 (blue) and T2
(red) input sets. The performance of front wall classification is
reliable. The recall of racquet classification is high on channels 1 and
2 in both sets. However, the performance of floor and glass
classifications is low.
to which x belongs to is computed by the following formula:
where C is the set of class labels without the class of false events and fk(x), cutk and preck are the confidence, the cutoff value and the precision of the best classifier in class k respectively.
Table 5. The classwise performance of the best classifiers.
Fig 8. Labelled audio signal.
1.77 second long samples from channel ch1 in Audio 2. Detected timestamps and the event classes are marked.Localization results
Based on the geometry of the court, the placement of the microphones and using the localization technique detailed in this study for each set of detection timestamps the 3-d position of the source of the event can be estimated. In case not all source channels provide a detection of the event localization is still possible. Four or more corresponding timestamps will yield a 3-d estimate, whereas with three timestamps the localization of events constrained on a surface (e.g. planes like wall or floor) remains possible.In Fig 9 the located events present in dataset Audio 1 are shown. In this measurement scenario the player was asked to hit different target areas on the front wall. It was a rapid exercise, as the ball was shot back at once. Only a few times the ball hit the floor, most of the sound is composed of alternating racquet and front wall events. In Fig 10 the front wall events are shown. The target areas can be seen clearly, and also it is visible the spots scatter a little more on the left. The reason could be the player being right handed or the fact the target area was hit later during the experiment and the player showed tiredness.
Fig 9. The position of impacts.
Visualize the localized events embedded in 3-d.
Fig 10. Front wall impacts.
Gray squares embrace the eight target areas.
Fig 11. The front wall offsets.
The distribution of the offset δ from the front wall (σ(δ) ≈ 0.02 m).Discussion
Our results support that in sports, where the relevant sound patterns are distinguishable, careful signal processing allows the localisation of shots. The described system is optimized for handling events and as a consequences the real-time analysis of data is possible, which is important to give an instant feedback. The framework can be extended to provide higher level statistics of events such as the evolution of shots types. From the wide range of possible applications we highlight three use cases. Firstly, during a match the players can get to know their precision in short time and if is necessary they can change their strategy. Secondly, during practice coaches can track the development of the players hit accuracy. Or thirdly, certain exercises can be defined, which can be automatically and objectively evaluated, without the need for the coach be present during the exercise.In this study our framework was adopted to squash. In theory it can be extended to any other sports where the stereotypical events are associated with a specific sound pattern. In those applications, where typical patterns are present but the surroundings introduce significant amount of noise, a solution could be to use additional microphones with possibly special characteristcs to record the noise allowing to subtract its contribution from all other input signals. For example in tennis games played in the open field.
Ethics statement
In this study human participants were instructed to carry out specific squash excercises. Participants were informed beforehands about the fact that during the excercise the sound is to be recorded. During the excercises the sound emerging mainly from the ball impacts was recorded. The recording itself do not contain any sensitive information. Along with the raw recordings no additional information about the participant is saved or published. Participants do not object that these recordings are made public.Acknowledgments
The hardware components enabling this study are installed at Gold Center’s squash court. We thank them for this opportunity and squash coach Shakeel Khan for the fruitful discussions.Authors thank the Hungarian National Research, Development and Innovation Office under Grant No. 125280 and the grant of Ericsson Ltd.
No comments:
Post a Comment