Feature Engineering and Dimensionality Reduction

Research 

After the pre-recorded data in the dataset was distinguished, in order to cater for other elements that are typical to affecting the discharge capacitance of supercapacitors, the group carried out research and literature review to determine the various other features that can be incorporated into the dataset. 

Some of these engineered features are illustrated in the table bellow:

1) Voltage decrease + IR drop	Calculated using the following formula:  When supercapacitors are rapidly charged and discharged it causes current to flow through its internal resistance and alternately cause a drop in the voltage which is known as IR drop [1]. This drop becomes more significant as the current increases. This can lead to a decrease in the voltage of the supercapacitor, which is known as voltage decrease. The voltage decrease can result in a reduction in the amount of energy that the supercapacitor can store and deliver
2) Fitting Coefficient	Computed using the formula given below:  In relation to supercapacitors, the fitting coefficient is a is a variable that is utilized to represent the behaviour of a supercapacitor. This is done by using this parameter in an equation that fits the resultant values acquired from a supercapacitor. The Ragone equation, for example, relates the energy density and power density of a supercapacitor to voltage and capacitance [2]. A fitting coefficient is used to adjust the equation and make it match the data when fitting experimental data to this equation. The fitting coefficient is usually determined through regression analysis, which involves fitting the equation to a set of experimental data points and determining the coefficient values that provide the best fit. The fitting coefficient is an important parameter for accurately modelling and optimizing the behaviour of a supercapacitor
3) Charge	Obtained using the formula given below: It is evident that a supercapacitors ability to store a certain amount of charge will decrease after long cycles of operation to the point where its inefficiency will outweigh its usability and force it to be deemed as a failed device [3]. Thus, this parameter is very important to be considered for representing the capacitance of the supercapacitor as its degradation is somewhat directly proportional to that of the capacitance.
4) Voltage Drop	Acquired using the equation:  The phenomenon of voltage drop in supercapacitors occurs when the voltage across the supercapacitor decreases over time as it discharges. This is due to the supercapacitor's internal resistance, which converts some of the electrical energy into heat and is lost rather than being delivered to the load (4).

After the above stated engineered features were added to the feature set, it was visualized to see its behaviour with changing capacitance values. 

Voltage Drop

Figure 1: Capacitance vs Voltage Drop

Charge

Figure 2: Capacitance vs Charge

The figure shown above illustrates the visualization of charge with respect to capacitance. 

Fitting Coefficient 

figure 3: Capacitance vs Fitting Coefficient

The graph sown above visualizes the relationship between the fitting coefficient and capacitance. 

Voltage decrease + IR drop

Figure 4: Capacitance vs Voltage decrease + IR drop

The data from 1 to 10000 cycles were divided into 4 parts which were denoted by the four different colours as shown in figure 4 to observe the changes in different stages of the supercapacitor’s life cycle.

After the engineered features were explored and logged into the feature set, it was very important to observed how each of these features related top each other in the sense how the change in one affected the change in the other.

Correlation Matrix 

Figure 5: Correlation Matrix in the form of a heat map

Figure 6: Correlation Plot of the different variables

The correlation plot above represented the correlation coefficients between each of the variables for one capacitor in batch 4. The leading diagonal subplot represented the plots as histograms with positive correlation. Note that the closer the coefficient values were to 1 the better relation they had. The subplots contained scatter plots for each variable and overlaid it with the plots of a different variable and compared it to a least-squares reference line. The slope of this resulting line was the coefficient value. If for example seen in cycle row and discharge capacity column, the coefficient value was -0.76 which mean they were negatively correlated while for charge capacity and discharge energy the coefficient yielded a positive value of 0.89 which meant they were positively correlated. However, if the coefficient value between any of the variables would have had been zero, it would indicate that their correlation was uncertain or could not be made out, as such, both would behave independently and not affect the nature of the other.

Determining the Intrinsic Dimensionality

Too much on our hands, isn't it......? 

If the regression models are trained using the entire feature set it might take a lot of proccesing time and power. In order to have faster and efficient training time we need to reduce the feature set to see which ones are the most important and hold majority of the variability over the overall data.

How can this be done? well simply using dimensionality reduction techniques such as PCA, CCA and t-SNE

PCA (Principal Component Analysis)

Figure 7: Pareto plot

Based on the pareto plot shown above in figure 7, it can be noticed that 97% of the variance (shown on the y-axis) is covered by the first three feature. This basically means that if the overall data was to be represented using the first 3 predictors, approximately 97% of it could be exhibited or resembled by these features, thus the intrinsic dimensionality can be observed to be 3. The strength of these three features along with the other 6 features can be better exhibited in the biplot given below in figure 8.

Biplot of the PCA transformed data 

Figure 8: Biplot of the PCA transformed data

The biplot above illustrates how much each of the features are related to each other in a 3D form. This can be used to select the most correlating variables and subject that to the final feature set that would later be used to train the regression models.

The loadings represent the variables' directions and strengths in each principal component, whereas the scores represent the positions of the observations in the principal component space while the length of the loading vector denotes the variable's importance in that principal component. Longer vectors represent variables that contribute more to that component. The angle between the loading vectors reveals information about variable relationships. Positive correlations are indicated by vectors that are closer together, while negative correlations are indicated by vectors that are farther apart.

The observations' positions in the biplot space show how they relate to one another based on the principal components. Close observations have similar patterns of variation, whereas far apart observations differ significantly. The important variables, clusters or patterns in the data, outliers or influential observations can be detected by analysing the biplot above

CCA (Curvilinear Component Analysis)

The CCA analysis was done to verify the PCA transformed data and see that the intrinsic dimensionality indeed can be 3.

In the CCA algorithm, the number of epochs was initialized as 100, alpha as 0.5 and lambda as 15. The lambda value can basically also be represented as 3*max(std(D)) in MATLAB. 

Input output mapping using dydx plot for the first 6 principal components

Figure 9: dydx Plot

As illustrated above in figure 9, the dydx plot indicates that the inputs and outputs can be well mapped onto each other with only slight variations. It was also noticed that the best fit was observed to be for the dydx plot with 3 principal components and contained no out lives since the cloud of data was well along the bisector line and this also implicates that the dimensionality of the output space is possible. 

t-SNE (t-Distributed Stochastic Neighbour Embedding)

Figure 5: dydx plot for the t-SNE transformed data 

From the dydx plot above it can be seen that there are no clustering thus there is nothing to unfold which is the main purpose of t-SNE. And since this is true, the input output mapping obtained for the t-SNE transformed data was seen to be inconclusive as it contained a lot of outlives.  

Summary of the dimensionality reduction process

Technique	Purpose	Linearity	Dimensionality Reduction	Suitable for Regression
PCA	Dimensionality Reduction	Linear	Yes, from high to low	Yes
CCA	Dimensionality Reduction	Non-linear	Yes from high to low	Possibly if relationship have some degree of non-linearity
t-SNE	Visualization	Non-linear	Yes, from high to low	No

From the findings it can clearly be seen that the PCA transformed data is the best one to proceed with to the model training stage

               PCA

References

[1] S. Tuukkanen, S. Lehtimäki, F. Jahangir, A. . -P. Eskelinen, D. Lupo and S. Franssila, "Printable                     and disposable supercapacitor from nanocellulose and carbon nanotubes," Proceedings of the                 5th Electronics System-integration Technology Conference (ESTC), Helsinki, Finland, 2014,                 pp. 1-6, doi: 10.1109/ESTC.2014.6962740.

[2] Y. Yan, Q. Li, W. Huang and W. Chen, "Operation Optimization and Control Method Based                             on Optimal Energy and Hydrogen Consumption for the Fuel Cell/Supercapacitor Hybrid Tram,"              in IEEE Transactions on Industrial Electronics, vol. 68, no. 2, pp. 1342-1352, Feb. 2021, doi:                 10.1109/TIE.2020.2967720.

[3] D. Linzen, S. Buller, E. Karden and R. W. De Doncker, "Analysis and evaluation of charge-                            balancing circuits on performance, reliability, and lifetime of supercapacitor systems," in IEEE              Transactions on Industry Applications, vol. 41, no. 5, pp. 1135-1141, Sept.-Oct. 2005, doi:                     10.1109/TIA.2005.853375.

[4] Y. Y. Yao, D. L. Zhang and D. G. Xu, "A Study of Supercapacitor Parameters and                                             Characteristics," 2006 International Conference on Power System Technology, Chongqing,                     China, 2006, pp. 1-4, doi: 10.1109/ICPST.2006.321487.

Edited by Shahil and Henal

S11172483@student.usp.ac.fj 

S11085370@student.usp.ac.fj