6.1 Motivation and Comparison to Related Work

Epidemiological studies serve as a basis for the identification of risk factors associated with a medical condition [3]–[5]. Machine learning is still relatively little used in epidemiology, mainly due to the hypothesis-driven nature of their research. However, examples of machine learning applications are the identification of health failure subtypes [169] and the discovery of factors (including biomarkers) that modulate a medical outcome [170], [171]. In longitudinal cohort studies, measurements are performed in multiple study waves; hence researchers obtain access to sequences of recordings. In the context of machine learning, extracting and leveraging the inherent temporal information from these sequences may increase model performance and, thus, the understanding of the medical condition of interest.

From a technical point of view, classification problems on timestamped data can be very different. They can be broadly summarized into three categories:

1. Each instance (patient, study participant) has a label at each time point \(t\). The goal is to assign the instance’s label at \(t\) + 1. Typically, this is a data stream classification problem [172], [173].
2. Each instance is associated with only one label for the all time points. Typically, this is a timeseries classification problem [174], [175].
3. There are no labels but the time point of an event is to be predicted. This problem is concerned with event prediction [176]–[178].

Our classification problem falls under (a), but we cannot use the advances of stream classification because we our data is unlabeled except for the last time point and our stream of three time points is tiny.

In clinical applications, temporal information is often exploited, predominantly for the analysis of patient records. For example, Pechenizkiy et al. [179] analyzed streams of recordings to predict rehospitalization because of heart failure events during remote patient management. Sun et al. [180] computed the similarity between streams of patients from patient monitoring data. Combi et al. [181] reported on streams of life signals, particularly on the temporal analysis of the timestamped medical records of hospital patients.

However, participants of an epidemiological, population-based study are not hospital patients – they are a random sample of the studied population, often with a skewed class distribution. In a longitudinal study of this kind, recordings for the same cohort member are made at each moment. Hielscher et al. [182] presented a feature engineering approach to extract temporal information from multiple but few patient recordings in a longitudinal epidemiological study. First, for each assessment, clusters of feature-value sequences associated with the target variable are found. Afterward, original and sequence features are used in conjunction for classification. Hielscher et al. [182] showed that classification performance increases when features with temporal information are incorporated into the feature space. Instead of modeling the individual change of measurement values, our approach involves deriving multivariate change descriptors.

Patient evolution with clustering was studied by Siddiqui et al. [183]. They proposed a method that predicts patient evolution from timestamped data by clustering them on similarity and predicting cluster movement in the multi-dimensional space. However, the patient data considered in [183] are labeled at each moment.

Our workflow combines labeled and unlabeled timestamped data from a longitudinal study to improve classification performance on skewed data. As the target variable, we study the multi-factorial disorder hepatic steatosis (fatty liver) on a sample of participants from the longitudinal population-based “Study of Health in Pomerania” (SHIP) [12], recall Section 2.2.1. For the SHIP cohort, the assessments (interviews, medical tests, etc.) were recorded in several moments (SHIP-0, SHIP-1, etc.), that are ca. 5 years apart. Temporal information is often used when analyzing patient data in a hospital, but there the time granularity is different. For example, in an intensive care unit, timestamped data are collected at a fast pace, i.e., every minute or even every second. In contrast, the participants of a longitudinal epidemiological study are monitored for months or even years. Hence, measurements of the same assessment in an epidemiological dataset are few and possibly far apart. The large period between two consecutive recordings complicates applying methods designed for data that arrive with a higher frequency. For example, a participant may exhibit alcohol abuse or become pregnant, stop smoking and start again, take antibiotics that affect the liver, or experience other lifestyle changes that turn the medical recordings taken five years ago irrelevant for learning the participant’s current health state. Another patient may have no lifestyle changes and no illnesses, so their past data reflect only aging.

A further challenge is that the label is unavailable for several waves. A reliable estimate of the fat accumulation in the liver was computed from magnetic resonance tomography images. In SHIP-0, MRT was unavailable. Instead, liver fat accumulation was derived from ultrasound – a procedure with lower clinical accuracy. In SHIP-1, the calculation was omitted altogether. Consequently, for a given SHIP participant, the class label is available in SHIP-2, no label in SHIP-1, and a partially reliable indicator in SHIP-0. Since hepatic steatosis is a reversible disorder, label imputation – using a growth model [184] – is not possible; participant evolution must be learned with only one moment with labeled data.

We address these challenges as follows. First, we group study participants at each moment on similarity, thus building clusters of cohort members with similar recordings at one of the three moments. Then, we connect the clusters across time, thus capturing each cluster’s transition from one study wave to the next. These transitions reflect the evolution of subpopulations, not individuals. Hence, next to the single labeled recording per cohort participant, we also exploit the earlier, unlabeled recordings, the description of the cluster they are assigned to, and information on how the clusters evolve. We show that this new, augmented dataset, combining labeled and unlabeled data on individuals and subpopulations, improves classification and delivers additional insights on some factors associated with hepatic steatosis.

6.2 Problem Formulation

We now provide some basic symbols and essential functions for our classification problem, summarized in Table 6.1.

Our longitudinal study comprises data for a participant \(x\in X\) for every feature \(f\in F\) and every time point \(t\in \{1,\ldots, T\}\) (also referred to as moment). The (single) value of \(x\) for feature \(f\) at moment \(t\) is denoted as \(v(x,f,t)\); the set of all values for \(x\) at \(t\), i.e., \(\{v(x,f,t)|\forall f\in{}F\}\), is denoted as \(obs(x,F,t)\). For some participant, \(v(\cdot)\) is NULL, i.e., there may be missing values. Our goal is to to predict the class label of \(x\) at moment \(T\), i.e., \(label(x,T)\) by using data from all observations in all previous moments \(t\in\{1\ldots T-1\}\). The class label of each participant is unavailable except for \(t=T\). We model this prediction task as conventional classification problem. We aim to improve classification performance by expanding the feature space with change descriptors, the so-called evolution features, presented next.

6.3 Evolution Features

We leverage latent temporal information of a longitudinal cohort study dataset by extracting informative features based on the individual change of participants and the transition of their respective clusters over time. For this purpose, we exploit the similarity among participants at each moment as a surrogate to the labels which are not available in the first two moments, assuming that similar participants evolve similarly. We call these new features “evolution features”. Our approach is illustrated in Figure 6.1 (a). We monitor the individual change of participants across the study waves, trace the clusters’ change separately, extract new features (from labeled and unlabeled data) and augment the original data space with our new descriptors of change. The complete classification workflow is depicted in Figure 6.1 (b).

In the following, we describe the clustering of study participants (Section 6.3.1), the generation of evolution features (Section 6.3.2), and our feature selection strategy to extract a subset of informative features as input for classification (Section 6.3.3), while dealing with class imbalance by undersampling the majority class.

6.4 Evaluation Setup

We evaluate our workflow with 10-fold cross-validation on four off-the-shelf classification algorithms: random forest [191] (RF), C4.5 decision tree [192], Naïve Bayes (NB), and k-nearest neighbor (kNN). First, we compare each algorithm’s generalization performance when used alone (baseline variant) vs. when incorporated into our workflow (workflow-enhanced variant). Further, we study the impact of different combinations of the three workflow components undersampling (U), feature selection (F), and incorporation of generated evolution features (G). Sensitivity (true positive rate), specificity (true negative rate), and F-measure (harmonic mean of precision and recall) serve as evaluation measures. As shown in Table 6.3, Baseline invokes only the classification algorithm; we use the classification algorithm which achieves the highest F-measure scores. The variant U-G performs undersampling and uses the generated evolution features for classification. Since we undersample only for feature selection and then build the classification models on the original dataset, so U-G is identical to --G and U-- is identical to the Baseline variant, so we omit to list U-G and U-- explicitly.

The main parameter is \(k\) which is the number of neighbors of a data point: we set \(minPts\) = \(k\) and use \(k\) to derive the values of the DBSCAN parameter \(eps\) (cf. Section 6.3.1) and of the parameters for the features same_kNN_\(k\)_t_1_t_2 and fraction_Of_POS_kNN_\(k\)_t (cf. Table 6.2). Further, the number of nearest neighbors for the k-NN classification algorithm is also set to \(k\). We vary \(k\) to measure its impact on classification performance.

Following the findings in Chapters 3 and 4 on the differences between female and male participants with respect to the outcome, we run the experiments on the whole dataset (PartitionAll) and on the partitions of female (PartitionF) and male (PartitionM) participants. Finally, we list the most important features found in PartitionAll and its two subsets.

6.5 Results

Figure 6.3 shows sensitivity (left), specificity (center), and F-measure (right) for the simple classifiers (gray curves) and their workflow-enhanced counterparts (same line style, colored) for different k.

Overall, each workflow-enhanced variant outperforms its simple counterpart with respect to sensitivity and F-measure, and outperforms or performs slightly worse in specificity. The workflow-enhanced Naive Bayes performs best concerning sensitivity for any k and best for k = 31. Decision trees exhibit the highest F-measure, with improvements on sensitivity and F-measure compared to its simple variant, albeit specificity being slightly worse; improvements are less for large k. Random Forests benefit the most from our workflow, with an absolute improvement in F-measure of over 30% (green vs. gray “+” curves in the right part of Figure 6.3). One explanation for the relatively low sensitivity of the simple RF variant is the large number of trees (100) learned on data samples containing very few positive examples: RF could have been trapped by the many majority class examples. This result is consistent with the specificity curve (almost straight line around 95%) of simple RF, while the F-measure is slightly above 40%. Our workflow improves RF sensitivity (63%) and F-measure (65%), while specificity remains high (90%). Overall, the impact of k on the three measures is limited for all algorithms except for the workflow-enhanced and the baseline k-NN, which is naturally affected stronger by the value of k than any other algorithm. Therefore, the workflow-enhanced variants outperform their simple counterparts in terms of sensitivity and F-measure. For some algorithms, our workflow prevents overfitting to the negative class.

The results of the workflow component-specific results in Figure 6.4 show that our complete workflow UFG and the variants UF- and --G outperform the other variants in sensitivity and F-measure. The variants -F- and -FG perform well only regarding specificity, suggesting that feature selection may not be beneficial without undersampling for datasets with class imbalance.

6.6 Identification of Important Evolution Features

The performance of the workflow variants, which include feature selection, indicates that a small number of features is sufficient for class separation. Hereafter, we report on the evolution features selected for classification and appearing among the top 15 features according to information gain for each partition; we term these features “important.”

Figure 6.5 shows that for PartitionAll 3 out of these 15 features are generated evolution features. The boxplots (a) and (c) in Figure 6.5 refer to differences between values recorded in two moments. The feature separationDelta_g_1_2 measures the difference in cluster separation for each participant based on the cluster assignment in moment 1 and 2, and corresponds to entry #9 in Table 6.2. Participants belonging to the positive class exhibit a higher median in separationDelta_g_1_2 than participants without the disorder, indicating that clusters harboring mostly positive participants cover larger, more sparse areas. The feature relative_Difference_som_huef_g_0_1 (#19) quantifies the difference in a participant’s hip circumference between SHIP-0 and SHIP-1, relative to the value in SHIP-0. On average, study participants from both classes lose weight when they grow older. Negative participants reduce more weight than positive participants (cf. Figure 6.5 (c)), which generally reflects differences in lifestyles. The mosaic chart in Figure 6.5 (b) for feature fraction_of_Positives_kNN_1_g_2 (#3) indicates that the “nearest neighbor” of a fatty liver participant is also more likely to exhibit the disorder than it is for a participant without fatty liver.

For PartitionF, 5 out of the top 15 features are evolution features, cf. Figure 6.6. Compared with female participants without the disorder, female subjects with fatty liver exhibit a larger distance to the centroid of their cluster in SHIP-1 (#2), a lower silhouette coefficient in SHIP-1 (#5), a higher difference in waist circumference between SHIP-0 and SHIP-2 (#19), and a lower relative difference in serum triglycerides concentration between SHIP-0 and SHIP-1 (#19).

For PartitionM, 2 out of the top 15 features are evolution features (Figure 6.7), including the relative difference in waist circumference between SHIP-1 and SHIP-2 (#19) and difference in cluster separation between SHIP-0 and SHIP-1 (#6). For both features, participants exhibiting the disorder have greater values.

6.7 Conclusion

We have extended our set of methods for static medical data presented in Part I by proposing a workflow for the classification of longitudinal cohort study data that exploits inherent temporal information by clustering the cohort participants at each moment, linking the clusters, and tracing participant evolution over the study’s moments. We extract evolution features from the clusters and their transitions, which are added to the feature space and subsequently used for classification. The workflow improves the generalization performance with respect to sensitivity and F-measure scores. The generated evolution features contribute to this improvement, even when used alone and without undersampling the skewed data. We have shown that the change of somatographic variables’ values and cluster quality indices over time are predictive.

A limitation concerns the assessment of a feature’s importance using information gain. The additional merit a feature has towards model predictions cannot be assessed being decoupled from the actual model. Furthermore, models may incorporate complex feature interactions that are not captured or subpopulation-specific differences in feature importance. Part III addresses post-hoc model interpretation, which includes more sophisticated approaches of measuring a feature’s attribution to a model.