9.1 Motivation and Comparison to Related Work

One example where the relationship between subpopulation membership on the target variable is not well understood yet is gender differences in tinnitus patients. Female gender has been identified as an important risk factor for psychological comorbidities in many studies: women show higher prevalence rates regarding depression, anxiety, and other psychosomatic diseases [280]–[286]. However, previous studies presented conflicting results on the relationship between gender and tinnitus severity and distress. Some studies found no gender differences with respect to tinnitus severity [287], annoyance [288], and similar tinnitus-related scores [289]. Others found higher tinnitus distress in women [290], higher loudness and annoyance in men [291], a high association of severe tinnitus with suicide attempts only in women [292], and a high correlation of tinnitus severity with life quality, depression, and stress only in men [293]. Overall, there is no consensus on gender-specific determinants of prevalence rates or accompanying symptoms of chronic tinnitus such as depression or anxiety. However, gender differences in psychological response profiles and coping strategies could substantially influence tinnitus chronification and treatment success rates [294]. Understanding gender differences may therefore facilitate a more detailed identification of symptom profiles, increase treatment response rates, and help provide access for vulnerable populations who may be less visible in the clinical setting.

9.2 Workflow

We present a workflow consisting of a modeling component and a post-modeling component, extending the approach presented in Chapter 8. More specifically, we (i) build machine learning models capable of predicting the values of a target variable while performing hyperparameter optimization in a cross-validation scheme, and (ii) use post-hoc interpretation mechanisms to identify variables that contributed most to the prediction of the best model. We deviate from the workflow presented earlier because we are interested in comparing a priori defined disjoint subpopulations instead of determining them.

We proceed as follows. First, we train a model for each population where subpopulation membership is the target variable. Then, we build models for each subpopulation separately and compare model reliance values between the best performing models. More specifically, we compare which variables appear to be important (i) for both subpopulations, (ii) for one of the two subpopulations, or (iii) for neither subpopulation. For example, we split a dataset into two non-overlapping based on the variable defining the subpopulations and validate our workflow by examining factors that discriminate between subpopulations and factors predictive for the response variable in both or either of the two subpopulations.

9.3 Comparing Differences in Feature Importance between Two Subpopulations

To measure feature-individual attributions to a model’s predictions and to compare them between two subpopulations, we use model reliance (MR; [258]), which is described in the context of iterative feature elimination in Section 8.2.3. Recall that \(MR(f,\zeta)\) is a permutation-based variable importance measure that calculates the increase in the error of a model \(\zeta\) when the values of the variable of interest \(f\) are randomly shuffled within the training set. If \(f\) is important for the prediction of \(\zeta\), \(MR(f,\zeta)\) > 1. If random permutation of the feature values leads to a higher performance of the model, then the feature’s attribute to model quality is low, whereupon \(MR\) < 1.

For models that predict subpopulation membership, we rank features by MR value and report on the most important features. For subpopulation-specific models, we use scatterplots (see Figure 9.1) depicting a feature’s contribution (as \(MR\) value) to the model Model_1 (x-axis), which is trained on one subpopulation, and to the model Model_2 (y-axis), trained on the other subpopulation. Features that contribute equally to both models are on the diagonal line, while higher \(MR\) values for one model are further from the diagonal. The variables with the highest average \(MR\) score or the highest difference magnitude between subpopulation-specific \(MR\) scores are colored and labeled.

9.4 Learning Tasks and Evaluation Setup

We validate our workflow on the Charité tinnitus patient dataset (CHA, Section 2.2.2) and the SHIP dataset (Section 2.2.1). We define five learning tasks (LT) for the following response variables and subpopulations:

• LT 1: gender (CHA)
• LT 2: tinnitus-related distress (CHA)
  • LT 2a: female subpopulation
  • LT 2b: male subpopulation
• LT 3: depression (CHA)
  • LT 3a: female subpopulation
  • LT 3b: male subpopulation
• LT 4: gender (SHIP)
• LT 5: liver fat concentration at second follow-up (SHIP)
  • LT 5a: female subpopulation
  • LT 5b: male subpopulation

LT 1 and LT 4 have gender as the target variable, and characteristics are identified that are predictive for one of the genders. For each learning task, we use variables from the first study as predictors. For the learning tasks LT 2, LT 3, and LT 5, we build separate models for each of the two gender subpopulations. We refer to these models as “F_model” and “M_model,” respectively; the learning task is explicitly stated if it cannot be inferred from the context.

Selection of algorithms and evaluation. Based on their encouraging performances relative to other classifiers in Chapter 8, we use the following five algorithms: least absolute shrinkage and selection operator (LASSO [243]), RIDGE [244], support vector machine (SVM [239]), random forest (RF [191]) and gradient boosted trees (GBT [24]), see Section 8.2.2 for a description. We use 10-fold cross-validation to evaluate model generalization performance and perform a grid search for hyperparameter selection (cf. listing of parameter candidates in Table 8.1). We choose the evaluation measures based on the type of the target variable. For LT 1, we employ accuracy and sensitivity for each gender. For LT 2, we use root mean squared error (RMSE) and the coefficient of determination R², defined as \(R^2=1-\frac{\sum_{i=1}^N (\hat{y}_i-y_i)^2}{\sum_{i=1}^N(\bar{y}-y_i)^2}\), where \(\bar{y}\) is the average response value. Higher values are better for all measures except RMSE.

For each learning task, we define a baseline performance. For the classification problem of LT 1 and LT 4, the baseline is equal to a model that always predicts the majority class over all training observations. Similarly, for the regression problems of LT 2, 3, and 5, the average value of a target variable over all training observations is used to predict every test instance’s response value.

Data preparation for CHA. For CHA, we use baseline data collected before the start of therapy. To make the learning tasks nontrivial, we remove variables from the same questionnaire as the response variable for each task. For example, for LT 2, we exclude all variables from the TQ questionnaire because the target variable is calculated from them. We include 1628 patients (828 female, 800 male) with complete data for the ADSL, PSQ, SF8, SOZK, TINSKAL, TQ, and TLQ questionnaires (cf. Table 2.1). The selection of these questionnaires is motivated to obtain a comprehensive assessment of tinnitus, comorbid conditions (e.g., depression), general quality of life, and socio-demographic data. For CHA and SHIP, multinomial variables, i.e., variables that take three or more symbolic values, such as reported gender, marital status, and education level, are encoded as dummy variables. For example, smoking status smoking_s0 can be one of the following values: 0 = never a smoker, 1 = ex-smoker, 2 = current smoker. The dummy variable smoking_s0_1 then indicates whether a study participant is an ex-smoker. To avoid multicollinearity, we remove the first dummy of each variable, leaving only n-1 dummies. For the smoking status example, we keep smoking_s0_1 and smoking_s0_2, and we remove smoking_s0_0. Finally, a total of 181 variables from the baseline measurements are used as predictors, including responses to individual questionnaire items, subscale scores, total scores, and, for each questionnaire, the average time taken to complete an item. Tinnitus-related distress is measured by the TQ total score (TQ_distress). The severity of depression is measured by the ADSL total score (ADSL_depression).

Data preparation for SHIP. For the SHIP data, we consider only the variables recorded in SHIP-0 and use the liver fat concentration (liverfat_s2) measured via MRT in SHIP-2. We use the same subset of 886 labeled participants as in Chapter 4, of which 460 are female and 426 male. We remove the variables related to the ultrasound diagnosis of hepatic steatosis, stea_s0 and stea_alt75_s0, because we have already identified their high correlation to the target in Chapters 3 and 4. Furthermore, we remove “near-zero” variance variables where the most frequent value occurs at least 19 times more frequently than the second most frequent value. Typical examples include the ATC_* medication variables where few participants report taking them. Variables with a variance near zero can lead to resampling problems because some of the resamples may have constant values for that variable. Besides, it is difficult to infer significant correlations from them, as it is unclear whether the measured effects can be generalized to the overall population or whether they are just an artifact of this small and thus unrepresentative sample.

Finally, we remove highly correlated features using the algorithm of Kuhn and Johnson [242]. We first compute the Pearson correlation coefficient for each pair of features. For feature pairs with an absolute correlation value of \(r \geq\) 0.9, we keep the feature that has the lower average correlation with the other features. We repeat this process until none of the correlation values exceed the specified threshold. Of the original 350 variables, 118 remain to be used for modeling. Since determining the appropriate type of missingness for each variable is beyond our workflow’s scope, we assume that missingness occurs completely at random (MCAR). Missing values are thus imputed by random sampling with replacement.

9.5 Validation on Two Datasets

9.6 Conclusion

We have presented a workflow to juxtapose the most important predictors between two a priori defined subpopulations based on black-box models. We have adapted model reliance to estimate a feature’s attribution regarding the model and to investigate subpopulation-specific differences in this respect. Compared to the workflow in Chapter 8, our goal was not to find a parsimonious model; hence we did not perform feature selection.

Our goals are related to causal inference, which aims to measure the true, unconfounded effect between variables. Current efforts include building methods to detect causal relationships in observational data [299], [300], opposed to a randomized clinical trial (RCT) in clinical research. RCTs are considered the gold standard for inferring causal effects of treatment [36]. In an RCT, individual patients of a patient population are randomly assigned to one of two subgroups: a treatment subgroup and a control subgroup. Only the former receives the treatment. Randomization of subgroup membership serves to minimize the effects of potential confounders and selection bias. The two subgroups are as similar as possible before the intervention so that it is possible to calculate the average treatment effect to quantify the true causal efficacy of the treatment [301]. Translated to our application, an appropriate causality question is: What is the effect of gender on tinnitus severity and depression? However, our goal was different: we wanted to exploratively examine similarities and differences between subpopulations regarding factors that are predictive of a response of interest. Thus, we were interested not only in the relationship of gender on the response, but also in differences between the female and male subpopulations in the predictability of other variables on the response. With the generation of new hypotheses as the main goal in mind, our focus is on providing exploratory methods for comparing differences between predefined subpopulations.