Load the required R packages (additional ones will be loaded as
needed)
Exploratory data analysis
Load and explore logged events
Transform time stamp data into the format suitable for processing
date and time data
We can now order the events, for each user, based on the time
stamp
Note: we will be using R’s pipe notation (|>) to make the code
easier to understand and follow
Let’s start by examining the time range the data is available for. It
should (roughly) coincide with the start and the end of the course
# start of the course
# end of the course
The course length (in weeks):
Since we want to make predictions based on the first couple of weeks
data, we need to add the week variable
Check the distribution of event counts across the course weeks
Also in proportions
Examine character variables that represent different types of actions
and logged events
Let’s examine the actions closer
Some of these actions refer to individual course topics, that is, to
the access to lecture materials on distinct course topics. These are:
General, Applications, Theory, Ethics, Feedback, La_types. We will
rename the actions to make the meaning clearer
# course_topics <- c("General", "Applications", "Theory", "Ethics", "Feedback", "La_types")
Examine also the log column
Load and examine grades data
Examine the summary statistics and distribution of the final
grade
Let’s add course_outcome as a binary variable indicating if
a student had a low grade. Students whose final grade is above 50th
percentile (median) will be considered as having good course outcome
(HIGH), the rest will be considered as having weak course outcome
(LOW)
Examine the distribution of the outcome variable (though we should
already know it)
Features
Two groups of action-based features will be computed and used for
prediction: (note: active days are days with at least one learning
action)
Features based on learning action counts: ** Total number of each
type of learning actions ** Average number of actions (of any type) per
day ** Entropy of daily action counts (considering active days
only)
Features based on number of active days ** Number of active days
** Average time distance between two consecutive active days
Since the idea is to create prediction models based on different
number of weeks data, we will also need to compute feature values for
different number of course weeks. Thus, we will create functions that
compute features based on the data for the given number of course weeks
(the input parameter).
To compute features based on counts per day, we need to add the date
variable
- Start with the total number of each type of learning actions
Note: to avoid having too many features (as action counts), we will
consider all actions related to access to the lecture materials on
different topics as one kind of action (‘Lecture’)
actions_tot_count <- function(events_data) {
}
Check the function with the data from the first two weeks of the
course
- Next, compute average number of actions (of any type) per day
avg_actions_per_day = function(events_data) {
}
Check the function with the data from the first two weeks of the
course
- Entropy of daily action counts
Entropy is a measure of disorder in a system. Here it is used as an
indicator of regularity of learning: lower the entropy, higher is the
regularity and vice versa. Note: A nice explanation of the intuition
behind the formula of Shannon entropy is given in this video.
Since we want to compute entropy of daily action counts, we need to
compute (approximate) the probability of action counts for each day. We
will do that by taking the proportion of daily action counts with
respect to the total action counts for the given student
entropy_of_action_counts = function(events_data) {
}
Check the function with the data from the first two weeks of the
course
- Number of active days (= days with at least one learning
action)
active_days_count = function(events_data) {
}
Check the function with the data from the first two weeks of the
course
- Average time distance between two consecutive active days
Note: for student with only 1 active day, avg_aday_dist will be NA.
To avoid losing students due to the missing value of this feature, we
will replace NAs with a large number (e.g., 2 x max distance), thus
indicating that a student rarely (if ever) got back to the course
activities
avg_dist_active_days = function(events_data) {
}
Check the function with the data from the first two weeks of the
course
Create feature set for 2 weeks of data and examine feature
relevance
Create a function that will allow for creating a feature set for any
(given) number of course weeks
create_feature_set = function(events_data) {
}
Create the feature set based on the first two weeks of data
Examine the feature set
Add the outcome variable
Examine the relevance of features for the prediction of the outcome
variable
Let’s first see how we can do it for one variable
Now, do for all at once
Note: the notation .data[[f]]
in the code below allow us
to access column from the ‘current’ data frame (in this case,
w2_data
) with the name given as the input variable of the
function (f
)
Predictive modeling
Load additional R packages required for model building and
evaluation
We will use decision tree (as implemented in the rpart package) as
the classification method, and will build a couple of decision tree (DT)
models, one for each of the first five weeks of the course. We will
build each model using the optimal value of the cp
hyper-parameter, identified through 10-fold cross-validation (as we did
before).
We will evaluate the models using the same metrics used before:
accuracy, precision, recall, F1
build_DT_model <- function(train_data) {
cp_grid <- expand.grid(.cp = seq(0.001, 0.1, 0.005))
ctrl <- trainControl(method = "CV",
number = 10,
classProbs = TRUE,
summaryFunction = twoClassSummary)
dt <- train(x = train_data |> select(-course_outcome),
y = train_data$course_outcome,
method = "rpart",
metric = "ROC",
tuneGrid = cp_grid,
trControl = ctrl)
dt$finalModel
}
get_evaluation_measures <- function(model, test_data) {
predicted_vals <- predict(model,
test_data |> select(-course_outcome),
type = 'class')
actual_vals <- test_data$course_outcome
cm <- table(actual_vals, predicted_vals)
# low achievement in the course is considered the positive class
TP <- cm[2,2]
TN <- cm[1,1]
FP <- cm[1,2]
FN <- cm[2,1]
accuracy = sum(diag(cm)) / sum(cm)
precision <- TP / (TP + FP)
recall <- TP / (TP + FN)
F1 <- (2 * precision * recall) / (precision + recall)
c(Accuracy = accuracy,
Precision = precision,
Recall = recall,
F1 = F1)
}
Create (classification) models for predicting course outcome, based
on progresively more weeks of events data
Starting from week 1, up to week 5, create predictive models and
examine their performance
models <- list()
eval_measures <- list()
for(k in 1:5) {
print(paste("Starting computations for week", k))
# create the dataset (features + outcome variable) for the given number of weeks (k)
# split the data into train and test sets
# build the model (through CV) and compute eval.measures
# add the model and its evaluation measures to the corresponding lists
}
Compare the models based on the evaluation measures
# transform the eval_measures list into a df
# embellish the evaluation report by:
# 1) adding the week column;
# 2) rounding the metric values to 4 digits;
# 3) rearranging the order of columns
Examine the importance of features in an early in the course model
with good performance
LS0tCnRpdGxlOiAiUHJlZGljdGl2ZSBtb2RlbGxpbmc6IHByZWRpY3RpbmcgY291cnNlIG91dGNvbWVzIGluIGEgYmxlbmRlZCBwb3N0Z3JhZHVhdGUgY291cnNlIgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCgpMb2FkIHRoZSByZXF1aXJlZCBSIHBhY2thZ2VzIChhZGRpdGlvbmFsIG9uZXMgd2lsbCBiZSBsb2FkZWQgYXMgbmVlZGVkKQpgYGB7ciBtZXNzYWdlPUZBTFNFfQoKCmBgYAoKIyMgRXhwbG9yYXRvcnkgZGF0YSBhbmFseXNpcwoKIyMjIExvYWQgYW5kIGV4cGxvcmUgbG9nZ2VkIGV2ZW50cwpgYGB7cn0KCmBgYAoKYGBge3J9CgpgYGAKClRyYW5zZm9ybSB0aW1lIHN0YW1wIGRhdGEgaW50byB0aGUgZm9ybWF0IHN1aXRhYmxlIGZvciBwcm9jZXNzaW5nIGRhdGUgYW5kIHRpbWUgZGF0YQpgYGB7cn0KCmBgYAoKYGBge3J9CgpgYGAKCldlIGNhbiBub3cgb3JkZXIgdGhlIGV2ZW50cywgZm9yIGVhY2ggdXNlciwgYmFzZWQgb24gdGhlIHRpbWUgc3RhbXAgCgpOb3RlOiB3ZSB3aWxsIGJlIHVzaW5nIFIncyBwaXBlIG5vdGF0aW9uICh8PikgdG8gbWFrZSB0aGUgY29kZSBlYXNpZXIgdG8gdW5kZXJzdGFuZCBhbmQgZm9sbG93IApgYGB7cn0KCmBgYAoKTGV0J3Mgc3RhcnQgYnkgZXhhbWluaW5nIHRoZSB0aW1lIHJhbmdlIHRoZSBkYXRhIGlzIGF2YWlsYWJsZSBmb3IuIApJdCBzaG91bGQgKHJvdWdobHkpIGNvaW5jaWRlIHdpdGggdGhlIHN0YXJ0IGFuZCB0aGUgZW5kIG9mIHRoZSBjb3Vyc2UKYGBge3J9CiMgc3RhcnQgb2YgdGhlIGNvdXJzZQoKYGBgCgpgYGB7cn0KIyBlbmQgb2YgdGhlIGNvdXJzZQoKCmBgYAoKVGhlIGNvdXJzZSBsZW5ndGggKGluIHdlZWtzKToKYGBge3J9CgpgYGAKClNpbmNlIHdlIHdhbnQgdG8gbWFrZSBwcmVkaWN0aW9ucyBiYXNlZCBvbiB0aGUgZmlyc3QgY291cGxlIG9mIHdlZWtzIGRhdGEsIHdlIG5lZWQgdG8gYWRkIHRoZSB3ZWVrIHZhcmlhYmxlIApgYGB7cn0KCgpgYGAKCkNoZWNrIHRoZSBkaXN0cmlidXRpb24gb2YgZXZlbnQgY291bnRzIGFjcm9zcyB0aGUgY291cnNlIHdlZWtzCmBgYHtyfQoKYGBgCgpBbHNvIGluIHByb3BvcnRpb25zCmBgYHtyfQoKYGBgCgpFeGFtaW5lIGNoYXJhY3RlciB2YXJpYWJsZXMgdGhhdCByZXByZXNlbnQgZGlmZmVyZW50IHR5cGVzIG9mIGFjdGlvbnMgYW5kIGxvZ2dlZCBldmVudHMKYGBge3J9CgoKYGBgCgpMZXQncyBleGFtaW5lIHRoZSBhY3Rpb25zIGNsb3NlcgpgYGB7cn0KCmBgYApTb21lIG9mIHRoZXNlIGFjdGlvbnMgcmVmZXIgdG8gaW5kaXZpZHVhbCBjb3Vyc2UgdG9waWNzLCB0aGF0IGlzLCB0byB0aGUgYWNjZXNzIHRvIGxlY3R1cmUgbWF0ZXJpYWxzIG9uIGRpc3RpbmN0IGNvdXJzZSB0b3BpY3MuIFRoZXNlIGFyZToKR2VuZXJhbCwgQXBwbGljYXRpb25zLCBUaGVvcnksICBFdGhpY3MsIEZlZWRiYWNrLCBMYV90eXBlcy4gCldlIHdpbGwgcmVuYW1lIHRoZSBhY3Rpb25zIHRvIG1ha2UgdGhlIG1lYW5pbmcgY2xlYXJlcgpgYGB7cn0KIyBjb3Vyc2VfdG9waWNzIDwtIGMoIkdlbmVyYWwiLCAiQXBwbGljYXRpb25zIiwgIlRoZW9yeSIsICAiRXRoaWNzIiwgIkZlZWRiYWNrIiwgIkxhX3R5cGVzIikKCgpgYGAKCmBgYHtyfQoKYGBgCgpFeGFtaW5lIGFsc28gdGhlIGxvZyBjb2x1bW4KYGBge3J9CgpgYGAKCgoKIyMjIExvYWQgYW5kIGV4YW1pbmUgZ3JhZGVzIGRhdGEKYGBge3J9CgpgYGAKCmBgYHtyfQoKYGBgCgpFeGFtaW5lIHRoZSBzdW1tYXJ5IHN0YXRpc3RpY3MgYW5kIGRpc3RyaWJ1dGlvbiBvZiB0aGUgZmluYWwgZ3JhZGUKYGBge3J9CgpgYGAKCmBgYHtyfQoKYGBgCgoKTGV0J3MgYWRkICpjb3Vyc2Vfb3V0Y29tZSogYXMgYSBiaW5hcnkgdmFyaWFibGUgaW5kaWNhdGluZyBpZiBhIHN0dWRlbnQgaGFkIGEgbG93IGdyYWRlLiAKU3R1ZGVudHMgd2hvc2UgZmluYWwgZ3JhZGUgaXMgYWJvdmUgNTB0aCBwZXJjZW50aWxlIChtZWRpYW4pIHdpbGwgYmUgY29uc2lkZXJlZCBhcyBoYXZpbmcgZ29vZCBjb3Vyc2Ugb3V0Y29tZSAoSElHSCksIHRoZSByZXN0IHdpbGwgYmUgY29uc2lkZXJlZCBhcyBoYXZpbmcgd2VhayBjb3Vyc2Ugb3V0Y29tZSAoTE9XKQpgYGB7cn0KCmBgYAoKRXhhbWluZSB0aGUgZGlzdHJpYnV0aW9uIG9mIHRoZSBvdXRjb21lIHZhcmlhYmxlICh0aG91Z2ggd2Ugc2hvdWxkIGFscmVhZHkga25vdyBpdCkKYGBge3J9CgpgYGAKCgojIyBGZWF0dXJlcwoKVHdvIGdyb3VwcyBvZiBhY3Rpb24tYmFzZWQgZmVhdHVyZXMgd2lsbCBiZSBjb21wdXRlZCBhbmQgdXNlZCBmb3IgcHJlZGljdGlvbjoKKG5vdGU6IGFjdGl2ZSBkYXlzIGFyZSBkYXlzIHdpdGggYXQgbGVhc3Qgb25lIGxlYXJuaW5nIGFjdGlvbikKCiogRmVhdHVyZXMgYmFzZWQgb24gbGVhcm5pbmcgYWN0aW9uIGNvdW50czoKKiogVG90YWwgbnVtYmVyIG9mIGVhY2ggdHlwZSBvZiBsZWFybmluZyBhY3Rpb25zIAoqKiBBdmVyYWdlIG51bWJlciBvZiBhY3Rpb25zIChvZiBhbnkgdHlwZSkgcGVyIGRheQoqKiBFbnRyb3B5IG9mIGRhaWx5IGFjdGlvbiBjb3VudHMgKGNvbnNpZGVyaW5nIGFjdGl2ZSBkYXlzIG9ubHkpCgoqIEZlYXR1cmVzIGJhc2VkIG9uIG51bWJlciBvZiBhY3RpdmUgZGF5cwoqKiBOdW1iZXIgb2YgYWN0aXZlIGRheXMKKiogQXZlcmFnZSB0aW1lIGRpc3RhbmNlIGJldHdlZW4gdHdvIGNvbnNlY3V0aXZlIGFjdGl2ZSBkYXlzCgpTaW5jZSB0aGUgaWRlYSBpcyB0byBjcmVhdGUgcHJlZGljdGlvbiBtb2RlbHMgYmFzZWQgb24gZGlmZmVyZW50IG51bWJlciBvZiB3ZWVrcyBkYXRhLCB3ZSB3aWxsIGFsc28gbmVlZCB0byBjb21wdXRlIGZlYXR1cmUgdmFsdWVzIGZvciBkaWZmZXJlbnQgbnVtYmVyIG9mIGNvdXJzZSB3ZWVrcy4gVGh1cywgd2Ugd2lsbCBjcmVhdGUgZnVuY3Rpb25zIHRoYXQgY29tcHV0ZSBmZWF0dXJlcyBiYXNlZCBvbiB0aGUgZGF0YSBmb3IgdGhlIGdpdmVuIG51bWJlciBvZiBjb3Vyc2Ugd2Vla3MgKHRoZSBpbnB1dCBwYXJhbWV0ZXIpLiAKClRvIGNvbXB1dGUgZmVhdHVyZXMgYmFzZWQgb24gY291bnRzIHBlciBkYXksIHdlIG5lZWQgdG8gYWRkIHRoZSBkYXRlIHZhcmlhYmxlCmBgYHtyfQoKYGBgCgooMSkgU3RhcnQgd2l0aCB0aGUgdG90YWwgbnVtYmVyIG9mIGVhY2ggdHlwZSBvZiBsZWFybmluZyBhY3Rpb25zIAoKTm90ZTogdG8gYXZvaWQgaGF2aW5nIHRvbyBtYW55IGZlYXR1cmVzIChhcyBhY3Rpb24gY291bnRzKSwgd2Ugd2lsbCBjb25zaWRlciBhbGwgYWN0aW9ucyByZWxhdGVkIHRvIGFjY2VzcyB0byB0aGUgbGVjdHVyZSBtYXRlcmlhbHMgb24gZGlmZmVyZW50IHRvcGljcyBhcyBvbmUga2luZCBvZiBhY3Rpb24gKCdMZWN0dXJlJykKYGBge3J9CmFjdGlvbnNfdG90X2NvdW50IDwtIGZ1bmN0aW9uKGV2ZW50c19kYXRhKSB7CiAgCn0KYGBgCgpDaGVjayB0aGUgZnVuY3Rpb24gd2l0aCB0aGUgZGF0YSBmcm9tIHRoZSBmaXJzdCB0d28gd2Vla3Mgb2YgdGhlIGNvdXJzZQpgYGB7cn0KCmBgYAoKKDIpIE5leHQsIGNvbXB1dGUgYXZlcmFnZSBudW1iZXIgb2YgYWN0aW9ucyAob2YgYW55IHR5cGUpIHBlciBkYXkKCmBgYHtyfQphdmdfYWN0aW9uc19wZXJfZGF5ID0gZnVuY3Rpb24oZXZlbnRzX2RhdGEpIHsKICAKICAKfQpgYGAKCkNoZWNrIHRoZSBmdW5jdGlvbiB3aXRoIHRoZSBkYXRhIGZyb20gdGhlIGZpcnN0IHR3byB3ZWVrcyBvZiB0aGUgY291cnNlCmBgYHtyfQoKYGBgCgooMykgRW50cm9weSBvZiBkYWlseSBhY3Rpb24gY291bnRzCgpFbnRyb3B5IGlzIGEgbWVhc3VyZSBvZiBkaXNvcmRlciBpbiBhIHN5c3RlbS4gSGVyZSBpdCBpcyB1c2VkIGFzIGFuIGluZGljYXRvciBvZiByZWd1bGFyaXR5IG9mIGxlYXJuaW5nOiBsb3dlciB0aGUgZW50cm9weSwgaGlnaGVyIGlzIHRoZSByZWd1bGFyaXR5IGFuZCB2aWNlIHZlcnNhLiAKTm90ZTogQSBuaWNlIGV4cGxhbmF0aW9uIG9mIHRoZSBpbnR1aXRpb24gYmVoaW5kIHRoZSBmb3JtdWxhIG9mIFNoYW5ub24gZW50cm9weSBpcyBnaXZlbiBpbiBbdGhpcyB2aWRlb10oaHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj0wR0NHYXcwUU9oQSkuCgpTaW5jZSB3ZSB3YW50IHRvIGNvbXB1dGUgZW50cm9weSBvZiBkYWlseSBhY3Rpb24gY291bnRzLCB3ZSBuZWVkIHRvIGNvbXB1dGUgKGFwcHJveGltYXRlKSB0aGUgcHJvYmFiaWxpdHkgb2YgYWN0aW9uIGNvdW50cyBmb3IgZWFjaCBkYXkuIFdlIHdpbGwgZG8gdGhhdCBieSB0YWtpbmcgdGhlIHByb3BvcnRpb24gb2YgZGFpbHkgYWN0aW9uIGNvdW50cyB3aXRoIHJlc3BlY3QgdG8gdGhlIHRvdGFsIGFjdGlvbiBjb3VudHMgZm9yIHRoZSBnaXZlbiBzdHVkZW50CmBgYHtyfQplbnRyb3B5X29mX2FjdGlvbl9jb3VudHMgPSBmdW5jdGlvbihldmVudHNfZGF0YSkgewogIAogIAp9CmBgYAoKQ2hlY2sgdGhlIGZ1bmN0aW9uIHdpdGggdGhlIGRhdGEgZnJvbSB0aGUgZmlyc3QgdHdvIHdlZWtzIG9mIHRoZSBjb3Vyc2UKYGBge3J9CgpgYGAKCig0KSBOdW1iZXIgb2YgYWN0aXZlIGRheXMgKD0gZGF5cyB3aXRoIGF0IGxlYXN0IG9uZSBsZWFybmluZyBhY3Rpb24pCgpgYGB7cn0KYWN0aXZlX2RheXNfY291bnQgPSBmdW5jdGlvbihldmVudHNfZGF0YSkgewogCiAgCn0KYGBgCgpDaGVjayB0aGUgZnVuY3Rpb24gd2l0aCB0aGUgZGF0YSBmcm9tIHRoZSBmaXJzdCB0d28gd2Vla3Mgb2YgdGhlIGNvdXJzZQpgYGB7cn0KCmBgYAoKKDUpIEF2ZXJhZ2UgdGltZSBkaXN0YW5jZSBiZXR3ZWVuIHR3byBjb25zZWN1dGl2ZSBhY3RpdmUgZGF5cwoKTm90ZTogZm9yIHN0dWRlbnQgd2l0aCBvbmx5IDEgYWN0aXZlIGRheSwgYXZnX2FkYXlfZGlzdCB3aWxsIGJlIE5BLiBUbyBhdm9pZCBsb3Npbmcgc3R1ZGVudHMgZHVlIHRvIHRoZSBtaXNzaW5nIHZhbHVlIG9mIHRoaXMgZmVhdHVyZSwgd2Ugd2lsbCByZXBsYWNlIE5BcyB3aXRoIGEgbGFyZ2UgbnVtYmVyIChlLmcuLCAyIHggbWF4IGRpc3RhbmNlKSwgdGh1cyBpbmRpY2F0aW5nIHRoYXQgYSBzdHVkZW50IHJhcmVseSAoaWYgZXZlcikgZ290IGJhY2sgdG8gdGhlIGNvdXJzZSBhY3Rpdml0aWVzCmBgYHtyfQphdmdfZGlzdF9hY3RpdmVfZGF5cyA9IGZ1bmN0aW9uKGV2ZW50c19kYXRhKSB7CiAgCiAgCn0KYGBgCgpDaGVjayB0aGUgZnVuY3Rpb24gd2l0aCB0aGUgZGF0YSBmcm9tIHRoZSBmaXJzdCB0d28gd2Vla3Mgb2YgdGhlIGNvdXJzZQpgYGB7cn0KCmBgYAoKIyMjIENyZWF0ZSBmZWF0dXJlIHNldCBmb3IgMiB3ZWVrcyBvZiBkYXRhIGFuZCBleGFtaW5lIGZlYXR1cmUgcmVsZXZhbmNlCgpDcmVhdGUgYSBmdW5jdGlvbiB0aGF0IHdpbGwgYWxsb3cgZm9yIGNyZWF0aW5nIGEgZmVhdHVyZSBzZXQgZm9yIGFueSAoZ2l2ZW4pIG51bWJlciBvZiBjb3Vyc2Ugd2Vla3MgCmBgYHtyfQpjcmVhdGVfZmVhdHVyZV9zZXQgPSBmdW5jdGlvbihldmVudHNfZGF0YSkgewogIAogIAp9CmBgYAoKQ3JlYXRlIHRoZSBmZWF0dXJlIHNldCBiYXNlZCBvbiB0aGUgZmlyc3QgdHdvIHdlZWtzIG9mIGRhdGEKYGBge3J9CgpgYGAKCkV4YW1pbmUgdGhlIGZlYXR1cmUgc2V0CmBgYHtyfQoKYGBgCgpBZGQgdGhlIG91dGNvbWUgdmFyaWFibGUKYGBge3J9CgoKYGBgCgpFeGFtaW5lIHRoZSByZWxldmFuY2Ugb2YgZmVhdHVyZXMgZm9yIHRoZSBwcmVkaWN0aW9uIG9mIHRoZSBvdXRjb21lIHZhcmlhYmxlCgpMZXQncyBmaXJzdCBzZWUgaG93IHdlIGNhbiBkbyBpdCBmb3Igb25lIHZhcmlhYmxlIApgYGB7cn0KCgpgYGAKCk5vdywgZG8gZm9yIGFsbCBhdCBvbmNlCgpOb3RlOiB0aGUgbm90YXRpb24gYC5kYXRhW1tmXV1gIGluIHRoZSBjb2RlIGJlbG93IGFsbG93IHVzIHRvIGFjY2VzcyBjb2x1bW4gZnJvbSB0aGUgJ2N1cnJlbnQnIGRhdGEgZnJhbWUgKGluIHRoaXMgY2FzZSwgYHcyX2RhdGFgKSB3aXRoIHRoZSBuYW1lIGdpdmVuIGFzIHRoZSBpbnB1dCB2YXJpYWJsZSBvZiB0aGUgZnVuY3Rpb24gKGBmYCkgCmBgYHtyfQoKCmBgYAoKCgoKIyMgUHJlZGljdGl2ZSBtb2RlbGluZwoKTG9hZCBhZGRpdGlvbmFsIFIgcGFja2FnZXMgcmVxdWlyZWQgZm9yIG1vZGVsIGJ1aWxkaW5nIGFuZCBldmFsdWF0aW9uIApgYGB7ciBtZXNzYWdlPUZBTFNFfQoKCmBgYAoKV2Ugd2lsbCB1c2UgZGVjaXNpb24gdHJlZSAoYXMgaW1wbGVtZW50ZWQgaW4gdGhlIHJwYXJ0IHBhY2thZ2UpIGFzIHRoZSBjbGFzc2lmaWNhdGlvbiBtZXRob2QsIGFuZCB3aWxsIGJ1aWxkIGEgY291cGxlIG9mIGRlY2lzaW9uIHRyZWUgKERUKSBtb2RlbHMsIG9uZSBmb3IgZWFjaCBvZiB0aGUgZmlyc3QgZml2ZSB3ZWVrcyBvZiB0aGUgY291cnNlLiBXZSB3aWxsIGJ1aWxkIGVhY2ggbW9kZWwgdXNpbmcgdGhlIG9wdGltYWwgdmFsdWUgb2YgdGhlIGBjcGAgaHlwZXItcGFyYW1ldGVyLCBpZGVudGlmaWVkIHRocm91Z2ggMTAtZm9sZCBjcm9zcy12YWxpZGF0aW9uIChhcyB3ZSBkaWQgYmVmb3JlKS4gCgpXZSB3aWxsIGV2YWx1YXRlIHRoZSBtb2RlbHMgdXNpbmcgdGhlIHNhbWUgbWV0cmljcyB1c2VkIGJlZm9yZTogYWNjdXJhY3ksIHByZWNpc2lvbiwgcmVjYWxsLCBGMQpgYGB7cn0KYnVpbGRfRFRfbW9kZWwgPC0gZnVuY3Rpb24odHJhaW5fZGF0YSkgewogIAogIGNwX2dyaWQgPC0gZXhwYW5kLmdyaWQoLmNwID0gc2VxKDAuMDAxLCAwLjEsIDAuMDA1KSkKICAKICBjdHJsIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiQ1YiLCAKICAgICAgICAgICAgICAgICAgICAgICBudW1iZXIgPSAxMCwKICAgICAgICAgICAgICAgICAgICAgICBjbGFzc1Byb2JzID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgICBzdW1tYXJ5RnVuY3Rpb24gPSB0d29DbGFzc1N1bW1hcnkpCiAgCiAgZHQgPC0gdHJhaW4oeCA9IHRyYWluX2RhdGEgfD4gc2VsZWN0KC1jb3Vyc2Vfb3V0Y29tZSksCiAgICAgICAgICAgICAgeSA9IHRyYWluX2RhdGEkY291cnNlX291dGNvbWUsCiAgICAgICAgICAgICAgbWV0aG9kID0gInJwYXJ0IiwKICAgICAgICAgICAgICBtZXRyaWMgPSAiUk9DIiwKICAgICAgICAgICAgICB0dW5lR3JpZCA9IGNwX2dyaWQsCiAgICAgICAgICAgICAgdHJDb250cm9sID0gY3RybCkKICAKICBkdCRmaW5hbE1vZGVsCn0KYGBgCgoKYGBge3J9CmdldF9ldmFsdWF0aW9uX21lYXN1cmVzIDwtIGZ1bmN0aW9uKG1vZGVsLCB0ZXN0X2RhdGEpIHsKICAKICBwcmVkaWN0ZWRfdmFscyA8LSBwcmVkaWN0KG1vZGVsLCAKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHRlc3RfZGF0YSB8PiBzZWxlY3QoLWNvdXJzZV9vdXRjb21lKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgIHR5cGUgPSAnY2xhc3MnKQogIGFjdHVhbF92YWxzIDwtIHRlc3RfZGF0YSRjb3Vyc2Vfb3V0Y29tZQogIAogIGNtIDwtIHRhYmxlKGFjdHVhbF92YWxzLCBwcmVkaWN0ZWRfdmFscykKICAKICAjIGxvdyBhY2hpZXZlbWVudCBpbiB0aGUgY291cnNlIGlzIGNvbnNpZGVyZWQgdGhlIHBvc2l0aXZlIGNsYXNzCiAgVFAgPC0gY21bMiwyXQogIFROIDwtIGNtWzEsMV0KICBGUCA8LSBjbVsxLDJdCiAgRk4gPC0gY21bMiwxXQoKICBhY2N1cmFjeSA9IHN1bShkaWFnKGNtKSkgLyBzdW0oY20pCiAgcHJlY2lzaW9uIDwtIFRQIC8gKFRQICsgRlApCiAgcmVjYWxsIDwtIFRQIC8gKFRQICsgRk4pCiAgRjEgPC0gKDIgKiBwcmVjaXNpb24gKiByZWNhbGwpIC8gKHByZWNpc2lvbiArIHJlY2FsbCkKICAKICBjKEFjY3VyYWN5ID0gYWNjdXJhY3ksIAogICAgUHJlY2lzaW9uID0gcHJlY2lzaW9uLCAKICAgIFJlY2FsbCA9IHJlY2FsbCwgCiAgICBGMSA9IEYxKQogIAp9CmBgYAoKCiMjIyBDcmVhdGUgKGNsYXNzaWZpY2F0aW9uKSBtb2RlbHMgZm9yIHByZWRpY3RpbmcgY291cnNlIG91dGNvbWUsIGJhc2VkIG9uIHByb2dyZXNpdmVseSBtb3JlIHdlZWtzIG9mIGV2ZW50cyBkYXRhCgpTdGFydGluZyBmcm9tIHdlZWsgMSwgdXAgdG8gd2VlayA1LCBjcmVhdGUgcHJlZGljdGl2ZSBtb2RlbHMgYW5kIGV4YW1pbmUgdGhlaXIgcGVyZm9ybWFuY2UKYGBge3Igd2FybmluZz1GQUxTRSwgbWVzc2FnZT1GQUxTRX0KbW9kZWxzIDwtIGxpc3QoKQpldmFsX21lYXN1cmVzIDwtIGxpc3QoKQoKZm9yKGsgaW4gMTo1KSB7CiAgCiAgcHJpbnQocGFzdGUoIlN0YXJ0aW5nIGNvbXB1dGF0aW9ucyBmb3Igd2VlayIsIGspKQogIAogICMgY3JlYXRlIHRoZSBkYXRhc2V0IChmZWF0dXJlcyArIG91dGNvbWUgdmFyaWFibGUpIGZvciB0aGUgZ2l2ZW4gbnVtYmVyIG9mIHdlZWtzIChrKSAKICAKICAKICAjIHNwbGl0IHRoZSBkYXRhIGludG8gdHJhaW4gYW5kIHRlc3Qgc2V0cwogIAoKICAjIGJ1aWxkIHRoZSBtb2RlbCAodGhyb3VnaCBDVikgYW5kIGNvbXB1dGUgZXZhbC5tZWFzdXJlcwogIAogIAogICMgYWRkIHRoZSBtb2RlbCBhbmQgaXRzIGV2YWx1YXRpb24gbWVhc3VyZXMgdG8gdGhlIGNvcnJlc3BvbmRpbmcgbGlzdHMgCiAgCn0KYGBgCgpDb21wYXJlIHRoZSBtb2RlbHMgYmFzZWQgb24gdGhlIGV2YWx1YXRpb24gbWVhc3VyZXMKYGBge3J9CiMgdHJhbnNmb3JtIHRoZSBldmFsX21lYXN1cmVzIGxpc3QgaW50byBhIGRmCgoKIyBlbWJlbGxpc2ggdGhlIGV2YWx1YXRpb24gcmVwb3J0IGJ5OiAKIyAxKSBhZGRpbmcgdGhlIHdlZWsgY29sdW1uOyAKIyAyKSByb3VuZGluZyB0aGUgbWV0cmljIHZhbHVlcyB0byA0IGRpZ2l0czsgCiMgMykgcmVhcnJhbmdpbmcgdGhlIG9yZGVyIG9mIGNvbHVtbnMgCgpgYGAKCgpFeGFtaW5lIHRoZSBpbXBvcnRhbmNlIG9mIGZlYXR1cmVzIGluIGFuIGVhcmx5IGluIHRoZSBjb3Vyc2UgbW9kZWwgd2l0aCBnb29kIHBlcmZvcm1hbmNlCmBgYHtyfQoKYGBg