This R Notebook is the complement to my blog post Pretrained Character Embeddings for Deep Learning and Automatic Text Generation.

This notebook is licensed under the MIT License. If you use the code or data visualization designs contained within this notebook, it would be greatly appreciated if proper attribution is given back to this notebook and/or myself. Thanks! :)

source("Rstart.R")

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Stackoverflow is a great place to get help:
http://stackoverflow.com/tags/ggplot2.
Registering fonts with R

Attaching package: ‘scales’

The following object is masked from ‘package:readr’:

    col_factor
library(tsne)

1 Visualize GloVe Vectors

df <- read_delim("glove.840B.300d-char.txt", col_names = F, delim=" ", quote = "—")
Parsed with column specification:
cols(
  .default = col_double(),
  X1 = col_character()
)
See spec(...) for full column specifications.
df[,1:6]

Assign colors by character type (warning: jank implementation for determining character type)

type <- ifelse(df$X1 %in% letters, "lowercase",
          ifelse(df$X1 %in% LETTERS, "uppercase",
            ifelse(df$X1 %in% c(0:9), "numeric", "punctuation")))
type <- factor(type, levels=c("lowercase", "uppercase", "numeric", "punctuation"))
type
 [1] punctuation punctuation punctuation numeric     numeric    
 [6] numeric     punctuation punctuation uppercase   uppercase  
[11] uppercase   uppercase   uppercase   uppercase   punctuation
[16] punctuation lowercase   lowercase   lowercase   lowercase  
[21] lowercase   lowercase   punctuation punctuation punctuation
[26] punctuation punctuation numeric     numeric     punctuation
[31] punctuation uppercase   uppercase   uppercase   uppercase  
[36] uppercase   uppercase   punctuation punctuation lowercase  
[41] lowercase   lowercase   lowercase   lowercase   lowercase  
[46] punctuation punctuation punctuation punctuation punctuation
[51] numeric     numeric     punctuation punctuation uppercase  
[56] uppercase   uppercase   uppercase   uppercase   uppercase  
[61] uppercase   punctuation lowercase   lowercase   lowercase  
[66] lowercase   lowercase   lowercase   lowercase   punctuation
[71] punctuation punctuation punctuation punctuation numeric    
[76] numeric     numeric     punctuation uppercase   uppercase  
[81] uppercase   uppercase   uppercase   uppercase   uppercase  
[86] punctuation lowercase   lowercase   lowercase   lowercase  
[91] lowercase   lowercase   lowercase   punctuation
Levels: lowercase uppercase numeric punctuation
perplexity = 7
initial_dims = 16
max_iter = 5000
set.seed(123)
df_reduce <- tsne(df %>% select(X2:X301) %>% data.matrix(), perplexity = perplexity,
                  initial_dims = initial_dims, max_iter = max_iter)
sigma summary: Min. : 0.1091 |1st Qu. : 0.2229 |Median : 0.3149 |Mean : 0.3803 |3rd Qu. : 0.4603 |Max. : 1.156 |
Epoch: Iteration #100 error is: 19.9008331270176
Epoch: Iteration #200 error is: 2.49154431736295
Epoch: Iteration #300 error is: 1.87324143576349
Epoch: Iteration #400 error is: 1.59075923235389
Epoch: Iteration #500 error is: 1.35222486231347
Epoch: Iteration #600 error is: 1.17576546267547
Epoch: Iteration #700 error is: 0.980337707923824
Epoch: Iteration #800 error is: 0.813922092570097
Epoch: Iteration #900 error is: 0.70708861397967
Epoch: Iteration #1000 error is: 0.523447573796321
Epoch: Iteration #1100 error is: 0.440259641018736
Epoch: Iteration #1200 error is: 0.390224409495345
Epoch: Iteration #1300 error is: 0.375597956673988
Epoch: Iteration #1400 error is: 0.340263006276789
Epoch: Iteration #1500 error is: 0.333036071217878
Epoch: Iteration #1600 error is: 0.327345734645549
Epoch: Iteration #1700 error is: 0.326405073212151
Epoch: Iteration #1800 error is: 0.325612512290506
Epoch: Iteration #1900 error is: 0.323755276828245
Epoch: Iteration #2000 error is: 0.32235501602663
Epoch: Iteration #2100 error is: 0.321176910848929
Epoch: Iteration #2200 error is: 0.320488027687165
Epoch: Iteration #2300 error is: 0.319912091494209
Epoch: Iteration #2400 error is: 0.316101284354926
Epoch: Iteration #2500 error is: 0.314703671412566
Epoch: Iteration #2600 error is: 0.313603700195101
Epoch: Iteration #2700 error is: 0.311729358032727
Epoch: Iteration #2800 error is: 0.290913056358454
Epoch: Iteration #2900 error is: 0.287556668655012
Epoch: Iteration #3000 error is: 0.285954703869886
Epoch: Iteration #3100 error is: 0.285499614053067
Epoch: Iteration #3200 error is: 0.285160223816836
Epoch: Iteration #3300 error is: 0.284873813057661
Epoch: Iteration #3400 error is: 0.284620577084481
Epoch: Iteration #3500 error is: 0.284375973817991
Epoch: Iteration #3600 error is: 0.28414429027049
Epoch: Iteration #3700 error is: 0.283928003398625
Epoch: Iteration #3800 error is: 0.283721128912677
Epoch: Iteration #3900 error is: 0.283513869146296
Epoch: Iteration #4000 error is: 0.283308973090469
Epoch: Iteration #4100 error is: 0.283112309796586
Epoch: Iteration #4200 error is: 0.2829246993822
Epoch: Iteration #4300 error is: 0.282755705105952
Epoch: Iteration #4400 error is: 0.282624585398679
Epoch: Iteration #4500 error is: 0.282504849665238
Epoch: Iteration #4600 error is: 0.282401249075633
Epoch: Iteration #4700 error is: 0.282316886030564
Epoch: Iteration #4800 error is: 0.282244244473652
Epoch: Iteration #4900 error is: 0.28218129535269
Epoch: Iteration #5000 error is: 0.282131537547469
df_reduce <- data.frame(char = df$X1, type = type, df_reduce) %>%
                tbl_df() %>%
                mutate(char = as.character(char))
df_reduce
plot <- ggplot(df_reduce, aes(x=X1, y=X2, label=char, color = type)) +
          geom_text(family="Source Code Pro Semibold") +
          theme_void(base_family = "Source Sans Pro", base_size=8) +
          scale_color_brewer(palette="Set1") + 
          labs(title = "Projection of 300D GloVe Character Vectors into 2D Space (16D, perplexity = 7)",
               subtitle = "Characters closer to each other are more similar in usage context.",
               color = '') +
          theme(plot.margin = unit(c(0.2,0.2,0.2,0.2),"cm"),
                plot.subtitle = element_text(family="Open Sans Condensed Bold", size=8, color="#666666"))
max_save(plot, "char-tsne", w=5, h=4, "Stanford NLP")
perplexity = 2
initial_dims = 64
max_iter = 5000
set.seed(123)
df_reduce <- tsne(df %>% select(X2:X301) %>% data.matrix(), perplexity = perplexity,
                  initial_dims = initial_dims, max_iter = max_iter)
sigma summary: Min. : 0.1142 |1st Qu. : 0.1725 |Median : 0.3115 |Mean : 0.3579 |3rd Qu. : 0.5363 |Max. : 1.088 |
Epoch: Iteration #100 error is: 25.8760793569182
Epoch: Iteration #200 error is: 3.40491184952478
Epoch: Iteration #300 error is: 2.82652095433813
Epoch: Iteration #400 error is: 2.61351919903927
Epoch: Iteration #500 error is: 2.44614053865498
Epoch: Iteration #600 error is: 2.37619368988545
Epoch: Iteration #700 error is: 2.23237319968277
Epoch: Iteration #800 error is: 2.28598472974399
Epoch: Iteration #900 error is: 2.28704907389129
Epoch: Iteration #1000 error is: 2.30782035770231
Epoch: Iteration #1100 error is: 2.2228642540803
Epoch: Iteration #1200 error is: 2.14152563933713
Epoch: Iteration #1300 error is: 2.21792771417654
Epoch: Iteration #1400 error is: 2.17480494303053
Epoch: Iteration #1500 error is: 2.14442841588378
Epoch: Iteration #1600 error is: 2.05000922881734
Epoch: Iteration #1700 error is: 2.04150963174296
Epoch: Iteration #1800 error is: 2.02794295152051
Epoch: Iteration #1900 error is: 2.00547829213824
Epoch: Iteration #2000 error is: 1.97400981512453
Epoch: Iteration #2100 error is: 1.95257091343749
Epoch: Iteration #2200 error is: 1.93701021733958
Epoch: Iteration #2300 error is: 1.91671100932913
Epoch: Iteration #2400 error is: 1.87794824921166
Epoch: Iteration #2500 error is: 1.82819422564111
Epoch: Iteration #2600 error is: 1.81097311237971
Epoch: Iteration #2700 error is: 1.80499353291699
Epoch: Iteration #2800 error is: 1.76040935313148
Epoch: Iteration #2900 error is: 1.70144530621949
Epoch: Iteration #3000 error is: 1.69555438130605
Epoch: Iteration #3100 error is: 1.68761496767178
Epoch: Iteration #3200 error is: 1.68414683514823
Epoch: Iteration #3300 error is: 1.68194197731156
Epoch: Iteration #3400 error is: 1.68043650776213
Epoch: Iteration #3500 error is: 1.67948578863723
Epoch: Iteration #3600 error is: 1.67856959435698
Epoch: Iteration #3700 error is: 1.67759489834003
Epoch: Iteration #3800 error is: 1.67618146680881
Epoch: Iteration #3900 error is: 1.66973732701943
Epoch: Iteration #4000 error is: 1.65942279809246
Epoch: Iteration #4100 error is: 1.65801595695891
Epoch: Iteration #4200 error is: 1.65691853224998
Epoch: Iteration #4300 error is: 1.6559144564949
Epoch: Iteration #4400 error is: 1.65518323967551
Epoch: Iteration #4500 error is: 1.65455328538897
Epoch: Iteration #4600 error is: 1.654179239483
Epoch: Iteration #4700 error is: 1.65378054679007
Epoch: Iteration #4800 error is: 1.65312648165102
Epoch: Iteration #4900 error is: 1.652384671507
Epoch: Iteration #5000 error is: 1.65204671206778
df_reduce <- data.frame(char = df$X1, type = type, df_reduce) %>%
                tbl_df() %>%
                mutate(char = as.character(char))
plot <- ggplot(df_reduce, aes(x=X1, y=X2, label=char, color = type)) +
          geom_text(family="Source Code Pro Semibold") +
          theme_void(base_family = "Source Sans Pro", base_size=8) +
          scale_color_brewer(palette="Set1") + 
          labs(title = "Projection of 300D GloVe Character Vectors into 2D Space (64D, perplexity = 2)",
               subtitle = "Characters closer to each other are more similar in usage context.",
               color = '') +
          theme(plot.margin = unit(c(0.2,0.2,0.2,0.2),"cm"),
                plot.subtitle = element_text(family="Open Sans Condensed Bold", size=8, color="#666666"))
max_save(plot, "char-tsne-2", w=5, h=4, "Stanford NLP")

2 Visualize Embedded Magic Characters

df_embed <- read_delim("char-embeddings.txt", col_names = F, delim=" ", quote = "—")
Parsed with column specification:
cols(
  .default = col_double(),
  X1 = col_character()
)
See spec(...) for full column specifications.
1 parsing failure.
row col    expected      actual                  file
 36  -- 301 columns 302 columns 'char-embeddings.txt'
df_embed <- na.omit(df_embed)   # removes space and newline since will not parse
df_embed[,1:6]
type_embed <- ifelse(df_embed$X1 %in% letters, "lowercase",
                ifelse(df_embed$X1 %in% LETTERS, "uppercase",
                 ifelse(df_embed$X1 %in% c(0:9), "numeric", "punctuation")))
type_embed <- factor(type_embed, levels=c("lowercase", "uppercase", "numeric", "punctuation"))
perplexity = 10
initial_dims = 30
max_iter = 5000
set.seed(123)
df_reduce <- tsne(df_embed %>% select(X2:X301) %>% data.matrix(), perplexity = perplexity,
                  initial_dims = initial_dims, max_iter = max_iter)
sigma summary: Min. : 0.3982 |1st Qu. : 0.534 |Median : 0.589 |Mean : 0.6118 |3rd Qu. : 0.6945 |Max. : 0.8683 |
Epoch: Iteration #100 error is: 19.8910592108382
Epoch: Iteration #200 error is: 2.33802134983821
Epoch: Iteration #300 error is: 1.90657874553873
Epoch: Iteration #400 error is: 1.67310138694167
Epoch: Iteration #500 error is: 1.60159203367983
Epoch: Iteration #600 error is: 1.54534214605539
Epoch: Iteration #700 error is: 1.47202400115262
Epoch: Iteration #800 error is: 1.36359179697377
Epoch: Iteration #900 error is: 1.29338901327261
Epoch: Iteration #1000 error is: 1.22449894038032
Epoch: Iteration #1100 error is: 1.18563239809868
Epoch: Iteration #1200 error is: 1.16394694407087
Epoch: Iteration #1300 error is: 1.14957125694423
Epoch: Iteration #1400 error is: 1.0890666358816
Epoch: Iteration #1500 error is: 1.0665950000023
Epoch: Iteration #1600 error is: 1.05740834572964
Epoch: Iteration #1700 error is: 1.04467688100733
Epoch: Iteration #1800 error is: 1.0277168013362
Epoch: Iteration #1900 error is: 1.01363810613918
Epoch: Iteration #2000 error is: 1.00711524892226
Epoch: Iteration #2100 error is: 1.00261733094675
Epoch: Iteration #2200 error is: 0.998513918731433
Epoch: Iteration #2300 error is: 0.990047086853415
Epoch: Iteration #2400 error is: 0.985935233982174
Epoch: Iteration #2500 error is: 0.981127594076246
Epoch: Iteration #2600 error is: 0.974665708237662
Epoch: Iteration #2700 error is: 0.971771107043645
Epoch: Iteration #2800 error is: 0.969903063157817
Epoch: Iteration #2900 error is: 0.963494917257623
Epoch: Iteration #3000 error is: 0.959103295799426
Epoch: Iteration #3100 error is: 0.956883967492001
Epoch: Iteration #3200 error is: 0.954526780938449
Epoch: Iteration #3300 error is: 0.953220835794314
Epoch: Iteration #3400 error is: 0.952100075685241
Epoch: Iteration #3500 error is: 0.950936349047178
Epoch: Iteration #3600 error is: 0.948268595283757
Epoch: Iteration #3700 error is: 0.946614033465507
Epoch: Iteration #3800 error is: 0.94473545110956
Epoch: Iteration #3900 error is: 0.943661074638806
Epoch: Iteration #4000 error is: 0.942714344630487
Epoch: Iteration #4100 error is: 0.941857686485872
Epoch: Iteration #4200 error is: 0.940628703186185
Epoch: Iteration #4300 error is: 0.939789045001551
Epoch: Iteration #4400 error is: 0.939497303537436
Epoch: Iteration #4500 error is: 0.939385356136726
Epoch: Iteration #4600 error is: 0.93926741472932
Epoch: Iteration #4700 error is: 0.939224899636872
Epoch: Iteration #4800 error is: 0.939195983783595
Epoch: Iteration #4900 error is: 0.939169260551808
Epoch: Iteration #5000 error is: 0.939139305300065
df_reduce <- data.frame(char = df_embed$X1, type = type_embed, df_reduce) %>%
                tbl_df() %>%
                mutate(char = as.character(char))
plot <- ggplot(df_reduce, aes(x=X1, y=X2, label=char, color = type)) +
          geom_text(family="Source Code Pro Semibold") +
          theme_void(base_family = "Source Sans Pro", base_size=8) +
          scale_color_brewer(palette="Set1") + 
          labs(title = "Projection of 300D Magic Card Character Vectors into 2D Space (30D, perplexity = 10)",
               subtitle = "Characters closer to each other are more similar in usage context.",
               color = '') +
          theme(plot.margin = unit(c(0.2,0.2,0.2,0.2),"cm"),
                plot.subtitle = element_text(family="Open Sans Condensed Bold", size=8, color="#666666"))
max_save(plot, "char-tsne-embed", w=5, h=4, "Keras Logging")

3 Training Perfomance

batches_per_epoch = 7850
df_train <- read_csv("log.csv") %>% filter(iteration <= 20) %>%
              mutate(cumbatch = (iteration-1) * batches_per_epoch + batch)
Parsed with column specification:
cols(
  iteration = col_integer(),
  batch = col_integer(),
  batch_loss = col_double(),
  epoch_loss = col_double(),
  elapsed_time = col_double()
)
df_train %>% head(200)
#df_train_reshape <- df_train %>% gather(key = loss_type, value = loss, batch_loss, epoch_loss) %>%
#                      mutate(loss_type = factor(loss_type))
#df_train_reshape %>% head(100)
plot <- ggplot(df_train, aes(x=cumbatch, y=batch_loss, color=factor(iteration))) +
          geom_line(size=0.1) +
          scale_y_sqrt(breaks=c(0, 0.25, 0.5, 1, 2, 4)) +
          scale_x_continuous(breaks = seq(0, max(df_train$cumbatch)+batches_per_epoch, by=batches_per_epoch), labels = c(0:20)) +
          fte_theme() +
          theme(panel.grid.major = element_line(size=0.1)) +
    labs(title = "Batch Loss Over Time While Training Magic Card Generator",
          x = "# Epoch",
          y = "Batch Loss (128 Samples per Batch)")
max_save(plot, "batch-losses", "Keras Logging")
plot <- ggplot(df_train, aes(x=cumbatch, y=epoch_loss, color=factor(iteration))) +
          geom_line(size=0.5) +
          scale_y_sqrt(breaks=c(0, 0.25, 0.5, 1, 2, 4)) +
          scale_x_continuous(breaks = seq(0, max(df_train$cumbatch)+batches_per_epoch, by=batches_per_epoch), labels = c(0:20)) +
          fte_theme() +
          theme(panel.grid.major = element_line(size=0.1)) +
    labs(title = "Epoch Loss Over Time While Training Magic Card Generator",
          x = "# Epoch",
          y = "Epoch Loss (Average Batch Loss During Epoch)")
max_save(plot, "epoch-losses", "Keras Logging")

4 LICENSE

The MIT License (MIT)

Copyright (c) 2017 Max Woolf

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

LS0tCnRpdGxlOiAiUHJldHJhaW5lZCBDaGFyYWN0ZXIgRW1iZWRkaW5ncyBmb3IgRGVlcCBMZWFybmluZyBhbmQgQXV0b21hdGljIFRleHQgR2VuZXJhdGlvbiIKYXV0aG9yOiAiTWF4IFdvb2xmIChAbWluaW1heGlyKSIKZGF0ZTogIjIwMTctMDQtMDQiCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgaGlnaGxpZ2h0OiB0YW5nbwogICAgbWF0aGpheDogbnVsbAogICAgbnVtYmVyX3NlY3Rpb25zOiB5ZXMKICAgIHRoZW1lOiBzcGFjZWxhYgogICAgdG9jOiB5ZXMKICAgIHRvY19mbG9hdDogeWVzCi0tLQoKVGhpcyBSIE5vdGVib29rIGlzIHRoZSBjb21wbGVtZW50IHRvIG15IGJsb2cgcG9zdCBbUHJldHJhaW5lZCBDaGFyYWN0ZXIgRW1iZWRkaW5ncyBmb3IgRGVlcCBMZWFybmluZyBhbmQgQXV0b21hdGljIFRleHQgR2VuZXJhdGlvbl0oaHR0cDovL21pbmltYXhpci5jb20vMjAxNy8wNC9jaGFyLWVtYmVkZGluZ3MvKS4KClRoaXMgbm90ZWJvb2sgaXMgbGljZW5zZWQgdW5kZXIgdGhlIE1JVCBMaWNlbnNlLiBJZiB5b3UgdXNlIHRoZSBjb2RlIG9yIGRhdGEgdmlzdWFsaXphdGlvbiBkZXNpZ25zIGNvbnRhaW5lZCB3aXRoaW4gdGhpcyBub3RlYm9vaywgaXQgd291bGQgYmUgZ3JlYXRseSBhcHByZWNpYXRlZCBpZiBwcm9wZXIgYXR0cmlidXRpb24gaXMgZ2l2ZW4gYmFjayB0byB0aGlzIG5vdGVib29rIGFuZC9vciBteXNlbGYuIFRoYW5rcyEgOikKCmBgYHtyfQpzb3VyY2UoIlJzdGFydC5SIikKbGlicmFyeSh0c25lKQpgYGAKCiMgVmlzdWFsaXplIEdsb1ZlIFZlY3RvcnMKCmBgYHtyfQpkZiA8LSByZWFkX2RlbGltKCJnbG92ZS44NDBCLjMwMGQtY2hhci50eHQiLCBjb2xfbmFtZXMgPSBGLCBkZWxpbT0iICIsIHF1b3RlID0gIuKAlCIpCmRmWywxOjZdCmBgYAoKQXNzaWduIGNvbG9ycyBieSBjaGFyYWN0ZXIgdHlwZSAod2FybmluZzogamFuayBpbXBsZW1lbnRhdGlvbiBmb3IgZGV0ZXJtaW5pbmcgY2hhcmFjdGVyIHR5cGUpCgpgYGB7cn0KdHlwZSA8LSBpZmVsc2UoZGYkWDEgJWluJSBsZXR0ZXJzLCAibG93ZXJjYXNlIiwKICAgICAgICAgIGlmZWxzZShkZiRYMSAlaW4lIExFVFRFUlMsICJ1cHBlcmNhc2UiLAogICAgICAgICAgICBpZmVsc2UoZGYkWDEgJWluJSBjKDA6OSksICJudW1lcmljIiwgInB1bmN0dWF0aW9uIikpKQoKdHlwZSA8LSBmYWN0b3IodHlwZSwgbGV2ZWxzPWMoImxvd2VyY2FzZSIsICJ1cHBlcmNhc2UiLCAibnVtZXJpYyIsICJwdW5jdHVhdGlvbiIpKQp0eXBlCmBgYAoKCmBgYHtyfQpwZXJwbGV4aXR5ID0gNwppbml0aWFsX2RpbXMgPSAxNgptYXhfaXRlciA9IDUwMDAKCnNldC5zZWVkKDEyMykKZGZfcmVkdWNlIDwtIHRzbmUoZGYgJT4lIHNlbGVjdChYMjpYMzAxKSAlPiUgZGF0YS5tYXRyaXgoKSwgcGVycGxleGl0eSA9IHBlcnBsZXhpdHksCiAgICAgICAgICAgICAgICAgIGluaXRpYWxfZGltcyA9IGluaXRpYWxfZGltcywgbWF4X2l0ZXIgPSBtYXhfaXRlcikKCmRmX3JlZHVjZSA8LSBkYXRhLmZyYW1lKGNoYXIgPSBkZiRYMSwgdHlwZSA9IHR5cGUsIGRmX3JlZHVjZSkgJT4lCiAgICAgICAgICAgICAgICB0YmxfZGYoKSAlPiUKICAgICAgICAgICAgICAgIG11dGF0ZShjaGFyID0gYXMuY2hhcmFjdGVyKGNoYXIpKQpkZl9yZWR1Y2UKYGBgCgpgYGB7cn0KcGxvdCA8LSBnZ3Bsb3QoZGZfcmVkdWNlLCBhZXMoeD1YMSwgeT1YMiwgbGFiZWw9Y2hhciwgY29sb3IgPSB0eXBlKSkgKwogICAgICAgICAgZ2VvbV90ZXh0KGZhbWlseT0iU291cmNlIENvZGUgUHJvIFNlbWlib2xkIikgKwogICAgICAgICAgdGhlbWVfdm9pZChiYXNlX2ZhbWlseSA9ICJTb3VyY2UgU2FucyBQcm8iLCBiYXNlX3NpemU9OCkgKwogICAgICAgICAgc2NhbGVfY29sb3JfYnJld2VyKHBhbGV0dGU9IlNldDEiKSArIAogICAgICAgICAgbGFicyh0aXRsZSA9ICJQcm9qZWN0aW9uIG9mIDMwMEQgR2xvVmUgQ2hhcmFjdGVyIFZlY3RvcnMgaW50byAyRCBTcGFjZSAoMTZELCBwZXJwbGV4aXR5ID0gNykiLAogICAgICAgICAgICAgICBzdWJ0aXRsZSA9ICJDaGFyYWN0ZXJzIGNsb3NlciB0byBlYWNoIG90aGVyIGFyZSBtb3JlIHNpbWlsYXIgaW4gdXNhZ2UgY29udGV4dC4iLAogICAgICAgICAgICAgICBjb2xvciA9ICcnKSArCiAgICAgICAgICB0aGVtZShwbG90Lm1hcmdpbiA9IHVuaXQoYygwLjIsMC4yLDAuMiwwLjIpLCJjbSIpLAogICAgICAgICAgICAgICAgcGxvdC5zdWJ0aXRsZSA9IGVsZW1lbnRfdGV4dChmYW1pbHk9Ik9wZW4gU2FucyBDb25kZW5zZWQgQm9sZCIsIHNpemU9OCwgY29sb3I9IiM2NjY2NjYiKSkKCm1heF9zYXZlKHBsb3QsICJjaGFyLXRzbmUiLCB3PTUsIGg9NCwgIlN0YW5mb3JkIE5MUCIpCmBgYAoKIVtdKGNoYXItdHNuZS5wbmcpCmBgYHtyfQpwZXJwbGV4aXR5ID0gMgppbml0aWFsX2RpbXMgPSA2NAptYXhfaXRlciA9IDUwMDAKCnNldC5zZWVkKDEyMykKZGZfcmVkdWNlIDwtIHRzbmUoZGYgJT4lIHNlbGVjdChYMjpYMzAxKSAlPiUgZGF0YS5tYXRyaXgoKSwgcGVycGxleGl0eSA9IHBlcnBsZXhpdHksCiAgICAgICAgICAgICAgICAgIGluaXRpYWxfZGltcyA9IGluaXRpYWxfZGltcywgbWF4X2l0ZXIgPSBtYXhfaXRlcikKCmRmX3JlZHVjZSA8LSBkYXRhLmZyYW1lKGNoYXIgPSBkZiRYMSwgdHlwZSA9IHR5cGUsIGRmX3JlZHVjZSkgJT4lCiAgICAgICAgICAgICAgICB0YmxfZGYoKSAlPiUKICAgICAgICAgICAgICAgIG11dGF0ZShjaGFyID0gYXMuY2hhcmFjdGVyKGNoYXIpKQoKcGxvdCA8LSBnZ3Bsb3QoZGZfcmVkdWNlLCBhZXMoeD1YMSwgeT1YMiwgbGFiZWw9Y2hhciwgY29sb3IgPSB0eXBlKSkgKwogICAgICAgICAgZ2VvbV90ZXh0KGZhbWlseT0iU291cmNlIENvZGUgUHJvIFNlbWlib2xkIikgKwogICAgICAgICAgdGhlbWVfdm9pZChiYXNlX2ZhbWlseSA9ICJTb3VyY2UgU2FucyBQcm8iLCBiYXNlX3NpemU9OCkgKwogICAgICAgICAgc2NhbGVfY29sb3JfYnJld2VyKHBhbGV0dGU9IlNldDEiKSArIAogICAgICAgICAgbGFicyh0aXRsZSA9ICJQcm9qZWN0aW9uIG9mIDMwMEQgR2xvVmUgQ2hhcmFjdGVyIFZlY3RvcnMgaW50byAyRCBTcGFjZSAoNjRELCBwZXJwbGV4aXR5ID0gMikiLAogICAgICAgICAgICAgICBzdWJ0aXRsZSA9ICJDaGFyYWN0ZXJzIGNsb3NlciB0byBlYWNoIG90aGVyIGFyZSBtb3JlIHNpbWlsYXIgaW4gdXNhZ2UgY29udGV4dC4iLAogICAgICAgICAgICAgICBjb2xvciA9ICcnKSArCiAgICAgICAgICB0aGVtZShwbG90Lm1hcmdpbiA9IHVuaXQoYygwLjIsMC4yLDAuMiwwLjIpLCJjbSIpLAogICAgICAgICAgICAgICAgcGxvdC5zdWJ0aXRsZSA9IGVsZW1lbnRfdGV4dChmYW1pbHk9Ik9wZW4gU2FucyBDb25kZW5zZWQgQm9sZCIsIHNpemU9OCwgY29sb3I9IiM2NjY2NjYiKSkKCm1heF9zYXZlKHBsb3QsICJjaGFyLXRzbmUtMiIsIHc9NSwgaD00LCAiU3RhbmZvcmQgTkxQIikKYGBgCgohW10oY2hhci10c25lLTIucG5nKQoKIyBWaXN1YWxpemUgRW1iZWRkZWQgTWFnaWMgQ2hhcmFjdGVycwoKYGBge3J9CmRmX2VtYmVkIDwtIHJlYWRfZGVsaW0oImNoYXItZW1iZWRkaW5ncy50eHQiLCBjb2xfbmFtZXMgPSBGLCBkZWxpbT0iICIsIHF1b3RlID0gIuKAlCIpCmRmX2VtYmVkIDwtIG5hLm9taXQoZGZfZW1iZWQpICAgIyByZW1vdmVzIHNwYWNlIGFuZCBuZXdsaW5lIHNpbmNlIHdpbGwgbm90IHBhcnNlCmRmX2VtYmVkWywxOjZdCmBgYAoKYGBge3J9CnR5cGVfZW1iZWQgPC0gaWZlbHNlKGRmX2VtYmVkJFgxICVpbiUgbGV0dGVycywgImxvd2VyY2FzZSIsCiAgICAgICAgICAgICAgICBpZmVsc2UoZGZfZW1iZWQkWDEgJWluJSBMRVRURVJTLCAidXBwZXJjYXNlIiwKICAgICAgICAgICAgICAgICBpZmVsc2UoZGZfZW1iZWQkWDEgJWluJSBjKDA6OSksICJudW1lcmljIiwgInB1bmN0dWF0aW9uIikpKQoKdHlwZV9lbWJlZCA8LSBmYWN0b3IodHlwZV9lbWJlZCwgbGV2ZWxzPWMoImxvd2VyY2FzZSIsICJ1cHBlcmNhc2UiLCAibnVtZXJpYyIsICJwdW5jdHVhdGlvbiIpKQpgYGAKCmBgYHtyfQpwZXJwbGV4aXR5ID0gMTAKaW5pdGlhbF9kaW1zID0gMzAKbWF4X2l0ZXIgPSA1MDAwCgpzZXQuc2VlZCgxMjMpCmRmX3JlZHVjZSA8LSB0c25lKGRmX2VtYmVkICU+JSBzZWxlY3QoWDI6WDMwMSkgJT4lIGRhdGEubWF0cml4KCksIHBlcnBsZXhpdHkgPSBwZXJwbGV4aXR5LAogICAgICAgICAgICAgICAgICBpbml0aWFsX2RpbXMgPSBpbml0aWFsX2RpbXMsIG1heF9pdGVyID0gbWF4X2l0ZXIpCgpkZl9yZWR1Y2UgPC0gZGF0YS5mcmFtZShjaGFyID0gZGZfZW1iZWQkWDEsIHR5cGUgPSB0eXBlX2VtYmVkLCBkZl9yZWR1Y2UpICU+JQogICAgICAgICAgICAgICAgdGJsX2RmKCkgJT4lCiAgICAgICAgICAgICAgICBtdXRhdGUoY2hhciA9IGFzLmNoYXJhY3RlcihjaGFyKSkKCnBsb3QgPC0gZ2dwbG90KGRmX3JlZHVjZSwgYWVzKHg9WDEsIHk9WDIsIGxhYmVsPWNoYXIsIGNvbG9yID0gdHlwZSkpICsKICAgICAgICAgIGdlb21fdGV4dChmYW1pbHk9IlNvdXJjZSBDb2RlIFBybyBTZW1pYm9sZCIpICsKICAgICAgICAgIHRoZW1lX3ZvaWQoYmFzZV9mYW1pbHkgPSAiU291cmNlIFNhbnMgUHJvIiwgYmFzZV9zaXplPTgpICsKICAgICAgICAgIHNjYWxlX2NvbG9yX2JyZXdlcihwYWxldHRlPSJTZXQxIikgKyAKICAgICAgICAgIGxhYnModGl0bGUgPSAiUHJvamVjdGlvbiBvZiAzMDBEIE1hZ2ljIENhcmQgQ2hhcmFjdGVyIFZlY3RvcnMgaW50byAyRCBTcGFjZSAoMzBELCBwZXJwbGV4aXR5ID0gMTApIiwKICAgICAgICAgICAgICAgc3VidGl0bGUgPSAiQ2hhcmFjdGVycyBjbG9zZXIgdG8gZWFjaCBvdGhlciBhcmUgbW9yZSBzaW1pbGFyIGluIHVzYWdlIGNvbnRleHQuIiwKICAgICAgICAgICAgICAgY29sb3IgPSAnJykgKwogICAgICAgICAgdGhlbWUocGxvdC5tYXJnaW4gPSB1bml0KGMoMC4yLDAuMiwwLjIsMC4yKSwiY20iKSwKICAgICAgICAgICAgICAgIHBsb3Quc3VidGl0bGUgPSBlbGVtZW50X3RleHQoZmFtaWx5PSJPcGVuIFNhbnMgQ29uZGVuc2VkIEJvbGQiLCBzaXplPTgsIGNvbG9yPSIjNjY2NjY2IikpCgptYXhfc2F2ZShwbG90LCAiY2hhci10c25lLWVtYmVkIiwgdz01LCBoPTQsICJLZXJhcyBMb2dnaW5nIikKYGBgCgohW10oY2hhci10c25lLWVtYmVkLnBuZykKCiMgVHJhaW5pbmcgUGVyZm9tYW5jZQoKYGBge3J9CmJhdGNoZXNfcGVyX2Vwb2NoID0gNzg1MAoKZGZfdHJhaW4gPC0gcmVhZF9jc3YoImxvZy5jc3YiKSAlPiUgZmlsdGVyKGl0ZXJhdGlvbiA8PSAyMCkgJT4lCiAgICAgICAgICAgICAgbXV0YXRlKGN1bWJhdGNoID0gKGl0ZXJhdGlvbi0xKSAqIGJhdGNoZXNfcGVyX2Vwb2NoICsgYmF0Y2gpCmRmX3RyYWluICU+JSBoZWFkKDIwMCkKYGBgCgpgYGB7cn0KI2RmX3RyYWluX3Jlc2hhcGUgPC0gZGZfdHJhaW4gJT4lIGdhdGhlcihrZXkgPSBsb3NzX3R5cGUsIHZhbHVlID0gbG9zcywgYmF0Y2hfbG9zcywgZXBvY2hfbG9zcykgJT4lCiMgICAgICAgICAgICAgICAgICAgICAgbXV0YXRlKGxvc3NfdHlwZSA9IGZhY3Rvcihsb3NzX3R5cGUpKQojZGZfdHJhaW5fcmVzaGFwZSAlPiUgaGVhZCgxMDApCmBgYAoKCmBgYHtyfQpwbG90IDwtIGdncGxvdChkZl90cmFpbiwgYWVzKHg9Y3VtYmF0Y2gsIHk9YmF0Y2hfbG9zcywgY29sb3I9ZmFjdG9yKGl0ZXJhdGlvbikpKSArCiAgICAgICAgICBnZW9tX2xpbmUoc2l6ZT0wLjEpICsKICAgICAgICAgIHNjYWxlX3lfc3FydChicmVha3M9YygwLCAwLjI1LCAwLjUsIDEsIDIsIDQpKSArCiAgICAgICAgICBzY2FsZV94X2NvbnRpbnVvdXMoYnJlYWtzID0gc2VxKDAsIG1heChkZl90cmFpbiRjdW1iYXRjaCkrYmF0Y2hlc19wZXJfZXBvY2gsIGJ5PWJhdGNoZXNfcGVyX2Vwb2NoKSwgbGFiZWxzID0gYygwOjIwKSkgKwogICAgICAgICAgZnRlX3RoZW1lKCkgKwogICAgICAgICAgdGhlbWUocGFuZWwuZ3JpZC5tYWpvciA9IGVsZW1lbnRfbGluZShzaXplPTAuMSkpICsKICAgIGxhYnModGl0bGUgPSAiQmF0Y2ggTG9zcyBPdmVyIFRpbWUgV2hpbGUgVHJhaW5pbmcgTWFnaWMgQ2FyZCBHZW5lcmF0b3IiLAogICAgICAgICAgeCA9ICIjIEVwb2NoIiwKICAgICAgICAgIHkgPSAiQmF0Y2ggTG9zcyAoMTI4IFNhbXBsZXMgcGVyIEJhdGNoKSIpCgptYXhfc2F2ZShwbG90LCAiYmF0Y2gtbG9zc2VzIiwgIktlcmFzIExvZ2dpbmciKQpgYGAKCiFbXShiYXRjaC1sb3NzZXMucG5nKQoKCmBgYHtyfQpwbG90IDwtIGdncGxvdChkZl90cmFpbiwgYWVzKHg9Y3VtYmF0Y2gsIHk9ZXBvY2hfbG9zcywgY29sb3I9ZmFjdG9yKGl0ZXJhdGlvbikpKSArCiAgICAgICAgICBnZW9tX2xpbmUoc2l6ZT0wLjUpICsKICAgICAgICAgIHNjYWxlX3lfc3FydChicmVha3M9YygwLCAwLjI1LCAwLjUsIDEsIDIsIDQpKSArCiAgICAgICAgICBzY2FsZV94X2NvbnRpbnVvdXMoYnJlYWtzID0gc2VxKDAsIG1heChkZl90cmFpbiRjdW1iYXRjaCkrYmF0Y2hlc19wZXJfZXBvY2gsIGJ5PWJhdGNoZXNfcGVyX2Vwb2NoKSwgbGFiZWxzID0gYygwOjIwKSkgKwogICAgICAgICAgZnRlX3RoZW1lKCkgKwogICAgICAgICAgdGhlbWUocGFuZWwuZ3JpZC5tYWpvciA9IGVsZW1lbnRfbGluZShzaXplPTAuMSkpICsKICAgIGxhYnModGl0bGUgPSAiRXBvY2ggTG9zcyBPdmVyIFRpbWUgV2hpbGUgVHJhaW5pbmcgTWFnaWMgQ2FyZCBHZW5lcmF0b3IiLAogICAgICAgICAgeCA9ICIjIEVwb2NoIiwKICAgICAgICAgIHkgPSAiRXBvY2ggTG9zcyAoQXZlcmFnZSBCYXRjaCBMb3NzIER1cmluZyBFcG9jaCkiKQoKbWF4X3NhdmUocGxvdCwgImVwb2NoLWxvc3NlcyIsICJLZXJhcyBMb2dnaW5nIikKYGBgCgohW10oZXBvY2gtbG9zc2VzLnBuZykKCiMgTElDRU5TRQoKVGhlIE1JVCBMaWNlbnNlIChNSVQpCgpDb3B5cmlnaHQgKGMpIDIwMTcgTWF4IFdvb2xmCgpQZXJtaXNzaW9uIGlzIGhlcmVieSBncmFudGVkLCBmcmVlIG9mIGNoYXJnZSwgdG8gYW55IHBlcnNvbiBvYnRhaW5pbmcgYSBjb3B5IG9mIHRoaXMgc29mdHdhcmUgYW5kIGFzc29jaWF0ZWQgZG9jdW1lbnRhdGlvbiBmaWxlcyAodGhlIOKAnFNvZnR3YXJl4oCdKSwgdG8gZGVhbCBpbiB0aGUgU29mdHdhcmUgd2l0aG91dCByZXN0cmljdGlvbiwgaW5jbHVkaW5nIHdpdGhvdXQgbGltaXRhdGlvbiB0aGUgcmlnaHRzIHRvIHVzZSwgY29weSwgbW9kaWZ5LCBtZXJnZSwgcHVibGlzaCwgZGlzdHJpYnV0ZSwgc3VibGljZW5zZSwgYW5kL29yIHNlbGwgY29waWVzIG9mIHRoZSBTb2Z0d2FyZSwgYW5kIHRvIHBlcm1pdCBwZXJzb25zIHRvIHdob20gdGhlIFNvZnR3YXJlIGlzIGZ1cm5pc2hlZCB0byBkbyBzbywgc3ViamVjdCB0byB0aGUgZm9sbG93aW5nIGNvbmRpdGlvbnM6CgpUaGUgYWJvdmUgY29weXJpZ2h0IG5vdGljZSBhbmQgdGhpcyBwZXJtaXNzaW9uIG5vdGljZSBzaGFsbCBiZSBpbmNsdWRlZCBpbiBhbGwgY29waWVzIG9yIHN1YnN0YW50aWFsIHBvcnRpb25zIG9mIHRoZSBTb2Z0d2FyZS4KClRIRSBTT0ZUV0FSRSBJUyBQUk9WSURFRCDigJxBUyBJU+KAnSwgV0lUSE9VVCBXQVJSQU5UWSBPRiBBTlkgS0lORCwgRVhQUkVTUyBPUiBJTVBMSUVELCBJTkNMVURJTkcgQlVUIE5PVCBMSU1JVEVEIFRPIFRIRSBXQVJSQU5USUVTIE9GIE1FUkNIQU5UQUJJTElUWSwgRklUTkVTUyBGT1IgQSBQQVJUSUNVTEFSIFBVUlBPU0UgQU5EIE5PTklORlJJTkdFTUVOVC4gSU4gTk8gRVZFTlQgU0hBTEwgVEhFIEFVVEhPUlMgT1IgQ09QWVJJR0hUIEhPTERFUlMgQkUgTElBQkxFIEZPUiBBTlkgQ0xBSU0sIERBTUFHRVMgT1IgT1RIRVIgTElBQklMSVRZLCBXSEVUSEVSIElOIEFOIEFDVElPTiBPRiBDT05UUkFDVCwgVE9SVCBPUiBPVEhFUldJU0UsIEFSSVNJTkcgRlJPTSwgT1VUIE9GIE9SIElOIENPTk5FQ1RJT04gV0lUSCBUSEUgU09GVFdBUkUgT1IgVEhFIFVTRSBPUiBPVEhFUiBERUFMSU5HUyBJTiBUSEUgU09GVFdBUkUu