Gabriel E. Cabrera Guzman: Hello World!

Gabriel E. Cabrera

First, since I don’t speak German, I think I should use the terms auto-value and auto-vector. Secondly, since there is no such a thing as auto-vector, we should really talk about auto-direction instead.

library(sparklyr)
library(sparklyr.deeperer)

# NOTE: the correct spark_home path to use depends on the configuration of the
# Spark cluster you are working with.
spark_home <- "/usr/lib/spark"
sc <- spark_connect(master = "yarn", spark_home = spark_home)

data_dir <- copy_images_to_hdfs()

# extract features from train- and test-data
image_data <- list()
for (x in c("train", "test")) {
  # import
  image_data[[x]] <- c("dogs", "cats") %>%
    lapply(
      function(label) {
        numeric_label <- ifelse(identical(label, "dogs"), 1L, 0L)
        spark_read_image(
          sc, dir = file.path(data_dir, x, label, fsep = "/")
        ) %>%
          dplyr::mutate(label = numeric_label)
      }
    ) %>%
      do.call(sdf_bind_rows, .)

  dl_featurizer <- invoke_new(
    sc,
    "com.databricks.sparkdl.DeepImageFeaturizer",
    random_string("dl_featurizer") # uid
  ) %>%
    invoke("setModelName", "InceptionV3") %>%
    invoke("setInputCol", "image") %>%
    invoke("setOutputCol", "features")
  image_data[[x]] <-
    dl_featurizer %>%
    invoke("transform", spark_dataframe(image_data[[x]])) %>%
    sdf_register()
}

Reuse

Text and figures are licensed under Creative Commons Attribution CC BY 4.0. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".

Citation

For attribution, please cite this work as

Cabrera (2020, Dec. 27). Gabriel E. Cabrera Guzman: Hello World!. Retrieved from http://gcabrerag.rbind.io/blog/site/posts/welcome/

BibTeX citation

@misc{cabrera2020hello,
  author = {Cabrera, Gabriel E.},
  title = {Gabriel E. Cabrera Guzman: Hello World!},
  url = {http://gcabrerag.rbind.io/blog/site/posts/welcome/},
  year = {2020}
}