/ Recent content in Home on Swimming the Data Lake Hugo -- gohugo.io en-us Thu, 23 May 2019 00:00:00 +0000 /post/2019/05/23/one-function-a-day-multiway-tables-with-ftable/ Thu, 23 May 2019 00:00:00 +0000 /post/2019/05/23/one-function-a-day-multiway-tables-with-ftable/ it´s usual to make confusion tables or cross tables when we have vectors of same lenght to compute and cross classify them, but some other issues arise when we have several dimensions (more than two). Since this is a common situation but it´s not so common to know the easy solution, so here comes ftable function from package stats. Let´s use Titanic dataset for an example: data(Titanic) ftable(Titanic) Survived No Yes Class Sex Age 1st Male Child 0 5 Adult 118 57 Female Child 0 1 Adult 4 140 2nd Male Child 0 11 Adult 154 14 Female Child 0 13 Adult 13 80 3rd Male Child 35 13 Adult 387 75 Female Child 17 14 Adult 89 76 Crew Male Child 0 0 Adult 670 192 Female Child 0 0 /post/2019/05/16/one-function-a-day-filldown/ Thu, 16 May 2019 00:00:00 +0000 /post/2019/05/16/one-function-a-day-filldown/ Sometimes we need to fill the values in a vector with previous values when data is missing. I wrote some convoluted code to do that but, as usual, someone solved the very same problem in an elegant way. Here comes the function filldown from package MESS library(MESS) myVector <- c(3, NA, NA, NA, 4, NA, NA, 7, NA, NA) filled.vector <- filldown(myVector) filled.vector [1] 3 3 3 3 4 4 4 7 7 7 and done! /post/2018/08/06/dockerized-spacemacs/ Mon, 06 Aug 2018 00:00:00 +0000 /post/2018/08/06/dockerized-spacemacs/ I was talking the other day of setting Spacemacs on Windows with no privileges. Sometimes it’s not even possible to do so, but you have Cygwin/mintty/babun + Docker engine on your machine for testing purposes. So here is the way of having a spacemacs running on Docker in no time. Main steps are: 1- install xinit and xorg on your *nix-terminal 2- set display and run dockerized spacemacs First step is easy, just install xinit and xorg. /post/2018/07/22/back-to-basics/ Sun, 22 Jul 2018 00:00:00 +0000 /post/2018/07/22/back-to-basics/ I think i’m a little bit “old school” or maybe sometimes you have to use the right tool for the task. Some time ago, i discovered I feel more productive staying away from the mouse, so terminals and text editors are my daily working environment from then. If you use linux, i’m preaching to the choir, and nearly the same if your work involved Mac OS. Sadly, sometimes you have to work using Windows. /post/2018/05/29/my-favourite-snippets/ Tue, 29 May 2018 00:00:00 +0000 /post/2018/05/29/my-favourite-snippets/ A hidden gem from Rstudio is snippets feature. A well known option in any other editor (Atom, VS Code, Notepad ++….) seems that for R people is not a very used tool. For what I know some developers tend to code a full Add-in for things that can be achieved easily just adding a snippet to your Rstudio configuration. Doing this is easy. The graphical way is getting to Rstudio Tools > Global Options > Code > Enable Code Snippets (Edit Snippets) /post/2018/05/09/using-inotify-for-long-running-r-scripts/ Wed, 09 May 2018 00:00:00 +0000 /post/2018/05/09/using-inotify-for-long-running-r-scripts/ Sometimes your Rscript takes a long time to run. Not for computation requirements but for the kind of processes involved, for instance when calling APIs with time limits and such (I’d use big data architectures otherwise). Last week I was working on one of such projects, and one of the requirements was: the user must provide an excel file via web interface to Rserver, and then computation begins. At the end, the results file is emailed to user. /post/2018/02/12/my-imposter-syndrome/ Mon, 12 Feb 2018 00:00:00 +0000 /post/2018/02/12/my-imposter-syndrome/ There’s a well known issue about the psychology of the majority of people working on Data Science. They feel a fraud. There’s no public study about it to my knowledge but informal guess in meetings and tech talks ended with several (if not all) hands up agreeing with the next sentence: “I think that I’m a fraud and don’t deserve the success I’ve achieved.” For me as a psychologist, it’s a very interesting area of investigation and would like to discover some serious study about it, but there’s some clear (and guessed) causes for such a feeling. /post/2018/01/09/spanish-my-package-finally-on-cran/ Tue, 09 Jan 2018 00:00:00 +0000 /post/2018/01/09/spanish-my-package-finally-on-cran/ Finally, my package “spanish” was ready for submission on CRAN and now it’s published! Not a great piece of work, just a place to gather a bunch of functions that I was unable to find anywhere else because are related to spanish data. Just one function right now, translating numerical quantities spelled in spanish to their integer counterparts. More functions in the coming months, related to Spanish Cadastral API, data cleaning, and such… /post/2017/12/05/packages-for-descriptive-step/ Tue, 05 Dec 2017 00:00:00 +0000 /post/2017/12/05/packages-for-descriptive-step/ On the first steps of any project, the most usual task is to take a glimpse, figure how our data is distributed, and as fast as possible, be ready for next steps (wrangling and imputation). In the old days, I’ve built my own functions to accomplish this task, until some other people did this very same work distributing packages with this target in mind. Many of them are awesome packages (really awesome), allowing you to plot missing values and data distribution, among other tasks, that makes very easy to figure which kind of data are you working with. /post/2017/12/04/explainers-when-your-r-model-needs-to-be-explained/ Mon, 04 Dec 2017 00:00:00 +0000 /post/2017/12/04/explainers-when-your-r-model-needs-to-be-explained/ When working on Machine Learning for classification and predictive models we tend to use the well known packages as randomforest, caret, xgboost, gbm and such. The issue is when the user needs explanation about how we get these results. The easy part is to explain a Tree. Pretty neat is you use the FFTrees package, that gives a cool output that needs a very short explanation. When working with more complex algorithms we have no such option so you must work hard to understand what a SVM or GBM does. /post/2017/11/21/geocoding-with-r-and-mapzen/ Tue, 21 Nov 2017 00:00:00 +0000 /post/2017/11/21/geocoding-with-r-and-mapzen/ The most widely used service for Geocoding is Google Maps, but you’d hit the limits early in your project if you don’t have a paid account. I often place data on a map, so I need precise geocoding. It’s not unlikely you had to test some other geocodig options ‘til you get the budget for a Google Maps paid account. I’ve been in that situation myself a couple of times. /post/2017/11/20/atom-for-r-keeping-settings-in-sync/ Mon, 20 Nov 2017 00:00:00 +0000 /post/2017/11/20/atom-for-r-keeping-settings-in-sync/ Last week I was working on some projects involving Posix and R code. I tend to use Atom Editor for such code instead of Rstudio, and sometimes I think the one an only valid contender against Rstudio is Atom Editor. With Hydrogen, Linter, and R language Syntax/autocomplete addins is a full featured editor. Plus, dark mode rocks! One of the most useful features Atom has and Rstudio hasn’t is setting syncs via Github. /post/2017/11/19/my-docker-images-for-r/ Sun, 19 Nov 2017 00:00:00 +0000 /post/2017/11/19/my-docker-images-for-r/ This month I reached the dreaded “2 hours limit” at Docker Hub on building my Docker images. The images are built but the message shown is “build cancelled”. You can still use it, but on the front desk, seems that I’m not doing a good job. Clearly, something to fix. So entered a hard refactoring week trying to make my images available again with no error message. My first attempt was to split TidyViz into several images, one for GIS, one for stats and ML, the other for Rmarkdown and reporting…the issue is I ended with a nightmare of unmantainable images with many dependencies not satisfied and as a result, several unusable images. /post/2017/11/04/easy-library-installing-and-loading/ Sat, 04 Nov 2017 00:00:00 +0000 /post/2017/11/04/easy-library-installing-and-loading/ I tend to use package managers for library load and management: pacman https://cran.r-project.org/web/packages/pacman/vignettes/Introduction_to_pacman.html With pacman::p_load() instead of 5 lines of code to load 5 common packages, like this: library(dplyr) library(tidyr) library(Hmisc) library(magrittr) library(janitor) You can write one line pacman::p_load(dplyr, tidyr, Hmisc, janitor, magrittr) if the package is not available on the system, it will first install it (through install.packages), and only then try to load it again. Same as library installr /post/2017/10/27/rocker-docker-for-rstats/ Fri, 27 Oct 2017 00:00:00 +0000 /post/2017/10/27/rocker-docker-for-rstats/ Last Tuesday (October 24th) I was at Madrid R User Group to give a tech talk about using Docker for automating our setup with Rstudio. Using Rocker images for easy and quick deployment. Here, the slides hoping someone will find it useful. https://github.com/verajosemanuel/slides_Docker_for_R_presentation/blob/master/Docker_for_Rstats-EN.Rmd /post/2017/10/16/spanish-stopwords-for-tidytext-package/ Mon, 16 Oct 2017 00:00:00 +0000 /post/2017/10/16/spanish-stopwords-for-tidytext-package/ A clever friend of mine says: the main programming language, is not Java or Python but English. So, when you’re searching for help, you’ll be much successful if searching in english. The main docs are in english, and if the topic is a bit obscure or not mainstream, you’ll be lucky if there’s some documents even in english. Searching for spanish documentation can lead you to bad translation if unlucky, or none at all at worst. /post/2017/08/29/selecting-packages-while-coding/ Tue, 29 Aug 2017 00:00:00 +0000 /post/2017/08/29/selecting-packages-while-coding/ Sometimes, you must choose a package to achieve some results while you’re coding. Let’s say, you’re wrangling data and noticed you need a quick way of getting the mode of a vector. Base R does not provides such a method/function, but we can get a package to make our life easier (mode is not a hard working function you can code, but let’s use it for the sake of the hint described here). /post/2017/08/12/calculating-date-diffs-per-grouped-events-with-dplyr/ Sat, 12 Aug 2017 00:00:00 +0000 /post/2017/08/12/calculating-date-diffs-per-grouped-events-with-dplyr/ Sometimes you’d need to calculate rolling date diffs from a vector or a data frame when you’re manipulating repeated events from different categories. It’s very easy and I’ll show you three different methods. Using dplyr and pipes, easy peasy! Take into account that lubridate has three classes or different ways, to define time spans. Duration: a span of time in seconds with no start time Interval: same as duration, but having an associated start time Period: time span between units larger than seconds (days, months, weeks, years…) We’ll be using three methods here: simple substraction base difftime lubridate %–% Periods operator library(magrittr) library(dplyr) library(lubridate) event <- rep(letters[1:4],3) event. /post/2017/08/07/how-to-navigate-the-more-than-11k-r-packages/ Mon, 07 Aug 2017 00:00:00 +0000 /post/2017/08/07/how-to-navigate-the-more-than-11k-r-packages/ One of the most powerful features of R is the vast package echosystem. But there’s more than 11.000 packages just in CRAN, and a thousands more if you take GIT and Bioconductor into account, so the issue is to find the proper package for your needs. There’s a lot of sites to search for help if you need a hand when programming in R to choose a package. Here, a list of some of them. /post/2017/07/31/docker-for-r-my-dockerfile/ Mon, 31 Jul 2017 00:00:00 +0000 /post/2017/07/31/docker-for-r-my-dockerfile/ Sometimes, you need to replace your hard disk, upgrade your o.s. or recently bought a new laptop. When this time comes, it’s a real pain to get your Rstudio up and running with all you favourite packages and settings. Solomon Hykes, gave birth to Docker thinking on apps unattached to a machine, readily available, working the same no matter how you interface with them. i’m not going to explain what Docker is ’cause there’s a lot of information to read on the internet. /post/2017/07/27/janitor-a-good-r-package-for-data-wrangling/ Thu, 27 Jul 2017 00:00:00 +0000 /post/2017/07/27/janitor-a-good-r-package-for-data-wrangling/ We all know the many hours spent cleaning and wrangling data. Sometimes I think my actual job is not “Data Scientist” but “Data Cleaner”. Data, as you surely know, is not often in the best shape, so for many people like me, one of the most appreciated tools is the one that makes cleaning easy. When I started working on what now is called data science I used python a lot for data wrangling but R is the “de facto best tool” for cleaning and munging right now due to the many good packages developed for that purpose. /post/2017/07/23/processing-mail-using-r/ Sun, 23 Jul 2017 00:00:00 +0000 /post/2017/07/23/processing-mail-using-r/ After a long time seeking for R packages to connect to a remote mailbox (not Gmail), I’ve had to admit that there’s no such feature right now in R. Tested a pair of Python scripts but too much convoluted to my needs. It’s an incredible issue that having more than 10 thousand packages available on CRAN, none of them was suitable for my needs. Some readers suggested tm.plugin.mail added to tm package, but my core feature is to connect to a remote mailbox. /post/2017/07/20/file-choose-empowering-users/ Thu, 20 Jul 2017 00:00:00 +0000 /post/2017/07/20/file-choose-empowering-users/ Sometimes when sharing your analysis, via Rmarkdown or the brand new NoteBook, the data file is located at the user’s computer, making unusable the default path from your own pc. But it’s easy to allow users to choose the data file to be imported and perform code execution using this file. it’s an easy command: myFile <- file.choose() a dialog will be opened for the user to browse file system and choose the file. /post/2017/07/19/3d-chart-using-rgl-library/ Wed, 19 Jul 2017 00:00:00 +0000 /post/2017/07/19/3d-chart-using-rgl-library/ Iris is one of the most used data set in R. We’ve seen it in many formats, and broadly used for data manipulation. You could say that there’s nothing new to learn if someone use Iris. I was wondering if there’s something new, something never done to it before. Just a couple of clicks and found a way to have an interactive animated 3D chart. Just use the mouse pointer to drag and change perspective and move cube. /post/2017/07/18/managing-installation-and-packages-in-r-installr-pacman/ Tue, 18 Jul 2017 00:00:00 +0000 /post/2017/07/18/managing-installation-and-packages-in-r-installr-pacman/ Mentioned yesterday the useful library pacman , so a brief comment about it is due. But I’m going to recommend installR for managing updates first (packages and R itself). With a console command, and just for windows users, it’s so easy to keep your Rstudio updated. install.packages(“installr”) library(installr) # with update command a wizard is opened, checking version and updating accordingly updateR() link to InstallR vignette I recommend to have a look at the vignette for many useful commands regarding this package. /post/2017/07/17/easy-rstudio-add-in-management-with-addinslist-package/ Mon, 17 Jul 2017 00:00:00 +0000 /post/2017/07/17/easy-rstudio-add-in-management-with-addinslist-package/ When started using Rstudio (some time ago) I had been wondering where the Rstudio Addins were located. There’s a menu option, but It was empty on my machine. Seeking an easy way to install some addins I’ve found “addinslist” install.packages (“addinslist”) Or using pacman: pacman::p_load(addinslist) After installation, a wizard is available for everybody to choose among a lot of add-ins for Rstudio. Some of them extremely useful. You can watch videos of some of them running. /post/2017/07/17/fourfoldplot-a-prettier-confusion-matrix-in-base-r/ Mon, 17 Jul 2017 00:00:00 +0000 /post/2017/07/17/fourfoldplot-a-prettier-confusion-matrix-in-base-r/ Working with R, it’s high likely you end with a table regarding to dichotomous variables in your datasets no matter the specific project you’re involved in. I like the ConfusionMatrix function from caret package, that calculates a cross-tabulation of observed and predicted classes. Here an example from caret vignette. library(caret) ## 2 class example lvs <- c(“normal”, “abnormal”) truth <- factor(rep(lvs, times = c(86, 258)), levels = rev(lvs)) pred <- factor( c(rep(lvs, times = c(54, 32)), rep(lvs, times = c(27, 231))), levels = rev(lvs)) xtab <- table(pred, truth) cm <- confusionMatrix(pred, truth) cm$table The confusion matrix renders as follows: /post/2017/07/15/some-essential-r-packages/ Sat, 15 Jul 2017 00:00:00 +0000 /post/2017/07/15/some-essential-r-packages/ For me, there’s a bunch of packages considered as “essential” ‘cause in the end, sooner or later I use them in any project that involved opening the RStudio regardless of the type of issue that I’m trying to address. At some point in the process I need to present preliminary data (knitr, rmarkdown, janitor), display graphs (ggplot2, ggthemes, ggThemeAssist, gridBase, grid, corrplot) or simply manipulate data (dplyr, tidyr, stringr). So I use the same snippet to load them at the beginning of any script. /post/2017/07/12/first-thing-first-thank-you-all/ Wed, 12 Jul 2017 00:00:00 +0000 /post/2017/07/12/first-thing-first-thank-you-all/ First things first. Question of etiquette, when starting a blog like this, mainly focused on Data Science with R, is to acknowledge all the people and teams that made possible that I’m writing this today. People from CRAN, Rstudio and the R consortium, for pushing forward the best language in the world for data analytics. A language I love from the first day. Today, many years ago, i still remember my goose bumps when writing R code for the first time. /about/ Mon, 01 Jan 0001 00:00:00 +0000 /about/ Started my career long time ago designing data architectures based on SQL Server and Oracle. Did some CRUD programming and database management applications with ASP and PHP, among other languages. Designing data architectures and modeling relational databases for complex applications lead to Certifications on Business Intelligence with Oracle BI an Microsoft BI. Early on my path discovered Linux and Open Source Community, and feeling very passionate about it, changed my path and never looked back. /vitae/ Mon, 01 Jan 0001 00:00:00 +0000 /vitae/ Started my career long time ago designing data architectures based on SQL Server and Oracle. Did some CRUD programming and database management applications with ASP and PHP, among other languages. Designing data architectures and modeling relational databases for complex applications lead to Certifications on Business Intelligence with Oracle BI an Microsoft BI. Early on my path discovered Linux and Open Source Community, and feeling very passionate about it, changed my path and never looked back.