Reading entire URL content is really easy using R

In my good old days, reading the entire content of a website is not easy. The process of web scraping and getting the required data requires lots of programming and a few tools. A friend of mine even developed and sold the tool which he called it (during the development) as myrobot. He developed using PHP.

Now, it is much easier and one of the many ways is using R.

Here are the steps (which requires you to write ONLY two lines of code)

1. Connect to the website using URL command
   con <- url ([the website url], “r”)
2. Then, read the website
   x <- readLines(con)
3. Do whatever you wish with the data. In this example, I print out the head of the website and also copy the whole content to a file.
   head(x)
   dput(x, “readFromUrlExample.html”)

There you go.

The sample source code can be retrieved at 

https://github.com/masteramuk/LearnR-Coursera/blob/master/sample-ReadFromUrl.R

Read More

Solving Committing Issue between R Studio and Github

---

Solving Committing Issue between R Studio and Github

In the normal development process, you will create a repo (the repo in this article is located at Github), followed by the cloning process or download as full directory into your localhost. It is much easier and straightforward. There won’t be any issues especially if your scrum master or release manager is a well trained person in handling branching, merging, and releasing code using git.

---

However, in most cases, especially for a full-stack developer who did everything on its own, you may encounter an issue if:

1. You created a project in your localhost first using R Studio and set Git as your SVN through your project setting
2. Then you created the repo at the GitHub
3. Finally, upon ready, you run command to sync with your GitHub

The following is the command that you use/execute and the result of running the command:

% git remote add origin [your GitHub report url]
% git pull origin master
warning: no common commits
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), done.
From [your GitHub report url]
* branch master -> FETCH_HEAD
* [new branch] master -> origin/master
fatal: refusing to merge unrelated histories

and you see the last sentence .. ERROR

---

Then, based on google, you followed with the following command

% git push -u origin master

and you get the following response (or similar)

To [your GitHub report url]
! [rejected] master -> master (non-fast-forward)
error: failed to push some refs to ‘[your GitHub report url]'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: ‘git pull …’) before pushing again.
hint: See the ‘Note about fast-forwards’ in ‘git push — help’ for details.

Next, you try to pull again to get the latest branch based on the previous error by running the command to pull again

% git pull origin master

And the result is still not promising 

From [your GitHub report url]
* branch master -> FETCH_HEAD
fatal: refusing to merge unrelated histories

What are you missing or wrongly done? I won’t be able to tell you the missing or wrong steps, but I’m sharing your step to overcoming the problem.

---

STEPS

1. Go to you localhost directory where you created the project
2. In that directory, you should find a file name .gitignore & folder .git
3. Delete both by running rm -fr (if you are using windows, the command might be different)
4. Next, init your project file again by running the command git init. You shall see the following message appear after executing the command — Initialized empty Git repository in [your project path]
5. Followed by adding the remote repo by running the command git remote add origin [your GitHub repo url]. 
6. The followed by git add . (make sure there is ‘.’ at the end of the command). It tells the git to add all directories in the remote repo to your local.
7. Followed by git pull origin master. If succeed, you shall be able to see the following result:
   remote: Enumerating objects: 3, done.
   remote: Counting objects: 100% (3/3), done.
   remote: Compressing objects: 100% (2/2), done.
   remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
   Unpacking objects: 100% (3/3), done.
   From [your GitHub repo url]
   * branch master -> FETCH_HEAD
   * [new branch] master -> origin/master
8. Finally, run git push -u origin master to verify again. You shall see the following result to indicate it is successfully integrated between your local repo and your Github repo and R Studio shall be able to interact perfectly with GitHub.
   Branch ‘master’ set up to track remote branch ‘master’ from ‘origin’. Everything up-to-date

Once you have done all the steps, go ahead to your R Studio and perform Stage -> Commit -> Commit Message -> Push to complete the process. Refresh your Github page and you shall see all of your local files at your GitHub repo.

If you find this useful, you can buy me a coffee :) @ https://www.buymeacoffee.com/masteramuk

Read More

Parameter in Tableau - Using available Dimension with additional 'All' for all data

I am writing (and still writing as of this article) a manual to assist me in teaching Tableau. One of the sections is Parameter. There are many examples and guidance provided by Tableau.

You may read/view the following articles/videos for the guide on parameter.

1. https://interworks.com/blog/interworks/2012/03/26/how-to-create-and-use-parameters-in-tableau
2. https://help.tableau.com/current/pro/desktop/en-us/parameters_create.htm
3. https://help.tableau.com/current/pro/desktop/en-us/changing-views-using-parameters.htm
4. https://www.youtube.com/watch?v=CrfEJ24FWpQ
5. https://www.youtube.com/watch?v=rJsaezoTVAE
6. https://www.youtube.com/watch?v=opfVV1maNVw

The only problem I faced was to have an 'All' data if the user did not select any or wishes to see all data. None of them help. I tried using Action and Filters, yet it only shows based on selection and all show nothing.

That is until I found this solution.

IFNULL([FilterField],’Null’) = IF [Paramter] != ‘All’ THEN [Parameter] ELSE IFNULL([FilterField],’Null’) END

In a more readable, it will be like this

IFNULL([FilterField],’Null’) = (

     IF [Parameter] != ‘All’ THEN 

          [Parameter] 

     ELSE 

          IFNULL([FilterField],’Null’) 

     END

)

Step to:

1. Create your parameter - In this case, you use any Dimension as the Parameter by choosing a List and choose the list by changing the Fixed section and click on Add Values From.
2. Don't forget to add 'All' as one additional list
3. Drag the Dimension that you used as the parameter from the Dimensions Card into Filters Card.
4. In the Filter option, choose Condition and choose by Formula, and insert the above solution into the formula. 
5. Don't forget to change the [FilterField] to the Dimension used for the filter and the [Parameter] to your created parameter. 
6. Do ensure that 'All' is the same as your definition in the parameter (the character case and spelling is equal)
7. And that shall do it. You are ready to go.

Thanks to the person posted here: https://tarsolutions.co.uk/blog/tableau-add-an-all-to-a-parameter/

There are also an alternative solution to this and posted here: http://www.vizwiz.com/2012/09/tableau-tip-adding-all-filter-option-to.html

Read More

Data Analysis - Use it !!

I was reading many developer's site and chat (telegram and whatsapp) when the government stated that they are looking for an app similar to Singapore apps to track the close contact of the Covid-19 positive patient.

In Singapore, they are using a technology which I presume is Bluetooth to ping close contact within the radiant of the tech and capture necessary data which then used to determine the contact and request them to perform screening. Here are some of the news:

1. https://www.thestar.com.my/tech/tech-news/2020/03/20/covid-19-singapore-launches-contact-tracing-mobile-app-to-track-coronavirus-infections
2. https://www.pymnts.com/coronavirus/2020/app-lets-singapore-track-virus-patients-movements/
3. https://www.nst.com.my/news/nation/2020/03/578445/smartphone-app-track-contacts-covid-19-patients
4. https://asia.nikkei.com/Spotlight/Coronavirus/Singapore-urges-citizens-to-sign-up-for-COVID-19-tracking-app

And the apps is available on Google Play and Apple Store

1. https://play.google.com/store/apps/details?id=sg.gov.tech.bluetrace&hl=en
2. https://apps.apple.com/sg/app/tracetogether/id1498276074

And as this article is written, there are many groups including international are coming with various hackathons for apps that can be used to track all COVID-19 patients and their close contact.

In Malaysia, since the announcement, many had gather groups to develop apps.

From my perspective, why must we reinvent the wheel? Why we need to develop many apps when we already have few that are potentially be used for it. 

For instance, D'scover by Favioriot was developed for a user to explore whatever the user likes but also close contact that uses the apps. I believed they can just tweak the apps to get close contact for COVID-19 and it is much faster than building and testing new apps. (by the way, this is not promoting them and I don't gain anything from it :))

Not just that apps, people have been using Google Maps, Waze, Grab, etc. All these apps collected millions of data and one of these data is people's location and whereabouts. On top of that, all telcos do have their customer's data location and track their movement. I attended the Big Data conference by Bigit a few years ago where one of the telcos presented their data analysis. They have shown the heatmap of their user and based on the communication tower.

I even had a discussion with a few telcos when they approach us (my previous company) to provide their services and wish for data sharing. I do request to have a set of data of their customer whereabouts too to ensure we can provide efficient services at the moment the customer approaches our station or hub, or at the time they are supposed to do so.

These data can be utilized to find close contact with COVID-19 patients. From these data, we know where they go, their ride and whom the came across with or pass by. Of course, these data are secured by all those companies for customer's safety and PDPA compliance. But, for the sake of government and to combat COVID-19, they can request minimal information limited to the phone number to call the respective COVID-19 contact. 

You just need a group of data scientists and data engineers to focus on the massaging and provide the relevant info to the government fast and secure. That's all :)

Don't REINVENT the wheel. Used It and MAXIMIZE the POTENTIAL.

* My personal opinion based on experience. Agreed to disagree :)

Read More

Running Pentaho Data Integration in Mac OS Catalina version 10.15 and above

---

If you haven’t download the application, you may access here https://community.hitachivantara.com/s/article/downloads

Running Pentaho Data Integration @ Spoon in Windows or Linux should be straight-forward. Either click on spoon.bat or spoon.sh or the Data Integration app icon.

Running Data Integration from the install folder

However, for Mac OS, especially with the latest version Catalina which only allowed certified and trusted 64-bit application to run, running Data Integration will be troublesome despite the security for the app was disabled and allowed for the application to run. 

Some of the guidance on enable security and allow the untrusted app to run are as follow:

1. https://edpflager.com/?p=3571
2. https://andres.jaimes.net/1388/running-pentaho-spoon-on-mac-osx/

I have tried both and after enabling the apps, click on the Data Integration icon still does not work and the application still did not run. 

---

Lastly, I had to try to run it through the terminal. For OSX Catalina and above, instead of normal bash, Apple brings in zsh and the behavior is totally different. Plus, if you are a developer and install many Java JDK, running the Pentaho Data Integration will not be as running the normal command. Here is the step to run on the terminal and it works for all OSX.

1. Open up your terminal

2. Navigate to your install path (where you install or unzip the data integration file)

3. Run ./spoon.sh (for bash or old scripting, running spoon.sh shall work without ‘./’)

4. If you run into an error such as JDK or java runtime error like below, do not panic. This maybe your current java JDK is set to be higher than supported by the application.

(base) MyMek @ MyEpal data-integration % ./spoon.sh

OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0

-Djava.endorsed.dirs=/Users/masteramuk/Documents/Apps/data-integration/system/karaf/lib/endorsed is not supported. Endorsed standards and standalone APIs

in modular form will be supported via the concept of upgradeable modules.

Error: Could not create the Java Virtual Machine.

Error: A fatal exception has occurred. Program will exit.

5. First, check your JDK version. Open a new terminal as the command need to be executed from the base (unless you set the JDK in your profile which may cause problem to run multiple JDK later). Run command java -version. You shall have similar to below. In my laptop, the current JDK version is set to JDK version 10.

(base) MyMek @ MyEpal data-integration % java -version

openjdk version "10.0.2" 2018-07-17

OpenJDK Runtime Environment AdoptOpenJDK (build 10.0.2+13)

OpenJDK 64-Bit Server VM AdoptOpenJDK (build 10.0.2+13, mixed mode)

6. Make sure you have multiple java JDK installed if you need to use the existing JDK version for your ‘other’ development. Refer https://www.devdungeon.com/content/install-multiple-jdk-windows-java-development to install multiple JDK. Refer here https://www.jenv.be/ to install jenv command tool.

7. For my laptop, I have a few versions of JDK and using jenv, I can set the selected JDK for global or local (only applied to the folder where we run jenv local command). Run jenv version to check on current JDK and jenv versions to list all JDK install. 

(base) MyMek @ MyEpal data-integration % jenv versions

  system

  1.8

  1.8.0.232

* 10.0 (set by /Users/masteramuk/Documents/Apps/data-integration/.java-version)

  10.0.2

  13.0

  13.0.1

  9

  openjdk64-1.8.0.232

  openjdk64-10.0.2

  openjdk64-13.0.1

  openjdk64-9

As of this manual written, Pentaho Data Integration @ Spoon supports up to JDK 1.8.

8. Use jenv to change the jdk to 1.9. Run command jenv local [JDK number]. In this example, I execute jenv local 1.8.0.232.
9. Finally, run again ./spoon.sh. The application shall run.

Running

Opening the application

The main screen

Read More

List of available data set for Data Scientist

I am exploring on data related to 2019nCoV and found the following list of dataset that can be used for data scientist, data scientist to be, student, etc to explore. Check it out

1. https://data.world/
2. https://quantumstat.com/dataset/dataset.html
3. https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType_s=Nucleotide&VirusLineage_ss=Wuhan%20seafood%20market%20pneumonia%20virus,%20taxid:2697049
4. https://www.ncbi.nlm.nih.gov/genbank/2019-ncov-seqs/?utm_campaign=wuhan_nCoV&utm_source=twitter&utm_medium=referral
5. https://www.ncbi.nlm.nih.gov/nuccore/MN908947,MN970003,MN970004,MN938384,MN938385,MN938386,MN938387,MN938388,MN938389,MN938390,MN975262,MN975263,MN975264,MN975265,MN975266,MN975267,MN975268,MN985325,MN988713,MN994467,MN994468,MN997409,MN988668,MN988669
6. https://researchguides.uic.edu/c.php?g=252253&p=1683071
7. https://archive.org/details/2019-search-engine-keyphrases.json

or you can just use google and search the entire internet through the engine
https://datasetsearch.research.google.com/

Read More

Tableau For Beginner

I'll be publishing an ebook on Visualizing using Tableau. To those interested, please PM ya. Here is the front page.

Read More

Big Data - we forgot to clean our data

Recently I have been working with lots of data coming from various business area such as maintenance, financial transaction, etc., and I found an interesting thought from much top management and young leaders whom don't have enough experience handling data from the source up till visualization.

The first thing comes out from their mind was can it be done in a few hours (or some of them thought it was in a blink of the eyes or in split seconds). Normal question was "When can we see the report or chart? Can we see it tomorrow" and the worst I get "I want it to be ready by today before noon".

My first reaction was WTF!! (but I won't say it loud). I will normally negotiate with them as most of them don't know the process and the data that they requested. Most of them are easily attracted/amazed by superb visualization presentation by Visualization Tool's Marketing team. The thought everything can be done easily as those marketing people said. They just forgot that in a business presentation, the data set used by those marketing people are prepared and cleaned before being utilized in the tool. For example, Tableau's presentation will always use Sales data for their sample.

It is true that many of visualization tools nowadays are capable to process and display any kind of data. With certain skills, you can massage and clean your data on the fly. I've done that and I know it can be done.  BUT... surely at a cost which from my perspective, it is no beneficial at all.

Why do I say so?

1. You cannot guarantee that the data you read is 100% clean. You might need to do lots of conversions, data massaging, replacements and calculations. This will definitely incur additional processing power and time during report population. I've come across with many data which either irrelevant, unclean (character in a supposed to be numerical column, date define as string, etc), or contain null values.
2. You may need to perform lots of table joint or union which can cause your report server or tool to be resource hungry.
3. You need to understand the data too. Each column and how it shall be visualize must be understood before you can present it correctly.

That's all from me..Adios

Read More

Lesson learn on R conference (ConfeRence 2018, ADAX Malaysia)

I attended R conferences organized @ ADAX Malaysia (ConfeRence) on October 20, 2018, recently. I do have an interest in AI, ML and Semantics Analysis. It started since 2008 when I further studies focusing on program analysis, specialized on static analysis on C overflow vulnerabilities.

There are key points which I would like to share based on the knowledge shared by local experts.

Ensemble Method

I'm quite new and just heard about this. The presenter shared that there are many methods but common are Bagging, Boosting and Stacking

Bagging

1. Can produce a Discrete or Continuous Result
2. Discrete (Classifications) - voting
3. Continuous - regression -> mean
4. Basically, the data is put into multiple "bags" in a random selection and you train the using one or various model. Finally, you find the mean of the result and produce the outcome from it.

For more understanding, check the video below

Boosting

1. From the result, you will get a significant error and continue the training process on the data until you have no error on the latest bag or based on a defined number of the model required.
2. 2 famous algorithm applied - GBM (Gradient Boosting Machine) & XGBoost - eXtreme Gradient Boosting Algorithm

To understand it, just check out the video below

Stacking (aka Ensemble Learners)

1. In Stacking, you still use various model and numbers of bags.
2. However, in stacking, you will use different algorithms such as KNN, LinReg, Decision Tree, SVM
3. The result of the training on each bag which applied different algorithm will be added together to find the mean of it.

Other methods are also applicable such as Random Forest.

Ensemble method in R -> You can use SuperLearner Package library in R Studio

There are issues with ensemble which the data might be highly accurate when it is trained in development. But when you applied in the actual world, the data may produce a different result. To avoid this (inaccuracy or over-fitting problem), you will need to implement elastic net on all model. 2 known algorithms in elastic net are ridge and lasso.

Text Clustering with R

1. Clustering in R can be calculated by using the algorithm like K-Means, KNN, Hierarchical Clustering, and DBScan.
2. For Text Clustering, you need to measure the distance to enable you to cluster the text. For instance, what is the distance between the word 'wheel' and 'tire'. 
3. Distance normally is measured using an algorithms known as Euclidean D
4. For text clustering, there is a library that can be utilized to measure the distance; Wu-palmer developed by. However, the library only available for Python and not for R yet (as of the information shared on Oct 20, 2018)
5. Available language lexical which can be used to measure distance using the algorithm is WordNet (https://wordnet.princeton.edu/)

Churn Prediction using SNA - Social Network Analysis

2 Library available and shall be used - igraph and R Markdown
Challenges in predicting - how to minimize loss due to algorithm implemented.

Some references:

1. https://en.wikipedia.org/wiki/Ensemble_learning
2. https://en.wikipedia.org/wiki/Cluster_analysis
3. WordNet English (Lexical English Language) https://wordnet.princeton.edu/
4. WordNet Bahasa Melayu (Lexical Malay Language) - http://wn-msa.sourceforge.net/index.eng.html
5. https://en.wikipedia.org/wiki/Semantic_analysis_(linguistics)
6. https://en.wikipedia.org/wiki/Latent_semantic_analysis
7. https://blog.thedigitalgroup.com/words-similarityrelatedness-using-wupalmer-algorithm
8. https://en.wikipedia.org/wiki/Euclidean_distance
9. https://www.researchgate.net/publication/310572659_A_modification_of_Wu_and_Palmer_Semantic_Similarity_Measure

Thanks to the team who share their knowledge. Check it out them at https://www.facebook.com/groups/MalaysiaRUserGroup/

Read More

List of Data Sciences and Machine Learning usefull link

Credit to: Shivam Panchal
As published by Shivam at Linked @ https://www.linkedin.com/pulse/data-science-machine-learning-beginners-path-shivam-panchal/

Platforms:

1. What Is Hadoop? Hadoop Tutorial For Beginners https://youtu.be/n3qnsVFNEIU
2. What is Apache Spark? The big data analytics platform explained http://www.techworld.com.au/article/629920/what-apache-spark-big-data-analytics-platform-explained/
3. Apache Spark Tutorial: ML with PySpark https://www.datacamp.com/community/tutorials/apache-spark-tutorial-machine-learning
4. A Beginner's Guide To Apache Pig https://hortonworks.com/tutorial/beginners-guide-to-apache-pig/
5. Realtime Event Processing in Hadoop with NiFi, Kafka and Storm https://hortonworks.com/tutorial/realtime-event-processing-in-hadoop-with-nifi-kafka-and-storm/

Math:

1. A Deep Dive Into Linear Algebra https://www.khanacademy.org/math/linear-algebra
2. An Introduction to Combinatorics & Graph Theory https://www.whitman.edu/mathematics/cgt_online/cgt.pdf

Tools & Framework:

1. TensorFlow Tutorial – Deep Learning Using TensorFlow https://youtu.be/yX8KuPZCAMo
2. A 6-part introduction to the MXNet API https://becominghuman.ai/an-introduction-to-the-mxnet-api-part-1-848febdcf8ab
3. Keras Tutorial: The Ultimate Beginner's Guide to Deep Learning in Python https://elitedatascience.com/keras-tutorial-deep-learning-in-python

Data Visualization:

1. Building Python Data Apps with Blaze and Bokeh https://youtu.be/1gD9LMqREDs
2. Matplotlib Tutorial: Python Plotting https://www.datacamp.com/community/tutorials/matplotlib-tutorial-python
3. Python Bokeh Tutorial - Creating Interactive Web Visualizations https://youtu.be/Mz1AXUE0nR4

Concepts:

1. Simple Linear Regression https://onlinecourses.science.psu.edu/stat501/node/250
2. Simple and Multiple Linear Regression in Python https://medium.com/towards-data-science/simple-and-multiple-linear-regression-in-python-c928425168f9
3. Linear Regression in R https://www.tutorialspoint.com/r/r_linear_regression.htm
4. An Introduction To Logistic Regression http://ufldl.stanford.edu/tutorial/supervised/LogisticRegression/
5. Building A Logistic Regression in Python, Step by Step by Susan Li https://medium.com/towards-data-science/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8
6. Supervised and Unsupervised Machine Learning Algorithms https://machinelearningmastery.com/supervised-and-unsupervised-machine-learning-algorithms/
7. 6 Easy Steps to Learn Naive Bayes Algorithm (with codes in Python and R) https://www.analyticsvidhya.com/blog/2017/09/naive-bayes-explained/
8. A Tutorial on Support Vector Machines for Pattern Recognition http://www.cs.northwestern.edu/~pardo/courses/eecs349/readings/support_vector_machines4.pdf
9. A Complete Tutorial on Tree Based Modeling from Scratch (in R & Python) https://www.analyticsvidhya.com/blog/2016/04/complete-tutorial-tree-based-modeling-scratch-in-python/

Python:

1. A Complete Tutorial to Learn Data Science with Python from Scratch https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-learn-data-science-python-scratch-2/
2. NumPy Tutorial: Data analysis with Python https://www.dataquest.io/blog/numpy-tutorial-python/
3. Scipy Tutorial: Vectors and Arrays (Linear Algebra) https://www.datacamp.com/community/tutorials/python-scipy-tutorial
4. Python Pandas Tutorial https://www.tutorialspoint.com/python_pandas/
5. Machine Learning with scikit learn Part 1 & 2 https://youtu.be/2kT6QOVSgSghttps://youtu.be/WLYzSas511I

CS:

1. A Thorough Overview of Computational Logic https://www.cs.utexas.edu/users/boyer/acl.pdf

Game Theory:

1. Game Theory - A 3 Part Introduction https://youtu.be/x8gOi7D6QeQ

Statistics:

1. Correlation & causality https://www.khanacademy.org/math/probability/scatterplots-a1/creating-interpreting-scatterplots/v/correlation-and-causality
2. Analysis of variance (ANOVA) https://www.khanacademy.org/math/statistics-probability/analysis-of-variance-anova-library
3. Understanding Hypothesis Tests: Significance Levels (Alpha) and P values in Statistics https://shar.es/1PANrc
4. Characteristics of Good Sample Surveys and Comparative Studies https://onlinecourses.science.psu.edu/stat100/node/3
5. Descriptive and Inferential Statistics https://www.thoughtco.com/differences-in-descriptive-and-inferential-statistics-3126224
6. Intro to Probability Theory https://youtu.be/f9XFM8YLccg
7. Introduction to Conditional Probability & Bayes theorem for data science https://www.analyticsvidhya.com/blog/2017/03/conditional-probability-bayes-theorem/
8. Central limit theorem https://www.khanacademy.org/math/statistics-probability/sampling-distributions-library/sample-means/v/central-limit-theorem

Read More