Reading entire URL content is really easy using R

In my good old days, reading the entire content of a website is not easy. The process of web scraping and getting the required data requires lots of programming and a few tools. A friend of mine even developed and sold the tool which he called it (during the development) as myrobot. He developed using PHP.

Now, it is much easier and one of the many ways is using R.

Here are the steps (which requires you to write ONLY two lines of code)

1. Connect to the website using URL command
   con <- url ([the website url], “r”)
2. Then, read the website
   x <- readLines(con)
3. Do whatever you wish with the data. In this example, I print out the head of the website and also copy the whole content to a file.
   head(x)
   dput(x, “readFromUrlExample.html”)

There you go.

The sample source code can be retrieved at 

https://github.com/masteramuk/LearnR-Coursera/blob/master/sample-ReadFromUrl.R

Read More

Solving Committing Issue between R Studio and Github

---

Solving Committing Issue between R Studio and Github

In the normal development process, you will create a repo (the repo in this article is located at Github), followed by the cloning process or download as full directory into your localhost. It is much easier and straightforward. There won’t be any issues especially if your scrum master or release manager is a well trained person in handling branching, merging, and releasing code using git.

---

However, in most cases, especially for a full-stack developer who did everything on its own, you may encounter an issue if:

1. You created a project in your localhost first using R Studio and set Git as your SVN through your project setting
2. Then you created the repo at the GitHub
3. Finally, upon ready, you run command to sync with your GitHub

The following is the command that you use/execute and the result of running the command:

% git remote add origin [your GitHub report url]
% git pull origin master
warning: no common commits
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), done.
From [your GitHub report url]
* branch master -> FETCH_HEAD
* [new branch] master -> origin/master
fatal: refusing to merge unrelated histories

and you see the last sentence .. ERROR

---

Then, based on google, you followed with the following command

% git push -u origin master

and you get the following response (or similar)

To [your GitHub report url]
! [rejected] master -> master (non-fast-forward)
error: failed to push some refs to ‘[your GitHub report url]'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: ‘git pull …’) before pushing again.
hint: See the ‘Note about fast-forwards’ in ‘git push — help’ for details.

Next, you try to pull again to get the latest branch based on the previous error by running the command to pull again

% git pull origin master

And the result is still not promising 

From [your GitHub report url]
* branch master -> FETCH_HEAD
fatal: refusing to merge unrelated histories

What are you missing or wrongly done? I won’t be able to tell you the missing or wrong steps, but I’m sharing your step to overcoming the problem.

---

STEPS

1. Go to you localhost directory where you created the project
2. In that directory, you should find a file name .gitignore & folder .git
3. Delete both by running rm -fr (if you are using windows, the command might be different)
4. Next, init your project file again by running the command git init. You shall see the following message appear after executing the command — Initialized empty Git repository in [your project path]
5. Followed by adding the remote repo by running the command git remote add origin [your GitHub repo url]. 
6. The followed by git add . (make sure there is ‘.’ at the end of the command). It tells the git to add all directories in the remote repo to your local.
7. Followed by git pull origin master. If succeed, you shall be able to see the following result:
   remote: Enumerating objects: 3, done.
   remote: Counting objects: 100% (3/3), done.
   remote: Compressing objects: 100% (2/2), done.
   remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
   Unpacking objects: 100% (3/3), done.
   From [your GitHub repo url]
   * branch master -> FETCH_HEAD
   * [new branch] master -> origin/master
8. Finally, run git push -u origin master to verify again. You shall see the following result to indicate it is successfully integrated between your local repo and your Github repo and R Studio shall be able to interact perfectly with GitHub.
   Branch ‘master’ set up to track remote branch ‘master’ from ‘origin’. Everything up-to-date

Once you have done all the steps, go ahead to your R Studio and perform Stage -> Commit -> Commit Message -> Push to complete the process. Refresh your Github page and you shall see all of your local files at your GitHub repo.

If you find this useful, you can buy me a coffee :) @ https://www.buymeacoffee.com/masteramuk

Read More

Tableau Public - Apple Mobility Data and Dark Mode

I've been working on 2 visualizations. 1 is based on Apply Mobility Data and the other is based on edited Sample Superstore.

Check it out here.

Mobility Data based on Apple's mobility data.

Grid on Dark Mode based on Tableau Sample Superstore (edited version)

All visual is available at https://public.tableau.com/profile/nurul.haszeli.ahmad#!/

All dataset is available at data.world @ https://data.world/haszeliahmad/data-analysis

Read More

List of available data set for Data Scientist

I am exploring on data related to 2019nCoV and found the following list of dataset that can be used for data scientist, data scientist to be, student, etc to explore. Check it out

1. https://data.world/
2. https://quantumstat.com/dataset/dataset.html
3. https://www.ncbi.nlm.nih.gov/labs/virus/vssi/#/virus?SeqType_s=Nucleotide&VirusLineage_ss=Wuhan%20seafood%20market%20pneumonia%20virus,%20taxid:2697049
4. https://www.ncbi.nlm.nih.gov/genbank/2019-ncov-seqs/?utm_campaign=wuhan_nCoV&utm_source=twitter&utm_medium=referral
5. https://www.ncbi.nlm.nih.gov/nuccore/MN908947,MN970003,MN970004,MN938384,MN938385,MN938386,MN938387,MN938388,MN938389,MN938390,MN975262,MN975263,MN975264,MN975265,MN975266,MN975267,MN975268,MN985325,MN988713,MN994467,MN994468,MN997409,MN988668,MN988669
6. https://researchguides.uic.edu/c.php?g=252253&p=1683071
7. https://archive.org/details/2019-search-engine-keyphrases.json

or you can just use google and search the entire internet through the engine
https://datasetsearch.research.google.com/

Read More

Tableau For Beginner

I'll be publishing an ebook on Visualizing using Tableau. To those interested, please PM ya. Here is the front page.

Read More

Improving JP with Improvised Prediction Model

Yesterday I wrote on low ridership (read here), of which out of many factors, information availability for journey planning contributes 12% from overall factors. However, the value is based on a survey on one location; that is Penang. Nonetheless, I believed, information availability is the key importance for a smooth journey planning, no matter of the services used or the impact to ridership. The reason for this is based on comments in Google Play for various Journey Planner such as Moovit, Transit Apps, SWIVL, SITS, etc, whereby many user stated their frustration on the accuracy of information displayed by the apps.

Beside information availability being a vital role for riders to plan their journey and services to use, based on the articles referred to in the post, the information must be also reliable, accurate and at real-time (or at least near real-time). If you received an information that was accurate a few minutes ago, there is a probability of the information to be inaccurate at the time of view or receive resulting in the inaccurate plan and action.

How to improve information accuracy, no matter how and when the information arrives at the user?
I did a quick review too on the following articles:

1. https://www.papercast.com/insights/predict-accurate-bus-arrival-journey-times/
2. https://core.ac.uk/download/pdf/82293981.pdf
3. https://escholarship.org/uc/item/51t364vz
4. http://gamma.cs.unc.edu/TROUTE/
5. https://www.researchgate.net/publication/274028208_Multimodal_Public_Transit_Trip_Planner_with_Real-Time_Transit_Data
6. https://repositorio-aberto.up.pt/bitstream/10216/6817/2/26915.pdf
7. https://jungleworks.com/predicting-accurate-arrival-time/
8. https://www.researchgate.net/publication/332342499_Survey_of_ETA_prediction_methods_in_public_transport_networks
9. https://dl.acm.org/citation.cfm?id=3219819.3219874
10. https://pdfs.semanticscholar.org/8c95/f20cd049e5f0d35466544958631e3e10c258.pdf
11. https://www.papercast.com/wp-content/uploads/2017/06/Papercast_A4_Better-ETA_2017.pdf
12. https://datascience.stackexchange.com/questions/10301/how-to-predict-eta-using-regression
13. https://ieeexplore.ieee.org/abstract/document/1212964
14. https://www.tandfonline.com/doi/abs/10.1080/15472450600981009
15. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8667.2004.00363.x
16. https://journals.sagepub.com/doi/abs/10.3141/1666-12
17. https://link.springer.com/chapter/10.1007/978-981-13-3393-4_29
18. https://patents.google.com/patent/US10254119B2/en
19. https://arxiv.org/abs/1904.05037
20. https://patents.google.com/patent/US20190130260A1/en
21. https://www.tandfonline.com/doi/abs/10.1080/19427867.2017.1366120
22. https://link.springer.com/article/10.1007/s12652-019-01198-1
23. https://arxiv.org/abs/1904.03444
24. https://patents.google.com/patent/US20190051154A1/en
25. https://ieeexplore.ieee.org/abstract/document/8691701

Based on the articles above, below is the gist of the findings:

1. Reliable and accurate information at real-time (or near real-time) is critical for smooth journey planning
2. Recent technology (BDA, AI, ML & IoT) has resulted in many new algorithms to predict accurate ETA to be used in JP
3. The most recent is KNN which requires lots of historical data and real-time tracking for accurate prediction

Recommendation:

1. To research and try-n-error all the algorithms to find the best algorithm to predict accurate ETA to be used in the Malaysian environment
2. To define the best algorithm based on time of request (peak or non-peak)
3. To come out with a new algorithm and flow to ensure the ETA for JP is 95% accuracy during non-peak and 90% accuracy during peak.

Read More