My experience on my daily works... helping others ease each other

Showing posts with label Data Integration. Show all posts
Showing posts with label Data Integration. Show all posts

Friday, May 29, 2020

Solving Committing Issue between R Studio and Github


Solving Committing Issue between R Studio and Github

In the normal development process, you will create a repo (the repo in this article is located at Github), followed by the cloning process or download as full directory into your localhost. It is much easier and straightforward. There won’t be any issues especially if your scrum master or release manager is a well trained person in handling branching, merging, and releasing code using git.


However, in most cases, especially for a full-stack developer who did everything on its own, you may encounter an issue if:

  1. You created a project in your localhost first using R Studio and set Git as your SVN through your project setting
  2. Then you created the repo at the GitHub
  3. Finally, upon ready, you run command to sync with your GitHub

The following is the command that you use/execute and the result of running the command:

% git remote add origin [your GitHub report url]
% git pull origin master
warning: no common commits
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), done.
From [your GitHub report url]
* branch master -> FETCH_HEAD
* [new branch] master -> origin/master
fatal: refusing to merge unrelated histories

and you see the last sentence .. ERROR


Then, based on google, you followed with the following command

% git push -u origin master

and you get the following response (or similar)

To [your GitHub report url]
! [rejected] master -> master (non-fast-forward)
error: failed to push some refs to ‘
[your GitHub report url]'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: ‘git pull …’) before pushing again.
hint: See the ‘Note about fast-forwards’ in ‘git push — help’ for details.

Next, you try to pull again to get the latest branch based on the previous error by running the command to pull again

% git pull origin master

And the result is still not promising 
From [your GitHub report url]
* branch master -> FETCH_HEAD
fatal: refusing to merge unrelated histories

What are you missing or wrongly done? I won’t be able to tell you the missing or wrong steps, but I’m sharing your step to overcoming the problem.


STEPS

  1. Go to you localhost directory where you created the project
  2. In that directory, you should find a file name .gitignore & folder .git
  3. Delete both by running rm -fr (if you are using windows, the command might be different)
  4. Next, init your project file again by running the command git init. You shall see the following message appear after executing the command — Initialized empty Git repository in [your project path]
  5. Followed by adding the remote repo by running the command git remote add origin [your GitHub repo url]
  6. The followed by git add . (make sure there is ‘.’ at the end of the command). It tells the git to add all directories in the remote repo to your local.
  7. Followed by git pull origin master. If succeed, you shall be able to see the following result:
    remote: Enumerating objects: 3, done.
    remote: Counting objects: 100% (3/3), done.
    remote: Compressing objects: 100% (2/2), done.
    remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
    Unpacking objects: 100% (3/3), done.
    From [your GitHub repo url]
    * branch master -> FETCH_HEAD
    * [new branch] master -> origin/master
  8. Finally, run git push -u origin master to verify again. You shall see the following result to indicate it is successfully integrated between your local repo and your Github repo and R Studio shall be able to interact perfectly with GitHub.
    Branch ‘master’ set up to track remote branch ‘master’ from ‘origin’. Everything up-to-date

Once you have done all the steps, go ahead to your R Studio and perform Stage -> Commit -> Commit Message -> Push to complete the process. Refresh your Github page and you shall see all of your local files at your GitHub repo.

If you find this useful, you can buy me a coffee :) @ https://www.buymeacoffee.com/masteramuk

Share:

Thursday, April 2, 2020

Data Analysis - Use it !!

I was reading many developer's site and chat (telegram and whatsapp) when the government stated that they are looking for an app similar to Singapore apps to track the close contact of the Covid-19 positive patient.

In Singapore, they are using a technology which I presume is Bluetooth to ping close contact within the radiant of the tech and capture necessary data which then used to determine the contact and request them to perform screening. Here are some of the news:

  1. https://www.thestar.com.my/tech/tech-news/2020/03/20/covid-19-singapore-launches-contact-tracing-mobile-app-to-track-coronavirus-infections
  2. https://www.pymnts.com/coronavirus/2020/app-lets-singapore-track-virus-patients-movements/
  3. https://www.nst.com.my/news/nation/2020/03/578445/smartphone-app-track-contacts-covid-19-patients
  4. https://asia.nikkei.com/Spotlight/Coronavirus/Singapore-urges-citizens-to-sign-up-for-COVID-19-tracking-app


And the apps is available on Google Play and Apple Store

  1. https://play.google.com/store/apps/details?id=sg.gov.tech.bluetrace&hl=en
  2. https://apps.apple.com/sg/app/tracetogether/id1498276074

And as this article is written, there are many groups including international are coming with various hackathons for apps that can be used to track all COVID-19 patients and their close contact.

In Malaysia, since the announcement, many had gather groups to develop apps.

From my perspective, why must we reinvent the wheel? Why we need to develop many apps when we already have few that are potentially be used for it. 

For instance, D'scover by Favioriot was developed for a user to explore whatever the user likes but also close contact that uses the apps. I believed they can just tweak the apps to get close contact for COVID-19 and it is much faster than building and testing new apps. (by the way, this is not promoting them and I don't gain anything from it :))

Not just that apps, people have been using Google Maps, Waze, Grab, etc. All these apps collected millions of data and one of these data is people's location and whereabouts. On top of that, all telcos do have their customer's data location and track their movement. I attended the Big Data conference by Bigit a few years ago where one of the telcos presented their data analysis. They have shown the heatmap of their user and based on the communication tower.

I even had a discussion with a few telcos when they approach us (my previous company) to provide their services and wish for data sharing. I do request to have a set of data of their customer whereabouts too to ensure we can provide efficient services at the moment the customer approaches our station or hub, or at the time they are supposed to do so.

These data can be utilized to find close contact with COVID-19 patients. From these data, we know where they go, their ride and whom the came across with or pass by. Of course, these data are secured by all those companies for customer's safety and PDPA compliance. But, for the sake of government and to combat COVID-19, they can request minimal information limited to the phone number to call the respective COVID-19 contact. 

You just need a group of data scientists and data engineers to focus on the massaging and provide the relevant info to the government fast and secure. That's all :)

Don't REINVENT the wheel. Used It and MAXIMIZE the POTENTIAL.

* My personal opinion based on experience. Agreed to disagree :)


Share:

Tuesday, February 25, 2020

Running Pentaho Data Integration in Mac OS Catalina version 10.15 and above


If you haven’t download the application, you may access here https://community.hitachivantara.com/s/article/downloads

Running Pentaho Data Integration @ Spoon in Windows or Linux should be straight-forward. Either click on spoon.bat or spoon.sh or the Data Integration app icon.
Running Data Integration from the install folder
However, for Mac OS, especially with the latest version Catalina which only allowed certified and trusted 64-bit application to run, running Data Integration will be troublesome despite the security for the app was disabled and allowed for the application to run. 





Some of the guidance on enable security and allow the untrusted app to run are as follow:
  1. https://edpflager.com/?p=3571
  2. https://andres.jaimes.net/1388/running-pentaho-spoon-on-mac-osx/
I have tried both and after enabling the apps, click on the Data Integration icon still does not work and the application still did not run. 

Lastly, I had to try to run it through the terminal. For OSX Catalina and above, instead of normal bash, Apple brings in zsh and the behavior is totally different. Plus, if you are a developer and install many Java JDK, running the Pentaho Data Integration will not be as running the normal command. Here is the step to run on the terminal and it works for all OSX.

1. Open up your terminal


2. Navigate to your install path (where you install or unzip the data integration file)


3. Run ./spoon.sh (for bash or old scripting, running spoon.sh shall work without ‘./’)

4. If you run into an error such as JDK or java runtime error like below, do not panic. This maybe your current java JDK is set to be higher than supported by the application.

(base) MyMek @ MyEpal data-integration % ./spoon.sh
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0
-Djava.endorsed.dirs=/Users/masteramuk/Documents/Apps/data-integration/system/karaf/lib/endorsed is not supported. Endorsed standards and standalone APIs
in modular form will be supported via the concept of upgradeable modules.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

5. First, check your JDK version. Open a new terminal as the command need to be executed from the base (unless you set the JDK in your profile which may cause problem to run multiple JDK later). Run command java -version. You shall have similar to below. In my laptop, the current JDK version is set to JDK version 10.

(base) MyMek @ MyEpal data-integration % java -version
openjdk version "10.0.2" 2018-07-17
OpenJDK Runtime Environment AdoptOpenJDK (build 10.0.2+13)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 10.0.2+13, mixed mode)

6. Make sure you have multiple java JDK installed if you need to use the existing JDK version for your ‘other’ development. Refer https://www.devdungeon.com/content/install-multiple-jdk-windows-java-development to install multiple JDK. Refer here https://www.jenv.be/ to install jenv command tool.

7. For my laptop, I have a few versions of JDK and using jenv, I can set the selected JDK for global or local (only applied to the folder where we run jenv local command). Run jenv version to check on current JDK and jenv versions to list all JDK install. 

(base) MyMek @ MyEpal data-integration % jenv versions
  system
  1.8
  1.8.0.232
* 10.0 (set by /Users/masteramuk/Documents/Apps/data-integration/.java-version)
  10.0.2
  13.0
  13.0.1
  9
  openjdk64-1.8.0.232
  openjdk64-10.0.2
  openjdk64-13.0.1
  openjdk64-9

As of this manual written, Pentaho Data Integration @ Spoon supports up to JDK 1.8.

8. Use jenv to change the jdk to 1.9. Run command jenv local [JDK number]. In this example, I execute jenv local 1.8.0.232.
9. Finally, run again ./spoon.sh. The application shall run.

Running


Opening the application

The main screen

Share:

About Me

Somewhere, Selangor, Malaysia
An IT by profession, a beginner in photography

Labels

Blog Archive

Blogger templates