My experience on my daily works... helping others ease each other

Showing posts with label Personal Opinion. Show all posts
Showing posts with label Personal Opinion. Show all posts

Thursday, April 2, 2020

Data Analysis - Use it !!

I was reading many developer's site and chat (telegram and whatsapp) when the government stated that they are looking for an app similar to Singapore apps to track the close contact of the Covid-19 positive patient.

In Singapore, they are using a technology which I presume is Bluetooth to ping close contact within the radiant of the tech and capture necessary data which then used to determine the contact and request them to perform screening. Here are some of the news:

  1. https://www.thestar.com.my/tech/tech-news/2020/03/20/covid-19-singapore-launches-contact-tracing-mobile-app-to-track-coronavirus-infections
  2. https://www.pymnts.com/coronavirus/2020/app-lets-singapore-track-virus-patients-movements/
  3. https://www.nst.com.my/news/nation/2020/03/578445/smartphone-app-track-contacts-covid-19-patients
  4. https://asia.nikkei.com/Spotlight/Coronavirus/Singapore-urges-citizens-to-sign-up-for-COVID-19-tracking-app


And the apps is available on Google Play and Apple Store

  1. https://play.google.com/store/apps/details?id=sg.gov.tech.bluetrace&hl=en
  2. https://apps.apple.com/sg/app/tracetogether/id1498276074

And as this article is written, there are many groups including international are coming with various hackathons for apps that can be used to track all COVID-19 patients and their close contact.

In Malaysia, since the announcement, many had gather groups to develop apps.

From my perspective, why must we reinvent the wheel? Why we need to develop many apps when we already have few that are potentially be used for it. 

For instance, D'scover by Favioriot was developed for a user to explore whatever the user likes but also close contact that uses the apps. I believed they can just tweak the apps to get close contact for COVID-19 and it is much faster than building and testing new apps. (by the way, this is not promoting them and I don't gain anything from it :))

Not just that apps, people have been using Google Maps, Waze, Grab, etc. All these apps collected millions of data and one of these data is people's location and whereabouts. On top of that, all telcos do have their customer's data location and track their movement. I attended the Big Data conference by Bigit a few years ago where one of the telcos presented their data analysis. They have shown the heatmap of their user and based on the communication tower.

I even had a discussion with a few telcos when they approach us (my previous company) to provide their services and wish for data sharing. I do request to have a set of data of their customer whereabouts too to ensure we can provide efficient services at the moment the customer approaches our station or hub, or at the time they are supposed to do so.

These data can be utilized to find close contact with COVID-19 patients. From these data, we know where they go, their ride and whom the came across with or pass by. Of course, these data are secured by all those companies for customer's safety and PDPA compliance. But, for the sake of government and to combat COVID-19, they can request minimal information limited to the phone number to call the respective COVID-19 contact. 

You just need a group of data scientists and data engineers to focus on the massaging and provide the relevant info to the government fast and secure. That's all :)

Don't REINVENT the wheel. Used It and MAXIMIZE the POTENTIAL.

* My personal opinion based on experience. Agreed to disagree :)


Share:

Saturday, November 3, 2018

Big Data - we forgot to clean our data

Recently I have been working with lots of data coming from various business area such as maintenance, financial transaction, etc., and I found an interesting thought from much top management and young leaders whom don't have enough experience handling data from the source up till visualization.

The first thing comes out from their mind was can it be done in a few hours (or some of them thought it was in a blink of the eyes or in split seconds). Normal question was "When can we see the report or chart? Can we see it tomorrow" and the worst I get "I want it to be ready by today before noon".

Image result for unclean dataMy first reaction was WTF!! (but I won't say it loud). I will normally negotiate with them as most of them don't know the process and the data that they requested. Most of them are easily attracted/amazed by superb visualization presentation by Visualization Tool's Marketing team. The thought everything can be done easily as those marketing people said. They just forgot that in a business presentation, the data set used by those marketing people are prepared and cleaned before being utilized in the tool. For example, Tableau's presentation will always use Sales data for their sample.

It is true that many of visualization tools nowadays are capable to process and display any kind of data. With certain skills, you can massage and clean your data on the fly. I've done that and I know it can be done.  BUT... surely at a cost which from my perspective, it is no beneficial at all.

Why do I say so?


  1. You cannot guarantee that the data you read is 100% clean. You might need to do lots of conversions, data massaging, replacements and calculations. This will definitely incur additional processing power and time during report population. I've come across with many data which either irrelevant, unclean (character in a supposed to be numerical column, date define as string, etc), or contain null values.
  2. You may need to perform lots of table joint or union which can cause your report server or tool to be resource hungry.
  3. You need to understand the data too. Each column and how it shall be visualize must be understood before you can present it correctly.


Related image

That's all from me..Adios









Share:

Friday, May 26, 2017

Kotlin + Netbeans IDE 8.2 - Don't use yet

I try the new language Kotlin with Netbeans 8.2.
http://plugins.netbeans.org/plugin/68590/kotlin

After download and install the plugin, it cause error on all projects currently on my list of projects and causing my netbeans to behave abnormally.

The error does not stop there. When I uninstall it, which you can't as it only appear in User Plugin and when you uninstall the plugin (just for the sake to remove kotlin), it causes many other problem.

So guys, use with extra careful ya.
Share:

Tuesday, April 26, 2016

Disrupting the world - The RAPID evolution of IT

In 1998, Kodak had 170,000 employees and sold 85% of all photo paper worldwide.
Within just a few years, their business model disappeared and they got bankrupt.
What happened to Kodak will happen in a lot of industries in the next 10 year - and most people don't see it coming. Did you think in 1998 that 3 years later you would never take pictures on paper film again?
Yet digital cameras were invented in 1975. The first ones only had 10,000 pixels, but followed Moore's law. So as with all exponential technologies, it was a disappointment for a long time, before it became way superiour and got mainstream in only a few short years. It will now happen with Artificial Intelligence, health, autonomous and electric cars, education, 3D printing, agriculture and jobs. Welcome to the 4th Industrial Revolution.
Welcome to the Exponential Age.

Software will disrupt most traditional industries in the next 5-10 years.
Uber is just a software tool, they don't own any cars, and are now the biggest taxi company in the world. Airbnb is now the biggest hotel company in the world, although they don't own any properties.

Artificial Intelligence: Computers become exponentially better in understanding the world. This year, a computer beat the best Go player in the world, 10 years earlier than expected. In the US, young lawyers already don't get jobs. Because of IBM Watson, you can get legal advice (so far for more or less basic stuff) within seconds, with 90% accuracy compared with 70% accuracy when done by humans. So if you study law, stop immediately. There will be 90% less laywyers in the future, only specialists will remain.
Watson already helps nurses diagnosing cancer, 4 time more accurate than human nurses. Facebook now has a pattern recognition software that can recognize faces better than humans. In 2030, computers will become more intelligent than humans.

Autonomous cars: In 2018 the first self driving cars will appear for the public. Around 2020, the complete industry will start to be disrupted. You don't want to own a car anymore. You will call a car with your phone, it will show up at your location and drive you to your destination. You will not need to park it, you only pay for the driven distance and can be productive while driving. Our kids will never get a driver's licence and will never own a car. It will change the cities, because we will need 90-95% less cars for that. We can transform former parking space into parks. 1,2 million people die each year in car accidents worldwide. We now have one accident every 100,000km, with autonomous driving that will drop to one accident in 10 million km. That will save a million lifes each year.

Most car companies might become bankrupt. Traditional car companies try the evolutionary approach and just build a better car, while tech companies (Tesla, Apple, Google) will do the revolutionary approach and build a computer on wheels. I spoke to a lot of engineers from Volkswagen and Audi; they are completely terrified of Tesla.

Insurance companies will have massive trouble because without accidents, the insurance will become 100x cheaper. Their car insurance business model will disappear.
Real estate will change. Because if you can work while you commute, people will move further away to live in a more beautiful neighborhood.
Electric cars will become mainstream until 2020. Cities will be less noisy because all cars will run on electric. Electricity will become incredibly cheap and clean: Solar production has been on an exponential curve for 30 years, but you can only now see the impact. Last year, more solar energy was installed worldwide than fossil. The price for solar will drop so much that all coal companies will be out of business by 2025.
With cheap electricity comes cheap and abundant water. Desalination now only needs 2kWh per cubic meter. We don't have scarce water in most places, we only have scarce drinking water. Imagine what will be possible if anyone can have as much clean water as he wants, for nearly no cost.

Health: The Tricorder X price will be announced this year. T here will be companies who will build a medical device (called the "Tricorder" from Star Trek) that works with you phone, which takes your retina scan, you blood sample and you breath into it. It then analyses 54 biomarkers that will identify nearly any disease. It will be cheap, so in a few years everyone on this planet will have access to world class medicine, nearly for free.

3D printing: The price of the cheapest 3D printer came down from 18,000$ to 400$ within 10 years. In the same time, it became 100 times faster. All major shoe companies started 3D printing shoes. Spare airplane parts are already 3D printed in remote airports. The space station now has a printer that eliminates the need for the large amout of spare parts they used to have in the past.
At the end of this year, new smartphones will have 3D scanning possibilities. You can then 3D scan your feet and print your perfect shoe at home. In China, they already 3D printed a complete 6-storey office building. By 2027, 10% of everything that's being produced will be 3D printed.

Business opportunities: If you think of a niche you want to go in, ask yourself: "in the future, do you think we will have that?" and if the answer is yes, how can you make that happen sooner? If it doesn't work with your phone, forget the idea. And any idea designed for success in the 20th century is doomed in to failure in the 21st century.

Work: 70-80% of jobs will disappear in the next 20 years. There will be a lot of new jobs, but it is not clear if there will be enough new jobs in such a small time.

Agriculture: There will be a 100$ agricultural robot in the future. Farmers in 3rd world countried can then become managers of their field instead of working all days on their fields. Aeroponics will need much less water. The first petri dish produced veal is now available and will be cheaper than cow produced veal in 2018. Right now, 30% of all agricultural surfaces is used for cows. Imagine if we don't need that space anymore. There are several startups who will bring insect protein to the market shortly. It contains more protein than meat. It will be labeled as "alternative protein source" (because most people still reject the idea of eating insects).

There is an app called "moodies" which can already tell in which mood you are. Until 2020 there will be apps that can tell by your facial expressions if you are lying. Imagine a political debate where it's being displayed when they are telling the truth and when not.

Bitcoin will become mainstream this year and might even become the default reserve currency.

Longevity: Right now, the average life span increases by 3 months per year. Four years ago, the life span used to be 79 years, now it's 80 years. The increase itself is increasing and by 2036, there will be more that one year increase per year. So we all might live for a long long time, probably way more than 100.

Education: The cheapest smartphones are already at 10$ in Africa and Asia. Until 2020, 70% of all humans will own a smartphone. That means, everyone has the same access to world class education. Every child can use Khan academy for everything a child learns at school in First World countries. We have already released our software in Indonesia and will release it in Arabic, Suaheli and Chinese this Summer, because I see an enormous potential. We will give the English app for free, so that children in Africa can become fluent in English within half a year.

p/s: Shared by my friend (Ijam).. Although the data/facts may inaccurate due to he did not provide the references, but looking to current trend and evolution, the potential of the statement to be accurate cannot be ignored.
Share:

Wednesday, November 4, 2015

MyISAM vs InnoDb - Full-text search versus Fail-safe capability

I in the midst of developing an applications for my wife to ease her daily life as lecturer and suddenly this versus thing trigger me :).


After some digging of information, I switches from InnoDb to MyISAM because I need to Full-text queries highly then insert/update process.

My conclusion:
1. Used MyISAM when your system is highly utilize for viewing and querying.
2. Used InnoDb when your system highly processing insert/update such as logging data or inserting raw data like track-and-trace system.
3. Used MyISAM if your application has the capability to roll-back or hold the data.
4. Used MyISAM if your data will grow beyond 1TB as at this point of time, query will cause problem for InnoDB.

References:
  1. http://www.thegeekstuff.com/2014/02/myisam-innodb-memory/
  2. http://dev.mysql.com/doc/refman/5.0/en/myisam-storage-engine.html
  3. http://dev.mysql.com/doc/refman/5.0/en/innodb-storage-engine.html
  4. http://tag1consulting.com/MySQL_Engines_MyISAM_vs_InnoDB#comment-115
  5. https://www.percona.com/blog/2007/01/08/innodb-vs-myisam-vs-falcon-benchmarks-part-1/
  6. http://www.rackspace.com/knowledge_center/article/mysql-engines-myisam-vs-innodb
  7. http://dba.stackexchange.com/questions/1/what-are-the-main-differences-between-innodb-and-myisam
  8. http://dba.stackexchange.com/questions/1/what-are-the-main-differences-between-innodb-and-myisam
  9. https://www.quora.com/How-are-deadlocks-resolved-in-Innodb-vs-MyISAM




Share:

Sunday, September 27, 2015

Don't post your log on screen @ web page

Programmers love to display log on their working page while developing the page. And normally, when they release the page or site, there are always pages that they forgot to disable of remove the log. When this happen, they are actually opening a small window to cyber threat @ hackers.

Take for example or a website below (click to enlarge)

The site show the IP address, the type of database and the path use for the database connection. Now, this may be good for programmer, but it is security loopholes that shall and can be easily avoided.

As an advice to programmers, if you need to log, PLEASE do log on a file which shall be truncated/replace (configured) not more than a week.

p/s: I'm also programmer with security interest :)
Share:

Monday, September 29, 2014

ATM's Machine - How can it get hacked?

Recently in Malaysia, just days ago (before I wrote this), few ATM machines was hacked and few bank losses for more than few millions RM. This is really serious and it is worst than traditional ways of stealing money from the machine. It shall be flagged as critical issue by the bankers and information security organization and community which request fast and firm action to prevent this. Just imagine if this people get to access and hacked the machine in one month from multiple locations. There will be billions losses and I can't imagine that (or rather afraid to think about it).

It was reported by The Star [3] on the same day and Harian Metero [1] and Utusan [2] on 30th September 2014 upon verification from respective agencies and company.

This is not the first as it happen previously in Montreal where 2 young teenagers unintended hacked into an Bank of Montreal ATM machines during their lunch break [8].

This won't be the last and neither the cases reported here in Malaysia. Regalado from Symantec [4], already raised the alarm since March upon his finding in end of 2013 [7].

As long as the machines is used software to operate, the issue will remain unless the community starts to built workable solutions to detect and prevent this from occurring. This is not possible until the community understand the real issue and root cause of it. People may see this as XP issue or software reside in the ATM machine and they will opt to change this two software. But in my case, these won't be the root cause. These two are merely the trigger button or one of the unlocked door waiting to be opened. The real root cause is from the initial step in software development lifecycle. This is where the community should start to implant their security measurements.

There are many ways to do this and one of the way is to increase the understanding of software developers in writing secure codes. I've wrote few papers which I hope that this will be a step moving towards having software that is harden and difficult to infiltrate:

  • Understanding Vulnerabilities by Refining Taxonomy (Proceeding and presented in IAS 2011. Appear in IEEE)
  • Classifications and Measurement on C Overflow Vulnerabilities Attack (Published in Journal IJNCAA)
  • Vulnerabilities and Exploitation in Computer System - Past, Present and Future. SiSKOM 2013 (ISBN 978-967-12088-0-9), Universiti Teknologi Mara, Shah Alam, Selangor, Malaysia, 3rd - 4th Sep 2013

I won't says that hacking will be impossible as there is no such thing as impossible when you have 'will'. But this is another step for us to create a stronger wall of security.
References:

  1. http://www.hmetro.com.my/articles/Trojan_sasarATM//Article
  2. http://utusan.com.my/utusan/Jenayah/20140930/je_01/Sindiket-guna-virus-lesap-wang-ATM
  3. http://www.thestar.com.my/News/Nation/2014/09/29/seventh-atm-hacked-into-al-rajhi-bank/
  4. http://www.hackerjournals.com/?p=23437
  5. http://www.hackersnewsbulletin.com/2014/03/windows-xp-flaw-hackers-withdraw-money-atm-just-sending-text-message.html
  6. http://securityaffairs.co/wordpress/23421/cyber-crime/rob-atms-couple-sms-messages.html
  7. http://www.deccanchronicle.com/140326/technology-latest/article/windows-xp-atms-being-hacked-simple-sms-symantec
  8. http://www.forbes.com/sites/jameslyne/2014/06/11/14-year-olds-hack-atm-in-lunch-hour-how-it-happened/
  9. http://thehackernews.com/2014/03/hacking-atm-machines-for-cash-with-just.html
  10. http://www.dailymail.co.uk/sciencetech/article-2655012/Teens-hack-cash-machine-lunch-break-Stunt-prompts-security-upgrade-Bank-Montreal-ATMs.html
Share:

Tuesday, August 13, 2013

Photos taken might pose security threat to you

How serious it is?


However, I found out that this can only affects the user IF:
1. You combine your photos with few other data such as Foursquare, Picasa, blogs, etc.
2. You did mention something that can be used to indicates your location or surrounding.
3. You used current photos capture capability (with sound and GPS setting)
4. And few others combination setting.
Share:

Thursday, April 18, 2013

HTTPS - Is it Safe (good discussion within members in LinkedIn)


Share:

About Me

Somewhere, Selangor, Malaysia
An IT by profession, a beginner in photography

Labels

Blog Archive

Blogger templates