The Data Science Trend and A Cloud-Based Crawler as a Platform for Intellectual Property and Competitive Intelligence Research
James H. Moeller
December 15, 2017
(Approximate Read-Time: 7 minutes, Word Count: 1,334.)
The Data Science Challenge
Lately it seems that the media is full of new reports on the changing landscape of jobs and how robotics, artificial intelligence, and data science, will be taking over many aspects of our lives. Examples include driving automobiles, taking restaurant orders, diagnosing medical issues, providing legal advice, and making finance and investment recommendations, just to name a few. Exactly when this will all happen is debatable. But if you’re gauging technology demonstrations and initial implementations, there’s no doubt that progress is happening quickly and that the technology is already being implemented in many businesses today.
Lately it seems that the media is full of new reports on the changing landscape of jobs and how robotics, artificial intelligence, and data science, will be taking over many aspects of our lives. Examples include driving automobiles, taking restaurant orders, diagnosing medical issues, providing legal advice, and making finance and investment recommendations, just to name a few. Exactly when this will all happen is debatable. But if you’re gauging technology demonstrations and initial implementations, there’s no doubt that progress is happening quickly and that the technology is already being implemented in many businesses today.
While some job sectors will be affected more dramatically than others, for me and my business, the current transition is really a continuation of technology adoption that I’ve been working with for decades. It’s a continual challenge of how to utilize new technology to make work easier, more efficient, and more effective. So, it’s with the pursuit of that challenge that I’m integrating more data science into my consulting practice with the intent of adding value to what I provide in terms of intellectual property and competitive intelligence research.
Recently I’ve been experimenting with an open-source Web crawler system named Scrapy (https://scrapy.org) and I’ve used that to implement a crawler data service available on my website (https://www.MoellerVentures.com). What I’ve designed is a cloud-based crawler with a browser-accessible, form front-end that works across all platforms (desktop, tablet, and mobile). It’s intended to be a focused, deep-dive keyword crawler, as compared to broader, keyword-oriented, search engines. Additionally, it can serve as a platform for future applications leveraging third-party capabilities and data sources. It can utilize the data science, AI and machine learning services from companies such as Amazon, Microsoft, Google, IBM and others. In combination with these services, it can also integration API-accessible data sources from organization like the U.S. Patent and Trademark Office (USPTO) and the Federal Communications Commission (FCC). Those will be topics of future blog posts.