Is Your Big Data Project Ready for Apache Ignite?

on December 26th, 2016
Big data

In the late 80s and early 90s, Commodore shipped the Amiga. The business world, particularly in the United States, wasn’t all that impressed and the line faded with little fanfare much to the chagrin of the rabid hobbyist fan base. One thing the business world lost by embracing Windows on Intel as the next big thing over the Commodore Amiga is the RAM drive.

On the old Amigas, you could copy an entire floppy disk over to the RAM drive and run the application significantly faster than running it straight from the floppy drive. Proving that even in the technology world what was once old is new again, a large amount of digital ink has spilled over the last year about in-memory computing, specifically in the data world. Just over a year ago, Apache’s in-memory computing project graduated from incubation to release under the name Apache Ignite.

Ignore the Man Behind the Curtain

Much as the RAM drive did for old Amiga programs, Apache Ignite brings big data to a new level. Whether transacting a dataset or analyzing and computing on it, Apache Ignite promises more speed. There have been a number of limitations on speed over the years. One limitation that even the average person has come to understand is that of the spinning hard drive. Only so much speed can be eked out of a system using physical hard drives. Most speed-conscious solutions have moved off of spinning drives and onto solid state drives.

Apache Ignite takes this a step further and moves the database to memory, specifically RAM. This isn’t just copying the database to RAM and running a background process to sync it back to the disk either. Ignite is an entire framework providing extremely fast, real-time access to data in a complete computational framework. It does this while stepping out of the way and letting you access it on any platform while keeping it in its existing form.

More than OLAP

In-memory data grids aren’t exactly new. What sets Apache Ignite apart is that it is useful for so much more than OLAP analysis. Ignite can be deployed right over the top of an already existing data store, whether RDBMS, NoSQL, or HDFS. While it’s been developed predominantly in Java it has support for .Net and C++ already, allowing it to work with almost any application you already have deployed.

It can do so much more than support faster reads and writes to a database too. With ANSI 99 SQL syntax support, there’s no need to rewrite existing queries. Additionally, Apache Ignite provides fully ACID compliant distributed transactions making it a valid platform regardless of your industry data requirements. If your big data projects have moved to Hadoop, Apache Ignite can integrate and provide the in-memory benefits and ease of use your solution may be lacking.

What’s the Catch?

Apache Ignite is big, powerful, and flexible. It’s also another framework sitting between your application and your data. It has the potential to increase big data analysis tenfold. The speed increases promise to be on par with moving from spinning drives to solid state drives. It’s not all roses though. As with any distributed system, there will be a point that latency between processes on different hardware may become a limiting factor. Of course, this limit is much less significant than a sudden drop in analysis time due to needing to scan a physical hard disk for data.

Just listing all the things that Ignite can do would fill this page. A product with such potential is hard to summarize. If your company is relying on big data for decisions, Ignite might be for you. If your business needs to make split-second decisions based on analysis of data, then Ignite is for you. If your business can’t respond any faster to getting data analyzed, Ignite may not be for you.

Whether you decide to try out Apache Ignite or not, follow us on Twitter and keep up with all the trends happening in big data today.