Moving Data [part 2]

So we've put off diving into the world of Apache Spark, purely because we were used to the other tools in industry. One could say, we were in a comfort zone with, for the most part, vendor specific load functions like LOAD DATA INFILE (MySQL), etc. We could move data, and we weren't interested in looking into the new ways of doing things.

However, since we've entered the world of transformations and actions with Apache Spark, it doesn't seem like we will be looking back.

It's a skill set that is rare at this stage in industry, and, it's a sought after skill set.

Come talk to us regarding moving terabytes and terabytes of data in the most efficient way possible.

www.silvafox.co.za
Location: Cape Town, South Africa

Comments

Post a Comment

Popular posts from this blog

"Bigness" data - some thoughts about the big data journey

The " Bigness " data. Ideas of Big Data If you are reading this blog, you are likely to be a nerd. Everywhere - Yes, data is everywhere, and the volumes are exponentially increasing by day. There exists a saying: "Where there is a will there is relatives". Are your measures trustworthy? Ensure their integrity, but at the same time give attention to the training needed to its users. Data Detectives - The cool part for me of data science, is the detecting fraud part. Fraudsters love lots and lots of data. The more the merrier. Its also easier to hide abnormalities when lots of data are present. Therefore, if you find your organization at the stage where you are IYO consuming and processing big data, then its time you consider employing a data detective. Machine Learning - This will only be useful if you actually start using what your system is learning about trends, etc OODA Loop - What came first? The egg or the chicken. This scenario gives ri
Read more

Code Reviews

Code Review Checklist Code reviews are a very important part of the development process, in any environment. It is another safety net in ensuring a software deliverable of high quality. These are some points to keep in mind about code reviews. Take note of what needs to happen before, during and after a code review: Make sure that your code is self-explanatory and has adequate amount of code documentation and comments. These comments needs to describe the problem it is addressing in a clear and concise manner. Automated tests are good to test chunks of functionality. Include automated tests with your changes to make it easier to validate that your code works. In a SQL heavy environment, calling the different SP with specific input parameter values and inspecting the return result, or db stored result is a good midway point in testing the code. Your code should be sitting in source control. Before a code review commences: Set a date and time for the review to commence, and sen
Read more

Moving Data [Apache Spark]

So I decided I'm going to use a real world example and do some transformations against such. I decided on http://dumps.wikimedia.org , so that I have some nice sized files which I'm able to really see the advantages Apache Spark brings to the table. My system is loaded with Apache Spark 1.6.0 and Scala 2.10.5. Lets do this : First open the spark shell: spark-shell Next load an SQLContext: val sqlContext = new org.apache.spark.sql.SQLContext(sc) //sc an existing Spark context  Next import the following packages into the shell session: import sqlContext.implicits._ import org.apache.spark.sql._ Now, you can start by loading the data from the " pagecounts-2011222 " file into a Resilient Distributed Dataset (RDD). RDDs have transformations and actions; the first() action returns the first element in the RDD  Now load the data from the file into a new RDD: val wikiHits = sc.textFile("/home/osboxes/Downloads/pagecounts-20151222") Do some
Read more