Moving Data [part 2]

So we've put off diving into the world of Apache Spark, purely because we were used to the other tools in industry. One could say, we were in a comfort zone with, for the most part, vendor specific load functions like LOAD DATA INFILE (MySQL), etc. We could move data, and we weren't interested in looking into the new ways of doing things.

However, since we've entered the world of transformations and actions with Apache Spark, it doesn't seem like we will be looking back.

It's a skill set that is rare at this stage in industry, and, it's a sought after skill set.

Come talk to us regarding moving terabytes and terabytes of data in the most efficient way possible.

www.silvafox.co.za
Location: Cape Town, South Africa

Comments

Post a Comment

Popular posts from this blog

Design Dimensional Model

Design Dimensional Model The art of designing a dimensional model is exactly that. An art. And, like any reputable artist will know, you can hone your artistic skills with regular exposure to alternative design approaches. Your goal is to hone your skills, and, that takes practice. So, if you know all the techniques in designing a dimensional model, my advice is: Practice this art some more.  For some this is a topic that is "old news". They say: "We know all about designing a dimensional model." I say, well, this can largely be true, because you've been working with facts and dimensions for 1 2 3 or more years.  However, when I start to worry about such ones, is when they use terms like: "We should actually use a SCD (slowly changing dimension) on this fact, and not this dimension". Some of you might see this statement as acceptable. However, for people who are purists when it comes to Ralph Kimball's design, that statement is an irritation o...
Read more

Code Reviews

Code Review Checklist Code reviews are a very important part of the development process, in any environment. It is another safety net in ensuring a software deliverable of high quality. These are some points to keep in mind about code reviews. Take note of what needs to happen before, during and after a code review: Make sure that your code is self-explanatory and has adequate amount of code documentation and comments. These comments needs to describe the problem it is addressing in a clear and concise manner. Automated tests are good to test chunks of functionality. Include automated tests with your changes to make it easier to validate that your code works. In a SQL heavy environment, calling the different SP with specific input parameter values and inspecting the return result, or db stored result is a good midway point in testing the code. Your code should be sitting in source control. Before a code review commences: Set a date and time for the review to commence, and sen...
Read more

Microsoft SQL Server and SSIS cheat sheet

Microsoft SQL Server and SSIS cheat sheet The below some short guidelines when regularly working with SSIS and MS SQL Server. Some of these can also be used for some knowledge reviews. Are you using: BULK INSERT Remember to ensure your text files are delimited with LF ( line feed ) only, and not CR ( carriage return ) characters. Open the file in Notepad++ to verify this. Alternately, do a search using Extended mode for "\r\n", and replace these with "\n" MERGE Very handy compared to separate INSERT / UPDATE statements. Using MERGE also means a combination of INSERT / UPDATE are atomic - a big plus. The gains far outweigh the cons regarding performance, which in my experience has been negligible   XACT_ABORT  A session variable that controls how processing should continue after an exception in your Stored Procedure is thrown. With this setting on, your entire batch is aborted, and processing jumps to the catch block from your try catch logic. With th...
Read more