Blog Feed

Design Dimensional Model The art of designing a dimensional model is exactly that. An art. And, like any reputable artist will know, you can hone your artistic skills with regular exposure to alternative design approaches. Your goal is to hone your skills, and, that takes practice. So, if you know all the techniques in designing a dimensional model, my advice is: Practice this art some more.  For some this is a topic that is "old news". They say: "We know all about designing a dimensional model." I say, well, this can largely be true, because you've been working with facts and dimensions for 1 2 3 or more years.  However, when I start to worry about such ones, is when they use terms like: "We should actually use a SCD (slowly changing dimension) on this fact, and not this dimension". Some of you might see this statement as acceptable. However, for people who are purists when it comes to Ralph Kimball's design, that statement is an irritation o...

Spark Streaming a local directory Introduction  For our streaming exercise, we will create two classes (one being a utility class that initially only handle the log4j properties for our application). We will manually compile our classes and use spark-submit to run our application. Note: For this example, we are creating the logging class inline into our main class.  Therefore this example helps you to slowly get to grips with the dynamics of Spark Scala programming, and, slowly grows your knowledge. We will not use the "spark/bin/run-example" script that most examples use, as I want you to understand step by step what you are doing. You are also more easily able to alter your code by doing it my way. Use Case  We are creating a streaming Apache Spark Scala program that reads a directory for new files and counts the amount of words in the file. Example  Create our class FileStream.scala (which contains the StreamingUtility class), which will for this e...

So I decided I'm going to use a real world example and do some transformations against such. I decided on http://dumps.wikimedia.org , so that I have some nice sized files which I'm able to really see the advantages Apache Spark brings to the table. My system is loaded with Apache Spark 1.6.0 and Scala 2.10.5. Lets do this : First open the spark shell: spark-shell Next load an SQLContext: val sqlContext = new org.apache.spark.sql.SQLContext(sc) //sc an existing Spark context  Next import the following packages into the shell session: import sqlContext.implicits._ import org.apache.spark.sql._ Now, you can start by loading the data from the " pagecounts-2011222 " file into a Resilient Distributed Dataset (RDD). RDDs have transformations and actions; the first() action returns the first element in the RDD  Now load the data from the file into a new RDD: val wikiHits = sc.textFile("/home/osboxes/Downloads/pagecounts-20151222") Do some ...

Have a look at your management settings in https://analytics.google.com/ on the Admin tab. This process is going slow, but it helps to know why blank ads are shown on your website: 1. Google does preliminary checks on your website to ensure it passes their checks for conformity, etc. 2. Your website pages includes the ad code generated by Adsense . After you have passed these steps and you notice your Ad Unit status being "New", you need to be patient. Google could take a few days to approve your application. You will receive a notification via email on outcome. At the same time, if approved, you website will start showing ads. That then means the cash can start rolling in.... Yippee!! Happy income generation peoples. www.silvafox.co.za

So you would like to add some dynamism to your static html pages. Well, ensure your hosted website service provider supports php scripting, which will be strange if they do not. Download the rss2html file where you will apply your changes to. Download from this location. Unzip this file and add it to the root of your website. Here are your checkpoints: Ensure your feed is reachable via following link: http:// <blogname> .blogspot.com/feeds/posts/default?alt=rss. Ensure you replace <blogname> with your blog name Modify your rss2html.php file as below: Configuration of RSS2html Unzip the folder.  Open the rss2html-docs.txt file in Notepad++ or other plain text editor and read the instructions.  Actually it's very simple to implement by modifying a few lines of code in the rss2html.php file: Tell the script where to find your RSS Feed:   $XMLfilename = "http:// your_blog .blogspot.com/feeds/posts/default?alt=rss"; Tell the script where to f...

The " Bigness " data. Ideas of Big Data If you are reading this blog, you are likely to be a nerd. Everywhere - Yes, data is everywhere, and the volumes are exponentially increasing by day. There exists a saying: "Where there is a will there is relatives". Are your measures trustworthy? Ensure their integrity, but at the same time give attention to the training needed to its users. Data Detectives - The cool part for me of data science, is the detecting fraud part. Fraudsters love lots and lots of data. The more the merrier. Its also easier to hide abnormalities when lots of data are present. Therefore, if you find your organization at the stage where you are IYO consuming and processing big data, then its time you consider employing a data detective. Machine Learning - This will only be useful if you actually start using what your system is learning about trends, etc OODA Loop - What came first? The egg or the chicken. This scenario gives ri...

Tasks or Containers should be run in parallel. The below Container A illustrates the parallelism we want to achieve. Container B illustrates sequential workflow: In the package property, you want to play with the property MaxConcurrentExecutables . With it, you can configure how many executables to run in parallel. Other things that are very important to keep in mind are : Reduce the number of columns - Yes, ensure to reference columns that you actually need. Lots of wasted space is saved, by following this golden rule Reduce the number of rows - If you are loading from a flat file, split the large files where possible Reduce column width - Here data type is VERY important. Overkill in data type usage is a common problem. The smaller the datatype (ensure you are not truncating important field information :-) ), the more efficient your processing will be Retrieve data using sp_executesql - When you are retrieving data, you may find you can directly using the Tabl...