What a delayed post but better late than never as they say.

Late last year, I started focusing this blog to a topic I am really passionate about: Software Development (yes I am still confused as to how it differs from Software Engineering but the latter sounds cooler). That meant I had to trash some posts I had which weren’t really related to the topic. Also, as I am taking graduate school, I noticed I have begun posting some theoretical bits which are just really rehashes of things you can find in any algorithms book.

All in all, I think the inadvertently general theme has been really messy. Ever seen a Filipino plate at a party? It’s literally a mix plate – a little bit of that and a little bit of this. It’s not uncommon to see spaghetti alongside beef stew and rice or even more bemusing three kinds of noodles and then sushi at the same time. Yes it’s fun to eat but no I don’t think a blog like that is fun to read.

However, blogging has been very helpful to me for organizing my thoughts and though I don’t promise to write more, at least I can promise to write with more focus.

What perfect time to reorganize but at the beginning of new year! So check out the new things I will be focusing on from now on:

1. Programming tips – This will revolve around the application of design patterns and refactorings in particular illustrating how I have used them at work or in school.

2. Software Development tools – Anything that will make software development more efficient like source control, frameworks. This year I am planning to look into continuous integration and Apache Solr or Elastic Search as well as create a catalog of commonly used subversion and git use cases.

3. Programming Philosophies – My reflections on approaches to programming like Test Driven Development or Dependency Injection.

4. Algorithms – Though I will admit I am just rehashing some topics that you could find in commonly used books, the purpose of these rehashes is to make the material less obscure. I have found that, unlike topics which you can find in Khan Academy, there is lack of straightforward and just as easy to digest material on advanced topics available on the web. When I can I will include links or practical exercises I myself have used to aid in understanding.

5. NEW! Code Aesthetics – Who hates spaghetti code? I certainly do. When I started reading open source code (for fun!) I learned that it is literally a mine of beautiful coding style. This will be my attempt to come up with a catalog of different styles.

I wanted to write a program that would find the top five frequent item sets among a set of baskets.

Consider the item set below:

D = [ ('milk', 'cheese', 'tuna'),
      ('cheese', 'mushroom'),
      ('cheese', 'eggs'),
      ('milk', 'cheese', 'mushroom'),
      ('milk', 'eggs'),
      ('cheese', 'eggs'),
      ('milk', 'cheese', 'mushroom', 'tuna'), 
      ('milk', 'cheese', 'eggs') ]

The goal I want for my program is to be able to induce that

(the presence of milk somehow implies the presence of cheese as well)

or

(milk, cheese and mushroom) somehow go together.

Just right now, I think I could also find a correlation between item sets and dates.

Anyway, as I have learned the key is to use map reduce and I was able to do this quite easily with a NoSQL database like Mongo DB.

A link to the code I wrote is available on github.

So far, I am able to generate the candidate item set for 2-combinations of the universal sets. This is another way of saying that the program I wrote is to count the number of times a two combination appears in all the baskets. For example, it is able to count that (‘milk’, ‘cheese’) occurs four times. Below is a screenshot of the script doing just that:

The next step after this is to generate the frequent item sets from the the candidate item set. This just means, I have to filter out non frequent 2-combinations. This can easily be done in two steps and is based on a support count that is defined before hand.

Let’s say a support count of 2 is defined, then the first thing to do is to check if the count of the 1 item sets that make up the two item sets have support count . Remember that the key to the apriori algorithm is that the subsets of a frequent item set must also be frequent.

This can easily be derived by running the script I wrote:

Now that I am able to generate the candidate item sets for any n-combination of the universal set, some next steps for my program would be to

1. Generate the frequent item set given a candidate item set.
2. Report the top 5 item sets with the highest interest and highest confidence.

Performance considerations also come to mind as my implementation of the algorithm has not been tested for a large data set.

So what are the differences?

• Little notation denotes a bound that is NOT asymptotically tight. – I used to think that was waaaay bigger compared to because come to think of it is bigger then right? Apparently not big enough. When I tried graphing it and zoomed out 100x, their lines were near each other.
• Little notation requires that the condition holds true for ALL positive constants c that is multiplied to – this is the most important distinction of them all. If one says that , then for ALL positive constants c and some , should be .

Phew, that was some definition and it wasn’t even the original one . I have always found myself hard pressed to remember and apply the definition above because I am lazy and I like a methodical quick fix (with emphasis on the word methodical).

So a technique I am using is graphing of both functions on a piece of paper. An easy starting point is to let where one can easily see from the graphs of both functions if indeed is for all . If it is, then the condition should hold true for .

One gotcha, however, is that c can also be anything between 1 and 0. Going back to my previous dilemma, is It seems at first that the answer is yes for but a closer look reveals that for which is the condition does not hold anymore. Hence, .

UPDATE: Since I have rarely encountered the use of the notation, one mnemonic I have devised today to help me quickly process a statement like ‘is is to quickly apply the form to the statement. As such, if I see a the statement like , it quickly registers as .

The Apriori algorithm is a basic method for finding frequent itemsets. The latter is used to generate association rules with high confidence and high interest.

Here is my summary of it along with a running example. The following set of baskets will be used:

D = [ ('milk', 'cheese', 'tuna'),
      ('cheese', 'mushroom'),
      ('cheese', 'eggs'),
      ('milk', 'cheese', 'mushroom'),
      ('milk', 'eggs'),
      ('cheese', 'eggs'),
      ('milk', 'cheese', 'mushroom', 'tuna'), 
      ('milk', 'cheese', 'eggs') ]

1. Generate ,is the set of subsets of D of size k. Generation of is done by

• first taking the cartesian product of the set to itself
• Joining the resulting sets.
• Checking to see if the subsets of the are frequent. A subset is frequent if it is in the set .

If , simply list all 1 itemsets.

one_itemset = ['milk', 'cheese', 'eggs', 'mushroom', 'tuna']

2. Generate , the frequent itsemsets by counting the number of times each element in occurs in . If an element in or the support threshold, it is qualified to be a member of . For and our example above

c1 = {'cheese': 7, 
      'tuna': 2, 
      'eggs': 6, 
      'mushroom': 2, 
      'milk': 6}

So far, I have tackled the base case which is . But what happens when ? Referring back to the step 1 we first,

Step 1a) Generate , the set of candidate itemsets of size k by taking the cartesian product of the set to itself

In this case we try to generate the set .

The next step is to

1b) Check to see if the subsets of the are frequent.

Since 1,2,3,4 and 5 are all frequent this is not an issue.

Unlike #1, there is no special case for step #2. We start by

Generating , the frequent itsemsets by counting the number of times each element in occurs in and then removing itemsets which are less than S, the support threshold.

This gives us the following

c1 = {
         ('milk', 'cheese'): 4,
         ('milk', 'eggs'): 4
         ('milk', 'tuna'): 2,
         ('cheese', 'eggs') : 4, 
         ('cheese', 'mushroom') : 2,
         ('cheese', 'tuna'): 2
     }

The algorithm can be ran as further and beyond until we find no more frequent itemsets.

Let me just add this gotcha about joining sets whose elements are also sets. Remember this is done at step 1 in the operation for

A join is performed on sets A and B if and only if

• their first k-2 elements are the same and
• their last element of A is B

For example, when joining to itself to create the sets and can be joined to form but the sets and cannot be joined to form a 3-itemset since the first elements of both sets e.g 1 and 3 are not equal.

1. The market basket model is probably the easiest introduction to anyone interested in data mining. The concept is simple. There are baskets and there are items in those baskets.

The market basket model: There are items and there baskets, also called itemsets, that hold those items.

2. Closely related to the the Market Basket model is the concept of frequent itemsets. Intuitively, a set of items is frequent if it occurs many times.

3. The following terms are used a lot when talking about frequent itemsets:

• Support count – is a term that refers to the number of times an itemset appears in a set of baskets. For example in the basket set
  

  [ ('cheese', 'milk', 'eggs'), ('milk'), ('milk', 'eggs', 'bread') ]

  the support count of the itemset (‘milk’, ‘eggs’) is 2 since it was a subset two times

• Support threshold – is a numerical limit that draws the line between a frequent itemset and non-frequent itemset. For example, in the basket set above, if one sets the support threshold at >= 2, then one can say that the itemset (‘milk’, ‘eggs’) is frequent.

4. Frequent itemsets are presented as an if-then rule like so: where I is a set of items and j is an item. This representation is called an association rule. In words, it can be said that if I appears in a basket then j is “likely” to appear as well.

5. In data mining parlance, the concept of ‘likely’ is more formally known as the confidence of the rule . Mathematically,

The insights behind this formula are that

• Baskets with in them cannot be more than baskets with I in them. Think about it. If one has I then there may or may not be the j around it in the same basket.
• Having said that, the more baskets with in them, the better. This makes the confidence in the rule stronger
• If is the same as the then that means that the confidence is 1 or 100%. In other words all the time!

6. In an association rule, interest is an indicator of how the item on the left affects the item on the right in . The formula is:

The insight behind this formula is that

• If the confidence outweighs the fraction of baskets with j, then it can be said that there is indeed a correlation between I and J and/or the presence of I somehow affects the presence of j.

• On the other hand, If there are significantly more baskets with j but not I, then the association rule isn’t really strong. It definitely is not the presence of I that implies the presence of j but something else. The instances when I and j are together in a basket can be said to be isolated

Despite being happy with my current job as a software engineer, I get bothered about it sometimes. It’s a great job but then I feel I am just like every other software engineer beside me. Every other engineer I have met seems to be good at the language they know whether it be PHP, Python or Ruby, Objective-C or Java for Android. Javascript, HTML and CSS is an expected skill. The thing is, these skills are easy to come by. It has in fact become cheap. All it takes is a little google search and copy and paste. To add to that, in my experience working as a developer in my country, I have observed that most developers and their clients wouldn’t even care if they’re not following the good practices in software development as long as their code works (but that is for another blog post). As a proud developer, I am always striving for loftier ideals when it comes software development. With the rise in outsourcing, I have always feared that without offering anything new, I would lose my job to someone who was willing to be paid at a lower price. One thing is for sure: a lower price and compromised ideals is something I could not afford to compete on.

Later on I would observe how developers went on to become managers. Though I have nothing against it, I lament that they had to do it because their options were limited. In a developing country like the Philippines, the challenges for a developer hit a wall and one comes full circle in just so many years. At that point it’s either one goes into management or sets up their own software outfit. Though the latter is a relishing challenge in itself, most developers I know pick up the same line of work from where they came from. The only difference is that they are doing it on their own. Looking at it, the only challenge I see at traditional software development is upgrading to new technologies every now and then. Some people are happy with that and that’s okay but that is not my idea of fun.

Hence, ever since, I have felt I needed to differentiate and offer something more as a software professional. At first, I thought writing clean code, following and advocating good practices will give me an edge. However, I found that for good engineers that is a basic given. Hence, being merely a good software engineer at best isn’t enough.

I have been browsing the career pages of tech companies I want to work for and I see a trend: That software engineers have specialties. A quick look at the career pages of facebook and twitter reveals a trend about being a software engineer: that each one has a specialty: Software Engineer in User Experience, Software Engineer in Tools, Software Engineer in Web Performance, Software Engineer in Mobile etc the specialties are to my mind like it is endless.

But one thing is for sure, everyone has a specialty, a focus as Steve Jobs said.

Ever since tablets and other mobile devices became popular I have realized how helpful they can be for learning. I used to lug a book stand with me to school to read in or put my laptop on. But with a tablet, I can store all my books and bring them with me not to mention ebooks are cheaper and in my opinion more eco-friendly. I also have a storage problem at home. Right now I think I have ten big books which I am having a hard time stowing away. Our library shelf is full and so I put them in my closet. But even as I am not a big fan of clothes I can see that my shirts are slowly telling my books to move away. I am also irritated to to look at my closet and see books instead of clothes.

Next to books, videos have also been one of my primary coursewares. Though in most of the videos I watch I need to be in front of Eclipse typing code away, in some videos I need to be in front of a pen and a notebook figuring algorithms and concepts. I have this need to store and easily watch this latter type of videos somewhere that’s not my laptop but with a big screen and sound system. Don’t get me started with Chromebook just yet because it still requires ubiquitous, high speed internet connection and that’s something we can’t afford in a third world country.

In a nutshell my requirements are,

• mobility.
• ebook reader.
• video playback.
• storage.

So the simple answer would be to buy a tablet. However one year or maybe even six months ago, tablets were way too much for those needs. I mean I obviously don’t need GPS, 3G as well as a camera. I usually use my smartphone for tasks where I would GPS, 3G and a camera.

So the closest thing that was available was the Nook Color but I wasn’t too keen on it because of the price. Then, the Kindle Fire came along. I think it’s perfect for my needs! However, that’s not what I bought last Wednesday. Instead, I settled for just the Kindle 4.

I think one of the big let downs for me was there was no way I could manage my Youtube videos. Though you could watch youtube via flash and it seemed to work based on the videos I saw but WTF come on man!

Some blog posts suggest downloading and then saving to the fire. Though this would work but I am too lazy to do it and I’d rather wait for an iteration of Fire that would allow seamless storing of videos from other sources like youtube. If they could do it for books then why not videos as well? I recently bought an ebook from the Pragmatic bookshelf, a store outside Amazon and my copy was sent directly to my Kindle. I wish the same could be done for Youtube videos on the Fire as a lot of course ware I watch is from there.

But with ecosystems like Apple and Amazon, you’ll never know. Still my fingers are crossed and I am hoping that Google or some Android developer who reads this post comes up with a Youtube app for the Fire that does that.

Right now, I here are some advantages I see that using Zend Form brings:

• One can package common forms like a login or signup into a library and reuse it
• Common error checking and validation code are available albeit this has to be done server side
• Error and validation messages also included!

Here are some challenges I see

• Hard to integrate with client side validation, most website nowadays are validated client side but for safety a.k.a when javascript is turned off, server side has to be there.
• Hard to create complex forms which require tables

I have read that decorators are all one needs for this but on the other hand, most developers find it a pain to learn. Hopefully I get a chance to implement a table form with Zend Form one day and put these fears to rest.

My application required me to indicate the, I quote, the “H Factor” of my referees. This can easily be looked up by doing a search on Microsoft Academic Search. Frankly, I didn’t expect something like that coming from Microsoft. Oh an by the way despite its usefulness I think it still sucks because some features only run on Silverlight WTF. Anyway, so the search engine would return a list of publications of a person queried. A link to that researcher’s profile is included at the top of the search results.

Here is my personal favorite as a Computer Science graduate.

The profile showcases a lot of interesting statistics such as citations by other people to a researchers work and of course the H Factor.

From where I come from, UP Diliman’s Department of Computer Science, the results are, to say the least, makes me sad and I say that with a smiley on my face and a little chuckle. It’s not surprising and it is the general trend in the Philippines anyway. In fact, an H Index heat map reveals that Singapore is the best in terms of research in South East Asia. You can verify this by viewing the Academic Map.

I did a quick scan of my professors and the highest H index is 2 from Sir Pros Naval. At 56 citations, his work is commendable. Guess what? He’s going to be my teacher in CS283 – Data Mining and I am really looking forward to the challenge after knowing his H index.

On the the less fun side, knowing about the H-index strikes me hard as it made me examine my objectives about going into grad school. I was a bit disappointed at this because some of my strongest referees come from the industry and of course they are not scholars or scientists. Maybe I am more cut out for the industry after all. I love learning and the occasional hard topic but right now I am not really sure that grad school is the right avenue to pursue it though there are still some fields that I would like to explore like Data Mining. So far, grad school has been an exploratory phase for me as I try to get to discover what my real interests are.

The first thing I did was played around with the controls but seeing that everything was so choppy on the new interface, I wrote it off at once! I see where Ubuntu is trying to go with all this and I really like how it is trying to bring in less technically inclined users to the FOSS world but for people it just won’t cut it.

One thing I noticed though was that loading times are faster than ever…literally in the blink of an eye I can go from the login screen to the desktop, or fire up chrome. Really cool. Oh and my favorite part: the terminal has a new makeover:

I haven’t really explored the version a lot but I hope to see some more cool new features and improvements.

UPDATE: I learned the hard way that upgrading really does take a lot of time. Whew. It took almost 6 hours for me with my download speed averaging at 150kbps.