Wednesday, December 26, 2012

Episode 23: Hadoop


Download

News
Tool of the Show
Book of the Show


Hadoop

History
  • Jeff Dean & Sanjay Ghemawat wrote the paper MapReduce
  • Created by Doug Cutting while he was at Yahoo!.
  • Intended to support Lucene (search engine reverse indexing).
  • Facebook announces their hadoop filesystem has grown to 100 petabytes. 
Features
  • HDFS: Hadoop Distributed Filesystem
  • HBase: A distributed, column-oriented database
  • Zookeeper: Distributed coordination service
  • Crunch: Simplified API for creating mapreduce pipelines.

        Strengths
        • Scale-free
        • Fault Tolerant
        • Can add/remove hardware in real-time.
        Weaknesses
        • Long spin up / spin down time.
          • Worker Pools
        • Excessive Serialization/deserialization
        • Excessive Materialization

        Tools
        • Avro: A serialization framework
        • Pig & Hive: querying and storing large datasets

        Uses
        • Storing/Manipulating Big Data.

        9 comments:

        1. 7 years later and I just now listened to this episode. I still use Emacs for software development. :-)

          ReplyDelete
        2. Aw, this was a very nice post. In thought I wish to put in writing like this additionally ?taking time and actual effort to make an excellent article?but what can I say?I procrastinate alot and on no account appear to get one thing done. https://royalcbd.com/product/cbd-oil-1000mg/

          ReplyDelete
        3. Great article with excellent idea!Thank you for such a valuable article. I really appreciate for this great information.. Driving test canada

          ReplyDelete
        4. 스포츠토토 Hi there, i read your blog from time to time and i own a similar one and i was just wondering if you get a lot of spam comments? If so how do you stop it, any plugin or anything you can suggest? I get so much lately it’s driving me crazy so any support is very much appreciated.|

          ReplyDelete
        5. I cherished up to you’ll receive performed right here. The caricature is tasteful, your authored subject matter stylish. nevertheless, you command get got an impatience over that you want be delivering the following. sick certainly come more in the past once more as precisely the similar just about very often inside case you shield this hike. 스포츠토토

          ReplyDelete
        6. 온라인카지노 Its like you read my mind! You seem to know a lot about this, like you wrote the book in it or something. I think that you can do with a few pics to drive the message home a bit, but instead of that, this is excellent blog. A great read. I’ll definitely be back.

          ReplyDelete
        7. Simply want to say your article is as astonishing. The clarity in your post is just great and i could assume you’re an expert on this subject. Well with your permission allow me to grab your RSS feed to keep updated with forthcoming post. Thanks a million and please continue the enjoyable work. 카지노사이트

          ReplyDelete