Hadoop requires every data type to be used as keys to implement Writable and Comparable interfaces and every data type to be used as values to implement Writable interface.
Writable interface
Writable interface provides a way to serialize and deserialize data, across network. Located in org.apache.hadoop.io package.
Hadoop uses Java's Comparable interface to facilitate the sorting process. Located in java.lang package.
For convenience Hadoop provides a WritableComparable interface which wraps both Writable and Comparable interfaces to a single interface. Located in org.apache.hadoop.io package.
Writable interface
Writable interface provides a way to serialize and deserialize data, across network. Located in org.apache.hadoop.io package.
public interface Writable { void write(DataOutput out) throws IOException; void readFields(DataInput in) throws IOException; }Comparable interface
Hadoop uses Java's Comparable interface to facilitate the sorting process. Located in java.lang package.
public interface Comparable<T> { public int compareTo(T o); }WritableComparable interface
For convenience Hadoop provides a WritableComparable interface which wraps both Writable and Comparable interfaces to a single interface. Located in org.apache.hadoop.io package.
public interface WritableComparable<T> extends Writable, Comparable<T> { }Hadoop provides a set of classes; Text, IntWritable, LongWritable, FloatWritable, BooleanWritable etc..., which implement WritableComparable interface, and therefore can be straightly used as key and value types.
Custom Data Types
ExampleAssume you want an object representation of a vehicle to be the type of your key or value. Three properties of a vehicle has taken in to consideration.
- Manufacturer
- Vehicle Identification Number(VIN)
- Mileage
A tab delimited file containing the observations of these three variables contains following sample data in it.
Sample Data
Toyota 1GCCS148X48370053 10000 Toyota 1FAPP64R1LH452315 40000 BMW WP1AA29P58L263510 10000 BMW JM3ER293470820653 60000 Nissan 3GTEC14V57G579789 10000 Nissan 1GNEK13T6YJ290558 25000 Honda 1GC4KVBG6AF244219 10000 Honda 1FMCU5K39AK063750 30000Custom Value Types
Hadoop provides the freedom of creating custom value types by implementing the Writable interface. Implementing the writable interface one should implement its two abstract methods write() and readFields(). In addition to that, in the following example java code, I have overridden the toString() method to return the text representation of the object.
package net.eviac.blog.datatypes.value; import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; import org.apache.hadoop.io.Writable; /** * @author pavithra * * Custom data type to be used as a value in Hadoop. * In Hadoop every data type to be used as values must implement Writable interface. * */ public class Vehicle implements Writable { private String model; private String vin; private int mileage; public void write(DataOutput out) throws IOException { out.writeUTF(model); out.writeUTF(vin); out.writeInt(mileage); } public void readFields(DataInput in) throws IOException { model = in.readUTF(); vin = in.readUTF(); mileage = in.readInt(); } @Override public String toString() { return model + ", " + vin + ", " + Integer.toString(mileage); } public String getModel() { return model; } public void setModel(String model) { this.model = model; } public String getVin() { return vin; } public void setVin(String vin) { this.vin = vin; } public int getMileage() { return mileage; } public void setMileage(int mileage) { this.mileage = mileage; } }Following MapReduce job, outputs total Mileage per Manufacturer, with using Vehicle as a custom value type.
package net.eviac.blog.datatypes.jobs; import java.io.IOException; import net.eviac.blog.datatypes.value.Vehicle; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.log4j.BasicConfigurator; import org.apache.log4j.Logger; /** * @author pavithra * */ public class ModelTotalMileage { static final Logger logger = Logger.getLogger(ModelTotalMileage.class); public static class ModelMileageMapper extends Mapper<Object, Text, Text, Vehicle>{ private Vehicle vehicle = new Vehicle(); private Text model = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { String var[] = new String[6]; var = value.toString().split("\t"); if(var.length == 3){ model.set(var[0]); vehicle.setModel(var[0]); vehicle.setVin(var[1]); vehicle.setMileage(Integer.parseInt(var[2])); context.write(model, vehicle); } } } public static class ModelTotalMileageReducer extends Reducer<Text,Vehicle,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<Vehicle> values, Context context ) throws IOException, InterruptedException { int totalMileage = 0; for (Vehicle vehicle : values) { totalMileage += vehicle.getMileage(); } result.set(totalMileage); context.write(key, result); } } public static void main(String[] args) throws Exception { BasicConfigurator.configure(); Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "Model Total Mileage"); job.setJarByClass(ModelTotalMileage.class); job.setMapperClass(ModelMileageMapper.class); job.setReducerClass(ModelTotalMileageReducer.class); job.setOutputKeyClass(Text.class); job.setMapOutputValueClass(Vehicle.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }Output
BMW 70000 Honda 40000 Nissan 35000 Toyota 50000Custom Key Types
Hadoop provides the freedom of creating custom key types by implementing the WritableComparable interface. In addition to using Writable interface, so they can be transmitted over the network, keys must implement Java's Comparable interface to facilitate the sorting process. Since outputs are sorted on keys by the framework, only the data type to be used as keys must implement Comparable interface.
Implementing Comparable interface a class should implement its abstract method compareTo(). A Vehicle object must be able to be compared to other vehicle objects, to facilitate the sorting process. Sorting process uses compareTo(), to determine how Vehicle objects should be sorted. For an instance, since VIN is a String and String implements Comparable, we can sort vehicles by VIN.
For partitioning process, it is important for key types to implement hashCode() as well, thus should override the equals() as well. hashCode() should use the same variable as equals() which in our case is VIN. If equals() method says two Vehicle objects are equal if they have the same VIN, Vehicle objects with the same VIN will have to return identical hash codes.
Following example provides a java code for complete custom key type.
package net.eviac.blog.datatypes.key; import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; import org.apache.hadoop.io.WritableComparable; /** * @author pavithra * * Custom data type to be used as a key in Hadoop. * In Hadoop every data type to be used as keys must implement WritableComparable interface. * */ public class Vehicle implements WritableComparable<Vehicle> { private String model; private String vin; private int mileage; public void write(DataOutput out) throws IOException { out.writeUTF(model); out.writeUTF(vin); out.writeInt(mileage); } public void readFields(DataInput in) throws IOException { model = in.readUTF(); vin = in.readUTF(); mileage = in.readInt(); } @Override public String toString() { return model + ", " + vin + ", " + Integer.toString(mileage); } public int compareTo(Vehicle o) { return vin.compareTo(o.getVin()); } @Override public boolean equals(Object obj) { if((obj instanceof Vehicle) && (((Vehicle)obj).getVin().equals(vin))){ return true; }else { return false; } } @Override public int hashCode() { int ascii = 0; for(int i=1;i<=vin.length();i++){ char character = vin.charAt(i); ascii += (int)character; } return ascii; } public String getModel() { return model; } public void setModel(String model) { this.model = model; } public String getVin() { return vin; } public void setVin(String vin) { this.vin = vin; } public int getMileage() { return mileage; } public void setMileage(int mileage) { this.mileage = mileage; } }Following MapReduce job, outputs Mileage per vehicle, with using Vehicle as a custom key type.
package net.eviac.blog.datatypes.jobs; import java.io.IOException; import net.eviac.blog.datatypes.key.Vehicle; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.log4j.BasicConfigurator; import org.apache.log4j.Logger; /** * @author pavithra * */ public class VehicleMileage { static final Logger logger = Logger.getLogger(VehicleMileage.class); public static class VehicleMileageMapper extends Mapper<Object, Text, Vehicle, IntWritable>{ private Vehicle vehicle = new Vehicle(); private IntWritable mileage = new IntWritable(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { String var[] = new String[6]; var = value.toString().split("\t"); if(var.length == 3){ mileage.set(Integer.parseInt(var[2])); vehicle.setModel(var[0]); vehicle.setVin(var[1]); vehicle.setMileage(Integer.parseInt(var[2])); context.write(vehicle, mileage); } } } public static class VehicleMileageReducer extends Reducer<Vehicle,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); private Text vin = new Text(); public void reduce(Vehicle key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int totalMileage = 0; for (IntWritable mileage : values) { totalMileage += mileage.get(); } result.set(totalMileage); vin.set(key.getVin()); context.write(vin, result); } } public static void main(String[] args) throws Exception { BasicConfigurator.configure(); Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "Model Total Mileage"); job.setJarByClass(VehicleMileage.class); job.setMapperClass(VehicleMileageMapper.class); job.setReducerClass(VehicleMileageReducer.class); job.setMapOutputKeyClass(Vehicle.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }Output
1FAPP64R1LH452315 40000 1FMCU5K39AK063750 30000 1GC4KVBG6AF244219 10000 1GCCS148X48370053 10000 1GNEK13T6YJ290558 25000 3GTEC14V57G579789 10000 JM3ER293470820653 60000 WP1AA29P58L263510 10000
Nice and informative, thanks for sharing.
ReplyDeleteHadoop training velachery
IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes. IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble.Final Year Projects for CSE
DeleteSpring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Spring Framework Corporate TRaining .
Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai
The Angular Training covers a wide range of topics including Angular Directives, Angular Services, and Angular programmability.Angular Training
Great post and informative blog.it was awesome to read, thanks for sharing this great content to my vision.
ReplyDeleteSeo job training in chennai
The advanced MapReduce features describe the execution and lower level details. In normal MapReduce programming, only knowing APIs and their usage are sufficient to write applications. But inner details of MapReduce are a must to understand the actual working details and gain confidence.
ReplyDeleteFor details : Hadoop Training in Chennai
There are lots of information about latest technology and how to get trained in them, like Best Hadoop Training In Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies Hadoop Training in Chennai By the way you are running a great blog. Thanks for sharing this blogs..
ReplyDeleteI found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing..
ReplyDeleteSalesForce Training in Chennai
Pretty article! I found some useful information in your blog, it was awesome to read,thanks for sharing this great content to my vision, keep sharing..
ReplyDeleteUnix Training In Chennai
This information is impressive..I am inspired with your post writing style & how continuously you describe this topic. After reading your post,thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic..
ReplyDeleteAndroid Training In Chennai In Chennai
I have read your blog and i got a very useful and knowledgeable information from your blog.You have done a great job.
ReplyDeleteSAP Training in Chennai
Oracle Training in chennai
ReplyDeleteThanks for sharing such a great information..Its really nice and informative..
Selenium Training in Chennai
ReplyDeleteWonderful blog.. Thanks for sharing informative blog.. its very useful to me..
Data warehousing Training in Chennai
ReplyDeleteI am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly..
Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly, but it’s no need to do it, because you have explained the concepts very well. It was crystal clear, keep sharing..
ReplyDeleteWebsphere Training in Chennai
Oracle DBA Training in Chennai
ReplyDeleteThanks for sharing this informative blog. I did Oracle DBA Certification in Greens Technology at Adyar. This is really useful for me to make a bright career..
There are lots of information about latest technology, like Hadoop cluster is a special type of computational cluster designed specifically for storing and analyzing huge amounts of unstructured data in a distributed computing environment. This information seems to be more unique and interesting. Thanks for sharing.
ReplyDeleteBig Data Training Chennai | Big Data Course in Chennai | Big Data Hadoop Training in Chennai
PLC Training in Chennai
ReplyDeletePLC Training Institute in Chennai
PLC Training Center in Chennai
PLC Training Centre in Chennai
PLC SCADA Training in Chennai
SCADA Training in Chennai
Best PLC Training in Chennai
Best PLC Training Institute in Chennai
DCS Training in Chennai
Automation Training in Chennai
Automation Training Institute in Chennai
Process Automation Training in Chennai
Industrial Automation Training in Chennai
HMI Training in Chennai
VFD Training in Chennai
Inplant Training in Chennai
Embedded Training in Chennai
VLSI Training in Chennai
IT Training in Chennai
AutoCad Training in Chennai
Web Designing Training in Chennai
Projects are Fully hands-on manner only in Wiztech Automation. In Final year projects 100% Practical methods only Followed by Wiztech to Improve the Student Knowledge in each and every part of the Projects. Wiztech Handle all domains in Engineering Field. We having the Latest IEEE Project and NON-IEEE Projects. So only Wiztech having the best Name among the Final Year Project Center .
ReplyDelete✔ Final Year Projects in Chennai
✔ Final Year Project centres in Chennai
✔ Final Year Projects in Chennai
✔ IEEE Projects in Chennai
✔ IEEE Project Centers in Chennai
✔ Best Final Year Project Centres in Chennai
✔ Mechanical Projects in Chennai
✔ Mechanical Project Centers in Chennai
✔ Mechanical Project Centres in Chennai
Your blog is really awesome and I get a lot of great information from this blog. Thanks for sharing such an informative post on blog. Keep posting. It’s really useful and interesting...
ReplyDeleteHealthcare Business Analysis Online Training
Tableau Online Training
Qlikview Online Training
ServiceNow Administrator Online Training
Peoplesoft HRMS Functional Online Training
Jharkhand Labour Department Recruitment 2016
ReplyDeleteI have visited this blog first time and i got a lot of informative data from here which is quiet helpful for me indeed.........
ReplyDeletetoo good piece of information, I had come to know about your site from my friend sajid, bangalore,i have read atleast 7 posts of yours by now, and let me tell you, your web-page gives the best and the most interesting information. This is just the kind of information that i had been looking for, i'm already your rss reader now and i would regularly watch out for the new post, once again hats off to you! Thanks a lot once again, Regards, ,servicenow training in hyderabad
new horizon security services manassas va I am really impressed with your efforts and really pleased to visit this post.
ReplyDeletehttp://www.crazybulkreviews.com.au/
ReplyDeleteits very tricky
Excellent post!!! Selenium automation testing tool makes your software validation process lot simpler. Keep on updating your blog with such awesome information. Selenium Training in Chennai | Selenium Course in Chennai | Best Selenium training institute in Chennai
ReplyDeleteHi Admin, I went through your article and it’s totally awesome. You can consider including RSS feed for easy content sharing, So that you can drive huge traffic to your blog. Hadoop Training in Chennai | Big Data Training in Chennai
ReplyDeleteinstagram takipçi satın al - instagram takipçi satın al - tiktok takipçi satın al - instagram takipçi satın al - instagram beğeni satın al - instagram takipçi satın al - instagram takipçi satın al - instagram takipçi satın al - instagram takipçi satın al - binance güvenilir mi - binance güvenilir mi - binance güvenilir mi - binance güvenilir mi - instagram beğeni satın al - instagram beğeni satın al - polen filtresi - google haritalara yer ekleme - btcturk güvenilir mi - binance hesap açma - kuşadası kiralık villa - tiktok izlenme satın al - instagram takipçi satın al - sms onay - paribu sahibi - binance sahibi - btcturk sahibi - paribu ne zaman kuruldu - binance ne zaman kuruldu - btcturk ne zaman kuruldu - youtube izlenme satın al - torrent oyun - google haritalara yer ekleme - altyapısız internet - bedava internet - no deposit bonus forex - erkek spor ayakkabı - webturkey.net - minecraft premium hesap - karfiltre.com - tiktok jeton hilesi - tiktok beğeni satın al - microsoft word indir - misli indir
ReplyDeleteyoutube abone satın al /n trendyol indirim kodu
ReplyDeletecami avizesi
cami avizeleri
avize cami
no deposit bonus forex 2021
takipçi satın al
takipçi satın al
takipçi satın al
takipcialdim.com/tiktok-takipci-satin-al/
instagram beğeni satın al
instagram beğeni satın al
btcturk
tiktok izlenme satın al
sms onay
youtube izlenme satın al
no deposit bonus forex 2021
tiktok jeton hilesi
tiktok beğeni satın al
binance
takipçi satın al
uc satın al
sms onay
sms onay
tiktok takipçi satın al
tiktok beğeni satın al
twitter takipçi satın al
trend topic satın al
youtube abone satın al
instagram beğeni satın al
tiktok beğeni satın al
twitter takipçi satın al
trend topic satın al
youtube abone satın al
takipcialdim.com/instagram-begeni-satin-al/
perde modelleri
instagram takipçi satın al
instagram takipçi satın al
takipçi satın al
instagram takipçi satın al
betboo
marsbahis
sultanbet
instagram takipçi satın al
ReplyDeleteucuz takipçi
takipçi satın al
https://takipcikenti.com
https://ucsatinal.org
instagram takipçi satın al
https://perdemodelleri.org
https://yazanadam.com
instagram takipçi satın al
balon perdeler
petek üstü perde
mutfak tül modelleri
kısa perde modelleri
fon perde modelleri
tül perde modelleri
https://atakanmedya.com
https://fatihmedya.com
https://smmpaketleri.com
https://takipcialdim.com
https://yazanadam.com
yasaklı sitelere giriş
aşk kitapları
yabancı şarkılar
sigorta sorgula
https://cozumlec.com
word indir ücretsiz
tiktok jeton hilesi
rastgele görüntülü sohbet
erkek spor ayakkabı
fitness moves
gym workouts
https://marsbahiscasino.org
http://4mcafee.com
http://paydayloansonlineare.com
Best software training institute in Chennai. Make your career development the best by learning software courses.
ReplyDeleterpa training in chennai
uipath training in chennai
cloud computing courses in chennai
I believe there are many more pleasurable opportunities ahead for
ReplyDeleteindividuals that looked at your site.
unix course in chennai
top software training institute in Chennai
seo fiyatları
ReplyDeletesaç ekimi
dedektör
instagram takipçi satın al
ankara evden eve nakliyat
fantezi iç giyim
sosyal medya yönetimi
mobil ödeme bozdurma
kripto para nasıl alınır
instagram beğeni satın al
ReplyDeleteyurtdışı kargo
seo fiyatları
saç ekimi
dedektör
fantazi iç giyim
sosyal medya yönetimi
farmasi üyelik
mobil ödeme bozdurma
bitcoin nasıl alınır
ReplyDeletetiktok jeton hilesi
youtube abone satın al
gate io güvenilir mi
referans kimliği nedir
tiktok takipçi satın al
bitcoin nasıl alınır
mobil ödeme bozdurma
mobil ödeme bozdurma
perde modelleri
ReplyDeletesms onay
mobil ödeme bozdurma
NFT NASİL ALINIR
ankara evden eve nakliyat
trafik sigortası
Dedektör
WEB SİTESİ KURMA
aşk kitapları