Jetty Server 9.3 Configuration for Gzip Handler

What is Gzip Handler?

The Jetty GzipHandler is a compression handler that you can apply to any dynamic resource (servlet). Gzip fixes many of the bugs in commonly available compression filters: it works with asynchronous servlets and handles all ways to set content length. It has been tested with Jetty continuations and suspending requests. 

Compressing the content can greatly improve the network bandwidth usage, but at the cost of memory and CPU cycles. The DefaultServlet is capable of serving pre-compressed static content, which saves memory and CPU. By default, the GzipHandler will check to see if pre-compressed content exists, and pass the request through to be handled by the DefaultServlet.

Gzip Rules: 

GzipHandler will gzip the content of a response if:

It is mapped to a matching path

The request method is configured to support gzip

The request is not from an excluded User-Agent

accept-encoding header is set to gzip

The response status code is >=200 and <300 p="">

The content length is unknown or more than the minGzipSize initParameter or the minGzipSize is 0(default)

The content-type does not match an excluded mime-type

No content-encoding is specified by the resource

Implementation:

1. Remove Gzip Filter from application web.xml file

First of all, comment any Gzip filter tag given in the deployed application web.xml file as Gzip filter is depreciated in Jetty Server v9.3: 

Location:

<jetyy-server>\webapps\ROOT\WEB-INF\web.xml

<!-- <filter>

    <filter-name>GzipFilter</filter-name>

    <filter-class>org.eclipse.jetty.servlets.GzipFilter</filter-class>

     <async-supported>true</async-supported>

  </filter>

  <filter-mapping>

    <filter-name>GzipFilter</filter-name>

    <url-pattern>/*</url-pattern>

  </filter-mapping>  -->

2. Add Gzip Handler parameter in jetty server 

Set Gzip Handler parameter true in servlet tag of Jetty Server’s webdefault.xml:
Location:

<jetty-server>\etc\webdefault.xml

  <init-param>
      <param-name>gzip</param-name>
      <param-value>true</param-value>
  </init-param>

3. Modify configuration of Jetty Server Gzip Handler

We can modify configuration of Gzip Handler through jetty-gzip.xml file available at jetty server as shown below:
Location:

<jetty-server>\etc\jetty-gzip.xml

<Configure id="Server" class="org.eclipse.jetty.server.Server">
  <Call name="insertHandler">
    <Arg>
      <New id="GzipHandler" class="org.eclipse.jetty.server.handler.gzip.GzipHandler">
 <Set name="minGzipSize"><Property name="jetty.gzip.minGzipSize" deprecated="gzip.minGzipSize" default="2048"/></Set>

 <Set name="checkGzExists"><Property name="jetty.gzip.checkGzExists" deprecated="gzip.checkGzExists" default="false"/></Set>

 <Set name="compressionLevel"><Property name="jetty.gzip.compressionLevel" deprecated="gzip.compressionLevel" 
default="-1"/>
</Set>
 <Set name="excludedAgentPatterns">
   <Array type="String">
     <Item><Property name="jetty.gzip.excludedUserAgent" deprecated="gzip.excludedUserAgent" default=".*MSIE.6\.0.*"/></Item>
   </Array>
 </Set>

<Set name="includedMethods">
   <Array type="String">
     <Item>GET</Item>
        <Item>POST</Item>
   </Array>
</Set>  
<Set name="includedPaths">
   <Array type="String">
     <Item>/*</Item>   
   </Array>
</Set>
</New>
</Arg>
       </Call>
   </Configure>

Where minGzipSize denotes
Content will only be compressed if content length is either unknown or greater than minGzipSize.
checkGzExists
True by default. If set to false, the handler will not check for pre-compressed content.
compressionLevel
The compression level used for deflate compression. (0-9).
includedMethods
List of HTTP methods to compress. If not set, only GET requests are compressed.
excludedAgentPatterns
A list of regex patterns for User-Agent names from which requests should not be compressed.
includedPaths
List of paths to consider for compression.


4. Activation of Gzip Module

In Jetty Server 9.3, Gzip module is given in the name of gzip.mod(jetty-server-location\modules\gzip.mod) which we need to activate first before Jetty server start. And for activation there is a command given --add-to-start through which you can activate any module of jetty server.
First open command prompt from jetty server installed location and hit below command:
Syntax: java -jar start.jar --add-to-start=gzip

Now we have activated the GzipHandler to configuration of the server successfully.

After that you can start jetty server as per your normal stat up process.

5. Listing Active Modules of Jetty Server

If you want to check which module is running or activated at jetty server, --list-modules command would be given at jetty server startup time.   

Syntax: java -jar start.jar --list-modules

Above listing shows activated modules on Jetty Server 9.3 and this will give a brief about every activated module as well.

6. Checking Gzip Handler at Network Layer

You can check gzip activation at Network tab of browser by just Click on User small request rows button like as shown below:

Now you can check that size of response data reduced from 67.9 MB to 1.4 MB as expected which due to gzip handler activation.

When you click on any request like connect above and select Headers Tab,

Content-Encoding is also activated with gzip.

Reference: http://www.eclipse.org/jetty/documentation/9.3.x/gzip-filter.html

Export data to MySQL database from HDFS using Sqoop Commands

Export data to MySQL database from HDFS using Sqoop:

1. First of all create table in mySql as it is mandatory to available in mySql db.

CREATE TABLE employee (id INT NOT NULL PRIMARY KEY,name VARCHAR(20),deg VARCHAR(20),salary INT,dept VARCHAR(10));

2. Now create a inputEmployee.txt file in local file system through vi Editor.

Like /home/cloudera/Export

Note: No space would be there after comma(,) saperator otherwise file will not export successfully.

1201,sumit,manager,50000,TP

1202,manisha,preader,50000,TP

1203,kalil,php dev,30000,AC

1204,prasanth,php dev,30000,AC

1205,kranthi,admin,20000,TP

1206,satish p,grp des,20000,GR

3.  Now place this input file into Hdfs file system by following command:

>hdfs dfs -put inputEmployee.txt /home/cloudera/exportResult/

4. Export this hdfs file into mysql through Sqoop:

>sqoop export --connect jdbc:mysql://localhost/test --username root --password cloudera --table employee --export-dir /home/cloudera/exportResult/

Note: No need to give input file name to export , only directory path is sufficient  to fetch the file.

All files would be fetched from this directory one by one to export.

5. Check the records in the table:

>select * form employee;

More Sqoop command:

1. List of databases getting from sqoop command:

sqoop list-databases --connect jdbc:mysql://localhost/ --username root -P

2. List of tables getting from sqoop command:

sqoop list-tables --connect jdbc:mysql://localhost/test --username root -P

3. We can also use following command for export:

sqoop export -m 1 –connect jdbc:MySQL://localhost/db1 –username sqoop –password root –table acad –export-dir /sqoop_msql/

To obtain a filtered map, we can use the following option:

–input-fields-terminated-by ‘/t’ –MySQL-delmiters

Where ‘/t’ denotes tab.

By default, Sqoop export creates INSERT statements. If the –update-key argument is stated, UPDATE statements will be created instead.

The -m argument sets the number of map jobs for reading the file splits from HDFS. Each mapper will have its own connection to the MySQL Server.

Import data in HDFS from mysql db using Sqoop Commands

Import data in HDFS from mysql db using Sqoop:

1. when target dir is not available:

>sqoop import --connect jdbc:mysql://localhost/retail_db --username root --password cloudera --table departments --m 1

2. To check departments part and success files:

>hdfs dfs -ls -R departments

3. To view content of departments part file:

>hdfs dfs -cat /departments/part-m-00000

=============

when target-dir is available:

1. >sqoop import --connect jdbc:mysql://localhost/retail_db --username root --password cloudera --table departments -m 1 --target-dir /home/cloudera/sqoopQueryResult1/

2. To check departments part and success files:
>hdfs dfs -ls -R departments

3. To view content of departments part file:
>hdfs dfs -cat /home/cloudera/sqoopQueryResult1/part-m-00000

==============

Import records sqoop command while applying where clause in the table:

>sqoop import --connect jdbc:mysql://localhost/retail_db --username root --password cloudera --table categories -m 1 --where "category_name like '%Golf%'" --target-dir /home/cloudera/sqoopQueryResult3/

=============

Incremental column value defines newly added rows in the table,

like here fetch the rows after the category_id last value is 57:

>sqoop import --connect jdbc:mysql://localhost/retail_db --username root --password cloudera --table categories -m 1 --incremental append --check-column category_id --last-value 57 --target-dir /home/cloudera/sqoopQueryResult4/

Note:

You can also specified the –class-name for the generated code and specified the –bindir where the compiled code and .jar file should be located. (Without these arguments, Sqoop would place the generated Java source file in your current working directory and the compiled .class file and .jar file in /tmp/sqoop-/compile.)

The class name simply derives from the table name unless you specify a name with the help of the –class-name command line argument (CLA). The –target-diris the location in HDFS where you want the imported table to be placed.

>sqoop import --connect jdbc:mysql://localhost/retail_db --username root --password cloudera --table departments -m 1 --class-name departmentsNew --target-dir /home/cloudera/sqoopQueryResult5/ --bindir /home/cloudera/sqoopQueryResult5/

This will create departmentsNew.class and departmentsNew.jar file in /home/cloudera/sqoopQueryResult5/ directory.

Class and Interface

Although I want to write full article for a topic in JAVA but today I felt, some times small tips are more important than a full page theory when you want to revise quickly like for an interview ;)

I’m writing here few notes which I have collected while reading Java Books and hope these would be helpful for you too.

Final Classes:

• String Class can’t be sub classed.
• Final class obliterates a key benefit of OOP – Extensibility.
• Final class uses for safety and security.

Abstract Classes:

• In abstract class, method marked abstract end in a semicolon(;) rather than curly braces.
• If you change a method from abstract to non-abstract then you need to change the semicolon at the end of the method declaration into a curly braces pair.

Interface:

• All the interface method must be implemented and must be public and abstract(implicitly it is already done).
• An interface is an 100% abstract class.
• But an abstract class can have abstract or non-abstract methods, while an interface can have only abstract methods (this is the difference).
• All variable defined in an interface must be public , static and final it means interface can declare only constants, not instance variables.
• Interface methods must not be static as interfaces defines instance methods.
• You can’t change the value of constant or variable defined in interface from implementing class, it will give compiler error.

Methods of Class Object:

• Boolean equals(object obj)
• void finalize()
• int hashcode()
• final void notify()
• final void notifyall()
• final void wait()
• String toString()

toString() method:

• This method simply spit out object state(or in other words) get the current values of the important instance variable.
• When you pass an object reference to System.out.println() method it will call object.toString() method implicitly.

Hope you liked the tips, I will give move tips in my next post so don’t forget to give your feedback/comments as it motivates me to write more.
C YA BUDDY!!!

Lambda Expression Java 8

After long time I again found my old craze to write on Technical things. So today’s topic is Lambda Expression Java 8 which is very much hyped now a days.
Here are some important points for Lambda Expression:

Lambda Expression Java 8

• The target type of Lambda Expression is the type of the context in which the Lambda Expression appears.
  For Example: A local variable that it’s assigned to or a method parameter that it’s gets passed into.
• Although you haven’t declare the variable as final you still can’t use them as non-final variable. If they are to be used in Lambda Expression. If you do use them as a non-final variables, then the compiler will show as error.
• Lambda Expression capture values not variables.
• Lambda Expression are statically typed, so lets investigate the types of Lambda Expressions themselves. These types are called Functional interfaces.
• A functional interface is an interface with a single abstract method that is used as the type of a Lambda Expression.

Important Functional Interfaces in Java

Interface	Arguments	Return
Predicate	T	Boolean
Consumer	T	Void
Function	T	R
Supplier	None	T
Unary Operator	T	T
Binary Operator	(T,T)	T

• In the same way that Java 7 allowed to leave out the generic types for a constructor. Java 8 allows to leave out the types for whole parameters of Lambda Expression.

Type Inference:
Javac looks for information close to your Lambda Expression and use this information to figure out what would be correct type. It still typed checked and provide all the safety that you are used to, but you don’t have to state or write the types explicitly. This is called Type Inference.
Okey, today I am trying to approach new fundamentals for technologies and delivering in the form of notes. Feel free to comments or ask questions it will motivate me to indulge more with you ;)

Hadoop Getting Started

Hadoop! Now a days very popular in handling Big data and its analytic implementation.
But before starting Hadoop we must know that why we need it.

• Facebook generates peta byte of data per day.
• An aircraft generates 10 tera bytes of data per second.
• Electricity transmission also generates big data.
• Trading data or stock exchange also produce large amount of data.

Now question arises…Previously too we had big data then How did we handle it ? Well it was handled by relational database.
But the drawback is, previously only particular set of data was analyzed which were available in the form of rows and columns.
We used Hydrogen Collider instrument which generated petabyte of data for every quantum of power.
As all data comes for example from facebook in log files which don’t have rows and columns that why we called unstructured data and here we need of some other tool like Hadoop.
If we talk data form then it would be divided in part:

• Volume
• Velocity
• Variety
• Value

That is-                Value = Volume + Velocity + Variety
Now for example if we want to read 1 TB of data by 1 machine which has 4 I/o channels and each channel speed is 100 MB/Second then it will take 45 min to read all data.
While 10 machine takes 4.5 min to read data.
Definition of Hadoop:

Apache Hadoop is a framework that allows for the distributed processing of large data set of across cluster (group) of commodity (normal PC) computers using a simple programming model.

Currently Adhaar Card implementation is bruning example of Hadoop uses in India.

Quartz Scheduler with Spring 3

Quartz Scheduler with Spring 3:

For achieving scheduled task in spring 3 with the help of Quartz, we need following jars :

Spring 3.0
Quartz 1.8.6

Step 1:

First, add defines quartz details in context file of spring i.e. spring-context.xml as shown below:

<bean id="jobA" class="com.rockmesolid.jobs.JobA" />

<!-- Quartz Job -->
<bean name="JobA" class="org.springframework.scheduling.quartz.JobDetailBean">
<property name="jobClass" value="com.rockmesolid.jobs.JobA" />
</bean>

<!-- Cron Trigger, run every 5 seconds -->
<bean id="cronTriggerJobA"
class="org.springframework.scheduling.quartz.CronTriggerBean">
<property name="jobDetail" ref="JobA" />
<property name="cronExpression" value="0/5 * * * * ?" />
</bean>

<bean class="org.springframework.scheduling.quartz.SchedulerFactoryBean">
<property name="triggers">
<list>
<ref bean="cronTriggerJobA" />
</list>
</property>
</bean>

As you can see tag defines the bean class for the JobA.

<bean name=”JobA” class=”org.springframework.scheduling.quartz.JobDetailBean”>

It says Quartz bean class which will work with JobA class of property name=”jobClass”.
Next step is to set CronTriggerBean which has property of cronExpression

<property name=”cronExpression” value=”0/5 * * * * ?” />

Here “0/5 * * * * ?” defines CronTrigger will hit JobA class at every 5 Second.
If you want to hit it at a particular time on everyday like 10:13 pm everyday then cronExpression would be like “*10 13 * ? *”

<bean class=”org.springframework.scheduling.quartz.SchedulerFactoryBean”>

 
This set a reference bean name as cronTriggerJobA for Quartz SchedulerFactory.

Step 2:
Now definition of JobA is:

package com.rockmesolid.jobs;

import org.quartz.JobExecutionContext;

import org.quartz.JobExecutionException;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.scheduling.quartz.QuartzJobBean;

import org.springframework.web.context.support.SpringBeanAutowiringSupport;

public class JobA extends QuartzJobBean {

@Override

protected void executeInternal(JobExecutionContext arg0) throws JobExecutionException {

SpringBeanAutowiringSupport.processInjectionBasedOnCurrentContext(this); // for calling or inject bean of current context of application

System.out.println("Job A is runing");

try {

// to perform some actions

} catch (Exception e) {

e.printStackTrace();

}

}

}

Here as we can see JobA extends QuartzJobBean for calling executeInternal method which passes JobExecutionContext as an argument.
SpringBeanAutowiringSupport.processInjectionBasedOnCurrentContext(this); is the most important line to call or inject bean of current context of application into the JobA class. After this line you can use any of your application bean class into JobA class and perform any action like business logic, db connection etc.
Hope it will help. Really it’s easy.