Postgresql JDBC enum and stored procedure

Simple way to submit postgresql enum from JDBC: Use type casting.

CallableStatement cs = conn.prepareCall("{call
sp_dummy(?::enum_result)}");
cs.setString(1, jobResult.toString());
cs.execute();

Maven simple java application

mvn archetype:generate -DgroupId={project-packaging} -DartifactId={project-name} -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

Work Breakdown Structure

Reference:  http://alvinalexander.com/fpa/cost-estimating/Estimating_using_Work_Break.shtml

Estimating using Work Breakdown Structure

Work Breakdown Structure (WBS) is probably how 98% of software development projects are estimated. WBS consists of someone sitting down and saying "Okay, how are we going to get this done?", then coming up with a long laundry list like this:

1. Setting up Development, Test, and Production environments - 8 hours
2. Creating the Login screen - 4 hours
3. Creating the Landing Page - 2 days
4. Developing the XYZ Page - 12 hours
5. etc., etc., etc.

You then continue with a long laundry list of every task you can imagine, with each task having an associated estimate. At the end you add up the time for all the tasks, and you have your overall time estimate.

Or do you?

WBS accuracy

As a practical matter, most WBS estimates I've seen under-estimate the total time required by a factor of 1.25 to 2.5. That is, if someone tells me that they did a WBS estimate and came up with 100 hours, the real answer is typically 125 to 250 hours.

IFPUG data also supports this. Though I don't have the exact data with me, I do remember that their data shows that estimates created using WBS alone under-estimate project effort by an average of about 50%. The reality is that this ratio actually varies quite a bit from person to person: some developers over-estimate, some under-estimate, some are very close. The good news is that people are consistent, and if they usually under-estimate by a factor of 2.5 they tend to always estimate by a similar ratio, unless ...

Use feedback to improve WBS estimates

If these developers are made to always review their estimates I believe they will eventually get better. For instance, let's say a developer told me a project would take 100 hours, but it really took 250 hours. After the project I would sit down with the developer and say "Okay, you estimated 100 hours, but it really took 250 hours. Now, I'm not here to beat you over the head. In fact, we like you quite a bit and want you on all of our future projects. But we need to take a little time and see where this estimate was off, okay?"

You do this behind closed doors, because everybody I know feels vulnerable when they're estimating. This stuff is hard, and it's not an exact science, I know that. But if you don't go through this process people won't get better.

Getting back to the WBS estimate, let's imagine that we worked through this process and we came up with an estimate of 1,600 hours. Given our team of four developers, this equates to 400 hours per developer. At 35 hours per week this is about 11.4 weeks of development effort.

Don't forget acceptance testing

Historically speaking, the people that I've seen take the WBS effort do not include final acceptance testing in their estimates, so I'll assume that's the case here. I'll use the same ratio of development time to final acceptance testing time that I used before (a 3:1 ratio), so if development takes 11.4 weeks, acceptance testing should take an additional 3.8 weeks (and the 1,600 hour estimate needs to be increased to 2,133 hours).

WBS estimate

Therefore, WBS predicts a total development time of 2,133 hours, or 15.2 weeks.

Jersey Maven setup Log4j2

1. In pom.xml, add:

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>2.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>2.0.1</version>
</dependency>

2. Add log4j configuration file to path: src/main/webapp/WEB-INF/classes
(create the classes folder manually)

3. Code:

Logger logger = LogManager.getLogger();
logger.info("info!!!!!!!!!!!!!!");

Serialize Object to Json indented string with Jersey

1. pom.xml - Use Jackson
        <dependency>
            <groupId>org.glassfish.jersey.media</groupId>
            <artifactId>jersey-media-json-jackson</artifactId>
        </dependency>   

2. Code:

        ObjectMapper mapper = new ObjectMapper();
        mapper.enable(SerializationFeature.INDENT_OUTPUT);
        String rJson = "";
        try {
            rJson = mapper.writeValueAsString(request);
            System.out.println(rJson);
        } catch (JsonProcessingException e1) {
            // TODO Auto-generated catch block
            e1.printStackTrace();
        }

Tomcat 7 connection pool with JNDI datasource and postgreSQL

1. Tomcat's content.xml

<Context>    <Resource name="jdbc/postgres" auth="Container"            type="javax.sql.DataSource" driverClassName="org.postgresql.Driver"            url="jdbc:postgresql://127.0.0.1:5432/mydb"            username="myuser" password="mypasswd" maxActive="20" maxIdle="10"  maxWait="-1"/>  </Context>

2. Web application's web.xml

<resource-ref>   <description>postgreSQL Datasource example</description>   <res-ref-name>jdbc/postgres</res-ref-name>   <res-type>javax.sql.DataSource</res-type>   <res-auth>Container</res-auth>  </resource-ref>

2. Code

    Connection conn = null;
        try {
            InitialContext cxt = new InitialContext();
            DataSource ds = (DataSource) cxt.lookup( "java:/comp/env/jdbc/postgres" );
            conn = ds.getConnection();
           
                 ...

                ...   
                while(rs.next()) {
                ...
                ...

                }
                rs.close();
            }
            conn.close();
        } catch (NamingException e1) {
            e1.printStackTrace();
        } catch (SQLException e) {
            e.printStackTrace();
        };

Maven Postgresql JDBC

1. Add dependency in pom.xml. e.g.
http://mvnrepository.com/artifact/org.postgresql/postgresql/9.3-1100-jdbc41
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.3-1100-jdbc41</version>
</dependency>

2. Coding:

Connection conn = null;
try
{
Class.forName("org.postgresql.Driver");
String url = "jdbc:postgresql://THE_HOST/THE_DATABASE";
conn = DriverManager.getConnection(url,"THE_USER", "THE_PASSWORD");
Statement st = conn.createStatement();
ResultSet rs = st.executeQuery("SELECT id, subject, permalink
FROM blogs ORDER BY id");
while ( rs.next() )
{
Blog blog = new Blog();
blog.id = rs.getInt ("id");
blog.subject = rs.getString ("subject");
blog.permalink = rs.getString ("permalink");
listOfBlogs.add(blog);
}
rs.close();
st.close();

}
catch (ClassNotFoundException e)
{
e.printStackTrace();
System.exit(1);
}
catch (SQLException e)
{
e.printStackTrace();
System.exit(2);
}

System server requirements

System hardware requirements
	CPU	File I/O	Database I/O	RAM
Management Web App	High	Low	High	Mid
Scoring Web App	Low	Low to Mid	Mid	Mid
Workers	High	High	Mid	High
PostgreSQL	Mid	High	N/A	High
DWH	N/A	N/A	High	N/A
ETL	High	Low to Mid	Mid	High
CRM	Mid	Low	Mid	Mid
Analytics Engine	High	Low	Mid	High
System Server Requirements
	File Server	Database Server	Web Server	Processing Server
Functionalities	SFTP, NFS	DWH, Operation DB	Web based application, CRM, …	Workers, ETL, Analysis Engine, …
CPU	Low (E3-1230v2 or existing)	Mid to High (E5-2420 dual / E52620 dual)	Mid (E3-1230v2/existing server)	High (E3-1230v2/existing server)
RAM	16G or above	32G or above	16G or above	16G or above
RAID	RAID 1/5/6	RAID 10	NO RAID / RAID 1	NO RAID
RAID Controller	Dell H710p or equivalent	Dell H710p or equivalent	N/A	N/A
HDD	TBD	1TB x 4	500GB	500GB
Quantity	x1	x1	xN	xN
Public IPs	x1	N/A	x2	N/A
Region	All	All	All	All
Scalability	Add drive for increase storage capacity.  Can add new server as required.	Single server can support up to 3 to 5 years.  Can add new server as required.	Scale Horizontally by adding new server as required.	Scale Horizontally by adding new server as required.
New Server Price	20K (Dell R220) to 40K (Dell R520)	50K+ HKD (Dell R520/R720)	12K HKD (Dell R220)	12K HKD (Dell R220)
Servers Allocation
Live site
	File Server	Database Server	Web Server	Processing Server
	New server Dell R220/R520	New server Dell R520/R720	HK
	20K/40K+ HKD - Stage 2+	50K+ HKD - Stage 2	existing server #1 - Management
		Support both all regions	existing server #2 - Scoring
		Database for Management, Scoring, Worker, DWH, etc.	Dell R210 - Workers, ETL, SFTP
			UK
			existing #3 - Management
			existing #4 - Scoring
	=R220, R210 #2 in Stage 1	= existing servers, R210 and R220 servers in Stage 1	Dell R220 #2 (12K HKD - Stage 1) - Worker, SFTP	=R210, R220 #2 in Stage 1
Standby site
	File Server	Database Server	Web Server	Processing Server
	New server Dell R220 RAID 1	New server Dell R220 with RAID 1	HK
	~20K HKD - Stage 3	(* Use low to mid-range standby server)	R220 #3 (12K HKD - Stage 1.5) - Management, Scoring, SFTP, …
		~20K HKD - Stage 3	1 x VM #1 (2K HKD - Stage 1.5) Worker
			UK
			R220 #3 (12K HKD - Stage 1.5) - Management, Scoring, SFTP, …
	=R220 #3, R220 #4, VM #1, VM #2 in Stage 1.5	=R220 #3, R220 #4, VM #1, VM #2 in Stage 1.5	1 x VM #2 (2K HKD - Stage 1.5) Worker	=VM #1, VM #2 in Stage 1.5

Linux, Ubuntu log files location

=> /var/log/daemon.log : Running services such as squid, ntpd and others
log message to this file

=> /var/log/dmesg : Linux kernel ring buffer log

=> /var/log/dpkg.log : All binary package log includes package
installation and other information

=> /var/log/faillog : User failed login log file

=> /var/log/kern.log : Kernel log file

=> /var/log/lpr.log : Printer log file

=> /var/log/mail.* : All mail server message log files

=> /var/log/mysql.* : MySQL server log file

=> /var/log/user.log : All userlevel logs

=> /var/log/xorg.0.log : X.org log file

=> /var/log/apache2/* : Apache web server log files directory

=> /var/log/lighttpd/* : Lighttpd web server log files directory

=> /var/log/fsck/* : fsck command log

=> /var/log/apport.log : Application crash report / log file

=> /var/log/messages : General log messages

=> /var/log/boot : System boot log

=> /var/log/debug : Debugging log messages

=> /var/log/auth.log : User login and authentication logs

Using pg_prove with pgTap

>psql -d my_database;
#> create extension pgtap;
>\q

>pg_prove -d my_database some_test_file.sql

install pg_prove for pgtap

https://github.com/theory/tap-parser-sourcehandler-pgtap

>sudo ./Build
>sudo ./Build installdeps
>sudo ./Build test
>sudo ./Build install

DONE

Postgresql test frame - pgTap

http://pgtap.org

Check Ubuntu ethernet interface speed.

>sudo ethtool eth0

Messaging tools - RabbitMQ

http://www.rabbitmq.com/

Measure the bandwidth between 2 servers

Computer A, B:
> sudo apt-get install iperf

Computer A:
> iperf -s

Computer B:
> iperf -c address_of_computer_A

Ubuntu Server reconfigure timezone

>date
>more /etc/timezone
>sudo dpkg-reconfigure tzdata
>sudo service cron stop
>sudo service cron start