Mar 10 2020

Standard Log Format

There isn’t a standard log format that spans all applications. There is a Common Log Format used by web servers, and there are logging frameworks that helps in the mechanics of producing logs. Unfortunately when it comes to composing the actual message that will be written out to log files the developer is left to his own devices.

Here are some best practices that I’ve found useful when adding logs to an application I’ve working on.

  • Each log event should be written to a single line.
  • Each log event should start with an event timestamp, preferably in UTC.
  • Include the log level in the log line.
  • Add a short description about the event that is being logged.
  • Included all pertinent data about the event in name=”value” pairs.
  • Quote all data values.
  • Use and log a transaction id as requests move through your applications.
  • Be consistent about field names used across applications.
  • Add a prefix to the field name to add context, for example for input or output data.
  • Mask out passwords, credit cards, and any other Personal Customer Information (PCI) data.
  • The log file name should include the application identifier and hostname.
  • Log files should be rotated out based on time and/or size.

Feb 4 2020

Software Development Checklist

During the summer, our team usually hires an intern. We find that the interns we hire are more than capable with the technical requirements but usually need some help on process. In order to help new team members adapt our process we’ve developed a number of useful checklists of some best practices. Here are some useful checklist to go through when working on new features.

Ticket Checklist

  • Add clear and concise title and description
  • Add reproducible steps
    • Use realistic sample data
  • Add specific acceptance criteria
  • Add related tickets

Code Checklist

  • Create your own feature branch
  • Check code into your branch often
  • Merge from master often
  • Run test suites on your own branch
  • Add and update test cases
  • And and update wiki documentation
  • Add comments to ticket

New Project Checklist

  • Source
    • Git Repository
    • Dependency Management
      • Maven
    • readme.md
  • Wiki documentation
    • Design Diagram
    • API Documentation
    • Common Configuration
    • AWS Permissions
    • Technical Specifications
      • Ports
      • Hardware
    • Sample log output
    • Sample Splunk queries
  • Splunk friendly logs
    • Splunk Dashboards
    • Splunk Alerts
  • Test Cases
    • JUnit
  • Release Management
    • Bamboo Plan
    • Ansible Playbook
    • Package
      • Docker Image
      • RPM Package

Jan 8 2020

Best Homebrew Formulae

Homebew is convenient package manager for the OS X. Once you install Homebrew on your Mac, you can then easily intall and uninstall tools, libraries, and applications through the brew command. The brew command accepts an install and unistall option, as well as list option that prints all currently installed applications.

You can install an application through a brew formula. There is a plathora of homebrew formulae.

The following homebrew formulae are the ones that I’ve found most useful as a software developer.

Best Homebrew Formulae

  • brew install mysql
    • Installs MySQL database
  • brew install awscli
    • Installs AWS CLI
  • brew install sqlite
    • Installs SQLite CLI
  • brew install groovy
    • Installs Groovy scripting language
  • brew install jq
    • Installs jq command to process JSON content
  • brew install jo
    • Installs jo command to generate JSON content
  • brew install bats
    • Installs test framework for Bash scripts
  • brew install wget
    • Installs wget command to fetch HTTP content
  • brew install markdown
    • Installs text-to-HTML converter
  • brew install hugo
    • Installs hugo static site generator


Jan 6 2020

POJOs Made Easy with Project Lombok

Every Java application has some sort of bean or Plain Old Java Object (POJO) to hold and manage state. Because of good Object Oriented Programming (OOP) principles we encapsulate field access with getter and setter methods. Most modern IDEs allow you to quickly create getters and getters. You can often find a Generate option under the Code menu from your favorite IDE. But generating getters and setters produces a lot of unnecessary boilerplate code.

With Project Lombok, you can reduce all that boilerplate getter and setter code with a few annotations.

The following bit of code will produce a Java class with the necessary getters and setters to access all the defined fields.

package com.juixe.core.pojo;

import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;

@ToString @EqualsAndHashCode
@Getter @Setter
public class Person {
   private Long id;
   private String firstName;
   private String lastName;
}

Given the Person class above, you can start coding the following.

   Person p = new Person();
   p.setFirstName("Boingo");
   p.setLastName("API");
   p.setId(1000l);

The getter and setter methods are added to the bytecode for the Person class by Project Lombok through the annotations. If you remove one of the annotations, like the @Setter annotation, then the setters methods will be deleted.

We just saw how a few annotation can generate getters, setters, equals, hashCode, and toString methods for simple Java classes. If you want all these methods generated for you, automatically, you can just use the @Data annotation like the following.

package com.juixe.core.pojo;

import lombok.Data;

@Data
public class Person {
   private Long id;
   private String firstName;
   private String lastName;
}

There is also a @Value annotation that is similar to @Data except that it is intended for immutable objects, so it doesn’t generate any setter methods.


Jan 2 2020

Human JSON

JSON is a versatile format to define data. I read and write JSON all the time but as a subset of JavaScript, it has a few limitations, most notably the lack of support for comments. Enter HJSON, or Human JSON. HJSON makes quotes around object names optional, it makes quotes around strings optional as well, it adds support for multi-line strings, and allows you to add comments in a number of ways. The following is valid HJSON.

{
  // use #, // or /**/ comments,
  // omit quotes for keys
  key: 1
  // omit quotes for strings
  contains: everything on this line
  // omit commas at the end of a line
  cool: {
    foo: 1
    bar: 2
  }
  // allow trailing commas
  list: [
    1,
    2,
  ]
  // and use multiline strings
  realist:
    '''
    My half empty glass,
    I will fill your empty half.
    Now you are half full.
    '''
}

There is an implementation of HJSON for most popular languages, like Java, Python, Go, Python, etc. I’ve used the hjson-java library to read HJSON file and converted it to JSON in a few lines.


Dec 27 2019

Create a HTTP Server in Java

Here is a quick tutorial on how to create a local embedded HTTP server in Java to serve JSON data. The whole code in less than 30 lines on Java.

package com.juixe.json.server;

import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;

import java.io.IOException;
import java.net.InetSocketAddress;

public class JSONServer implements HttpHandler {

    public static void main(String[] args) throws IOException  {
        HttpServer server = HttpServer.create(new InetSocketAddress(8080), 0);
        server.createContext("/data", new JSONServer());
        server.setExecutor(null);
        server.start();
    }

    @Override
    public void handle(HttpExchange he) throws IOException {
        String response = "{\"message\": \"Hello, HTTP Client\"}";

        he.getResponseHeaders().set("Content-Type", "application/json;charset=UTF-8");
        he.sendResponseHeaders(200, response.length());
        he.getResponseBody().write(response.getBytes());
        he.close();
    }
}

The HttpServer and HttpHandler classes are part of the JRE, so no additional libraries are required. Java also includes a HttpsServer which supports SSL. In the HttpHandler, you can return any data but for this example we are returning a simple JSON response.