AWS SAM local start-api, connecting to db/service when both are running in Docker Desktop

I am current building/testing a node/NestJS based API, that I am deploying to AWS Lambda.
I chose AWS SAM for the deployment tooling, as the integration is simple, lightweight and declarative.

Running the service locally, npm run start:dev, works perfectly and connects to the DB, mongo, hosted in Docker without a problem

Before deploying, I webpack the project to 1 file and run it in a lambda equivalent environment, which also runs on Docker, using "sam local start-api" and here is where the problem appears... 

My service is unable to connect to mongo, when both are running in Docker.

After the usual questions... is MongoDB running, can I connect to it, etc, I knew there was an issue.

Initially all I got was:

Function 'xxxxxx' timed out after 3 seconds

Really helpful.

After some messing around, I eventually reset the SAM timeout, retried and got:

MongoServerSelectionError: connect EHOSTUNREACH 127.0.0.1:27017

Now we are getting somewhere... it is unable to reach mongo.

To cut a long story short, after trying many different things, I eventually found the correct IP to call Mongo running in Docker. It's right down the bottom of the config if you are interested.

More importantly, I then looked up how to extract it directly...

docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' mongodb

Call that, stick the IP address in your environment variables, and tada.. bob's your aunties brother.

Should also work for any other db / service that you need to connect to, when running it all in Docker.

SE-Linux: check and deactivate

I quite often run into security problems when testing stuff on my machine.
This is to remind me how to deactivate SE-Linux, so I can see if it is causing the problem...

sudo getenforce
> Enforcing

If "sudo getenforce" returns "Enforcing", you have SE-Linux, so turn it off.

sudo setenforce 0

then

sudo getenforce
> Permissive

... now test the thing...

making Makefile Self documenting

I have use 'make' on the last few projects, just to control all the CMD line build tools we now need to use.

I found a neat trick a little while ago, that automatically shows you what CMDs are available.

#
# Why use Makefile?
# because you get help lists & auto-complete on complex commands
#

help:           ## Show this help.
 @fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/\\$$//' | sed -e 's/##//'

# linebreak
: ## ======================================================================

# make all output silent - ie: no CMDs shown
#.SILENT:

gitStatus: ## show GIT status
  @git status -b --column -s

Simple copy this into the top of your Makefile, and add comments as shown using ## after the cmd name.
Now when you call 'make' without any arguments, you will see a list of available commands with their descriptions.

A brilliant tool for projects with loads of cmd line tools to remember.

How to get AWS EC2 instance metadata from Java

I have recently found that I can access EC2 machine instance metadata using curl.
Which is brilliantly useful.

However, I wanted to get the same data inside my Java applications.
So I build a utility to make it available...

package com.mendeley.weblet.oauth.utility;

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;
import com.sun.jersey.api.client.config.ClientConfig;
import com.sun.jersey.client.apache4.config.DefaultApacheHttpClient4Config;

import java.net.URI;


/**
 * Utility class to access EC2 instance Meta-data
 *
 * @See (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
 */
public class EC2Metadata {

    /**
     * Some of the Metadata types
     */
    public enum Type{
        ami_id("ami-id"),
        ami_launch_index("ami-launch-index"),
        ami_manifest_path("ami-manifest-path"),
        block_device_mapping("block-device-mapping/"),
        hostname("hostname"),
        instance_action("instance-action"),
        instance_id("instance-id"),
        instance_type("instance-type"),
        kernel_id("kernel-id"),
        local_hostname("local-hostname"),
        local_ipv4("local-ipv4"),
        mac("mac"),
        network("network/"),
        placement("placement/"),
        public_hostname("public-hostname"),
        public_ipv4("public-ipv4"),
        public_keys("public-keys/"),
        reservation_id("reservation-id"),
        security_groups("security-groups"),
        services("services/");

        private String name;

        private Type(String name){
            this.name = name;
        }
    }


    /**
     * Get metadata using Type enum
     *
     * @param type
     * @param timeout
     * @param defaultValue
     * @return
     */
    public static String retrieveMetadata(Type type, int timeout,String defaultValue) {
        return retrieveMetadata(type.toString(),timeout,defaultValue);
    }

    /**
     * Get metadata by String value
     * Allows further metadata to be retrieved.
     * See AWS documentation for more info.
     *
     * @param type
     * @param timeout
     * @param defaultValue
     * @return
     */
    public static String retrieveMetadata(String type, int timeout,String defaultValue) {
        try{
            URI uri = URI.create("http://169.254.169.254/latest/meta-data/" + type);
            System.out.println(uri.toString());

            ClientConfig config = new DefaultApacheHttpClient4Config();
            config.getProperties().put(ClientConfig.PROPERTY_CONNECT_TIMEOUT,timeout);

            Client client = Client.create(config);
            WebResource webResource = client.resource(uri);

            ClientResponse response = webResource.get(ClientResponse.class);

            String results = response.getEntity(String.class);
            return results;

        }catch(Throwable t){
            return defaultValue;
        }
    }


    public static void main(String[] args) {
        String myEC2Id = retrieveMetadata(Type.instance_id,1000, "null");
        System.out.println("The Instance Id is " + myEC2Id + " .");
    }

}

I will try to get it into Github at some point, until then, please use it as you see fit.
Obviously I take no responsibility for the end of the world as you know it, should it occur.

Using ENYOjs to build a Chrome App, a few pointers

As per usual, I am trying something new... discovered a few problems, then solutions, and need to remind myself about the solutions for the next time.

I am building a Chrome App using ENYO, partly because I actually need an app right now, and partly because I think ENYO will make a perfect quick build solution for the next time.

Anyway...

Problem 1: document.write error
Solution: use renderInto.

var app = new UMLEditor({name: "app"});

app.renderInto(document.body);        

Problem 2: localstorage warning
Solution: refactor to use chrome.local.storage
   Open: source/data/sources/localStorage.js
   Change: e.localstorage to chrome.local.storage


Simple problems, simple solutions, but annoying to look up again next time.

Groovy Eclipse compiler versions and ShortTypeHandling

Just a short note to remind myself.

If you come across a classNotFoundException: org.codehaus.groovy.runtime.typehandling.ShortTypeHandling

Then the chances are you have 2 versions of Groovy being used.
1 pre Groovy 2.3.5, and 1 after it.

Reminder: use this to investigate (remember to look at parents too)
mvn dependency:tree -Dverbose

I believe this is a Java 8 compatibility change, but would need to investigate to be sure.

Anyway, go here to find out more:
http://glaforge.appspot.com/article/groovy-2-3-5-out-with-upward-compatibility 

UML Diagram Editor is ready

I have been working on a small chrome App, a tool for building UML diagrams from text. It leverages PlantUML, which I have found to be a great tool, but seriously lacks a decent desktop app to use it. So I build one, or at least the start of one.

You can find it here:
UML Diagram Editor

I have been teaching my colleagues to create Sequence Diagrams to plan agile stories. They can capture, communicate and confirm all the tasks required for a story with all the participants and interested parties, before even creating the tickets to achieve it.

As far as clear communication goes, I now see sequence diagrams as equal importance to using BDD style acceptance criteria. As both allow everyone up and down the technical/business chains to understand exactly what it planned, and what is required.

To use sequence diagrams for you project...

Keep it simple, only one feature per story.
It is more likely to finish on time, and easier to describe fully.
Any scope creep goes into a new story, to be prioritized into the backlog.

When planning the story, simply create a sequence diagram describing the flow that will occur when the feature is finished. Include UIs, acceptance criteria, validation, message formats, button clicks, and anything else you believe is needed to describe the solution you are going to build.

Now just look at your completed diagram, the tasks should fall out of it very readily, separated by component, and position in the flow. Dependencies will also be very obvious, so those tasks can be done in order.


I have found this to be a superb solution to the issue of communication within a team, as well as externally, as it provides an easy point of focus and discussion. When it is easy to edit or modify the UML diagram, it becomes a live document, and the only one needed if the tasks are kept small.