Swagger implementation for Go HTTP

The most important thing when building the web API is creating API documentation. With API documentation, it will be easier for another programmer to integrate their application with your web service.

In this post, I will show how to integrate Go net/http package and swagger using http-swagger package.

Installation

First, we have to install swag package.

$ go get github.com/swaggo/swag/cmd/swag

Initialization

Then we have to generate first `docs` folder, so we could import it to the application. Please run it on root folder of the project.

$ swag init

After generate docs folder, we import the package and generated docs. In this case, my project name is goweb.

import (
    httpSwagger "github.com/swaggo/http-swagger"
    _ "goweb/docs"
)

We have to initialize the general info for the documentation. It will be placed before main function. It describes the name, description, contact, etc. For the detail, we could see from the swag documentation.

// @title Go Restful API with Swagger
// @version 1.0
// @description Simple swagger implementation in Go HTTP

// @contact.name Linggar Primahastoko
// @contact.url http://linggar.asia
// @contact.email x@linggar.asia

// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name Authorization

// @host localhost:8082
// @BasePath /
func main() {
	...
}

We add these blocks to tell swagger that we will be using token in header for the authentication. So it means, if we want directly access some pages that using authentication, we have to pass Authorization: Basic xxxyyytokenzzzaaa

// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name Authorization

Implementation

We will write the documentation on each function and placed before the function itself. For the parameter and response, we just have to write the struct name on the documentation because we already have defined the structs. This is the part I like the most, because we don’t have to define the structure again in the documentation part.

type authParam struct {
	Username string `json:"username"`
	Password string `json:"password"`
}

type authResp struct {
	Token string `json:"token"`
}

// authLogin godoc
// @Summary Auth Login
// @Description Auth Login
// @Tags auth
// @ID auth-login
// @Accept  json
// @Produce  json
// @Param authLogin body main.authParam true "Auth Login Input"
// @Success 200 {object} main.authResp
// @Router /auth/login [post]
func authLogin(w http.ResponseWriter, r *http.Request) {
    ...
}

That is the login function, so we don’t have to pass the token to access the resource. But what if we want to authorize with token in swagger? Yes, we only have to add // @Security ApiKeyAuth.

// userProfile godoc
// @Summary User Profile
// @Description User Profile
// @Tags users
// @ID user-profile
// @Accept  json
// @Produce  json
// @Success 200 {object} main.meResp
// @Router /users/profile [get]
// @Security ApiKeyAuth
func userProfile(w http.ResponseWriter, r *http.Request) {
    ...
}

Then finally we have to add the routing and swagger function in the main function, so we could access it in the browser.

func main() {
	http.HandleFunc("/swagger/", httpSwagger.WrapHandler)
        ...
	http.ListenAndServe(":8082", nil)
}

After we setup all the components, we have to generate the documentation with swag command again.

$ swag init
2019/01/29 20:29:34 Generate swagger docs....
2019/01/29 20:29:34 Generate general API Info
2019/01/29 20:29:34 Generating main.authParam
2019/01/29 20:29:34 Generating main.authResp
2019/01/29 20:29:34 Generating main.meResp
2019/01/29 20:29:34 create docs.go at  docs/docs.go

Finally, we could build and run the application. We could access the documentation in http://<host>:<port>/swagger/index.html.

Example

This is the full implementation to integrate swagger with Go net/http package. This is only an example. It is not a perfect code that you could just copy and paste into your web app, because actually, we could write with better handler, wrapper, and middleware. For the detail of username, password, and token, you could get that information from the code. So, please enjoy! 😀

main.go

package main

import (
	"encoding/json"
	httpSwagger "github.com/swaggo/http-swagger"
	"log"
	"net/http"
	"strings"
	_ "goweb/docs"
)

type errorResp struct {
	Error string `json:"error"`
}

type indexResp struct {
	Message string `json:"message"`
}

type meResp struct {
	Username string `json:"username"`
	Fullname string `json:"fullname"`
}

type authResp struct {
	Token string `json:"token"`
}

type authParam struct {
	Username string `json:"username"`
	Password string `json:"password"`
}


// writeJSON provides function to format output response in JSON
func writeJSON(w http.ResponseWriter, code int, payload interface{}) {
	resp, err := json.Marshal(payload)
	if err != nil {
		log.Println("Error Parsing JSON")
	}

	w.Header().Set("Content-type", "application/json")
	w.WriteHeader(code)
	w.Write(resp)
}

// basicAuthMW is middleware function to check whether user is authenticated or not
// actually you could write better code for this function
func basicAuthMW(w http.ResponseWriter, r *http.Request) map[string]string {
	errorAuth := errorResp{
		Error: "Unauthorized access",
	}

	header := r.Header.Get("Authorization")
	if header == "" {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	apiKey := strings.Split(header, " ")

	if len(apiKey) != 2 {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	if apiKey[0] != "Basic" {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	users := map[string]map[string]string{
		"28b662d883b6d76fd96e4ddc5e9ba780": map[string]string{
			"username": "linggar",
			"fullname": "Linggar Primahastoko",
		},
	}

	if _ , ok := users[apiKey[1]]; !ok {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	return users[apiKey[1]]
}

func decodePost(r *http.Request, structure interface{}) {
	decoder := json.NewDecoder(r.Body)
	err := decoder.Decode(structure)
	if err != nil {
		log.Println("Error parsing post data")
	}
}

// authLogin godoc
// @Summary Auth Login
// @Description Auth Login
// @Tags auth
// @ID auth-login
// @Accept  json
// @Produce  json
// @Param authLogin body main.authParam true "Auth Login Input"
// @Success 200 {object} main.authResp
// @Router /auth/login [post]
func authLogin(w http.ResponseWriter, r *http.Request) {
	var param authParam
	decodePost(r, &amp;param)

	if param.Username == "linggar" &amp;&amp; param.Password == "linggar" {
		respAuth := authResp{
			Token: "28b662d883b6d76fd96e4ddc5e9ba780",
		}
		writeJSON(w, 200, respAuth)
	} else {
		failResp := errorResp{
			Error: "Wrong username/password",
		}
		writeJSON(w, 401, failResp)
	}
}

// userProfile godoc
// @Summary User Profile
// @Description User Profile
// @Tags users
// @ID user-profile
// @Accept  json
// @Produce  json
// @Success 200 {object} main.meResp
// @Router /users/profile [get]
// @Security ApiKeyAuth
func userProfile(w http.ResponseWriter, r *http.Request) {
	info := basicAuthMW(w, r)

	if len(info) == 0 {
		return
	}

	respMe := meResp{
		Username: info["username"],
		Fullname: info["fullname"],
	}

	writeJSON(w, 200, respMe)
}

// @title Go Restful API with Swagger
// @version 1.0
// @description Simple swagger implementation in Go HTTP

// @contact.name Linggar Primahastoko
// @contact.url http://linggar.asia
// @contact.email x@linggar.asia

// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name Authorization

// @host localhost:8082
// @BasePath /
func main() {
	http.HandleFunc("/swagger/", httpSwagger.WrapHandler)

	http.HandleFunc("/auth/login", authLogin)
	http.HandleFunc("/users/profile", userProfile)

	http.ListenAndServe(":8082", nil)
}

Continue Reading

Kubectl, Master NotReady

When I setup the kubernetes at the first time, after execute kubeadm init and apply the pod networks plugin, flannel or calico, I see that the master status is not ready.

linx@node-1:~$ kubectl get nodes
NAME      STATUS     ROLES    AGE     VERSION
node-1   NotReady   master   3m44s   v1.13.0

After reading some articles, I found out that we also need apply weave-kube plugin. I try to apply weave kube plugin, then the master status becomes ready 😀

linx@node-1:~$ kubectl apply -f https://git.io/weave-kube-1.6
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.extensions/weave-net created
linx@node-1:~$ kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
node-1   Ready    master   8m55s   v1.13.0
Continue Reading

Delete Index With PyArango

I am using Python to create some background services and web service in the office. For the database, we are using Arangodb for our main database, so we use PyArango to connect to the database.

At a time, I need to delete index in the database collection, but I cannot find any example related to index deletion. So after reading the code and API documentation, I will share the example here.

Connect to database

from pyArango.connection import Connection

conn = Connection(arangoURL='http://<host>:<port>', username='user', password='pass')
db = conn['dbname']

collname = db['collname']

To get index list, we can use getIndexes()

print(collname.getIndexes())

We will get the index list and the index object

{'primary': {'collname/0': <pyArango.index.Index object at 0x7f44404b85c0>}, 'hash': {'collname/7490651': <pyArango.index.Index object at 0x7f44404b8860>, 'collname/7490654': <pyArango.index.Index object at 0x7f44404b8828>}, 'skiplist': {}, 'geo': {}, 'fulltext': {}}

Before we continue, we could see the detail of index class at the github page https://github.com/tariqdaouda/pyArango/blob/master/pyArango/index.py. Then we will try list the attributes of index object.

print(dir(collname.getIndexes()['hash']['collname/7490654']))

Index attributes:

['URL', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_create', 'collection', 'connection', 'delete', 'indexesURL', 'infos']

We could see the index information with collname.getIndexes()[‘hash’][‘collname/7490654’].infos

{'deduplicate': True, 'fields': ['keyTime'], 'id': 'collname/7490654', 'selectivityEstimate': 0.007465129984938773, 'sparse': False, 'type': 'hash', 'unique': False}

If we want to delete the index, just call the delete method.

collname.getIndexes()['hash']['collname/7490654'].delete()

We will get any success response after deleting the index.

Continue Reading

Removing Stop Words

I have a task to do word counting for some articles. The detail of the task is getting the list of id from Elasticsearch, get the content from ArangoDB, then do some text processing to clean the content and counting the word frequency.

After did it with Scala, Go, and Python, I found out that it is very slow when I am doing it with Python. Doing it with Scala and Go only take around 3-4 seconds to process 12,563 articles. But when we do with Python, it takes around 15-18 seconds. And after do some profiling, finally I found out that it is very slow to remove any stopwords from big number of articles. I am using common method in Python to remove the stopwords.

stopwords = open('stopwords.txt').read().splitlines()
word_count = Counter([i for i in all_words.split() if len(i) > 4 if i not in stopwords])

It needs around 16 seconds to do the stopwords removing and counting with Counter function.

$ time python counting.py --start 2018-09-01 --end 2018-09-30
'get_es_data'  802.65 ms
'get_arango_data'  449.94 ms
'clean_texts'  1286.84 ms
'word_count'  13980.26 ms
Total articles: 12563
Top words: ['jokowi', 'presiden', 'indonesia', 'ketua', 'partai', 'prabowo', 'widodo', 'jakarta', 'negara', 'games']
Total time execution (in seconds): 16.5261652469635

real    0m16.647s
user    0m15.668s
sys     0m0.288s

Then after reading some methods about how some people do text processing, I found this good article. It is said that Python dictionaries use hash tables, this means that a lookup operation (e.g., if x in y) is O(1). A lookup operation in a list means that the entire list needs to be iterated, resulting in O(n) for a list of length n.

So I try to convert the stopwords list to dictionaries.

stopwords = open('stopwords.txt').read().splitlines()
stop_dicts = dict.fromkeys(stopwords, True)
words = Counter([i for i in words.split() if len(i) > 4 if i not in stop_dicts])

Then we will get the faster result than the previous one with the list type.

$ time python counting.py --start 2018-09-01 --end 2018-09-30
'get_es_data'  787.50 ms
'get_arango_data'  461.97 ms
'clean_texts'  1311.56 ms
'word_count'  785.08 ms
Total articles: 12563
Top words: ['jokowi', 'presiden', 'indonesia', 'ketua', 'partai', 'prabowo', 'widodo', 'jakarta', 'negara', 'games']
Total time execution (in seconds): 3.3524017333984375

real    0m3.503s
user    0m2.560s
sys     0m0.220s

It only takes 3 seconds to do all process. It is very fast, isn’t it? 😀

Continue Reading
1 2 3 18