Mongo Go Driver Custom Decoder

In this article, I will show you how to custom the decode function of mongo go driver. We use it to decode the query result from mongodb to struct that we have already set.

Right now I have a task to migrate the web app, from Scala to Go. Everything went well until we found one problem, when we save the datetime data in mongodb and we want to display the data on unix timestamp.

Actually, before this pull request, it will show the unix time on the json response when we use primitive.DateTime type on the struct. But now, there is an override function for marshalling.

Anyway I use this sample of data.

{
        "_id" : ObjectId("607041f4576f81145415bc16"),
        "name" : "Linggar",
        "createdAt" : ISODate("2016-03-03T08:00:00Z")
}

I tried to create function to override json response, to convert the data from datetime type to integer.

type Timestamp time.Time

func (t Timestamp) MarshalJSON() ([]byte, error) {
	return []byte(fmt.Sprint(t.Unix())), nil
}

type DataObj struct {
	ID        primitive.ObjectID `json:"_id" bson:"_id"`
	Name      string             `json:"name" bson:"name"`
	CreatedAt Timestamp          `json:"createdAt" bson:"createdAt"`
}

But when I decode the query result using this function.

cur.Decode(&result)

I got this error response.

error decoding key createdAt: cannot decode UTC datetime into a main.Timestamp

It seems mongo go driver cannot map the variable to the new type we create when it meet datetime type. But if we directly set the type to datetime on the struct, we didn’t find any error.

Actually we can set 2 structs to solve this problem, one struct for receiving the data from the query result, and one struct to reformat the data to the type we want. But I am just too lazy to write all struct twice 😛

After stuck with this problem for couple hours, finally I found the solution from the documentation! We can create custom decoder for one specific type. We create the registry and add the custom rule when unmarshalling the data with this function: bson.UnmarshalWithRegistry. To use that function, we have to decode to the data first to the bytes format with function DecodeBytes().

This is the code I use to decode and convert the data. First, we create the custom registry.

type Timestamp int64

type DataObj struct {
	ID        primitive.ObjectID `json:"_id" bson:"_id"`
	Name      string             `json:"name" bson:"name"`
	CreatedAt Timestamp          `json:"createdAt" bson:"createdAt"`
}

func createCustomRegistry() *bsoncodec.RegistryBuilder {
	var primitiveCodecs bson.PrimitiveCodecs
	rb := bsoncodec.NewRegistryBuilder()
	bsoncodec.DefaultValueEncoders{}.RegisterDefaultEncoders(rb)
	bsoncodec.DefaultValueDecoders{}.RegisterDefaultDecoders(rb)
	// register our new type
	myNumberType := reflect.TypeOf(Timestamp(0))
	// read the datetime type and convert to integer
	rb.RegisterTypeDecoder(
		myNumberType,
		bsoncodec.ValueDecoderFunc(func(_ bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
			// this is the function when we read the datetime format
			read, err := vr.ReadDateTime()
			if err != nil {
				return err
			}
			val.SetInt(read)
			return nil
		}),
	)
	primitiveCodecs.RegisterPrimitiveCodecs(rb)
	return rb
}

Then we decode the query and map the result.

var result DataObj
decoded, _ := cur.DecodeBytes()
var customRegistry = createCustomRegistry().Build()
err = bson.UnmarshalWithRegistry(customRegistry, []byte(decoded), &result)

And we will get the unix timestamp on the result!

$ go run main.go 
{"_id":"607041f4576f81145415bc16","name":"Linggar","createdAt":1456992000000}

Conclusion

Mongo go driver has so many functions to map and convert the data. We can create custom decoder and encoder to map the data to the type we want, without creating so many structs to decode and reformat the data.

Finally, if you can’t find any tutorial to solve your problem, please don’t give up, perhaps you can find your solution on the documentation 😀

Continue Reading

Nginx Reverse Proxy Flask-restx Swagger Configuration

Few days ago, I was setting up nginx reverse proxy for my project. I was using gunicorn for the HTTP Server with flask-restx for the web framework. Actually this was the first time for me using this combination. I usually use uwsgi for the deployment.

It was quite simple to deploy flask with gunicorn. You can use port binding.

gunicorn --bind 0.0.0.0:9999 app:app

Or you can use unix sock.

gunicorn --workers 2 --bind unix:sockname.sock -m 644 app:app

I was using the second option, using unix sock, so this was the configuration I use for nginx reverse proxy.

location / {
    root /path/to/project/;
    proxy_pass http://unix:/path/to/unix.sock;
}

After finishing the configuration and testing the service, although I could access all function of the web service, I found out a problem when accessing the documentation.

The documentation page was still getting the swagger data from localhost, because we only set the reverse proxy option to the project.

Then, I realized that we have to config the proxy header also inside the configuration, so the documentation page will request the data from the domain we setup, instead of localhost. So this is my nginx configuration now.

location / {
    root /path/to/project/;
    proxy_pass http://unix:/path/to/unix.sock;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
}

We use proxy_set_header Host $host to set the proxy request to same name as the server_name directive, so it will request the domain name, not the original host where we deploy the service.

I also use proxy_set_header X-Forwarded-Proto $scheme because I am using SSL for the domain. If you don’t use that option, the request will be blocked, because of mixed content issue (domain with SSL request to non SSL domain).

Then, after adding that two lines and restarting the service, we can load the data from the swagger 😁

Well, maybe this is all I can share. If you found any problem when using gunicorn and flask restx with nginx as reverse proxy, you can try this solution. Thank you!

Continue Reading

Hot Score!

Few weeks ago, my office had a sharing session. My friend shared how he created a tinder bot to do automatic swiping using selenium. Actually the main point of the presentation is how to use selenium (GUI or script) to finish your daily tasks automatically. But, I was interested with the tinder bot. I though, instead of doing mass swiping to the right or left, we can choose whether he/she is based on our preference or not. If he/she is based on our preference, then swipe right.

After googling some materials, I found some projects about beauty facial prediction, but all projects mostly used SCUT-FBP5500 dataset. The SCUT-FBP5500 dataset has totally 5500 frontal faces with diverse properties, labelled by 60 volunteers. But, what I want to do is to predict the image based on my preference, not by others.

Then, I found this BeautyPredict. This project is quite simple. I tried to understand the process: preparing the data, training, predicting the score, and of course the dataset pattern. After understanding the process, finally I have 2 tasks to be done:

  • Create the dataset tool.
  • Refactor the BeautyPredict because there are some deprecated function.

Create the dataset tool

I am using flask to build the web app. And, because i am not good at design, thanks to adminlte for helping me with the design. Of course, it is mobile friendly.

And for images data, I have around 3000 face data in my storage, actually I got that data from my research few years ago. Finally, that data has a purpose!

Refactor the beauty predict

Although the project is quite simple, but there are some deprecated functions, so it will be failed if we run it with newer version of the python libraries.

Time to predict

After labeling around 1000 images, I tried to process the images and training the data, then I picked random image from instagram and predict it, it’s quite good and actually the result was as I expected.

Sorry, I have to blur the image 😛

Conclusion

So, with this project, we can rate someone whether he/she is based on our preference. If you want to know more about how the system score the image, you can visit the BeautyPredict project on Github.

And I just want to say this on the conclusion, that I believe, all men are handsome and all women are beautiful, but all people have their own preference.

Anyway, this was just my weekend research and it is not finished yet, so I didn’t write the complete instruction. But if you want to know more about the project, you can visit here.

Continue Reading

Swagger implementation for Go HTTP

The most important thing when building the web API is creating API documentation. With API documentation, it will be easier for another programmer to integrate their application with your web service.

In this post, I will show how to integrate Go net/http package and swagger using http-swagger package.

Installation

First, we have to install swag package.

$ go get github.com/swaggo/swag/cmd/swag

Initialization

Then we have to generate first `docs` folder, so we could import it to the application. Please run it on root folder of the project.

$ swag init

After generate docs folder, we import the package and generated docs. In this case, my project name is goweb.

import (
    httpSwagger "github.com/swaggo/http-swagger"
    _ "goweb/docs"
)

We have to initialize the general info for the documentation. It will be placed before main function. It describes the name, description, contact, etc. For the detail, we could see from the swag documentation.

// @title Go Restful API with Swagger
// @version 1.0
// @description Simple swagger implementation in Go HTTP

// @contact.name Linggar Primahastoko
// @contact.url https://linggar.asia
// @contact.email x@linggar.asia

// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name Authorization

// @host localhost:8082
// @BasePath /
func main() {
	...
}

We add these blocks to tell swagger that we will be using token in header for the authentication. So it means, if we want directly access some pages that using authentication, we have to pass Authorization: Basic xxxyyytokenzzzaaa

// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name Authorization

Implementation

We will write the documentation on each function and placed before the function itself. For the parameter and response, we just have to write the struct name on the documentation because we already have defined the structs. This is the part I like the most, because we don’t have to define the structure again in the documentation part.

type authParam struct {
	Username string `json:"username"`
	Password string `json:"password"`
}

type authResp struct {
	Token string `json:"token"`
}

// authLogin godoc
// @Summary Auth Login
// @Description Auth Login
// @Tags auth
// @ID auth-login
// @Accept  json
// @Produce  json
// @Param authLogin body main.authParam true "Auth Login Input"
// @Success 200 {object} main.authResp
// @Router /auth/login [post]
func authLogin(w http.ResponseWriter, r *http.Request) {
    ...
}

That is the login function, so we don’t have to pass the token to access the resource. But what if we want to authorize with token in swagger? Yes, we only have to add // @Security ApiKeyAuth.

// userProfile godoc
// @Summary User Profile
// @Description User Profile
// @Tags users
// @ID user-profile
// @Accept  json
// @Produce  json
// @Success 200 {object} main.meResp
// @Router /users/profile [get]
// @Security ApiKeyAuth
func userProfile(w http.ResponseWriter, r *http.Request) {
    ...
}

Then finally we have to add the routing and swagger function in the main function, so we could access it in the browser.

func main() {
	http.HandleFunc("/swagger/", httpSwagger.WrapHandler)
        ...
	http.ListenAndServe(":8082", nil)
}

After we setup all the components, we have to generate the documentation with swag command again.

$ swag init
2019/01/29 20:29:34 Generate swagger docs....
2019/01/29 20:29:34 Generate general API Info
2019/01/29 20:29:34 Generating main.authParam
2019/01/29 20:29:34 Generating main.authResp
2019/01/29 20:29:34 Generating main.meResp
2019/01/29 20:29:34 create docs.go at  docs/docs.go

Finally, we could build and run the application. We could access the documentation in http://<host>:<port>/swagger/index.html.

Example

This is the full implementation to integrate swagger with Go net/http package. This is only an example. It is not a perfect code that you could just copy and paste into your web app, because actually, we could write with better handler, wrapper, and middleware. For the detail of username, password, and token, you could get that information from the code. So, please enjoy! 😀

main.go

package main

import (
	"encoding/json"
	httpSwagger "github.com/swaggo/http-swagger"
	"log"
	"net/http"
	"strings"
	_ "goweb/docs"
)

type errorResp struct {
	Error string `json:"error"`
}

type indexResp struct {
	Message string `json:"message"`
}

type meResp struct {
	Username string `json:"username"`
	Fullname string `json:"fullname"`
}

type authResp struct {
	Token string `json:"token"`
}

type authParam struct {
	Username string `json:"username"`
	Password string `json:"password"`
}


// writeJSON provides function to format output response in JSON
func writeJSON(w http.ResponseWriter, code int, payload interface{}) {
	resp, err := json.Marshal(payload)
	if err != nil {
		log.Println("Error Parsing JSON")
	}

	w.Header().Set("Content-type", "application/json")
	w.WriteHeader(code)
	w.Write(resp)
}

// basicAuthMW is middleware function to check whether user is authenticated or not
// actually you could write better code for this function
func basicAuthMW(w http.ResponseWriter, r *http.Request) map[string]string {
	errorAuth := errorResp{
		Error: "Unauthorized access",
	}

	header := r.Header.Get("Authorization")
	if header == "" {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	apiKey := strings.Split(header, " ")

	if len(apiKey) != 2 {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	if apiKey[0] != "Basic" {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	users := map[string]map[string]string{
		"28b662d883b6d76fd96e4ddc5e9ba780": map[string]string{
			"username": "linggar",
			"fullname": "Linggar Primahastoko",
		},
	}

	if _ , ok := users[apiKey[1]]; !ok {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	return users[apiKey[1]]
}

func decodePost(r *http.Request, structure interface{}) {
	decoder := json.NewDecoder(r.Body)
	err := decoder.Decode(structure)
	if err != nil {
		log.Println("Error parsing post data")
	}
}

// authLogin godoc
// @Summary Auth Login
// @Description Auth Login
// @Tags auth
// @ID auth-login
// @Accept  json
// @Produce  json
// @Param authLogin body main.authParam true "Auth Login Input"
// @Success 200 {object} main.authResp
// @Router /auth/login [post]
func authLogin(w http.ResponseWriter, r *http.Request) {
	var param authParam
	decodePost(r, &param)

	if param.Username == "linggar" && param.Password == "linggar" {
		respAuth := authResp{
			Token: "28b662d883b6d76fd96e4ddc5e9ba780",
		}
		writeJSON(w, 200, respAuth)
	} else {
		failResp := errorResp{
			Error: "Wrong username/password",
		}
		writeJSON(w, 401, failResp)
	}
}

// userProfile godoc
// @Summary User Profile
// @Description User Profile
// @Tags users
// @ID user-profile
// @Accept  json
// @Produce  json
// @Success 200 {object} main.meResp
// @Router /users/profile [get]
// @Security ApiKeyAuth
func userProfile(w http.ResponseWriter, r *http.Request) {
	info := basicAuthMW(w, r)

	if len(info) == 0 {
		return
	}

	respMe := meResp{
		Username: info["username"],
		Fullname: info["fullname"],
	}

	writeJSON(w, 200, respMe)
}

// @title Go Restful API with Swagger
// @version 1.0
// @description Simple swagger implementation in Go HTTP

// @contact.name Linggar Primahastoko
// @contact.url https://linggar.asia
// @contact.email x@linggar.asia

// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name Authorization

// @host localhost:8082
// @BasePath /
func main() {
	http.HandleFunc("/swagger/", httpSwagger.WrapHandler)

	http.HandleFunc("/auth/login", authLogin)
	http.HandleFunc("/users/profile", userProfile)

	http.ListenAndServe(":8082", nil)
}

Continue Reading