Hot Score!

Few weeks ago, my office had a sharing session. My friend shared how he created a tinder bot to do automatic swiping using selenium. Actually the main point of the presentation is how to use selenium (GUI or script) to finish your daily tasks automatically. But, I was interested with the tinder bot. I though, instead of doing mass swiping to the right or left, we can choose whether he/she is based on our preference or not. If he/she is based on our preference, then swipe right.

After googling some materials, I found some projects about beauty facial prediction, but all projects mostly used SCUT-FBP5500 dataset. The SCUT-FBP5500 dataset has totally 5500 frontal faces with diverse properties, labelled by 60 volunteers. But, what I want to do is to predict the image based on my preference, not by others.

Then, I found this BeautyPredict. This project is quite simple. I tried to understand the process: preparing the data, training, predicting the score, and of course the dataset pattern. After understanding the process, finally I have 2 tasks to be done:

  • Create the dataset tool.
  • Refactor the BeautyPredict because there are some deprecated function.

Create the dataset tool

I am using flask to build the web app. And, because i am not good at design, thanks to adminlte for helping me with the design. Of course, it is mobile friendly.

And for images data, I have around 3000 face data in my storage, actually I got that data from my research few years ago. Finally, that data has a purpose!

Refactor the beauty predict

Although the project is quite simple, but there are some deprecated functions, so it will be failed if we run it with newer version of the python libraries.

Time to predict

After labeling around 1000 images, I tried to process the images and training the data, then I picked random image from instagram and predict it, it’s quite good and actually the result was as I expected.

Sorry, I have to blur the image 😛

Conclusion

So, with this project, we can rate someone whether he/she is based on our preference. If you want to know more about how the system score the image, you can visit the BeautyPredict project on Github.

And I just want to say this on the conclusion, that I believe, all men are handsome and all women are beautiful, but all people have their own preference.

Anyway, this was just my weekend research and it is not finished yet, so I didn’t write the complete instruction. But if you want to know more about the project, you can visit here.

Continue Reading

Implement Chained Dropdown List in Django Admin

It is not difficult to create chained dropdown list in Django with custom template, but in one project, I need to create it in Django admin page.

I need to create chained dropdown list of area in my app. For your information, my country, Indonesia, has four levels of area.

And this is my area model.

from django.db import models


class Provinsi(models.Model):
    name = models.CharField(max_length=255, null=True, blank=True)
    code = models.IntegerField(null=True, blank=True)

    class Meta:
        db_table = 'provinsi'
        verbose_name_plural = 'provinsi'

    def __str__(self):
        return self.name


class Kabupaten(models.Model):
    name = models.CharField(max_length=255, null=True, blank=True)
    code = models.IntegerField(null=True, blank=True)
    provinsi_code = models.IntegerField(null=True, blank=True)
    provinsi = models.ForeignKey(
        Provinsi,
        on_delete=models.CASCADE
    )

    class Meta:
        db_table = 'kabupaten'
        verbose_name_plural = 'kabupaten'

    def __str__(self):
        return self.name


class Kecamatan(models.Model):
    name = models.CharField(max_length=255, null=True, blank=True)
    code = models.IntegerField(null=True, blank=True)
    kabupaten_code = models.IntegerField(null=True, blank=True)
    kabupaten = models.ForeignKey(
        Kabupaten,
        on_delete=models.CASCADE
    )

    class Meta:
        db_table = 'kecamatan'
        verbose_name_plural = 'kecamatan'

    def __str__(self):
        return self.name


class Kelurahan(models.Model):
    name = models.CharField(max_length=255, null=True, blank=True)
    code = models.BigIntegerField(null=True, blank=True)
    kecamatan_code = models.IntegerField(null=True, blank=True)
    kecamatan = models.ForeignKey(
        Kecamatan,
        on_delete=models.CASCADE
    )

    class Meta:
        db_table = 'kelurahan'
        verbose_name_plural = 'kelurahan'

    def __str__(self):
        return self.name

I have another app called warehouse. This app has some foreign key columns related to area model (one to many relationship). This is the warehouse model.

from django.db import models
...
from area.models import Provinsi, Kabupaten, Kecamatan, Kelurahan


class Warehouse(models.Model):
    ...
    provinsi = models.ForeignKey(
        Provinsi,
        on_delete=models.CASCADE,
        related_name='warehouse_provinsi',
    )
    kabupaten = models.ForeignKey(
        Kabupaten,
        on_delete=models.CASCADE,
        related_name='warehouse_kabupaten',
    )
    kecamatan = models.ForeignKey(
        Kecamatan,
        on_delete=models.CASCADE,
        related_name='warehouse_kecamatan',
    )
    kelurahan = models.ForeignKey(
        Kelurahan,
        on_delete=models.CASCADE,
        related_name='warehouse_kelurahan',
    )
   ...

When I generated the model and register the app to the admin area with admin.site.register(app), we can see the warehouse app management (CRUD) registered to the admin area. But, by default when it comes to dropdown list, Django admin will load all data to the selection form. The problem is, I have around 80,000 rows in fourth level of area (kelurahan), so it will be slowing down warehouse add page. This is the reason why I need to create chained dropdown list.

After reading some materials, we need some elements to do this.

  • Create endpoints to list the area data
  • Create ajax function to load area data from the endpoints when user click the parent area
  • Create custom form to override area selection form in warehouse module
  • Override default form in warehouse admin

Create Endpoints to List the Area Data

I created views.py in area module and register it to the router. This is my views.py file in the area module.

from django.http import JsonResponse
from django.contrib.auth.decorators import login_required
from area.models import Provinsi, Kabupaten, Kecamatan, Kelurahan


@login_required
def provinsi_list(request):
    provinsi = Provinsi.objects.all()
    return JsonResponse({'data': [{'id': p.id, 'name': p.name} for p in provinsi]})


@login_required
def kabupaten_list(request, provinsi_id):
    kabupaten = Kabupaten.objects.filter(provinsi=provinsi_id)
    return JsonResponse({'data': [{'id': k.id, 'name': k.name} for k in kabupaten]})


@login_required
def kecamatan_list(request, kabupaten_id):
    kecamatan = Kecamatan.objects.filter(kabupaten_id=kabupaten_id)
    return JsonResponse({'data': [{'id': k.id, 'name': k.name} for k in kecamatan]})


@login_required
def kelurahan_list(request, kecamatan_id):
    kelurahan = Kelurahan.objects.filter(kecamatan_id=kecamatan_id)
    return JsonResponse({'data': [{'id': k.id, 'name': k.name} for k in kelurahan]})

Do not forget to register the function to the main router.

Create Ajax Function to Load Area Data from The Endpoints when User Click The Parent Area

Django admin has already included older version of jquery (of course I still use jquery!), so we only create the ajax function to get the area data. I create a file in static/js/chained-area.js then load it to admin.py in warehouse app. Do not forget to set your static files folder in Django settings.

function getKabupaten(prov_id) {
    let $ = django.jQuery;
    $.get('/area/kabupaten/' + prov_id, function (resp){
        let kabupaten_list = '<option value="" selected="">---------</option>'
        $.each(resp.data, function(i, item){
           kabupaten_list += '<option value="'+ item.id +'">'+ item.name +'</option>'
        });
        $('#id_kabupaten').html(kabupaten_list);
    });
}

function getKecamatan(kabupaten_id) {
    let $ = django.jQuery;
    $.get('/area/kecamatan/' + kabupaten_id, function (resp){
        let kecamatan_list = '<option value="" selected="">---------</option>'
        $.each(resp.data, function(i, item){
           kecamatan_list += '<option value="'+ item.id +'">'+ item.name +'</option>'
        });
        $('#id_kecamatan').html(kecamatan_list);
    });
}

function getKelurahan(kecamatan_id) {
    let $ = django.jQuery;
    $.get('/area/kelurahan/' + kecamatan_id, function (resp){
        let kelurahan_list = '<option value="" selected="">---------</option>'
        $.each(resp.data, function(i, item){
           kelurahan_list += '<option value="'+ item.id +'">'+ item.name +'</option>'
        });
        $('#id_kelurahan').html(kelurahan_list);
    });
}

Then, I load it in admin.py of warehouse module.

class WarehouseAdmin(admin.ModelAdmin):
    ...
    class Media:
        js = (
            'js/chained-area.js',
        )

Create Custom Form To Override Area Selection Form in Warehouse Module

I create one file inside warehouse app, it is forms.py. I only override all fields related to the area, so it won’t affect the other fields. And because it will be implemented to add and edit form, I use kwargs[‘instance’] to detect whether it is a new object or existing object. There will be instance key in kwargs if it is an existing object or return value of error (error when adding data). If it is a new object, there won’t be instance key.

from django import forms
from warehouse.models import Warehouse
from area.models import Provinsi, Kabupaten, Kecamatan, Kelurahan


class WarehouseForm(forms.ModelForm):
    class Meta:
        model = Warehouse

    def __init__(self, *args, **kwargs):
        super(WarehouseForm, self).__init__(*args, **kwargs)

        # when there is instance key, select the default value
        # Provinsi always loaded for initial data, because Provinsi is on the first level 
        try:
            self.initial['provinsi'] = kwargs['instance'].provinsi.id
        except:
            pass
        provinsi_list = [('', '---------')] + [(i.id, i.name) for i in Provinsi.objects.all()]

        # Kabupaten, Kecamatan, and Kelurahan is on the child level, it will be loaded when user click the parent level
        try:
            self.initial['kabupaten'] = kwargs['instance'].kabupaten.id
            kabupaten_init_form = [(i.id, i.name) for i in Kabupaten.objects.filter(
                provinsi=kwargs['instance'].provinsi
            )]
        except:
            kabupaten_init_form = [('', '---------')]

        try:
            self.initial['kecamatan'] = kwargs['instance'].kecamatan.id
            kecamatan_init_form = [(i.id, i.name) for i in Kecamatan.objects.filter(
                kabupaten=kwargs['instance'].kabupaten
            )]
        except:
            kecamatan_init_form = [('', '---------')]

        try:
            self.initial['kelurahan'] = kwargs['instance'].kelurahan.id
            kelurahan_init_form = [(i.id, i.name) for i in Kelurahan.objects.filter(
                kecamatan=kwargs['instance'].kecamatan
            )]
        except:
            kelurahan_init_form = [('', '---------')]

        # Override the form, add onchange attribute to call the ajax function
        self.fields['provinsi'].widget = forms.Select(
            attrs={
                'id': 'id_provinsi',
                'onchange': 'getKabupaten(this.value)',
                'style': 'width:200px'
            },
            choices=provinsi_list,
        )
        self.fields['kabupaten'].widget = forms.Select(
            attrs={
                'id': 'id_kabupaten',
                'onchange': 'getKecamatan(this.value)',
                'style': 'width:200px'
            },
            choices=kabupaten_init_form
        )
        self.fields['kecamatan'].widget = forms.Select(
            attrs={
                'id': 'id_kecamatan',
                'onchange': 'getKelurahan(this.value)',
                'style': 'width:200px'
            },
            choices=kecamatan_init_form
        )
        self.fields['kelurahan'].widget = forms.Select(
            attrs={
                'id': 'id_kelurahan',
                'style': 'width:200px'
            },
            choices=kelurahan_init_form
        )

Override Default Form in Warehouse Admin

After setting the ajax function and custom form, we override the form variable in warehouse/admin.py.

from django.contrib import admin
from warehouse.models import Warehouse
from warehouse.forms import WarehouseForm
...


class WarehouseAdmin(admin.ModelAdmin):
    form = WarehouseForm
    ...

admin.site.register(Warehouse, WarehouseAdmin)

Okay, we’re all set. Let’s test the function.

From the recorded video, we can see that Django admin didn’t load all area data for each form selection. The form selection will load the data after the user choose the parent area.

Conclusions

Django admin is one of the magic we can find in Django, it is a very powerful feature. But, because it is only for standard CRUD, sometimes we need to modify it based on our needs, mostly for optimization purpose.

Actually, we can create our own admin page with custom template, but trust me, if you want to make simple website (mostly CRUD) for only couple of hours, you should try Django and its admin page 😁

Continue Reading

Swagger implementation for Go HTTP

The most important thing when building the web API is creating API documentation. With API documentation, it will be easier for another programmer to integrate their application with your web service.

In this post, I will show how to integrate Go net/http package and swagger using http-swagger package.

Installation

First, we have to install swag package.

$ go get github.com/swaggo/swag/cmd/swag

Initialization

Then we have to generate first `docs` folder, so we could import it to the application. Please run it on root folder of the project.

$ swag init

After generate docs folder, we import the package and generated docs. In this case, my project name is goweb.

import (
    httpSwagger "github.com/swaggo/http-swagger"
    _ "goweb/docs"
)

We have to initialize the general info for the documentation. It will be placed before main function. It describes the name, description, contact, etc. For the detail, we could see from the swag documentation.

// @title Go Restful API with Swagger
// @version 1.0
// @description Simple swagger implementation in Go HTTP

// @contact.name Linggar Primahastoko
// @contact.url https://linggar.asia
// @contact.email x@linggar.asia

// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name Authorization

// @host localhost:8082
// @BasePath /
func main() {
	...
}

We add these blocks to tell swagger that we will be using token in header for the authentication. So it means, if we want directly access some pages that using authentication, we have to pass Authorization: Basic xxxyyytokenzzzaaa

// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name Authorization

Implementation

We will write the documentation on each function and placed before the function itself. For the parameter and response, we just have to write the struct name on the documentation because we already have defined the structs. This is the part I like the most, because we don’t have to define the structure again in the documentation part.

type authParam struct {
	Username string `json:"username"`
	Password string `json:"password"`
}

type authResp struct {
	Token string `json:"token"`
}

// authLogin godoc
// @Summary Auth Login
// @Description Auth Login
// @Tags auth
// @ID auth-login
// @Accept  json
// @Produce  json
// @Param authLogin body main.authParam true "Auth Login Input"
// @Success 200 {object} main.authResp
// @Router /auth/login [post]
func authLogin(w http.ResponseWriter, r *http.Request) {
    ...
}

That is the login function, so we don’t have to pass the token to access the resource. But what if we want to authorize with token in swagger? Yes, we only have to add // @Security ApiKeyAuth.

// userProfile godoc
// @Summary User Profile
// @Description User Profile
// @Tags users
// @ID user-profile
// @Accept  json
// @Produce  json
// @Success 200 {object} main.meResp
// @Router /users/profile [get]
// @Security ApiKeyAuth
func userProfile(w http.ResponseWriter, r *http.Request) {
    ...
}

Then finally we have to add the routing and swagger function in the main function, so we could access it in the browser.

func main() {
	http.HandleFunc("/swagger/", httpSwagger.WrapHandler)
        ...
	http.ListenAndServe(":8082", nil)
}

After we setup all the components, we have to generate the documentation with swag command again.

$ swag init
2019/01/29 20:29:34 Generate swagger docs....
2019/01/29 20:29:34 Generate general API Info
2019/01/29 20:29:34 Generating main.authParam
2019/01/29 20:29:34 Generating main.authResp
2019/01/29 20:29:34 Generating main.meResp
2019/01/29 20:29:34 create docs.go at  docs/docs.go

Finally, we could build and run the application. We could access the documentation in http://<host>:<port>/swagger/index.html.

Example

This is the full implementation to integrate swagger with Go net/http package. This is only an example. It is not a perfect code that you could just copy and paste into your web app, because actually, we could write with better handler, wrapper, and middleware. For the detail of username, password, and token, you could get that information from the code. So, please enjoy! 😀

main.go

package main

import (
	"encoding/json"
	httpSwagger "github.com/swaggo/http-swagger"
	"log"
	"net/http"
	"strings"
	_ "goweb/docs"
)

type errorResp struct {
	Error string `json:"error"`
}

type indexResp struct {
	Message string `json:"message"`
}

type meResp struct {
	Username string `json:"username"`
	Fullname string `json:"fullname"`
}

type authResp struct {
	Token string `json:"token"`
}

type authParam struct {
	Username string `json:"username"`
	Password string `json:"password"`
}


// writeJSON provides function to format output response in JSON
func writeJSON(w http.ResponseWriter, code int, payload interface{}) {
	resp, err := json.Marshal(payload)
	if err != nil {
		log.Println("Error Parsing JSON")
	}

	w.Header().Set("Content-type", "application/json")
	w.WriteHeader(code)
	w.Write(resp)
}

// basicAuthMW is middleware function to check whether user is authenticated or not
// actually you could write better code for this function
func basicAuthMW(w http.ResponseWriter, r *http.Request) map[string]string {
	errorAuth := errorResp{
		Error: "Unauthorized access",
	}

	header := r.Header.Get("Authorization")
	if header == "" {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	apiKey := strings.Split(header, " ")

	if len(apiKey) != 2 {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	if apiKey[0] != "Basic" {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	users := map[string]map[string]string{
		"28b662d883b6d76fd96e4ddc5e9ba780": map[string]string{
			"username": "linggar",
			"fullname": "Linggar Primahastoko",
		},
	}

	if _ , ok := users[apiKey[1]]; !ok {
		writeJSON(w, 401, errorAuth)
		return map[string]string{}
	}

	return users[apiKey[1]]
}

func decodePost(r *http.Request, structure interface{}) {
	decoder := json.NewDecoder(r.Body)
	err := decoder.Decode(structure)
	if err != nil {
		log.Println("Error parsing post data")
	}
}

// authLogin godoc
// @Summary Auth Login
// @Description Auth Login
// @Tags auth
// @ID auth-login
// @Accept  json
// @Produce  json
// @Param authLogin body main.authParam true "Auth Login Input"
// @Success 200 {object} main.authResp
// @Router /auth/login [post]
func authLogin(w http.ResponseWriter, r *http.Request) {
	var param authParam
	decodePost(r, &param)

	if param.Username == "linggar" && param.Password == "linggar" {
		respAuth := authResp{
			Token: "28b662d883b6d76fd96e4ddc5e9ba780",
		}
		writeJSON(w, 200, respAuth)
	} else {
		failResp := errorResp{
			Error: "Wrong username/password",
		}
		writeJSON(w, 401, failResp)
	}
}

// userProfile godoc
// @Summary User Profile
// @Description User Profile
// @Tags users
// @ID user-profile
// @Accept  json
// @Produce  json
// @Success 200 {object} main.meResp
// @Router /users/profile [get]
// @Security ApiKeyAuth
func userProfile(w http.ResponseWriter, r *http.Request) {
	info := basicAuthMW(w, r)

	if len(info) == 0 {
		return
	}

	respMe := meResp{
		Username: info["username"],
		Fullname: info["fullname"],
	}

	writeJSON(w, 200, respMe)
}

// @title Go Restful API with Swagger
// @version 1.0
// @description Simple swagger implementation in Go HTTP

// @contact.name Linggar Primahastoko
// @contact.url https://linggar.asia
// @contact.email x@linggar.asia

// @securityDefinitions.apikey ApiKeyAuth
// @in header
// @name Authorization

// @host localhost:8082
// @BasePath /
func main() {
	http.HandleFunc("/swagger/", httpSwagger.WrapHandler)

	http.HandleFunc("/auth/login", authLogin)
	http.HandleFunc("/users/profile", userProfile)

	http.ListenAndServe(":8082", nil)
}

Continue Reading

Kubectl, Master NotReady

When I setup the kubernetes at the first time, after execute kubeadm init and apply the pod networks plugin, flannel or calico, I see that the master status is not ready.

linx@node-1:~$ kubectl get nodes
NAME      STATUS     ROLES    AGE     VERSION
node-1   NotReady   master   3m44s   v1.13.0

After reading some articles, I found out that we also need apply weave-kube plugin. I try to apply weave kube plugin, then the master status becomes ready 😀

linx@node-1:~$ kubectl apply -f https://git.io/weave-kube-1.6
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.extensions/weave-net created
linx@node-1:~$ kubectl get nodes
NAME      STATUS   ROLES    AGE     VERSION
node-1   Ready    master   8m55s   v1.13.0
Continue Reading