Implement Chained Dropdown List in Django Admin

It is not difficult to create chained dropdown list in Django with custom template, but in one project, I need to create it in Django admin page.

I need to create chained dropdown list of area in my app. For your information, my country, Indonesia, has four levels of area.

And this is my area model.

from django.db import models


class Provinsi(models.Model):
    name = models.CharField(max_length=255, null=True, blank=True)
    code = models.IntegerField(null=True, blank=True)

    class Meta:
        db_table = 'provinsi'
        verbose_name_plural = 'provinsi'

    def __str__(self):
        return self.name


class Kabupaten(models.Model):
    name = models.CharField(max_length=255, null=True, blank=True)
    code = models.IntegerField(null=True, blank=True)
    provinsi_code = models.IntegerField(null=True, blank=True)
    provinsi = models.ForeignKey(
        Provinsi,
        on_delete=models.CASCADE
    )

    class Meta:
        db_table = 'kabupaten'
        verbose_name_plural = 'kabupaten'

    def __str__(self):
        return self.name


class Kecamatan(models.Model):
    name = models.CharField(max_length=255, null=True, blank=True)
    code = models.IntegerField(null=True, blank=True)
    kabupaten_code = models.IntegerField(null=True, blank=True)
    kabupaten = models.ForeignKey(
        Kabupaten,
        on_delete=models.CASCADE
    )

    class Meta:
        db_table = 'kecamatan'
        verbose_name_plural = 'kecamatan'

    def __str__(self):
        return self.name


class Kelurahan(models.Model):
    name = models.CharField(max_length=255, null=True, blank=True)
    code = models.BigIntegerField(null=True, blank=True)
    kecamatan_code = models.IntegerField(null=True, blank=True)
    kecamatan = models.ForeignKey(
        Kecamatan,
        on_delete=models.CASCADE
    )

    class Meta:
        db_table = 'kelurahan'
        verbose_name_plural = 'kelurahan'

    def __str__(self):
        return self.name

I have another app called warehouse. This app has some foreign key columns related to area model (one to many relationship). This is the warehouse model.

from django.db import models
...
from area.models import Provinsi, Kabupaten, Kecamatan, Kelurahan


class Warehouse(models.Model):
    ...
    provinsi = models.ForeignKey(
        Provinsi,
        on_delete=models.CASCADE,
        related_name='warehouse_provinsi',
    )
    kabupaten = models.ForeignKey(
        Kabupaten,
        on_delete=models.CASCADE,
        related_name='warehouse_kabupaten',
    )
    kecamatan = models.ForeignKey(
        Kecamatan,
        on_delete=models.CASCADE,
        related_name='warehouse_kecamatan',
    )
    kelurahan = models.ForeignKey(
        Kelurahan,
        on_delete=models.CASCADE,
        related_name='warehouse_kelurahan',
    )
   ...

When I generated the model and register the app to the admin area with admin.site.register(app), we can see the warehouse app management (CRUD) registered to the admin area. But, by default when it comes to dropdown list, Django admin will load all data to the selection form. The problem is, I have around 80,000 rows in fourth level of area (kelurahan), so it will be slowing down warehouse add page. This is the reason why I need to create chained dropdown list.

After reading some materials, we need some elements to do this.

  • Create endpoints to list the area data
  • Create ajax function to load area data from the endpoints when user click the parent area
  • Create custom form to override area selection form in warehouse module
  • Override default form in warehouse admin

Create Endpoints to List the Area Data

I created views.py in area module and register it to the router. This is my views.py file in the area module.

from django.http import JsonResponse
from django.contrib.auth.decorators import login_required
from area.models import Provinsi, Kabupaten, Kecamatan, Kelurahan


@login_required
def provinsi_list(request):
    provinsi = Provinsi.objects.all()
    return JsonResponse({'data': [{'id': p.id, 'name': p.name} for p in provinsi]})


@login_required
def kabupaten_list(request, provinsi_id):
    kabupaten = Kabupaten.objects.filter(provinsi=provinsi_id)
    return JsonResponse({'data': [{'id': k.id, 'name': k.name} for k in kabupaten]})


@login_required
def kecamatan_list(request, kabupaten_id):
    kecamatan = Kecamatan.objects.filter(kabupaten_id=kabupaten_id)
    return JsonResponse({'data': [{'id': k.id, 'name': k.name} for k in kecamatan]})


@login_required
def kelurahan_list(request, kecamatan_id):
    kelurahan = Kelurahan.objects.filter(kecamatan_id=kecamatan_id)
    return JsonResponse({'data': [{'id': k.id, 'name': k.name} for k in kelurahan]})

Do not forget to register the function to the main router.

Create Ajax Function to Load Area Data from The Endpoints when User Click The Parent Area

Django admin has already included older version of jquery (of course I still use jquery!), so we only create the ajax function to get the area data. I create a file in static/js/chained-area.js then load it to admin.py in warehouse app. Do not forget to set your static files folder in Django settings.

function getKabupaten(prov_id) {
    let $ = django.jQuery;
    $.get('/area/kabupaten/' + prov_id, function (resp){
        let kabupaten_list = '<option value="" selected="">---------</option>'
        $.each(resp.data, function(i, item){
           kabupaten_list += '<option value="'+ item.id +'">'+ item.name +'</option>'
        });
        $('#id_kabupaten').html(kabupaten_list);
    });
}

function getKecamatan(kabupaten_id) {
    let $ = django.jQuery;
    $.get('/area/kecamatan/' + kabupaten_id, function (resp){
        let kecamatan_list = '<option value="" selected="">---------</option>'
        $.each(resp.data, function(i, item){
           kecamatan_list += '<option value="'+ item.id +'">'+ item.name +'</option>'
        });
        $('#id_kecamatan').html(kecamatan_list);
    });
}

function getKelurahan(kecamatan_id) {
    let $ = django.jQuery;
    $.get('/area/kelurahan/' + kecamatan_id, function (resp){
        let kelurahan_list = '<option value="" selected="">---------</option>'
        $.each(resp.data, function(i, item){
           kelurahan_list += '<option value="'+ item.id +'">'+ item.name +'</option>'
        });
        $('#id_kelurahan').html(kelurahan_list);
    });
}

Then, I load it in admin.py of warehouse module.

class WarehouseAdmin(admin.ModelAdmin):
    ...
    class Media:
        js = (
            'js/chained-area.js',
        )

Create Custom Form To Override Area Selection Form in Warehouse Module

I create one file inside warehouse app, it is forms.py. I only override all fields related to the area, so it won’t affect the other fields. And because it will be implemented to add and edit form, I use kwargs[‘instance’] to detect whether it is a new object or existing object. There will be instance key in kwargs if it is an existing object or return value of error (error when adding data). If it is a new object, there won’t be instance key.

from django import forms
from warehouse.models import Warehouse
from area.models import Provinsi, Kabupaten, Kecamatan, Kelurahan


class WarehouseForm(forms.ModelForm):
    class Meta:
        model = Warehouse

    def __init__(self, *args, **kwargs):
        super(WarehouseForm, self).__init__(*args, **kwargs)

        # when there is instance key, select the default value
        # Provinsi always loaded for initial data, because Provinsi is on the first level 
        try:
            self.initial['provinsi'] = kwargs['instance'].provinsi.id
        except:
            pass
        provinsi_list = [('', '---------')] + [(i.id, i.name) for i in Provinsi.objects.all()]

        # Kabupaten, Kecamatan, and Kelurahan is on the child level, it will be loaded when user click the parent level
        try:
            self.initial['kabupaten'] = kwargs['instance'].kabupaten.id
            kabupaten_init_form = [(i.id, i.name) for i in Kabupaten.objects.filter(
                provinsi=kwargs['instance'].provinsi
            )]
        except:
            kabupaten_init_form = [('', '---------')]

        try:
            self.initial['kecamatan'] = kwargs['instance'].kecamatan.id
            kecamatan_init_form = [(i.id, i.name) for i in Kecamatan.objects.filter(
                kabupaten=kwargs['instance'].kabupaten
            )]
        except:
            kecamatan_init_form = [('', '---------')]

        try:
            self.initial['kelurahan'] = kwargs['instance'].kelurahan.id
            kelurahan_init_form = [(i.id, i.name) for i in Kelurahan.objects.filter(
                kecamatan=kwargs['instance'].kecamatan
            )]
        except:
            kelurahan_init_form = [('', '---------')]

        # Override the form, add onchange attribute to call the ajax function
        self.fields['provinsi'].widget = forms.Select(
            attrs={
                'id': 'id_provinsi',
                'onchange': 'getKabupaten(this.value)',
                'style': 'width:200px'
            },
            choices=provinsi_list,
        )
        self.fields['kabupaten'].widget = forms.Select(
            attrs={
                'id': 'id_kabupaten',
                'onchange': 'getKecamatan(this.value)',
                'style': 'width:200px'
            },
            choices=kabupaten_init_form
        )
        self.fields['kecamatan'].widget = forms.Select(
            attrs={
                'id': 'id_kecamatan',
                'onchange': 'getKelurahan(this.value)',
                'style': 'width:200px'
            },
            choices=kecamatan_init_form
        )
        self.fields['kelurahan'].widget = forms.Select(
            attrs={
                'id': 'id_kelurahan',
                'style': 'width:200px'
            },
            choices=kelurahan_init_form
        )

Override Default Form in Warehouse Admin

After setting the ajax function and custom form, we override the form variable in warehouse/admin.py.

from django.contrib import admin
from warehouse.models import Warehouse
from warehouse.forms import WarehouseForm
...


class WarehouseAdmin(admin.ModelAdmin):
    form = WarehouseForm
    ...

admin.site.register(Warehouse, WarehouseAdmin)

Okay, we’re all set. Let’s test the function.

From the recorded video, we can see that Django admin didn’t load all area data for each form selection. The form selection will load the data after the user choose the parent area.

Conclusions

Django admin is one of the magic we can find in Django, it is a very powerful feature. But, because it is only for standard CRUD, sometimes we need to modify it based on our needs, mostly for optimization purpose.

Actually, we can create our own admin page with custom template, but trust me, if you want to make simple website (mostly CRUD) for only couple of hours, you should try Django and its admin page 😁

Continue Reading

Syntaxnet with GPU Support

After compiling Syntaxnet (old version with bazel 0.5.x) with GPU support, you will find this error message.

E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate xxxG (xxxxxxx bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
E tensorflow/stream_executor/cuda/cuda_driver.cc:965] failed to allocate xxxG (xxxxxxx bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

No, no. It is not because your GPU memory is not enough. It is because sometimes tensorflow eats all of your GPU memory. So, what do you have to do? You just have to modify the main function of this file.

models/research/syntaxnet/syntaxnet/parser_eval.py

gpu_opt = tf.GPUOptions(allow_growth=True)
with tf.Session(config=tf.ConfigProto(gpu_options=gpu_opt)) as sess:
    Eval(sess)

Or you can download the patch here.

Now you only use about 300 MB of your GPU memory to run Syntaxnet 😀

Reference: Github

Continue Reading

Toml Config Value by Name (Go)

I use https://github.com/BurntSushi/toml for parsing the configuration file. Because the library use the struct type for the configuration variable, we have to access the struct fields with a dot to get a value.

config.Database.Username

But I want to do it dynamically. I want to create a function that receive a string type key and return the configuration value. So I do it with this way.

package main

import (
	"fmt"
	"os"
	"reflect"
	"strings"
	"github.com/BurntSushi/toml"
)

type tomlConfig struct {
	Database databaseInfo
	Title    string
}

type databaseInfo struct {
	Username string
	Password string
}

// Function to get struct type variable by index name
func GetField(t *tomlConfig, field string) string {
	r := reflect.ValueOf(t)
	f := reflect.Indirect(r)

	splitField := strings.Split(field, ".")
	for _, s := range splitField {
		f = f.FieldByName(strings.Title(s))
	}

	return f.String()
}

// Function to get config by string
func GetConfig(key string) string {
	var config tomlConfig
	if _, err := toml.Decode(
		`
		title = "Text title"

		[database]
		username = "root"
		password = "password"
		`, &config); err != nil {
		fmt.Println("Please check your configuration file")
		os.Exit(1)
	}

	configValue := GetField(&config, key)

	return configValue
}

func main() {
	fmt.Println(GetConfig("testKey"))
	fmt.Println(GetConfig("database.testKey"))
	fmt.Println(GetConfig("title"))
	fmt.Println(GetConfig("database.username"))
	fmt.Println(GetConfig("database.password"))
}

Result:

$ go run main.go 
<invalid Value>
<invalid Value>
Text title
root
password
Continue Reading

Gogstash – Logstash Alternative

I am now doing some projects that need a monitoring application to monitor the webservice. After having some chit and chat, we decide to use ELK (Elasticsearch, Logstash, and Kibana). If you want to know what ELK is, just search on Google and there will be so many articles related to it.

If you have already read some articles about ELK, you will know that ELK is the application to monitor and analyze all types of log.

  • Elasticsearch: indexing the data.
  • Logstash: log processing / parsing.
  • Kibana: visualize the data.

But after trying to configure and run ELK, I found out that Logstash is heavy to be run on the server with small specification. Because of this reason, I am trying to find some Logstash alternatives, and finally I found Gogstash, Logstash like, written in Golang.

While reading the documentation, I found out that there are some differences between Gogstash and Logstash when using the filter (I am using grok filter in Logstash). I tried to apply same pattern in Gogstash but it didn’t work. After all these things, I decide to use another filter. I am using gonx filter.

Although grok pattern and gonx pattern is different, it is not so difficult to create the configuration for gonx filter. And after some modification, Gogstash run smoothly. For your information, I am using flask for building the webservice, and this is an example line of the application log.

192.168.100.57 - - [05/Dec/2017 16:27:27] "GET / HTTP/1.1" 200 -

There are two types of Gogstash configuration, json and yml format. Here is my yml configuration.

input:
  - type: file
    path: '/home/linggar/webapp/nohup.out'

filter:
  - type: gonx
    format: '$clientip - - [$date $time] "$full_request" $status -'
    source: message
  - type: gonx
    format: '$verb $request HTTP/$httpversion'
    source: full_request

output:
  - type: elastic
    url: 'http://127.0.0.1:9200'
    index: gogstash_log
    document_type: testtype

And that’s it. Although Gogstash is not as powerful as Logstash, it is very light and one of so many Logstash alternatives which you could try.

Continue Reading