In this article, I will show you how to custom the decode function of mongo go driver. We use it to decode the query result from mongodb to struct that we have already set.
Right now I have a task to migrate the web app, from Scala to Go. Everything went well until we found one problem, when we save the datetime data in mongodb and we want to display the data on unix timestamp.
Actually, before this pull request, it will show the unix time on the json response when we use primitive.DateTime type on the struct. But now, there is an override function for marshalling.
I tried to create function to override json response, to convert the data from datetime type to integer.
type Timestamp time.Time
func (t Timestamp) MarshalJSON() ([]byte, error) {
return []byte(fmt.Sprint(t.Unix())), nil
}
type DataObj struct {
ID primitive.ObjectID `json:"_id" bson:"_id"`
Name string `json:"name" bson:"name"`
CreatedAt Timestamp `json:"createdAt" bson:"createdAt"`
}
But when I decode the query result using this function.
cur.Decode(&result)
I got this error response.
error decoding key createdAt: cannot decode UTC datetime into a main.Timestamp
It seems mongo go driver cannot map the variable to the new type we create when it meet datetime type. But if we directly set the type to datetime on the struct, we didn’t find any error.
Actually we can set 2 structs to solve this problem, one struct for receiving the data from the query result, and one struct to reformat the data to the type we want. But I am just too lazy to write all struct twice 😛
After stuck with this problem for couple hours, finally I found the solution from the documentation! We can create custom decoder for one specific type. We create the registry and add the custom rule when unmarshalling the data with this function: bson.UnmarshalWithRegistry. To use that function, we have to decode to the data first to the bytes format with function DecodeBytes().
This is the code I use to decode and convert the data. First, we create the custom registry.
type Timestamp int64
type DataObj struct {
ID primitive.ObjectID `json:"_id" bson:"_id"`
Name string `json:"name" bson:"name"`
CreatedAt Timestamp `json:"createdAt" bson:"createdAt"`
}
func createCustomRegistry() *bsoncodec.RegistryBuilder {
var primitiveCodecs bson.PrimitiveCodecs
rb := bsoncodec.NewRegistryBuilder()
bsoncodec.DefaultValueEncoders{}.RegisterDefaultEncoders(rb)
bsoncodec.DefaultValueDecoders{}.RegisterDefaultDecoders(rb)
// register our new type
myNumberType := reflect.TypeOf(Timestamp(0))
// read the datetime type and convert to integer
rb.RegisterTypeDecoder(
myNumberType,
bsoncodec.ValueDecoderFunc(func(_ bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
// this is the function when we read the datetime format
read, err := vr.ReadDateTime()
if err != nil {
return err
}
val.SetInt(read)
return nil
}),
)
primitiveCodecs.RegisterPrimitiveCodecs(rb)
return rb
}
Then we decode the query and map the result.
var result DataObj
decoded, _ := cur.DecodeBytes()
var customRegistry = createCustomRegistry().Build()
err = bson.UnmarshalWithRegistry(customRegistry, []byte(decoded), &result)
And we will get the unix timestamp on the result!
$ go run main.go
{"_id":"607041f4576f81145415bc16","name":"Linggar","createdAt":1456992000000}
Conclusion
Mongo go driver has so many functions to map and convert the data. We can create custom decoder and encoder to map the data to the type we want, without creating so many structs to decode and reformat the data.
Finally, if you can’t find any tutorial to solve your problem, please don’t give up, perhaps you can find your solution on the documentation 😀
Few days ago, I was setting up nginx reverse proxy for my project. I was using gunicorn for the HTTP Server with flask-restx for the web framework. Actually this was the first time for me using this combination. I usually use uwsgi for the deployment.
It was quite simple to deploy flask with gunicorn. You can use port binding.
After finishing the configuration and testing the service, although I could access all function of the web service, I found out a problem when accessing the documentation.
The documentation page was still getting the swagger data from localhost, because we only set the reverse proxy option to the project.
Then, I realized that we have to config the proxy header also inside the configuration, so the documentation page will request the data from the domain we setup, instead of localhost. So this is my nginx configuration now.
We use proxy_set_header Host $host to set the proxy request to same name as the server_name directive, so it will request the domain name, not the original host where we deploy the service.
I also use proxy_set_header X-Forwarded-Proto $scheme because I am using SSL for the domain. If you don’t use that option, the request will be blocked, because of mixed content issue (domain with SSL request to non SSL domain).
Then, after adding that two lines and restarting the service, we can load the data from the swagger 😁
Well, maybe this is all I can share. If you found any problem when using gunicorn and flask restx with nginx as reverse proxy, you can try this solution. Thank you!
Few weeks ago, my office had a sharing session. My friend shared how he created a tinder bot to do automatic swiping using selenium. Actually the main point of the presentation is how to use selenium (GUI or script) to finish your daily tasks automatically. But, I was interested with the tinder bot. I though, instead of doing mass swiping to the right or left, we can choose whether he/she is based on our preference or not. If he/she is based on our preference, then swipe right.
After googling some materials, I found some projects about beauty facial prediction, but all projects mostly used SCUT-FBP5500 dataset. The SCUT-FBP5500 dataset has totally 5500 frontal faces with diverse properties, labelled by 60 volunteers. But, what I want to do is to predict the image based on my preference, not by others.
Then, I found this BeautyPredict. This project is quite simple. I tried to understand the process: preparing the data, training, predicting the score, and of course the dataset pattern. After understanding the process, finally I have 2 tasks to be done:
Create the dataset tool.
Refactor the BeautyPredict because there are some deprecated function.
Create the dataset tool
I am using flask to build the web app. And, because i am not good at design, thanks to adminlte for helping me with the design. Of course, it is mobile friendly.
And for images data, I have around 3000 face data in my storage, actually I got that data from my research few years ago. Finally, that data has a purpose!
Refactor the beauty predict
Although the project is quite simple, but there are some deprecated functions, so it will be failed if we run it with newer version of the python libraries.
Time to predict
After labeling around 1000 images, I tried to process the images and training the data, then I picked random image from instagram and predict it, it’s quite good and actually the result was as I expected.
Conclusion
So, with this project, we can rate someone whether he/she is based on our preference. If you want to know more about how the system score the image, you can visit the BeautyPredict project on Github.
And I just want to say this on the conclusion, that I believe, all men are handsome and all women are beautiful, but all people have their own preference.
Anyway, this was just my weekend research and it is not finished yet, so I didn’t write the complete instruction. But if you want to know more about the project, you can visit here.
It is not difficult to create chained dropdown list in Django with custom template, but in one project, I need to create it in Django admin page.
I need to create chained dropdown list of area in my app. For your information, my country, Indonesia, has four levels of area.
And this is my area model.
from django.db import models
class Provinsi(models.Model):
name = models.CharField(max_length=255, null=True, blank=True)
code = models.IntegerField(null=True, blank=True)
class Meta:
db_table = 'provinsi'
verbose_name_plural = 'provinsi'
def __str__(self):
return self.name
class Kabupaten(models.Model):
name = models.CharField(max_length=255, null=True, blank=True)
code = models.IntegerField(null=True, blank=True)
provinsi_code = models.IntegerField(null=True, blank=True)
provinsi = models.ForeignKey(
Provinsi,
on_delete=models.CASCADE
)
class Meta:
db_table = 'kabupaten'
verbose_name_plural = 'kabupaten'
def __str__(self):
return self.name
class Kecamatan(models.Model):
name = models.CharField(max_length=255, null=True, blank=True)
code = models.IntegerField(null=True, blank=True)
kabupaten_code = models.IntegerField(null=True, blank=True)
kabupaten = models.ForeignKey(
Kabupaten,
on_delete=models.CASCADE
)
class Meta:
db_table = 'kecamatan'
verbose_name_plural = 'kecamatan'
def __str__(self):
return self.name
class Kelurahan(models.Model):
name = models.CharField(max_length=255, null=True, blank=True)
code = models.BigIntegerField(null=True, blank=True)
kecamatan_code = models.IntegerField(null=True, blank=True)
kecamatan = models.ForeignKey(
Kecamatan,
on_delete=models.CASCADE
)
class Meta:
db_table = 'kelurahan'
verbose_name_plural = 'kelurahan'
def __str__(self):
return self.name
I have another app called warehouse. This app has some foreign key columns related to area model (one to many relationship). This is the warehouse model.
from django.db import models
...
from area.models import Provinsi, Kabupaten, Kecamatan, Kelurahan
class Warehouse(models.Model):
...
provinsi = models.ForeignKey(
Provinsi,
on_delete=models.CASCADE,
related_name='warehouse_provinsi',
)
kabupaten = models.ForeignKey(
Kabupaten,
on_delete=models.CASCADE,
related_name='warehouse_kabupaten',
)
kecamatan = models.ForeignKey(
Kecamatan,
on_delete=models.CASCADE,
related_name='warehouse_kecamatan',
)
kelurahan = models.ForeignKey(
Kelurahan,
on_delete=models.CASCADE,
related_name='warehouse_kelurahan',
)
...
When I generated the model and register the app to the admin area with admin.site.register(app), we can see the warehouse app management (CRUD) registered to the admin area. But, by default when it comes to dropdown list, Django admin will load all data to the selection form. The problem is, I have around 80,000 rows in fourth level of area (kelurahan), so it will be slowing down warehouse add page. This is the reason why I need to create chained dropdown list.
After reading some materials, we need some elements to do this.
Create endpoints to list the area data
Create ajax function to load area data from the endpoints when user click the parent area
Create custom form to override area selection form in warehouse module
Override default form in warehouse admin
Create Endpoints to List the Area Data
I created views.py in area module and register it to the router. This is my views.py file in the area module.
from django.http import JsonResponse
from django.contrib.auth.decorators import login_required
from area.models import Provinsi, Kabupaten, Kecamatan, Kelurahan
@login_required
def provinsi_list(request):
provinsi = Provinsi.objects.all()
return JsonResponse({'data': [{'id': p.id, 'name': p.name} for p in provinsi]})
@login_required
def kabupaten_list(request, provinsi_id):
kabupaten = Kabupaten.objects.filter(provinsi=provinsi_id)
return JsonResponse({'data': [{'id': k.id, 'name': k.name} for k in kabupaten]})
@login_required
def kecamatan_list(request, kabupaten_id):
kecamatan = Kecamatan.objects.filter(kabupaten_id=kabupaten_id)
return JsonResponse({'data': [{'id': k.id, 'name': k.name} for k in kecamatan]})
@login_required
def kelurahan_list(request, kecamatan_id):
kelurahan = Kelurahan.objects.filter(kecamatan_id=kecamatan_id)
return JsonResponse({'data': [{'id': k.id, 'name': k.name} for k in kelurahan]})
Do not forget to register the function to the main router.
Create Ajax Function to Load Area Data from The Endpoints when User Click The Parent Area
Django admin has already included older version of jquery (of course I still use jquery!), so we only create the ajax function to get the area data. I create a file in static/js/chained-area.js then load it to admin.py in warehouse app. Do not forget to set your static files folder in Django settings.
function getKabupaten(prov_id) {
let $ = django.jQuery;
$.get('/area/kabupaten/' + prov_id, function (resp){
let kabupaten_list = '<option value="" selected="">---------</option>'
$.each(resp.data, function(i, item){
kabupaten_list += '<option value="'+ item.id +'">'+ item.name +'</option>'
});
$('#id_kabupaten').html(kabupaten_list);
});
}
function getKecamatan(kabupaten_id) {
let $ = django.jQuery;
$.get('/area/kecamatan/' + kabupaten_id, function (resp){
let kecamatan_list = '<option value="" selected="">---------</option>'
$.each(resp.data, function(i, item){
kecamatan_list += '<option value="'+ item.id +'">'+ item.name +'</option>'
});
$('#id_kecamatan').html(kecamatan_list);
});
}
function getKelurahan(kecamatan_id) {
let $ = django.jQuery;
$.get('/area/kelurahan/' + kecamatan_id, function (resp){
let kelurahan_list = '<option value="" selected="">---------</option>'
$.each(resp.data, function(i, item){
kelurahan_list += '<option value="'+ item.id +'">'+ item.name +'</option>'
});
$('#id_kelurahan').html(kelurahan_list);
});
}
Then, I load it in admin.py of warehouse module.
class WarehouseAdmin(admin.ModelAdmin):
...
class Media:
js = (
'js/chained-area.js',
)
Create Custom Form To Override Area Selection Form in Warehouse Module
I create one file inside warehouse app, it is forms.py. I only override all fields related to the area, so it won’t affect the other fields. And because it will be implemented to add and edit form, I use kwargs[‘instance’] to detect whether it is a new object or existing object. There will be instance key in kwargs if it is an existing object or return value of error (error when adding data). If it is a new object, there won’t be instance key.
from django import forms
from warehouse.models import Warehouse
from area.models import Provinsi, Kabupaten, Kecamatan, Kelurahan
class WarehouseForm(forms.ModelForm):
class Meta:
model = Warehouse
def __init__(self, *args, **kwargs):
super(WarehouseForm, self).__init__(*args, **kwargs)
# when there is instance key, select the default value
# Provinsi always loaded for initial data, because Provinsi is on the first level
try:
self.initial['provinsi'] = kwargs['instance'].provinsi.id
except:
pass
provinsi_list = [('', '---------')] + [(i.id, i.name) for i in Provinsi.objects.all()]
# Kabupaten, Kecamatan, and Kelurahan is on the child level, it will be loaded when user click the parent level
try:
self.initial['kabupaten'] = kwargs['instance'].kabupaten.id
kabupaten_init_form = [(i.id, i.name) for i in Kabupaten.objects.filter(
provinsi=kwargs['instance'].provinsi
)]
except:
kabupaten_init_form = [('', '---------')]
try:
self.initial['kecamatan'] = kwargs['instance'].kecamatan.id
kecamatan_init_form = [(i.id, i.name) for i in Kecamatan.objects.filter(
kabupaten=kwargs['instance'].kabupaten
)]
except:
kecamatan_init_form = [('', '---------')]
try:
self.initial['kelurahan'] = kwargs['instance'].kelurahan.id
kelurahan_init_form = [(i.id, i.name) for i in Kelurahan.objects.filter(
kecamatan=kwargs['instance'].kecamatan
)]
except:
kelurahan_init_form = [('', '---------')]
# Override the form, add onchange attribute to call the ajax function
self.fields['provinsi'].widget = forms.Select(
attrs={
'id': 'id_provinsi',
'onchange': 'getKabupaten(this.value)',
'style': 'width:200px'
},
choices=provinsi_list,
)
self.fields['kabupaten'].widget = forms.Select(
attrs={
'id': 'id_kabupaten',
'onchange': 'getKecamatan(this.value)',
'style': 'width:200px'
},
choices=kabupaten_init_form
)
self.fields['kecamatan'].widget = forms.Select(
attrs={
'id': 'id_kecamatan',
'onchange': 'getKelurahan(this.value)',
'style': 'width:200px'
},
choices=kecamatan_init_form
)
self.fields['kelurahan'].widget = forms.Select(
attrs={
'id': 'id_kelurahan',
'style': 'width:200px'
},
choices=kelurahan_init_form
)
Override Default Form in Warehouse Admin
After setting the ajax function and custom form, we override the form variable in warehouse/admin.py.
from django.contrib import admin
from warehouse.models import Warehouse
from warehouse.forms import WarehouseForm
...
class WarehouseAdmin(admin.ModelAdmin):
form = WarehouseForm
...
admin.site.register(Warehouse, WarehouseAdmin)
Okay, we’re all set. Let’s test the function.
From the recorded video, we can see that Django admin didn’t load all area data for each form selection. The form selection will load the data after the user choose the parent area.
Conclusions
Django admin is one of the magic we can find in Django, it is a very powerful feature. But, because it is only for standard CRUD, sometimes we need to modify it based on our needs, mostly for optimization purpose.
Actually, we can create our own admin page with custom template, but trust me, if you want to make simple website (mostly CRUD) for only couple of hours, you should try Django and its admin page 😁