Django Ajax Selects Edit Button

Last year, my wife started open a shopping service for entrusted goods. When she started getting overwhelmed managing the orders, I built a simple website for her, so she can manage the products and the orders through the website. By the way, I was using Django and Django Admin to build the website.

Few days ago, she asked me to add the autocomplete function when searching the product on the create order page. She asked for it because she already had a lot of products. Until now, she has around 190 products to manage. You can imagine that it would be difficult to choose a product when using mobile browser.

By the way, the table structure between the order and the product is many to many. So, I am using admin.TabularInline on the create order page. After googling some materials, I found the library to add the autocomplete function easily to Django Admin form.

Django Ajax Selects

From the github page of Django Ajax Selects, it is very easy to add autocomplete function. We just have to install the library, add the library to INSTALLED_APPS on settings.py, create lookup function, create custom form, and register the form to the admin. We can see the screenshot below after implement the function.

Now, she can directly type the product and choose the product she want. But, I realized that there is no add and edit button for the product. When you use Django Admin admin.TabularInline, you can find some buttons beside the select form. When you click that button, it will show the pop up for add new item or edit the existing item.

Add and Edit Button

After reading the documentation, it only shows how to implement the add button. Because I am using the tabular line, you just have to add AjaxSelectAdminTabularInline to the inline class.

from .models import OrderProduct
from ajax_select.admin import AjaxSelectAdminTabularInline

...
class OrderProductInline(AjaxSelectAdminTabularInline):
    model = OrderProduct

Then you will see the add button beside the form.

If we click the add button, it will pop up the add product page. But, where is the edit button? Because when we choose the product, it only show the trash button to remove the item.

From the documentation, I cannot find any resource to add the edit button. But, after reading it again, I think I can create the custom return for the format_item_display function inside the lookups.py file. Previously, it only return the item name.

def format_item_display(self, item):
    return u"<span class='tag'>%s</span>" % item.name

Then I tried to modify the return template.

def format_item_display(self, item):
    return """
        <span class='product'>{}</span> 
        <a 
            id='change_id_order_product_{}'
            class='related-widget-wrapper-link change-related' 
            data-href-template='/admin/products/product/__fk__/change/?_to_field=id&_popup=1'
            href='/admin/products/product/{}/change/?_to_field=id&_popup=1'
        >
            <img src="/static/admin/img/icon-changelink.svg" alt="Change">
        </a>""".format(item.name, item.id, item.id)

I just clone the behaviour from another form that has a button to add and edit function. Then, voila, we can see the edit button now.

Do not forget to add the id, so the pop up can be closed automatically after we submitted the edit button.

Conclusion

You can see that it is very easy to add autocomplete function to the select form in Django Admin. Actually I found another library, Django Autocomplete Light. It is a great library, but after reading the documentation, I chose Django Ajax Selects, because it was easier for me to implement the autocomplete function for inline form.

Continue Reading

Twitter Media Upload with go-twitter

On the previous post, I said that I was doing the web service migration in the company, porting the code from Scala to Go. One of the task was integrating the web service with Twitter: getting the data, posting the status, and uploading the media.

By the way, I am using go-twitter to do the integration. Everything ran smoothly until we want to upload the image to the twitter. There is no method related to the media in the library. After reading some issues, we found this pull request. We can see that there is a method for uploading any media inside the commit log, but we don’t know why the owner hasn’t merged them.

Because of that matter, we have 2 options: Use the forked version, or use the original repository and find out the way to upload the media. After reading the function and the documentation in the original repository, finally we choose the second option, because I think it is quite simple. I will explain it step by step.

Using go-twitter library

This is the sample to use the user-auth method with the app and user keys (Consumer Key, Consumer Secret, Access Key, and Access Secret).

config := oauth1.NewConfig("consumerKey", "consumerSecret")
token := oauth1.NewToken("accessToken", "accessSecret")
httpClient := config.Client(oauth1.NoContext, token)
client := twitter.NewClient(httpClient)

The client variable is a wrapper function that already prepared with functions to do anything on the twitter. We can see the method in twitter.go

// NewClient returns a new Client.
func NewClient(httpClient *http.Client) *Client {
   base := sling.New().Client(httpClient).Base(twitterAPI)
   return &Client{
      sling:          base,
      Accounts:       newAccountService(base.New()),
      DirectMessages: newDirectMessageService(base.New()),
      Favorites:      newFavoriteService(base.New()),
      Followers:      newFollowerService(base.New()),
      Friends:        newFriendService(base.New()),
      Friendships:    newFriendshipService(base.New()),
      Lists:          newListService(base.New()),
      RateLimits:     newRateLimitService(base.New()),
      Search:         newSearchService(base.New()),
      PremiumSearch:  newPremiumSearchService(base.New()),
      Statuses:       newStatusService(base.New()),
      Streams:        newStreamService(httpClient, base.New()),
      Timelines:      newTimelineService(base.New()),
      Trends:         newTrendsService(base.New()),
      Users:          newUserService(base.New()),
   }
}

Using the signed HTTP Client

From the NewClient function, we find that it receives http.Client parameter. Then, we check this part.

httpClient := config.Client(oauth1.NoContext, token)

We take a deeper look at the Client function. We can find it in config.go.

// Client returns an HTTP client which uses the provided ctx and access Token.
func (c *Config) Client(ctx context.Context, t *Token) *http.Client {
return NewClient(ctx, c, t)
}

// NewClient returns a new http Client which signs requests via OAuth1.
func NewClient(ctx context.Context, config *Config, token *Token) *http.Client {
transport := &Transport{
Base: contextTransport(ctx),
source: StaticTokenSource(token),
auther: newAuther(config),
}
return &http.Client{Transport: transport}
}

We can see that actually the httpClient variable is an http client that already signed with oauth and we can use it directly with the Twitter endpoint. So, this is the code to upload the media and post the status using go-twitter http client.

import (
   "bytes"
   "encoding/json"
   "fmt"
   "github.com/dghubble/oauth1"
   "io"
   "io/ioutil"
   "mime/multipart"
   "net/url"
   "os"
   "strconv"
)

type MediaUpload struct {
   MediaId int `json:"media_id"`
}

func main() {
   // authenticate
   config := oauth1.NewConfig("consumerKey", "consumerSecret") 
   token := oauth1.NewToken("accessToken", "accessSecret") 
   httpClient := config.Client(oauth1.NoContext, token)

   // create body form
   b := &bytes.Buffer{}
   form := multipart.NewWriter(b)

   // create media paramater
   fw, err := form.CreateFormFile("media", "file.jpg")
   if err != nil {
      panic(err)
   }

   // open file
   opened, err := os.Open("/path/to/file.jpg")
   if err != nil {
      panic(err)
   }

   // copy to form
   _, err = io.Copy(fw, opened)
   if err != nil {
      panic(err)
   }

   // close form
   form.Close()

   // upload media
   resp, err := httpClient.Post("https://upload.twitter.com/1.1/media/upload.json?media_category=tweet_image", form.FormDataContentType(), bytes.NewReader(b.Bytes()))
   if err != nil {
      fmt.Printf("Error: %s\n", err)
   }
   defer resp.Body.Close()

   // decode response and get media id
   m := &MediaUpload{}
   _ = json.NewDecoder(resp.Body).Decode(m)
   mid := strconv.Itoa(m.MediaId)

   // post status with media id
   resp, err = httpClient.PostForm("https://api.twitter.com/1.1/statuses/update.json", url.Values{"status": {"Post the status!"}, "media_ids": {mid}})
   // parse response
   body, err := ioutil.ReadAll(resp.Body)
   if err != nil {
      fmt.Printf("Error: %s\n", err)
   }

   fmt.Printf("Response: %s\n", body)
}
Continue Reading

Mongo Go Driver Custom Decoder

In this article, I will show you how to custom the decode function of mongo go driver. We use it to decode the query result from mongodb to struct that we have already set.

Right now I have a task to migrate the web app, from Scala to Go. Everything went well until we found one problem, when we save the datetime data in mongodb and we want to display the data on unix timestamp.

Actually, before this pull request, it will show the unix time on the json response when we use primitive.DateTime type on the struct. But now, there is an override function for marshalling.

Anyway I use this sample of data.

{
        "_id" : ObjectId("607041f4576f81145415bc16"),
        "name" : "Linggar",
        "createdAt" : ISODate("2016-03-03T08:00:00Z")
}

I tried to create function to override json response, to convert the data from datetime type to integer.

type Timestamp time.Time

func (t Timestamp) MarshalJSON() ([]byte, error) {
	return []byte(fmt.Sprint(t.Unix())), nil
}

type DataObj struct {
	ID        primitive.ObjectID `json:"_id" bson:"_id"`
	Name      string             `json:"name" bson:"name"`
	CreatedAt Timestamp          `json:"createdAt" bson:"createdAt"`
}

But when I decode the query result using this function.

cur.Decode(&result)

I got this error response.

error decoding key createdAt: cannot decode UTC datetime into a main.Timestamp

It seems mongo go driver cannot map the variable to the new type we create when it meet datetime type. But if we directly set the type to datetime on the struct, we didn’t find any error.

Actually we can set 2 structs to solve this problem, one struct for receiving the data from the query result, and one struct to reformat the data to the type we want. But I am just too lazy to write all struct twice 😛

After stuck with this problem for couple hours, finally I found the solution from the documentation! We can create custom decoder for one specific type. We create the registry and add the custom rule when unmarshalling the data with this function: bson.UnmarshalWithRegistry. To use that function, we have to decode to the data first to the bytes format with function DecodeBytes().

This is the code I use to decode and convert the data. First, we create the custom registry.

type Timestamp int64

type DataObj struct {
	ID        primitive.ObjectID `json:"_id" bson:"_id"`
	Name      string             `json:"name" bson:"name"`
	CreatedAt Timestamp          `json:"createdAt" bson:"createdAt"`
}

func createCustomRegistry() *bsoncodec.RegistryBuilder {
	var primitiveCodecs bson.PrimitiveCodecs
	rb := bsoncodec.NewRegistryBuilder()
	bsoncodec.DefaultValueEncoders{}.RegisterDefaultEncoders(rb)
	bsoncodec.DefaultValueDecoders{}.RegisterDefaultDecoders(rb)
	// register our new type
	myNumberType := reflect.TypeOf(Timestamp(0))
	// read the datetime type and convert to integer
	rb.RegisterTypeDecoder(
		myNumberType,
		bsoncodec.ValueDecoderFunc(func(_ bsoncodec.DecodeContext, vr bsonrw.ValueReader, val reflect.Value) error {
			// this is the function when we read the datetime format
			read, err := vr.ReadDateTime()
			if err != nil {
				return err
			}
			val.SetInt(read)
			return nil
		}),
	)
	primitiveCodecs.RegisterPrimitiveCodecs(rb)
	return rb
}

Then we decode the query and map the result.

var result DataObj
decoded, _ := cur.DecodeBytes()
var customRegistry = createCustomRegistry().Build()
err = bson.UnmarshalWithRegistry(customRegistry, []byte(decoded), &result)

And we will get the unix timestamp on the result!

$ go run main.go 
{"_id":"607041f4576f81145415bc16","name":"Linggar","createdAt":1456992000000}

Conclusion

Mongo go driver has so many functions to map and convert the data. We can create custom decoder and encoder to map the data to the type we want, without creating so many structs to decode and reformat the data.

Finally, if you can’t find any tutorial to solve your problem, please don’t give up, perhaps you can find your solution on the documentation 😀

Continue Reading

Nginx Reverse Proxy Flask-restx Swagger Configuration

Few days ago, I was setting up nginx reverse proxy for my project. I was using gunicorn for the HTTP Server with flask-restx for the web framework. Actually this was the first time for me using this combination. I usually use uwsgi for the deployment.

It was quite simple to deploy flask with gunicorn. You can use port binding.

gunicorn --bind 0.0.0.0:9999 app:app

Or you can use unix sock.

gunicorn --workers 2 --bind unix:sockname.sock -m 644 app:app

I was using the second option, using unix sock, so this was the configuration I use for nginx reverse proxy.

location / {
    root /path/to/project/;
    proxy_pass http://unix:/path/to/unix.sock;
}

After finishing the configuration and testing the service, although I could access all function of the web service, I found out a problem when accessing the documentation.

The documentation page was still getting the swagger data from localhost, because we only set the reverse proxy option to the project.

Then, I realized that we have to config the proxy header also inside the configuration, so the documentation page will request the data from the domain we setup, instead of localhost. So this is my nginx configuration now.

location / {
    root /path/to/project/;
    proxy_pass http://unix:/path/to/unix.sock;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
}

We use proxy_set_header Host $host to set the proxy request to same name as the server_name directive, so it will request the domain name, not the original host where we deploy the service.

I also use proxy_set_header X-Forwarded-Proto $scheme because I am using SSL for the domain. If you don’t use that option, the request will be blocked, because of mixed content issue (domain with SSL request to non SSL domain).

Then, after adding that two lines and restarting the service, we can load the data from the swagger 😁

Well, maybe this is all I can share. If you found any problem when using gunicorn and flask restx with nginx as reverse proxy, you can try this solution. Thank you!

Continue Reading
1 2 3 12