Easily Manage Go Pkgsets with GVM

On these few months, I’m focusing on doing a Go project. Instead of working on a project, I also have some Go projects to maintain, so I’m using GVM to manage multiple versions of Go. Besides using the multiple versions, I also separated pkgsets between the projects. About how to manage the versions and pkgsets, we can read the introduction here.

GVM is easy to use. I usually use these commands below when working on Go project.

List all available versions

$ gvm listall

List installed versions

$ gvm list

Install specific version

$ gvm install <go-specific-version>

Create pkgset

$ gvm use <go-specific-version>
$ gvm pkgset create <pkgset-name>

Switch to specific pkgset

$ gvm use <go-specific-version>
$ gvm pkgset use <pkgset-name>

Actually GVM is very easy to use. We can switch between the versions or pkgsets installed on the laptop. But, there is one thing that disturb me. I’m used to using virtualenvwrapper when doing a Python project. The thing is, I can easily switch to the environment with one line of command.

Creating project environment

$ mkvirtualenv -p python3.10 my-project

Switch the environment

$ workon my-project

Yes, I can switch to the specific environment without selecting the python version first. The autocomplete also works great.

Unlike virtualenvwrapper, on gvm, we have to select the version first, then we can select the pkgset we want to working on. Then, I think I have to find virtualenvwrapper like for GVM version, to easily create and select the pkgset. Unfortunately, I cannot find any tools or library that can fulfill my purpose, so it’s time to invent the wheel.

CREATING USE-GVM AND CREATE-GVM COMMAND

To simplify the commands, I created 2 scripts to create and switch the version managers and pkgsets.

use-gvm.sh

#!/bin/bash

help() {
    echo -e "$0 helps you easily to switch to existing go version and pkgset." && \
    echo -e "Usage: $0 <pkgSetName>|<goVersion>" && \
    echo -e "Example: $0 elastic-go|go1.21.4"
}

# check gvm root variable
if [[ "$GVM_ROOT" == "" ]];
then echo "Please set GVM_ROOT environment!" && exit
fi

if [ -z "$1" ];
then help && exit
fi

PKGSET_DIR=$GVM_ROOT/pkgsets

TEMP_ARGS=${1//\// }
ARGS=(${TEMP_ARGS})

# check go version
GO_VERSION=${ARGS[1]}
GO_VERSION_DIR=$PKGSET_DIR/$GO_VERSION
if test ! -d $GO_VERSION_DIR
then echo -e "go version not exist!" && exit
fi

# check go version
GO_PKGSET=${ARGS[0]}
FULL_PKGSET_DIR=$GO_VERSION_DIR/$GO_PKGSET
if test ! -d $FULL_PKGSET_DIR
then echo "pkgset not exist!" && exit
fi

# load gvm scripts to detect subcommand
source  $GVM_ROOT/scripts/gvm
gvm use $GO_VERSION && gvm pkgset use $GO_PKGSET

create-gvm.sh

#!/bin/bash

help() {
    echo "$0 helps you create a pkgset in one line"
    echo -e "Usage: $0 -n <pkgSetName> -g <goVersion>" && \
    echo -e "Example: $0 -n elastic-go -g go1.21.4"
}


# check gvm root variable
if [[ "$GVM_ROOT" == "" ]];
then echo "Please set GVM_ROOT environment!" && exit
fi

if [ -z "$1" ];
then help && exit
fi

while getopts g:n: flag
do
    case "${flag}" in
        g) goversion=${OPTARG};;
        n) name=${OPTARG};;
    esac
done

if [[ $goversion == "" ]];
then echo -e "-g is empty, go version is required!" && exit
fi

if [[ $name == "" ]];
then echo -e "-n is empty, pkgset name is required!" && exit
fi

FULL_GO_VERSION_DIR="$GVM_ROOT/gos/$goversion"
if test ! -d $FULL_GO_VERSION_DIR
then echo -e "cannot find go version, please install it first!" && exit
fi

FULL_PKGSET_DIR="$GVM_ROOT/pkgsets/$goversion/$name"
if test -d $FULL_PKGSET_DIR
then echo -e "pkgset $name is already exist with selected go version!" && exit
fi

# load gvm scripts to detect subcommand
source  $GVM_ROOT/scripts/gvm
gvm use $goversion && gvm pkgset create $name && gvm use $goversion && gvm pkgset use $name

use-gvm.sh is a script to switch between pkgsets, and create-gvm.sh is a script to select version and create pkgset. Then we need to use dot command to implement the environment to the current session. To do that, we have to add alias command to call the original script. We can add these lines to our .bashrc

use-gvm(){
    . /usr/local/bin/use-gvm.sh $1
}

create-gvm(){
    . /usr/local/bin/create-gvm.sh $@
}

For the last step, I created completion script to help the command autocomplete.

easy-gvm-completion.bash

#!/bin/bash
_usegvm_completions()
{
    # check gvm root variable
    if [[ "$GVM_ROOT" == "" ]];
    then echo "Please set GVM_ROOT environment!" && exit
    fi

    PKGSET_DIR="$GVM_ROOT/pkgsets"
    GO_INSTALLED=`ls $GVM_ROOT/gos`

    GO_PKGSET_LIST=""
    # list go installed
    for i in ${GO_INSTALLED// / }
    do
        # list pkgset installed
        PKGSET_INSTALLED_DIR=`ls $PKGSET_DIR/$i`
        for j in ${PKGSET_INSTALLED_DIR// / }
        do
            GO_PKGSET_LIST+="$j/$i "
        done
    done

    COMPREPLY=($(compgen -W "$GO_PKGSET_LIST" "${COMP_WORDS[1]}"))
}

_creategvm_completions()
{
    COMPREPLY=()
    local cur=${COMP_WORDS[COMP_CWORD]}
    local prev=${COMP_WORDS[COMP_CWORD-1]}
    opts="-g -n"

    if [[ ${cur} == -* ]]
    then
        COMPREPLY=($(compgen -W "$opts" -- $cur ) )
        return 0
    fi

    # check gvm root variable
    if [[ "$GVM_ROOT" == "" ]];
    then echo "Please set GVM_ROOT environment!" && exit
    fi

    GO_INSTALLED=`ls $GVM_ROOT/gos`

    # # list go installed
    GO_VERSIONS=""
    for i in "${GO_INSTALLED// / }"
    do
        GO_VERSIONS+=$i
    done


    case "${prev}" in
        -g)
            COMPREPLY=( $( compgen -W "$GO_VERSIONS" -- $cur ) )
            return 0
            ;;
        -n)
            return 0
            ;;
        *)
            ;;
    esac
}


complete -F _usegvm_completions use-gvm
complete -F _creategvm_completions create-gvm

Load the completion script in .bashrc

source /path/to/easy-gvm-completion.bash

After finishing the steps, we can open the new session and try the command. We can also see the suggestion when pressing <tab>

When we want to create pkgset, we will see the suggestion for the go version when using -g flag.

Then, when we create pkgset, it will automatically use the new pkgset.

Conclusion

Actually, this is just for simplifying gvm command, to create and switch the pkgset. So the function is very specific to fulfill my own purpose. There are still so many commands we can create and help us to easier our job. Hopefully this article could have helped you. Thank you! 😀

Continue Reading

Django Admin Custom Page

In the previous post, I told you that I built a website to help my wife manage the orders for her shopping service. This year, she registered to a marketplace to gain more customers. She can manage her goods and orders in that marketplace website. But, the marketplace website is still on development, there are some important features we cannot access, such as calculating the sales and profit. We have to calculate it manually and propose to withdraw the money.

In this post, I won’t tell you how I created the feature in my website (built with Django) to integrate with the marketplace service, getting the data and process it automatically. But, I will tell you how to add the custom page in Django Admin.

Django Admin Page

Before we add the custom page, I assume you can run the initial migration and create the user to login to Django Admin page.

Setting Up Templates Directory

First, we have to add the templates directory setting for the custom templates. Create the templates directory in the project root directory, then we add the directory in the ProjectName/settings.py

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

Then we create the admin directory under templates directory.

Create New App

After setting up the templates directory, we can create the new app for the custom app.

$ python manage.py startapp CustomApp

Create views.py inside CustomApp for the view function.

from django.template.response import TemplateResponse


def custom_app_view(request):
    context = {'title': 'Custom App View', 'site_title': 'Custom App View', 'site_header': 'Custom App Header',
               'index_title': 'Custom App', 'text': 'Custom App Text'}
    return TemplateResponse(request, 'admin/custom_app_view.html', context)

Then we create the template for custom app view in templates/admin/custom_app_view.html

{% extends 'admin/change_list.html' %}

{% block pagination %}{% endblock %}
{% block filters %}{% endblock filters %}
{% block object-tools %}{% endblock object-tools %}
{% block search %}{% endblock %}

{% block breadcrumbs %}
<div class="breadcrumbs">
  <a href="{% url 'admin:index' %}">Home</a>
  {% if page_name %} &rsaquo; {{ page_name }}{% endif %}
</div>
{% endblock %}

{% block result_list %}

{{ text }}
{% endblock result_list %}

Override the Admin Template

We have to override the admin template by creating the custom index for Django Admin. First, we need to create the admin.py inside the CustomApp.

from django.contrib import admin
from django.urls import path
from .views import custom_app_view


class CustomAppAdmin(admin.AdminSite):
    index_template = 'admin/custom_index.html'
    site_title = site_header = index_title = 'Custom App'

    def get_urls(
        self,
    ):
        return [
            path(
                "custom-app/",
                self.admin_view(custom_app_view),
                name="custom_app_index",
            ),
        ] + super().get_urls()


admin_site = CustomAppAdmin()

With get_urls function, we can add the custom url to the existing django urls. Then, we create html files related to admin view. We create templates/admin/custom_index.html

{% extends "admin/index.html" %}

{% block content %}
<div id="content-main">
  {% include "admin/custom_app_list.html" with app_list=app_list show_changelinks=True %}
</div>
{% endblock %}

We also create templates/admin/custom_app_list.html to override the app list, we add custom app menu in this file.

<div id="extra_links_wrapper" class="module">
    <table>
        <caption>
            <a class="section" title="Custom App">Custom App</a>
        </caption>
        <tr>
            <th scope="row">
                <a href="{% url 'admin:custom_app_index' %}">
                    Custom App
                </a>
            </th>
            <td></td>
        </tr>
    </table>
</div>

{% include 'admin/app_list.html' %}

Register New Admin Site

To register new admin site, we have to add the admin url inside ProjectName/urls.py

from django.urls import path
from CustomApp.admin import admin_site

urlpatterns = [
    path('admin/', admin_site.urls),
]

Custom Admin Index

After all those configurations, we can see that the admin index page has already changed.

We can also see the app page.

But how about any other modules that already registered in the original Django Admin? Of course, we can re-register the existing apps. We just have to add register function in ProjectName/urls.py

from django.contrib import admin
from django.urls import path
from CustomApp.admin import admin_site

# register all registered module in admin
for model, _ in admin.site._registry.items():
    admin_site.register(model)


urlpatterns = [
    path('admin/', admin_site.urls),
]

Then we can see the auth module again.

With registering the existing module to the new admin site, we can choose to register the new app directly to the django admin or custom admin.

Example

That’s all I can share about adding the custom page in Django Admin. If you want to see the example, you can find it on my Github.

Optional: Older Version

When creating this post, I was using Django version 4.1.3, but when I use the older version (I use version 3.0.7 to build my wife’s order management site), I got this error.

The reason why I got the error is because at that specific version, Django has no app_list.html file. So, you have to create it manually in templates/admin/app_list.html

{% load i18n %}

{% if app_list %}
  {% for app in app_list %}
    <div class="app-{{ app.app_label }} module{% if app.app_url in request.path|urlencode %} current-app{% endif %}">
      <table>
        <caption>
          <a href="{{ app.app_url }}" class="section" title="{% blocktrans with name=app.name %}Models in the {{ name }} application{% endblocktrans %}">{{ app.name }}</a>
        </caption>
        {% for model in app.models %}
          <tr class="model-{{ model.object_name|lower }}{% if model.admin_url in request.path|urlencode %} current-model{% endif %}">
            {% if model.admin_url %}
              <th scope="row"><a href="{{ model.admin_url }}"{% if model.admin_url in request.path|urlencode %} aria-current="page"{% endif %}>{{ model.name }}</a></th>
            {% else %}
              <th scope="row">{{ model.name }}</th>
            {% endif %}

            {% if model.add_url %}
              <td><a href="{{ model.add_url }}" class="addlink">{% trans 'Add' %}</a></td>
            {% else %}
              <td></td>
            {% endif %}

            {% if model.admin_url and show_changelinks %}
              {% if model.view_only %}
                <td><a href="{{ model.admin_url }}" class="viewlink">{% trans 'View' %}</a></td>
              {% else %}
                <td><a href="{{ model.admin_url }}" class="changelink">{% trans 'Change' %}</a></td>
              {% endif %}
            {% elif show_changelinks %}
              <td></td>
            {% endif %}
          </tr>
        {% endfor %}
      </table>
    </div>
  {% endfor %}
{% else %}
  <p>{% trans 'You don’t have permission to view or edit anything.' %}</p>
{% endif %}

Notes: some syntaxes such as trans and blocktrans are for specific Django version: 3.0.7 and below.

Continue Reading

Django Ajax Selects Edit Button

Last year, my wife started open a shopping service for entrusted goods. When she started getting overwhelmed managing the orders, I built a simple website for her, so she can manage the products and the orders through the website. By the way, I was using Django and Django Admin to build the website.

Few days ago, she asked me to add the autocomplete function when searching the product on the create order page. She asked for it because she already had a lot of products. Until now, she has around 190 products to manage. You can imagine that it would be difficult to choose a product when using mobile browser.

By the way, the table structure between the order and the product is many to many. So, I am using admin.TabularInline on the create order page. After googling some materials, I found the library to add the autocomplete function easily to Django Admin form.

Django Ajax Selects

From the github page of Django Ajax Selects, it is very easy to add autocomplete function. We just have to install the library, add the library to INSTALLED_APPS on settings.py, create lookup function, create custom form, and register the form to the admin. We can see the screenshot below after implement the function.

Now, she can directly type the product and choose the product she want. But, I realized that there is no add and edit button for the product. When you use Django Admin admin.TabularInline, you can find some buttons beside the select form. When you click that button, it will show the pop up for add new item or edit the existing item.

Add and Edit Button

After reading the documentation, it only shows how to implement the add button. Because I am using the tabular line, you just have to add AjaxSelectAdminTabularInline to the inline class.

from .models import OrderProduct
from ajax_select.admin import AjaxSelectAdminTabularInline

...
class OrderProductInline(AjaxSelectAdminTabularInline):
    model = OrderProduct

Then you will see the add button beside the form.

If we click the add button, it will pop up the add product page. But, where is the edit button? Because when we choose the product, it only show the trash button to remove the item.

From the documentation, I cannot find any resource to add the edit button. But, after reading it again, I think I can create the custom return for the format_item_display function inside the lookups.py file. Previously, it only return the item name.

def format_item_display(self, item):
    return u"<span class='tag'>%s</span>" % item.name

Then I tried to modify the return template.

def format_item_display(self, item):
    return """
        <span class='product'>{}</span> 
        <a 
            id='change_id_order_product_{}'
            class='related-widget-wrapper-link change-related' 
            data-href-template='/admin/products/product/__fk__/change/?_to_field=id&_popup=1'
            href='/admin/products/product/{}/change/?_to_field=id&_popup=1'
        >
            <img src="/static/admin/img/icon-changelink.svg" alt="Change">
        </a>""".format(item.name, item.id, item.id)

I just clone the behaviour from another form that has a button to add and edit function. Then, voila, we can see the edit button now.

Do not forget to add the id, so the pop up can be closed automatically after we submitted the edit button.

Conclusion

You can see that it is very easy to add autocomplete function to the select form in Django Admin. Actually I found another library, Django Autocomplete Light. It is a great library, but after reading the documentation, I chose Django Ajax Selects, because it was easier for me to implement the autocomplete function for inline form.

Continue Reading

Twitter Media Upload with go-twitter

On the previous post, I said that I was doing the web service migration in the company, porting the code from Scala to Go. One of the task was integrating the web service with Twitter: getting the data, posting the status, and uploading the media.

By the way, I am using go-twitter to do the integration. Everything ran smoothly until we want to upload the image to the twitter. There is no method related to the media in the library. After reading some issues, we found this pull request. We can see that there is a method for uploading any media inside the commit log, but we don’t know why the owner hasn’t merged them.

Because of that matter, we have 2 options: Use the forked version, or use the original repository and find out the way to upload the media. After reading the function and the documentation in the original repository, finally we choose the second option, because I think it is quite simple. I will explain it step by step.

Using go-twitter library

This is the sample to use the user-auth method with the app and user keys (Consumer Key, Consumer Secret, Access Key, and Access Secret).

config := oauth1.NewConfig("consumerKey", "consumerSecret")
token := oauth1.NewToken("accessToken", "accessSecret")
httpClient := config.Client(oauth1.NoContext, token)
client := twitter.NewClient(httpClient)

The client variable is a wrapper function that already prepared with functions to do anything on the twitter. We can see the method in twitter.go

// NewClient returns a new Client.
func NewClient(httpClient *http.Client) *Client {
   base := sling.New().Client(httpClient).Base(twitterAPI)
   return &Client{
      sling:          base,
      Accounts:       newAccountService(base.New()),
      DirectMessages: newDirectMessageService(base.New()),
      Favorites:      newFavoriteService(base.New()),
      Followers:      newFollowerService(base.New()),
      Friends:        newFriendService(base.New()),
      Friendships:    newFriendshipService(base.New()),
      Lists:          newListService(base.New()),
      RateLimits:     newRateLimitService(base.New()),
      Search:         newSearchService(base.New()),
      PremiumSearch:  newPremiumSearchService(base.New()),
      Statuses:       newStatusService(base.New()),
      Streams:        newStreamService(httpClient, base.New()),
      Timelines:      newTimelineService(base.New()),
      Trends:         newTrendsService(base.New()),
      Users:          newUserService(base.New()),
   }
}

Using the signed HTTP Client

From the NewClient function, we find that it receives http.Client parameter. Then, we check this part.

httpClient := config.Client(oauth1.NoContext, token)

We take a deeper look at the Client function. We can find it in config.go.

// Client returns an HTTP client which uses the provided ctx and access Token.
func (c *Config) Client(ctx context.Context, t *Token) *http.Client {
return NewClient(ctx, c, t)
}

// NewClient returns a new http Client which signs requests via OAuth1.
func NewClient(ctx context.Context, config *Config, token *Token) *http.Client {
transport := &Transport{
Base: contextTransport(ctx),
source: StaticTokenSource(token),
auther: newAuther(config),
}
return &http.Client{Transport: transport}
}

We can see that actually the httpClient variable is an http client that already signed with oauth and we can use it directly with the Twitter endpoint. So, this is the code to upload the media and post the status using go-twitter http client.

import (
   "bytes"
   "encoding/json"
   "fmt"
   "github.com/dghubble/oauth1"
   "io"
   "io/ioutil"
   "mime/multipart"
   "net/url"
   "os"
   "strconv"
)

type MediaUpload struct {
   MediaId int `json:"media_id"`
}

func main() {
   // authenticate
   config := oauth1.NewConfig("consumerKey", "consumerSecret") 
   token := oauth1.NewToken("accessToken", "accessSecret") 
   httpClient := config.Client(oauth1.NoContext, token)

   // create body form
   b := &bytes.Buffer{}
   form := multipart.NewWriter(b)

   // create media paramater
   fw, err := form.CreateFormFile("media", "file.jpg")
   if err != nil {
      panic(err)
   }

   // open file
   opened, err := os.Open("/path/to/file.jpg")
   if err != nil {
      panic(err)
   }

   // copy to form
   _, err = io.Copy(fw, opened)
   if err != nil {
      panic(err)
   }

   // close form
   form.Close()

   // upload media
   resp, err := httpClient.Post("https://upload.twitter.com/1.1/media/upload.json?media_category=tweet_image", form.FormDataContentType(), bytes.NewReader(b.Bytes()))
   if err != nil {
      fmt.Printf("Error: %s\n", err)
   }
   defer resp.Body.Close()

   // decode response and get media id
   m := &MediaUpload{}
   _ = json.NewDecoder(resp.Body).Decode(m)
   mid := strconv.Itoa(m.MediaId)

   // post status with media id
   resp, err = httpClient.PostForm("https://api.twitter.com/1.1/statuses/update.json", url.Values{"status": {"Post the status!"}, "media_ids": {mid}})
   // parse response
   body, err := ioutil.ReadAll(resp.Body)
   if err != nil {
      fmt.Printf("Error: %s\n", err)
   }

   fmt.Printf("Response: %s\n", body)
}
Continue Reading
1 2 3 12