Titbits — 1 comment
11
Dec 11

Quick Shell

You can go straight forward to GitHub, download and start using quick-shell… then read further.

I still like PHP for its simplicity of deployment and management – just copy files and voila! And for simple webapps you need the cheapest web hosting like the one I have to host a WordPress installation for this blog. Everything simple and easy, but… sloooow! I mean, being restricted to FTP access (normally, you don’t get SSH access with a cheap PHP hosting) only, transfering files and later managing them can be tedious and ineffective. Fortunately, there’s a bunch of file management application (written in PHP) that can help you in performing file tasks. But they use your disk space, can be slow and require to much clicking with often page refreshes… So, ideally would be to have that silver bullet SSH access (though it’s not ideal as you may not be able to use SSH from everywhere). Just compare, what is faster: to type “cd …” and “rm …” (“unzip …”, “untar …” etc) and press ENTER, or to click through the tree of files with page refresh between each, or even worse – to perform file manipulations via FTP? For hackers like us, the answer is obvious – CLI. ;)
php-shellSo, I though it could be easy to create a web command line application that would execute system commands in my hosting environment… This thinking resulted in a simple and very lightweight PHP application that gives me all I need in terms of file management on my php hosting – it’s called quick-shell. Besides that it offers easy ajax-based file upload into a current directory (implemented with jQuery AjaxFileUpload plugin) – so you can do pretty much everything from your browser. Feel free to download/fork it from GitHub, use it and let me know how you find it.

P.S. Very important! Don’t forget to create .htpasswd file with your password to protect your files from anonymous access.

Databases / Optimisation / Titbits — No comments
14
Aug 11

Hierarchical data fetched in one query

This is a very typical situation when you have a hierarchical data structure of the type like forum posts or comments:

class Comment {
    long id;
    String comment;
    Comment parent;
}

Let’s store it in a relational database (e.g. MySQL). For this we need to create a table:

CREATE TABLE Comment(
  id int PRIMARY KEY,
  comment text,
  parent_id int,
  FOREIGN KEY (parent_id) REFERENCES Comment(id)
);

And you need to show it in a tree view, e.g.:

/Comment#1
   /Comment#2 (Reply to Comment#1)
   /Comment#4 (Reply to Comment#1)
      /Comment#8 (Reply to Comment#4)
   /Comment#7 (Reply to Comment#1)
/Comment#3
/Comment#5
   /Comment#6 (Reply to Comment#5)
      /Comment#9 (Reply to Comment#6)

You can see that comments added in an arbitrary order (by looking at the comment id sequence).

So how are you going to select all the data to build the tree by SQL means? Standard SQL does not provide hierarchical select feature (like Oracle’s CONNECT BY). One would probably end up doing multiple queries, which sounds like a very inefficient thing to do (imagine fetching a 10-level depth tree of 1000 comments). Fortunately, there is a better idea. I would say even more: a solution with one simple select query! :) Excited? It’s very simple, actually.

The idea is to construct a select statement which will return results sorted in an order we need (say, in the tree above each node is a row in the query result set). On the other hand it is very similar to the structure of directories and files in the filesystem. So, we will use a property of files here, which is the path. When creating a new Comment instance we will set its path, and update when setting a parent Comment:

class Comment {
   String path;
   …
   Comment(long id) {
     this.id = id;
     this.path = "/" + id;
   }

   void setParent(Comment parent) {
      this.parent = parent;
      this.path = (parent == null ? "" : parent.path) + "/" + id;
   }
}

If we apply this transformation to the example above the data in the table will look like:

id  parent_id  path
————————————-
1   NULL       /1
2   1          /1/2
3   NULL       /3
4   1          /1/4
5   NULL       /5
6   5          /5/6
7   1          /1/7
8   4          /1/4/8
9   6          /5/6/9

Now we can use the path field to retrieve our comments ordered accordingly. By executing

SELECT * FROM comment ORDER BY path

we will get this:

id  parent_id  path
————————————-
1   NULL       /1
2   1          /1/2
4   1          /1/4
8   4          /1/4/8
7   1          /1/7
3   NULL       /3
5   NULL       /5
6   5          /5/6
9   6          /5/6/9

The rest is only a matter of formatting the results in order to get nicely looking comments tree output. (Note: the level can be computed as a number of slashes, ‘/’, in the path string).

Also, when using ORM like Hibernate (or JPA in general), you are not forced to use SQL and can get your job done with simple criteria-based or HQL queries.

As a free bonus, you get pagination working properly without extra tweaking as you would have to do in case of multiple queries.

P.S. One piece that is missing in the above example is that the sorting for strings based on numbers will be done in the alphabetical order, e.g. 11 will appear before 2 (which is obviously not what we want). So to fix this problem you can prefix each part of the path with a length of the numbers in characters. It will result in smth like: 12, 211 (or 1.2, 2.11)…

Databases / Optimisation — No comments
13
Aug 11

Server-side cache vs client-side cache of prepared statements

I’ve been asked to explain the difference between server-side and client-side caching of database prepared statements. So here is a brief overview.

First of all, query plans are not cached on the client-side (it would be a bit foolish, because it would not give any performance boost, because we would have to send the query plan over the network each time before execution, and it contains more structured information than just a SQL string).

The parsed query plan is only stored on the server side, in a common session cache, so all sessions can reuse it. This comprises the biggest part of queries optimisation by the database server (in terms of usage of both CPU time and memory).

What is cached on the client side? – It is an identifier of the parsed query (cache then looks like a map of [sql => id]). Another big difference from the server-side cache is that the client-side cache of prepared statements resides in scope of a session (connection). (I’m not sure about the rest of the JDBC drivers, but in the Oracle implementation query parameters are also cached, which would create a memory “leak” problem when pooled connections are actively used for some period of time, hence they don’t get closed, and the cache doesn’t get released.)

In general, the prepared statement execution looks like (in terms on commands sent to the database server):

1. id = PREPARE (sql) <- here on the server there will be an attempt to get the query plan from the cache.
2. EXECUTE (id, params1)
3. (optional) EXECUTE (id, params2) …

You can see that PREPARE for the same query is executed only once throughout the session (the lifetime of the connection). Considering that in most scenarios connection pool is used, it allows you to save relevant amount of time by not performing an additional PREPARE request on the network before each consecutive EXECUTE.

Design patterns — No comments
06
Jul 11

Optimistic locking vs Pessimistic locking explained concisely

I’ve found this funny and interesting explanation of Optimistic vs Pessimistic locking through the analogy with the toilet door and could not stand not to share :)

Optimistic locking is like leaving the door to the toilet open: you have a number of toilets, select one, and you are optimistic that nobody will come to use the same toilet, too. If someome comes, it will of course be embarassing, an exception is raised, and he must abort his try, but you hope that these conflicts are the execption.

Pessimistic locking is like always locking the door to the toilet: although there are a number of different toilets, you are expecting every time that somebody else will come to use one, too, and that the only safe option is to lock all toilet doors (which corresponds to page locking). If someone comes and wants to use a toilet, he cannot enter and must wait before the locked door.

Found at Optimistic vs. Pessimistic Locking.

Optimisation — No comments
05
Jul 11

mod-pagespeeding up your site

A beautiful and simple way to speed up your site without applying any magic in your code is to install mod_pagespeed module for Apache web server.

Installation:

  • Create /etc/yum.repos.d/mod-pagespeed.repo yum config file for mod_pagespeed with the following content:
    [mod-pagespeed]
    name=mod-pagespeed
    baseurl=http://dl.google.com/linux/mod-pagespeed/rpm/stable/x86_64
    enabled=1
    gpgcheck=0
  • Then install mod-pagespeed:
    yum install mod-pagespeed
  • Restart httpd service to get it all working:
    service httpd restart

Later, I will update this post with the comparison of performance with and without mod_pagespeed (with lori add-on).

Grails / Titbits — 2 comments
27
Apr 11

Empty img src or Freaking repeated request problem

Hi! This is a victorious and quick post I have to make. (: And here is why. It took me two days of debugging and googling to find out the cause of a freaking problem: browser kept up sending same request for one of the pages of a Grails app I am working on now. The reason is an empty ‘src’ attribute of an <img> element (actually, it can be any element referring to loaded resource like a .js or .css file or any media content). You can read more about the problem itself here: Empty image src can destroy your site. I myself would probably never face this problem if didn’t have captcha (generated for each request to the problematic URL) and an optional image.

And here’s the solution:
1) in server code

<img ${ !url?:’src=”$url”‘ }/>

(in Grails terms meaning: don’t render the ‘src’ attribute if the url is empty)

2) in client (javascript) code:

imgElement.src = null; // don’t set img.src to empty string, though null is fine

So first, never ever leave your ‘src’ attribute empty! And second, never ever set ‘src’ value to empty string programatically or by any other means. Third, it’s ok to do this when it’s officially fixed either html5 is fully supported by all major browsers (hopefully).

Have a good one!

Uncategorized — 3 comments
24
Apr 11

Plesk + Apache + mod_jk + Tomcat = Good relationship

1. Create an application/domain user:

$ useradd supersite

2. Install Tomcat by copying and unzipping an archive:

$ wget http://apache.mirror.anlx.net/tomcat/tomcat-7/v7.0.12/bin/apache-tomcat-7.0.12.zip
$ unzip apache-tomcat-7.0.12.zip -d tomcat7

3. Create a service configuration for Tomcat 7:

$ cd /etc/init.d
$ edit tomcat7-supersite
#!/bin/bash
#
# tomcat7-supersite
#
# chkconfig: – 81 21
#
# description: Tomcat 7 (supersite) startup script
#

# Source function library.
. /etc/init.d/functions

#Test root privilages
RETVAL=$?
test $EUID = 0  || exit 4

CATALINA_HOME=/home/supersite/tomcat7
JAVA_HOME=/usr/lib/jvm/jdk1.6.0_24
TOMCAT_OWNER=supersite
CATALINA_PID=$CATALINA_HOME/bin/tomcat.pid

export CATALINA_HOME
export JAVA_HOME
export TOMCAT_OWNER
export CATALINA_PID

start() {
        rm -f $CATALINA_PID
        echo -n "Starting Tomcat for $TOMCAT_OWNER:  "
        su $TOMCAT_OWNER -c $CATALINA_HOME/bin/startup.sh
        sleep 2
}

stop() {
        echo -n "Stopping Tomcat for $TOMCAT_OWNER: "
        su $TOMCAT_OWNER -c $CATALINA_HOME/bin/shutdown.sh
}
restart() {
        # Restart
        stop
        echo "Sleeping for 20 seconds …"
        sleep 20
        start
}

status() {
    RETVAL="1"
    if [ -f "${CATALINA_PID}" ]; then
        read kpid < $CATALINA_PID
        if checkpid $kpid 2>&1; then
            echo "$0 is already running (${kpid})"
            RETVAL="0"
        else
            echo "lock file found but no process running for pid $kpid"
        fi
    else
        pid="$(pgrep -u ${TOMCAT_OWNER} java)"
        if [ -n "$pid" ]; then
            echo "$0 running (${pid}) but no PID file exists"
            RETVAL="0"
        else
            echo "$0 is stopped"
        fi
    fi
    return $RETVAL
}

# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart)
        restart
        ;;
  status)
        status
        ;;
  version)
        parseOptions
        "${JAVA_HOME}/bin/java" \
            -classpath "${CATALINA_HOME}/server/lib/catalina.jar" \
            org.apache.catalina.util.ServerInfo
        ;;
  *)
        echo $"Usage: tomcat {start|stop|restart}"
        exit 1
esac

exit $RETVAL

4. Verify the service config with commands:

$ service tomcat7-supersite start
$ service tomcat7-supersite status
$ service tomcat7-supersite restart
$ service tomcat7-supersite stop

5. Configure an AJP connector in $TOMCAT_HOME/conf/service.xml:

<Connector port="8010" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8"/>

If you don’t want your Tomcat to be exposed for direct HTTP requests comment out HTTP/1.1 connector section.

6. Configure Tomcat to respond to the supersite domain (e.g., supersite.com):

<Engine name="Catalina" defaultHost="localhost">
      <Host name="supersite.com" appBase="webapps" unpackWARs="true" autoDeploy="true" xmlValidation="false" xmlNamespaceAware="false">
            <Alias>www.supersite.com</Alias>
      </Host>
    </Engine>

7. Update your mod_jk to the latest version by getting an appropriate for your system mod_jk*.so file from http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/:

$ cd /etc/httpd/modules
$ mv mod_jk.so mod_jk.bk
$ wget -O mod_jk.so http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/linux/jk-1.2.31/x86_64/mod_jk-1.2.31-httpd-2.2.x.so
$ chmod 755 mod_jk.so

8. Create a new domain (supersite.com) using Plesk panel.

9. Configure your Apache to use mod_jk for the supersite domain:

$ cd /var/www/vhosts/supersite.com/conf
$ edit vhost.conf
ServerAlias www.supersite.com
JkMount /* ajp-veneve

Then execute the following to ask Plesk to include your new vhost.conf into httpd.include:

$ /usr/local/psa/admin/sbin/websrvmng -u –vhost-name=supersite.com

Create jk.conf:

$ cd /etc/httpd/conf.d
$ edit jk.conf
LoadModule jk_module /usr/lib64/httpd/modules/mod_jk.so
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info
JkLogStampFormat “[%a %b %d %H:%M:%S %Y]“

Create workers.properties in /etc/httpd/conf:

worker.list=ajp-supersite
worker.ajp-supersite.port=8010
worker.ajp-supersite.host=localhost
worker.ajp-supersite.type=ajp13

10. Restart Apache:

$ service httpd restart

11. Deploy you application to Tomcat by copying war file to /home/supersite/tomcat7/webapps. You can also deploy your application as a ROOT application by either renaming the war file to ROOT.war or replacing content of the existing ROOT directory with the war content.

12. Verify if the Tomcat it responding by going to http://supersite.com (also verify http://www.supersite.com).

13. Check Tomcat logs in $TOMCAT_HOME/logs/catalina.out if your application failed to start. This will help to diagnose a possible issue with the application itself.

Java — No comments
28
Mar 11

Ruthless but Fair Task Executor

Facebook executionGoal: To implement fair mechanism for execution of tasks in parallel with fair usage of resources by each task.
Proposed implementation is going to be ruthless so as fairly long running tasks will be simply killed, so they don’t block execution of other tasks in the queue.






Ingridients: WatchedFutureTask – a monitored task, which should rather be nimble and swift.

public class WatchedFutureTask extends FutureTask<Object> {
    private long startTime;

    public WatchedFutureTask(Runnable runnable) {
        super(runnable, null);
    }
   
    public void run() {
        startTime = System.currentTimeMillis();
        super.run();
    }

    public boolean isTimedOut(int timeout) {
        return !isCancelled() && !isDone() && System.currentTimeMillis() > (startTime + timeout);
    }
}

TaskMonitor – the ruthless and thread-safe killer.

public class TaskMonitor extends Thread {
    private final Queue<WatchedFutureTask> runningTasks = new ConcurrentLinkedQueue<WatchedFutureTask>();
    private final long timeout;

    public TaskMonitor(long timeout) {
        super();
        this.timeout = timeout;
    }

    public void watch(WatchedFutureTask task) {
        runningTasks.add(task);
    }

    public void run() {
        while (!isInterrupted()) {
            for (Iterator<WatchedFutureTask> it = runningTasks.iterator(); it.hasNext();) {
                WatchedFutureTask task = it.next();
                if (task.isTimedOut(timeout)) task.cancel(true);
                if (task.isDone() || task.isCancelled()) it.remove();
            }
            try {
                Thread.sleep(5000); // run every 5 second
            } (InterruptedException e) {
                // should exit naturally
            }
        }
    }
}

ExecutorService provided by java.util.concurrent package.
And now putting it all together we have got TaskExecutor:

public class TaskExecutor {
    private TaskMonitor monitor;
    private ExecutorService executor;

    public TaskExecutor(int threadPoolSize, long taskTimeout) {
        ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
        monitor = new TaskMonitor(taskTimeout);
        monitor.start();
    }

    public void submit(Runnable task) {
        WatchedFutureTask wtask = new WatchedFutureTask(task);
        executor.execute(wtask);
        monitor.watch(wtask);
    }
}

Leveraging capability of Future to hold task execution result (of type T), we could eloborate and arrange collection of the tasks execution results. To achieve this, you would replace usages of Runnable with usages of Callable and implement appropriate logic to store the outcomes.

Projects — No comments
10
Mar 11

Веришь? или Не Веришь?

Spill the beans Вот уже немного больше двух месяцев ведётся работа над новым сервисом ВеНеВе.ру (“Верю! Не Верю!”). Я решил написать о нём, так как мы делаем хороший прогрес, и версия 1.0 бета уже скоро увидит свет.

Немного о самом сервисе и его цели. Проект несёт две связанные между собой миссии: 1) свободное участие в развитии событий вокруг нас и 2) повышение качества и достоверности информации в Интернете.

Каждый задаётся резонным вопросом (мы тоже им задавались): Как пользователи могут участвовать?.. Ок, если совсем просто, то на сайте это участие выражено простым выбором и нажатием на кнопку “Верю!” или кнопку “Не верю!“. Если немного сложнее, то здесь замешана интуиция: интуиция никогда не подводит, в отличие от логических рассуждений. Также слово “верю” – это выражение интуиции, в отличие от “знаю” (как можно наверняка знать, что будет в даже очень близком будущем?), и таким образом, оно несёт в себе, если могу так выразиться, личное отношение каждого из вас к куску информации или интуитивное знание. Те, кто, сейчас читая, приписывают мистику этому процесу (и думают “что за бред он тут пишет!?”), уже выражают своё мнение через “не верю“…

Говоря о достоверности и качестве информации, я НЕ имею в виду известные или “справочные” данные, а скорее информацию, сложно проверяемую или связанную с событиями в будущем (через час или завтра – это тоже будущее). К тому же разные новостные сайты могут подавать информацию по-разному, с разной степенью достоверности (дайте знать, оставив комментарий, если Вы верите/не верите в то, что я только что написал).

Конечно, сайт не будет состоять только из двух кнопок… Мы стараемся сделать его максимально интуитивным, удобным и полезным, и мы захотим услышать ваше мнение!

Вот я и, как говорится, пролил немного бобов, а вам решать верить или не верить. :)

Java — 1 comment
28
Jan 11

Java tricks. Implementing annotations

Say, as it is popular now, you want to have possibility of either static (annotation-based) or dynamic (xml/db-based) configuration of your piece of programming mastership. For instance, your application should support some basic privilege-based security, so, we’ll define @Privilege annotation:

package annotations;
 
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Privilege {
    String code();
}

And a class using it:

@Privilege(code="manage.account")
public class AccountController { … }

Abstracting from the details of implementation, the handling of the annotation might look like this:

public class SecurityInterceptor {
 
    public Object intercept(User user, Object controller) {
        annotations.Privilege priv = getPrivilege(controller);
        user.hasPrivilege(priv.code());
    }
 
    protected annotations.Privilege getPrivilege(Object controller) {
        return controller.getClass().getAnnotation(annotations.Privilege.class);
    }
}

Now, we want to be able to configure another controller, OrderController, in a more dynamic way, e.g. by loading the following configuration:

<controller class="OrderController">
    <privilege>manage.account</privilege>
</controller>

So we need to have an object representation for privilege:

public class Privilege {
    public String getCode() { return code; }
    private String code;
}

and a privilege-aware controller:

public class SecuredController {
    public SecuredController(Privilege priv) { this.privilege = priv; }
    public Privilege getPrivilege() { return privilege; }
}
 
public class OrderController extends SecuredController {
    public OrderController(Privilege priv) { super(priv); }
}

See how Privilege and @Privilege are similar? (yeh, I tried to make it obvious ;) )… And in the declaration of annotations.Privilege there is an interface keyword… hm, does it mean we can implement the annotation? Why not! So here we go:

public class Privilege implements annotations.Privilege {
    public String getCode() { return code; }
    public String code() { return getCode(); }
    private String code;
}

And the last stroke is to slightly modify SecurityInterceptor‘s getPrivilege method (of course, being developers of good manners, we shall follow Open-Close Principle):

public class EnhancedSecurityInterceptor extends SecurityInterceptor {
 
    protected annotations.Privilege getPrivilege(Object controller) {
        return (controller instanceof SecuredController) ?
            ((SecuredController) controller).getPrivilege() :
            controller.getClass().getAnnotation(annotations.Privilege.class);
    }
}

Note: the java compiler will threaten you with “Annotation is used as super interface” warning, but you should remain brave and confident!

Moral of the story: reuse code and reduce your maintenance effort!

Feel free to comment, criticise and share your feelings.

← Older Entries