Yak Shaving

This is the code that I  use to merge multiple Lucene indexes into one. There are many reasons to merge multiple indexes into one like:

• Speed
• Ease of management
• Space – the size of the merged index is less than the sum of non-merged indexes

Here is the code of an Index Merger:

package ucy.cs.hpcl.minerSoft.indexmanipulation;

/*This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

Author: Asterios Katsifodimos (http://www.asteriosk.gr)
*/
import java.io.File;
import java.io.IOException;
import java.util.Date;

import org.apache.lucene.analysis.standard.StandardAnalyzer;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;

public class IndexMerger {

	/** Index all text files under a directory. */
	public static void main(String[] args) {

		if(args.length != 2){
			System.out.println("Usage: java -jar IndexMerger.jar " +
					           "existing_indexes_dir merged_index_dir");
			System.out.println(" existing_indexes_dir: A directory where the " +
					             "indexes that have to merged exist");
			System.out.println("   e.g. indexes/");
			System.out.println("   e.g.         index1");
			System.out.println("   e.g.         index2");
			System.out.println("   e.g.         index3");
			System.out.println(" merged_index_dir: A directory where the merged " +
					                               "index will be stored");
			System.out.println("   e.g. merged_indexes");
			System.exit(1);
		}

		File INDEXES_DIR  = new File(args[0]);
		File INDEX_DIR    = new File(args[1]);

		INDEX_DIR.mkdir();

		Date start = new Date();

		try {
			IndexWriter writer = new IndexWriter(INDEX_DIR,
												new StandardAnalyzer(),
												true);
			writer.setMergeFactor(1000);
			writer.setRAMBufferSizeMB(50);

			Directory indexes[] = new Directory[INDEXES_DIR.list().length];

			for (int i = 0; i < INDEXES_DIR.list().length; i++) {
				System.out.println("Adding: " + INDEXES_DIR.list()[i]);
				indexes[i] = FSDirectory.getDirectory(INDEXES_DIR.getAbsolutePath()
													+ "/" + INDEXES_DIR.list()[i]);
			}

			System.out.print("Merging added indexes...");
			writer.addIndexes(indexes);
			System.out.println("done");

			System.out.print("Optimizing index...");
			writer.optimize();
			writer.close();
			System.out.println("done");

			Date end = new Date();
			System.out.println("It took: "+((end.getTime() - start.getTime()) / 1000)
											+ "\"");

		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}

You can play with these values to gain some more performance. My settings are pretty generic.

writer.setMergeFactor(1000);
writer.setRAMBufferSizeMB(50);

Downloads

Here you can download an Index Merger that takes an argument with the folder containing the indexes to be merged and an output directory where it is going to store the merged indexes.

• IndexMerger.zip, Contains: a standalone jar file (with Lucene bundled), and the source code of the IndexMerger.

Leave a comment if you find something bad on this code!

TeXlipse is an Eclipse Plugin that allows you to have control of your Tex files, compile them, and convert them to PDF. It runs through Eclipse and its very easy to use. Using Eclipse as a tool for Tex is nice because you can use all the other Eclipse’s tools to be more productive(CVS, SVN, Mylin etc).

The only requirement to install Texlipse, is to have a Tex toolchain already installed on your Mac. The recommended way to have the whole toolchain, is to install MacTex. In order to install it, just visit http://www.tug.org/mactex/ and install it(the file you are about to download is about 1.2 GBs!).

After you have installed MacTex, you have to install TeXlipse.

Read the rest of this article »

Suppose that you want to access a remote port in a machine that runs a service on port 3306. Also suppose that the remote machine has restricted access to that port only for requests coming from the host “localhost”. You will have to create a tunnel to that machine and tunnel all your requests from you local computer’s port e.g. 2000 to the remote host’s port 3306.

ssh  -L 2000:localhost:3306  root@asteriosk.gr

After doing this, every request to localhost:2000 will be redirected(tunneled) to the remote machine at port 3306 through a secure channel! The remote machine, will accept all requests coming from the tunnel like if they were coming from localhost.

For me, this was a very nice way to access my MySQL database from my computer with the Sequel Pro client that does not support SOCKS proxies. I tell Sequel to connect to 127.0.0.1:2000 and all the requests that I make, are being redirected to my host (asteriosk.gr) so that MySQL thinks I am a local user and lets me in.

Let me know if there is something not clear here!