Installing tesseract for python on Ubuntu 14.04

Building and installing tesseract for python on Ubuntu 14.04.

root@server:/home/user/tesseract# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.1 LTS"

Install packages

sudo apt-get install python-distutils-extra tesseract-ocr tesseract-ocr-eng libopencv-dev libtesseract-dev libleptonica-dev python-all-dev swig libcv-dev python-opencv python-numpy python-setuptools build-essential subversion

sudo apt-get install autoconf automake libtool

sudo apt-get install libpng12-dev libjpeg62-dev libtiff4-dev zlib1g-dev

For tesseract training install the next packages:

sudo apt-get install libicu-dev libpango1.0-dev libcairo2-dev

Download leptonica

wget http://www.leptonica.com/source/leptonica-1.71.tar.gz

tar xvf leptonica-1.71.tar.gz

and build it

cd leptonica-1.71

./configure

make

make install

Download tesseract-ocr

wget https://bitbucket.org/3togo/python-tesseract/downloads/tesseract-3.03-rc1.tar.gz

tar xvf tesseract-3.03-rc1.tar.gz

and build it

cd tesseract-3.03

./autogen.sh

./configure

make

sudo make install

sudo ldconfig

Download (checkout) python-tesseract

svn checkout http://python-tesseract.googlecode.com/svn/trunk/src python-tesseract

I've used 659 revistion.

and build it

cd python-tesseract

python setup.py clean

python setup.py build

python setup.py install

After that try to run your python example.

If you'll get such error:

Error opening data file ./tessdata/eng.traineddata

Please make sure the TESSDATA_PREFIX environment variable is set to the parent directory of your "tessdata" directory.

Failed loading language 'eng'

Tesseract couldn't load any languages!

AdaptedTemplates != NULL:Error:Assert failed:in file adaptmatch.cpp, line 174

Segmentation fault (core dumped)

You could fix it by patching "mainblk.cpp" file inside tesseract-3.03\ccutil\ folder the next way:

In the "mainblk.cpp" file code:

  if (argv0 != NULL) {

    datadir = argv0;

  } else {

    if (getenv("TESSDATA_PREFIX")) {

      datadir = getenv("TESSDATA_PREFIX");

    } else {

#ifdef TESSDATA_PREFIX

#define _STR(a) #a

#define _XSTR(a) _STR(a)

    datadir = _XSTR(TESSDATA_PREFIX);

#undef _XSTR

#undef _STR

#endif

    }

  }

  // insert code here

  // datadir may still be empty:

  if (datadir.length() == 0) {

    datadir = "./";

add:

  if (getenv("TESSDATA_PREFIX")) {

      datadir = getenv("TESSDATA_PREFIX");

  } else {

    // check dir with tessdata

    struct stat sb;

    if (stat("/usr/share/tesseract-ocr/tessdata", &sb) == 0 && S_ISDIR(sb.st_mode)) {    

      datadir = "/usr/share/tesseract-ocr";

    }

  }

and include:

#include <sys/stat.h>

And rebuild and reinstall tesseract-ocr:

cd tesseract-3.03

make

sudo make install

So, after that, if you have TESSDATA_PREFIX env variable, it will be loaded, and if you have tessdata folder with files in /usr/share/tesseract-ocr/ it will be loaded, otherwise directory with your python example module (./) will be checked for tessdata folder.

P.S. Take a look at the repo, by the way, there are already built deb package: https://bitbucket.org/3togo/python-tesseract/

Update:

If you have the next error message when importing tesseract module in Python:
Traceback (most recent call last):
  File "test_module.py", line 5, in
    from tesseract_ocr import TesseractOCR
  File "/home/user/ocr/module/tesseract_ocr.py", line 9, in
    import tesseract
  File "/usr/local/lib/python2.7/dist-packages/python_tesseract-0.9-py2.7-linux-x86_64.egg/tesseract.py", line 28, in
    _tesseract = swig_import_helper()
  File "/usr/local/lib/python2.7/dist-packages/python_tesseract-0.9-py2.7-linux-x86_64.egg/tesseract.py", line 24, in swig_import_helper
    _mod = imp.load_module('_tesseract', fp, pathname, description)
ImportError: /usr/local/lib/python2.7/dist-packages/python_tesseract-0.9-py2.7-linux-x86_64.egg/_tesseract.so: undefined symbol: cvSetData

Check that opencv library is linked in the _tesseract.so, because cvSetData is opencv's function.

ldd _tesseract.so | grep libopencv

If the output is empty try to build _tesseract.so using this command:

sudo c++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/tesseract_wrap.o build/temp.linux-x86_64-2.7/main.o -lstdc++ -ltesseract -llept -lopencv_superres -lopencv_video -lopencv_videostab -lopencv_ml -lopencv_ocl -lopencv_contrib -lopencv_flann -lopencv_calib3d -lopencv_imgproc -lopencv_core -lopencv_legacy -lopencv_stitching -lopencv_features2d -lopencv_photo -lopencv_ts -lopencv_objdetect -lopencv_highgui -lopencv_gpu -o build/lib.linux-x86_64-2.7/_tesseract.so

NB: The command python setup.py build must be executed before the above one, otherwise next errors will be printed:

c++: error: build/temp.linux-x86_64-2.7/tesseract_wrap.o: No such file or directory

c++: error: build/temp.linux-x86_64-2.7/main.o: No such file or directory

After successful build of _tesseract.so copy it to python-tesseract directory:

sudo cp ./build/lib.linux-x86_64-2.7/_tesseract.so .

And check again that opencv library is linked in the _tesseract.so.

ldd _tesseract.so | grep libopencv

Should be something like:

libopencv_core.so.2.4 => /usr/local/lib/libopencv_core.so.2.4 (0x000070d310313370)

Now install python-tesseract:

python setup.py install

After that the problem "undefined symbol: cvSetData" will be solved.

If you have the next error:
>>> import tesseract
Traceback (most recent call last):
  File "", line 1, in
  File "/usr/lib/python2.7/dist-packages/tesseract.py", line 28, in
    _tesseract = swig_import_helper()
  File "/usr/lib/python2.7/dist-packages/tesseract.py", line 24, in swig_import_helper
    _mod = imp.load_module('_tesseract', fp, pathname, description)
ImportError: /usr/lib/python2.7/dist-packages/_tesseract.x86_64-linux-gnu.so: undefined symbol: pixGenerateFlateData

Try to use old python-tesseract svn revision (e.g. 659 or 660).
https://code.google.com/p/python-tesseract/source/detail?r=659

Update: 
I've added a new tutorial Installing tesseract for python on Ubuntu 15.10.

Useful tools for CTF

I've selected useful and must-have tools for CTF games and computer security competitions. Most of this tools are often indispensable during the games (especially task-based/jeopardy CTF games).

I've combined tools by categories just like in CTF games: Reverse, Steganography, Networking, Forensics, Cryptography, Scripting.
Most of tools are cross-platform, but some of them are only for Windows or Linux.

Here the light and dark editions of cheat sheets/posters with tools:

Утилиты, программы и тулзы для CTF игр

This is the first version of useful CTF tools cheat sheets. I'm planning to update them with new useful tools.

Thanks to shr for a good advice to add the links for tools. Here are the links to the tools from cheat sheets:

Reverse Engineering:
GDB - http://www.gnu.org/software/gdb/download/
IDA Pro - https://www.hex-rays.com/products/ida/support/download.shtml
Immunity Debugger - http://debugger.immunityinc.com/
OllyDbg - http://www.ollydbg.de/
radare2 - http://www.radare.org/y/?p=download
Hopper - http://www.hopperapp.com/download.html
nm - unix/linux tool
objdump - linux tool
strace - linux tool
ILSpy - http://ilspy.net/
JD-GUI - http://jd.benow.ca/#jd-gui-overview
FFDec - http://www.free-decompiler.com/flash/download.html
dex2jar - http://code.google.com/p/dex2jar/
uncompyle2 - https://github.com/wibiti/uncompyle2
Hex editors:
Windows:
HxD - http://mh-nexus.de/en/hxd/
Neo - http://www.new-hex-editor.com/hex-editor-downloads.html
Linux:
Bless - http://home.gna.org/bless/downloads.html
wxHexEditor - http://www.wxhexeditor.org/download.php
Exe unpackers - Unpacking Kit 2012 - http://forum.exetools.com/showthread.php?t=13610

Networking:
Wireshark, tshark - https://www.wireshark.org/download.html
OpenVPN - https://openvpn.net/
OpenSSL - https://www.openssl.org/related/binaries.html
tcpdump - http://www.tcpdump.org/
netcat - http://netcat.sourceforge.net/
nmap - http://nmap.org/download.html

Steganography:
OpenStego - http://www.openstego.info/
OutGuess - http://www.outguess.org/download.php
SilentEye - http://www.silenteye.org/download.html
Steghide - http://steghide.sourceforge.net/download.php
StegFS - http://sourceforge.net/projects/stegfs/
pngcheck - http://www.libpng.org/pub/png/apps/pngcheck.html
GIMP - http://www.gimp.org/downloads/
Audacity - http://audacity.sourceforge.net/download/
MP3Stego - http://www.petitcolas.net/steganography/mp3stego/
ffmpeg (for video analysis) - https://www.ffmpeg.org/download.html

Forensics:
dd - unix/linux tool
strings - unix/linux tool
scalpel - https://github.com/sleuthkit/scalpel
TrID - http://mark0.net/soft-trid-e.html
binwalk - http://binwalk.org/
foremost - http://foremost.sourceforge.net/
ExifTool - http://www.sno.phy.queensu.ca/~phil/exiftool/
Digital Forensics Framework (DFF) - http://www.digital-forensic.org/download/
Computer Aided INvestigative Environment (CAINE) Linux forensics live distribution - http://www.caine-live.net/
The Sleuth Kit (TSK) - http://www.sleuthkit.org/sleuthkit/download.php
Volatility - http://code.google.com/p/volatility/

Scripting / PPC (Professional Programming and Coding):
Text editors:
Sublime Text - http://www.sublimetext.com/
Notepad++ - http://notepad-plus-plus.org/
vim - http://www.vim.org/
emacs - http://www.gnu.org/software/emacs/

Crypto:
Cryptool - https://www.cryptool.org/
hashpump - https://github.com/bwall/HashPump
Sage - http://www.sagemath.org/
John the Ripper - http://www.openwall.com/john/
xortool - https://github.com/hellman/xortool
Online tools:
http://www.crypo.com/
http://www.cryptool-online.org/
http://rumkin.com/tools/cipher/
Modules for python - pycrypto - https://www.dlitz.net/software/pycrypto/

SU-CTF Quals 2014 - Write-ups

This weekend I've played Sharif University CTF (SU-CTF) Quals 2014.
Here are some of my write-ups:

What is this [20]
We have two images. Need to find the flag! It's steganography task.
Using the next simple Python code we'll get difference between the two images.

from PIL import Image, ImageChops
image1 = Image.open('pic1.jpg')
image2 = Image.open('pic2.jpg')
diff = ImageChops.difference(image1, image2)
diff.save('diff.jpg')

And the resulting image is with a flag:

Flag: AZADI TOWER

Guess the number [30]
It's Reverse task. Need to guess the number and find the flag.
We have guess.jar file. 
Decompile the file.

import java.io.PrintStream;
import java.math.BigInteger;

public class guess
{

    public guess()
    {
    }

    static String XOR(String _str_one, String _str_two)
    {
        BigInteger i1 = new BigInteger(_str_one, 16);
        BigInteger i2 = new BigInteger(_str_two, 16);
        BigInteger res = i1.xor(i2);
        String result = res.toString(16);
        return result;
    }

    public static void main(String args[])
    {
        int guess_number = 0;
        int my_num = 0x14d8f707;
        int my_number = 0x5c214f6c;
        int flag = 0x149b861a;
        if(args.length > 0)
        {
            try
            {
                guess_number = Integer.parseInt(args[0]);
                if(my_number / 5 == guess_number)
                {
                    String str_one = "4b64ca12ace755516c178f72d05d7061";
                    String str_two = "ecd44646cfe5994ebeb35bf922e25dba";
                    my_num += flag;
                    String answer = XOR(str_one, str_two);
                    System.out.println((new StringBuilder("your flag is: ")).append(answer).toString());
                } else
                {
                    System.err.println("wrong guess!");
                    System.exit(1);
                }
            }
            catch(NumberFormatException e)
            {
                System.err.println("please enter an integer \nexample: java -jar guess 12");
                System.exit(1);
            }
        } else
        {
            System.err.println("wrong guess!");
            int num = 0xf4240;
            num++;
            System.exit(1);
        }
    }
}

So to get a flag we just need to xor two hex numbers.

print '%x' % (0x4b64ca12ace755516c178f72d05d7061 ^ 0xecd44646cfe5994ebeb35bf922e25dba)

Flag: a7b08c546302cc1fd2a4d48bf2bf2ddb

Recover deleted file [40]
Forensics task. Need to recover the disk and find the flag.

After a quick analysis of disk-image. I've found that elf file is present in the image.
After cutting this file out, and using radare2 to disassemble it, I've got a flag:

Convert bytes to chars using Python:

print ''.join(map(chr, [0x20, 0x64, 0x65, 0x36, 0x38, 0x33, 0x38, 0x32, 0x35, 0x32, 0x66, 0x39, 0x35, 0x64, 0x33, 0x62, 0x39, 0x65, 0x38, 0x30, 0x33, 0x62, 0x32, 0x38, 0x64, 0x66, 0x33, 0x33, 0x62, 0x34, 0x62, 0x61, 0x61, 0x00]))

Flag: de6838252f95d3b9e803b28df33b4baa

Hidden Message [40]
Steganography task. What is the hidden message?
We have captured network traffic (80 packets) in pcap file.

Ok. I've assumed that 80 packets are 80 bits, or 10 bytes.
And if source port is 3401 then it is "0" bit, otherwise (if port is 3400) - add "1" bit.
And finally I've got the next binary sequence:
01001000011001010110100101110011011001010110111001100010011001010111001001100111
After splitting into 8-bit chunks (bytes) and converting each byte to ascii, I've got a flag.
Here is my code for getting flag from hidden-message.pcap file.

# read data
data = []
with open('hidden-message.pcap', 'rb') as f:
 data = f.read()

# get bits
bits = ''
for i in xrange(75, len(data), 81):
 bits += '0' if data[i:i+1]=='I' else '1'
# convert to chars
flag = ''
for i in xrange(0, len(bits), 8):
 flag += chr(int(bits[i:i+8], 2))
print flag

Flag: Heisenberg

Sudoku image encryption [40]
Need to solve sudoku to get a flag.

Solved sudoku:
964127538
712385694
385496712
491578263
238614975
576239841
627843159
153962487
849751326

Need to set appropriate image in the row, according to the number (column index). Repeat this operation for each row.
Using the next Python script, I've got the flag:

from PIL import Image

# solved sudoku
s = '''
964127538
712385694
385496712
491578263
238614975
576239841
627843159
153962487
849751326
'''
s = s.replace('\n', '')

image = Image.open('image.png').convert('RGB')
out_image = Image.new('RGB', image.size)
for j in xrange(0, 9):
 for i in xrange(0, 9):
  img_cell = image.crop((i * 50, j * 50, i * 50 + 50, j * 50 + 50))
  # get pos (column)
  column = (int(s[j * 9 + i]) - 1) * 50
  out_image.paste(img_cell, (column, j * 50))
out_image.save('out_image.png')

The image with the flag:

Flag: d41d8cd98f00b204e9800998ecf8427e3

Hear With Your Eyes [100]
It's steganography task.
We have sound.wav file.
I've opened a spectrogram view of this file in Audacity and got the flag.

Flag: e5353bb7b57578bd4da1c898a8e2d767

Cafe-1 [150]
Recon task.
Nice coffee shop, much better than Starbucks! Do you know a professional Logo Designer?!
Flag: md5(Nickname)
We have a photo cafeteria.jpg with QR code.

I've a little bit modified QR code image (rotated and scaled). Then I've tried to draw random points in the missing area until it recognized the QR code:

After successful scanning of QR code, I've got the next data from it:
GodForgives,AllYouHaveToDoIsAsk
suctf.com/cce1/d393
Good! I've opened this page. Here it is:

After that I've looked inside the logo:
http://suctf.com/cce1/d393/images/sharifnosh.png
But no useful info about designer was there.
After that I've checked the parent directory and opened the next url:
http://suctf.com/cce1/
Only logo (logo_6a47d39b77f1b182147fa51fe444ac58.png) of coffee shop was on this page, so I've looked inside of it and found this:
Designed by: nooneonemore
For: http://suctf.com/cce1/d393
Good! Flag is found. md5(nooneonemore)
Flag: 8fb02613df9e1ff165ccc161329fa661

Cafe-2 [150]
Recon task.
What was the quote of the day? (see cafe-1)
Flag:md5(Quote)
Firstly I've checked QR code on site About page. There was the next data:
A friend cannot be considered a friend until he is tested in three occasions: in timeof need, behind your back, and after your death.
http://www.suctf.com/cce1ef795f762eab3fde5eeb2e6293d5/d3930f9805f8ba9d2a81a96bec3
But I've checked this quote, it's incorrect.
Then I've checked the quote from first task QR code:
GodForgives,AllYouHaveToDoIsAsk
And it was correct! So flag is md5 from this quote.
Flag: 9ff2cda61c87fedf687efd9ec84d0600

Python docstrings

Yesterday my colleague had some problems with updating Python documentation strings. Is __doc__ immutable for classes and methods? I decided to look into the matter.
So let's check this strange Python docstrings immutability.

Let's take the next Python code:

class A(object): 
    """Class A"""
    def foo(self):
        """Function foo"""
        pass

A.__doc__ = 'A class'

We will catch an exception:

AttributeError: attribute '__doc__' of 'type' objects is not writable.

The reason could be found in Python sources.
Here is part of code from Python 2.7 sources "Objects/typeobject.c" file:

static PyGetSetDef type_getsets[] = {
    ...
    {"__doc__", (getter)type_get_doc, NULL, NULL},
    ...
}

As you can see a setter function is NULL. So the problem is here.

Let's check the same for an instance of the class A.

a = A()
a.__doc__ = 'A'
print a.__doc__
A
print A.__doc__
Class A

So it is possible to change docstring for instance objects of a class, but the docstring of class remains unchanged.

To update docstring of classes it is possible to use metaclasses. Here is simple trick, that creates a copy of class with updated docstring.

class A(object):
    """Class A"""

    def foo(self):
        """Function foo"""
        print 'A.foo()'

def update_class_doc(cls, new_doc):
    new_dict = dict(cls.__dict__)
    new_dict['__doc__'] = new_doc
    new_cls = type(cls.__name__, A.__bases__, new_dict)
    return new_cls

A = update_class_doc(A, 'Updated docstring of class A')

print A.__doc__

Updated docstring of class A

Now let's try to change docstring for methods.

A.foo.__doc__ = 'Updated docstring of foo'

But exception will be caught again:
AttributeError: attribute '__doc__' of 'instancemethod' objects is not writable.

The reason could be found in Python "Objects/classobject.c" sources:

static PyGetSetDef instancemethod_getset[] = {
    {"__doc__", (getter)instancemethod_get_doc, NULL, NULL},
    {0}
};

As you can see a setter function is NULL again.

But it's possible to bypass this limitation, knowing that instancemethod gets docstring from its __func__ method. The __doc__ attribute of function is writeable.

print A.foo.__doc__
Function foo

A.foo.__func__.__doc__ = 'Updated docstring of foo'
print A.foo.__doc__
Updated docstring of foo

what is the same as:

A.foo.im_func.__doc__ = 'Updated docstring of foo 2'
print A.foo.__doc__
Updated docstring of foo 2

From Python 2.7 documentation:
"Changed in version 2.6: For Python 3 forward-compatibility, im_func is also available as __func__, and im_self as __self__."

In Python 3.3 issue 12773 was fixed, and now it's possible to set the __doc__ attribute of classes.

static PyGetSetDef type_getsets[] = {
    ...
    {"__doc__", (getter)type_get_doc, (setter)type_set_doc, NULL},
    ...
}

So now, using Python version >3.3, you can just write the next for a class A:

A.__doc__ = 'A class'

And no exceptions will be caught.

Python hash calculation algorithms

I've decided to understand how Python calculates hash values for built-in objects. The blessing that Python sources are opened. So I've analyzed C code of hash functions for built-in data types and rewrote equivalent functions in Python.
Here is my code of hash functions for built-in data types (str, int, float, complex, bool, object, tuple and frozenset). Tested on both 32-bit and 64-bit versions of Python 2.7 and also on Python 3.2, 3.5 (CPython implementation of Python).
As you can see, from source code, Python 3 uses updated versions of int and float hash calculation algorithms. Moreover starting from version 3.4, SipHash algorithm replaced the Fowler-Noll-Vo (FNV) algorithm as default string and bytes hash algorithm.
N.B.: None is an object, so it is necessary to use "object_hash" function to calculate its hash.
P.S.: If you find some test data where built-in hash functions' results will differ from equivalent rewritten functions' results, please let me know.

Picture collage maker using Python

Yesterday I decided to make a program for creation a collage from images. 

I've selected the next important input parameters: images list, output resulting collage image, width of resulting collage and the value of images' initial height (this value is approximate and will change, to fit the line of images in given width).

I've chosen the next algorithm:

1) Try to add images (resized to initial height) one by one to the first line:

And when the sum of their widths will go beyond the width value, start the next line.

2) After all images were added to the line, need to fit their size to width value. Here the height of images in the line will be changed from initial height.

3) Resize images to fit the width in all lines.

4) If one of the lines will contain only one image - need to repeat all steps from the beginning again, reducing the initial height value, to compact lines and to find a place for mentioned "alone" image.

I've also added shuffle of images, to get different collages each time.

The result of the program is the next:

Source code for collage maker is available on my github: https://github.com/delimitry/collage_maker

It's written in Python and uses PIL (Python Imaging Library).

Solar System in JavaScript

Snapshot of a first version of Solar System visualization in JavaScript and HTML5 Canvas.