Saturday, December 26, 2015

R2D2-style sound generator in Python

I've wrote a StarWars' R2-D2-style sound generator in Python. A link to my repo on github is:
To play wav file uses sound-playing interface for Windows (module winsound).

Monday, December 14, 2015

Python 3 check using type([1.0 for i in [1]][0]) is type([1 for i in [1]][0])

Yesterday I've stumbled on a very interesting tweet: "guess why: (by arigo) so a way to know "are we on python 3" is: type([1.0 for i in [1]][0]) is type([1 for i in [1]][0])". Original link is

In Python 3 the result is:
>>> type([1.0 for i in [1]][0]) is type([1 for i in [1]][0])
>>> True
but in Python 2:
>>> type([1.0 for i in [1]][0]) is type([1 for i in [1]][0])
>>> False
It is interesting that items' types in Python 3 are both float:
>>> type([1.0 for i in [1]][0]), type([1 for i in [1]][0])
>>> (<class 'float'>, <class 'float'>)
but in Python 2 they are different:
>>> type([1.0 for i in [1]][0]), type([1 for i in [1]][0])
>>> (<type 'float'>, <type 'int'>)
The first part of a secret is that in Python 3 list comprehensions have their own scope, but in Python 2 they haven't. Guido van Rossum wrote about this "dirty little secret" here.
In Python 3 for list comprehensions a special code object listcomp was created.

And the second part of a secret is that both listcomp code objects from this example are located at the same address, because declared on the one line in code.
Here is a bytecode (I've used dis module to get is):
 6 LOAD_CONST       1 (<code object <listcomp> at 0x7fd91d983270, file "", line 3>)
35 LOAD_CONST       1 (<code object <listcomp> at 0x7fd91d983270, file "", line 3>)
List comprehensions [1.0 for i in [1]] and [1 for i in [1]] are on one line have the same address (address of the first listcomp code object), because their listcomp code objects are the same.
Update: As Rhomboid correctly noticed in the discussion on reddit, constants for current code object are stored in dictionary (although displayed as tuple) and the same code object constants are folded.
As hash values for [1.0 for i in [1]][0] and [1 for i in [1]][0] are the same, when addition to the dict is performed - items' values are compared using code_richcompare methods (in this way python dict resolves collisions).

Here is how hash for code object is calculated (from codeobject.c):
static Py_hash_t
code_hash(PyCodeObject *co)
    Py_hash_t h, h0, h1, h2, h3, h4, h5, h6;
    h0 = PyObject_Hash(co->co_name);
    if (h0 == -1) return -1;
    h1 = PyObject_Hash(co->co_code);
    if (h1 == -1) return -1;
    h2 = PyObject_Hash(co->co_consts);
    if (h2 == -1) return -1;
    h3 = PyObject_Hash(co->co_names);
    if (h3 == -1) return -1;
    h4 = PyObject_Hash(co->co_varnames);
    if (h4 == -1) return -1;
    h5 = PyObject_Hash(co->co_freevars);
    if (h5 == -1) return -1;
    h6 = PyObject_Hash(co->co_cellvars);
    if (h6 == -1) return -1;
    h = h0 ^ h1 ^ h2 ^ h3 ^ h4 ^ h5 ^ h6 ^
        co->co_argcount ^ co->co_kwonlyargcount ^
        co->co_nlocals ^ co->co_flags;
    if (h == -1) h = -2;
    return h;
Here is how code objects are compared (from codeobject.c):
static PyObject *
code_richcompare(PyObject *self, PyObject *other, int op)
    PyCodeObject *co, *cp;
    int eq;
    PyObject *res;

    if ((op != Py_EQ && op != Py_NE) ||
        !PyCode_Check(self) ||
        !PyCode_Check(other)) {

    co = (PyCodeObject *)self;
    cp = (PyCodeObject *)other;

    eq = PyObject_RichCompareBool(co->co_name, cp->co_name, Py_EQ);
    if (eq <= 0) goto unequal;
    eq = co->co_argcount == cp->co_argcount;
    if (!eq) goto unequal;
    eq = co->co_kwonlyargcount == cp->co_kwonlyargcount;
    if (!eq) goto unequal;
    eq = co->co_nlocals == cp->co_nlocals;
    if (!eq) goto unequal;
    eq = co->co_flags == cp->co_flags;
    if (!eq) goto unequal;
    eq = co->co_firstlineno == cp->co_firstlineno;
    if (!eq) goto unequal;
    eq = PyObject_RichCompareBool(co->co_code, cp->co_code, Py_EQ);
    if (eq <= 0) goto unequal;
    eq = PyObject_RichCompareBool(co->co_consts, cp->co_consts, Py_EQ);
    if (eq <= 0) goto unequal;
    eq = PyObject_RichCompareBool(co->co_names, cp->co_names, Py_EQ);
    if (eq <= 0) goto unequal;
    eq = PyObject_RichCompareBool(co->co_varnames, cp->co_varnames, Py_EQ);
    if (eq <= 0) goto unequal;
    eq = PyObject_RichCompareBool(co->co_freevars, cp->co_freevars, Py_EQ);
    if (eq <= 0) goto unequal;
    eq = PyObject_RichCompareBool(co->co_cellvars, cp->co_cellvars, Py_EQ);
    if (eq <= 0) goto unequal;

    if (op == Py_EQ)
        res = Py_True;
        res = Py_False;
    goto done;

    if (eq < 0)
        return NULL;
    if (op == Py_NE)
        res = Py_True;
        res = Py_False;

    return res;
For the first list comprehension code object variables are the next:
co_name = <listcomp>
co_argcount = 1
co_kwonlyargcount = 0
co_nlocals = 2
co_flags = 83
co_firstlineno = 3
co_code = b'g\x00\x00|\x00\x00]\x0c\x00}\x01\x00d\x00\x00\x91\x02\x00q\x06\x00S' 
co_consts = (1.0,) 
co_names = () 
co_varnames = ('.0', 'i') 
co_freevars = () 
co_cellvars = () 
And for the second one code object variables are the same, except co_consts:
co_consts = (1,)
But as 1.0 == 1, and tuples (1.0,) and (1,) are equal. Therefore Python considers the second code object as duplicate of the first one, and its address is the same. So identity operator "is" returns True for the same objects.

And that is why in Python 3 the next expressions will also be valid.
>>> a = type([1.0 for i in [1]][0]); b = type([1 for i in [1]][0])
>>> print(a, b)
>>> <class 'float'> <class 'float'>

>>> type([1 for i in [1]][0]) is type([True for i in [1]][0])
>>> True
See also: how hash values are calculated in Python read in my article Python hash calculation algorithms

Thursday, November 12, 2015

Just for Fun

Recently I've finished a book "Just for Fun: The Story of an Accidental Revolutionary" by Linus Torvalds and David Diamond. One more easy-reading autobiography of Linus Torvalds about Linux creation history, free software, open source, meaning of life and Linus' law of life.
Along with a technical information about Linux and its creation, in the book there are a lot of interesting facts about life in Finland and US, history and reminiscences. By the way, first public version on Linux, which was uploaded to internet is still there:
I highly recommend it to Linuxoids, IT-people and biography books lover.

Sunday, November 1, 2015

Spritesheet cutter

Yesterday in our SPb Python telegram chat a question "how to automatically cut spritesheet into several sprites" was asked. An example of spritesheet image is here:
Several solutions were proposed, and today I've decided to write my own solution.

So the problem is to cut each sprite out and save to separate file. As a parameter could be passed the color of border. By the way by default it could be fetched from top left pixel. I've coded a Python script that takes input spritesheet image, border color if known, and output folder to save found and cutted out sprites. Output sprite images are grouped by size.

Thursday, October 22, 2015

Gracker game

Gracker - it's a good game for beginners to develop skills in searching and exploiting vulnerabilities.
To start the game connect using SSH to with password level0.
After solving each level you could read solution (recap) from the author.

I recommend this game if you are interested in CTF and want to improve your skills. Check also smashthestack (if you haven't seen yet).

You can also read about creating this game on the author's site:

Wednesday, September 16, 2015

Python ftplib storbinary hanging

Today I've spent several hours to find why python program is hangs sometimes.
Using gdb I've found that sometimes ftplib's storbinary hangs at the end of file transfer.

gdb has shown that it hangs at:
... in __libc_recv (fd=?, buf=?, n=8192, flags=-1) at ../sysdeps/unix/sysv/linux/x86_64/recv.c:33

I've added callback argument to storbinary to print last data block (by default blocksize is 8192):
def print_last(data):
    if data < 8192:
        print len(data)

ftp.storbinary('STOR %s' % fn, f, callback=print_last)
And after that I've found that program hangs at the end of file transfer.
I also found that EOFError happens after creating some number of ftp sessions (sockets). In my case after creating >46 (or 47) simultaneous ftp connection.

Solution was simple - to add timeout argument on ftp creation.
ftp = ftplib.FTP(, self.username, self.password, timeout=10)

Tuesday, September 15, 2015

Images resizer in Go

I've published my small tool, used to study Go (golang). Source code is available on my github:
This program includes different ways to resize images - one non-concurrent and two concurrent:
first - uses just goroutines and channels, second - uses pool of workers to limit simultaneously running goroutines. More information is in readme on github.

Thursday, September 3, 2015

Working with text files in Python 2.7

Reading text files in Python 2.7 at first sight is simple, but without correct opening, a work with them can lead to unexpected results.
For example you have a file, looks like ascii-encoded and reading it line by line saving line number and line data. But if in this file some non-ascii symbols are hidden, you will have problems with encoding. In Python such problem could be reproduced as UnicodeDecodeError:
UnicodeDecodeError: 'ascii' codec can't decode byte 0x?? in position ??: ordinal not in range(128)
Moreover if line endings are nonuniform for single platform (i.e. '\r\n' or '\r' will be met along with '\n') you could have problems with lines number. And if you open your log file in text editor - line numbers in it and in your Python script results will not correspond.
Let's prepare the test logfile:
filename = 'logfile.log'
with open(filename, 'w') as f:
    f.write('line 1' + '\n')
    f.write('line 2' + '\n')
    f.write('line 3' + '\r')
    f.write('line 4' + '\r\n')
    f.write('line 5' + '\n')
    f.write('line 6' + '\n')
    f.write('line 7')
If you open it in text editor, the result will be the following:
1 line 1
2 line 2
3 line 3
4 line 4

6 line 5
7 line 6
8 line 7
Now for example we needed to get the line number with string "line 5".
filename = 'logfile.log'
with open(filename, 'r') as f:
    for line_num, line in enumerate(f):
        if line.startswith('line 5'):
            print '%s: %s' % (line_num + 1, line.strip())
the output is:
4: line 5
Here we have a wrong line number 4. The line number in text editor is 6.

Fortunately in python there are several possible ways to open files with different useful arguments.
1) open(name[, mode[, buffering]])
2), mode[, encoding[, errors[, buffering]]])
3), mode='r', buffering=-1, encoding=None, errors=None, newline=None, closefd=True)
The way 1 we've already used, let's try to use 2 and 3.
import io
import codecs

filename = 'logfile.log'
with, 'r') as f:
    for line_num, line in enumerate(f):
        if line.startswith('line 5'):
            print '%s: %s' % (line_num + 1, line.strip())

with, 'r') as f:
    for line_num, line in enumerate(f):
        if line.startswith('line 5'):
            print '%s: %s' % (line_num + 1, line.strip())

and the output is:
4: line 5
6: line 5
Here gave the same result as a simple open, but gave the expected result.
Due to its argument newline, it is possible to enable universal newlines mode and read the lines regardless line ending format (Windows, Unix, Mac OS up to version 9), i.e. lines can end with '\r\n', '\n', or '\r'. This mode is enabled by default (when newline=None). Therefore lines are split correctly and we get the line numbers the same as in text editors.
In Python 3.x is the default interface to access files and streams. and functions have encoding argument that specifies the encoding which is to be used for the file. Also they have errors argument - an optional string that specifies how encoding and decoding errors are to be handled. The difference is that handles line endings differently.
After the additions of arguments:
with, 'r', encoding='utf-8', errors='replace') as f:
    for line_num, line in enumerate(f):
        if line.startswith('line 5'):
            print '%s: %s' % (line_num + 1, line.strip())

with, 'r', encoding='utf-8', errors='replace') as f:
    for line_num, line in enumerate(f):
        if line.startswith('line 5'):
            print '%s: %s' % (line_num + 1, line.strip())
the output is:
6: line 5
6: line 5

Tuesday, July 21, 2015

Python 2.7 tricks

1) map() with an initial argument of None is similar to zip():

>>> a = (1,)
>>> b = (2, 3, 4)
>>> c = (5, 6)

>>> map(None, a, b, c)
 (1, 2, 5), 
 (None, 3, 6), 
 (None, 4, None)

>>> zip(a, b, c)
[(1, 2, 5)]

2) startswith() and endswith() accept tuples as suffix:

>>> ext.endswith(('.exe', '.dll'))
>>> filename.startswith(('img', 'pic', 'logo'))

3)[, fillchar])

>>> 'text'.center(10)
'   text   '
>>> 'text'.center(10, '-')


4) filter(function, iterable) with None as a function is the same as bool as a function.

>>> filter(None, [1, 0, 2, 3, 4, None, 5])
[1, 2, 3, 4, 5]

>>> filter(bool, [1, 0, 2, 3, 4, None, 5])
[1, 2, 3, 4, 5]

5) str.split([sep[, maxsplit]) with sep as None or not specified all consecutive whitespaces are regarded as a single separator:

>>> 'This  is  a string'.split(None)
['This', 'is', 'a', 'string']
>>> 'This  is  a string'.split()
['This', 'is', 'a', 'string']
But if sep is ' ':
>>> 'This  is  a string'.split(' ')
['This', '', 'is', '', 'a', 'string']

6) Operator <> is the same as !=

7) It is possible to use several context managers on one line:

with open('in_file_a', 'r') as in_file_a, open('out_file_a', 'w') as out_file_a, open('out_file_c', 'a') as out_file_c:

Saturday, April 25, 2015

ASCII canvas in Python

Python modules that allow to draw primitives using terminal as canvas. I called it ASCII canvas.
I've added support of colors. So now it is possible to draw colored ASCII-images.

Using script to convert RGB pixels from image to terminal colors, I've got terminal art:

Supported colors test table. For Linux:
Colors in terminal Linux
And for Windows:
Colors in terminal Windows

Sunday, April 5, 2015

ASCII Clock in Terminal using Python

Today I've created Python script that prints a clock in ASCII art style to the terminal.
To perform ASCII drawing a special Python class (ASCII Canvas) with several functions was created.
Source code is available on my GitHub:
Here are the result:
Also I've provided resizing of clock. So if the size of clock if small, week day and day are hidden.
Added scale by x-axis to equalize the width and height of the clock, because fonts' height is more than their width.
For linux version added a set of standard chars to draw the clock items with. Here you can see the clock under linux:

Monday, March 30, 2015

Solving the problems with pycallgraph under Windows

I've installed Python Call Graph (pycallgraph) using pip:

pip install pycallgraph

Then I've tried to run it:

pycallgraph graphviz -- ./

But I've got a message:
"pycallgraph" is not recognized as an internal or external command, operable program or batch file.

Ok, then I found pycallgraph location and executed it:

python c:\Python27\Scripts\pycallgraph graphviz -- ./

But alas, I've got error message:

E:\>python c:\Python27\Scripts\pycallgraph graphviz -- ./
Traceback (most recent call last):
  File "c:\Python27\Scripts\pycallgraph", line 25, in
    with __pycallgraph.PyCallGraph(config=__config):
  File "C:\Python27\lib\site-packages\pycallgraph\", line 32, in __init__
  File "C:\Python27\lib\site-packages\pycallgraph\", line 53, in reset
  File "C:\Python27\lib\site-packages\pycallgraph\", line 97, in prepare_output
  File "C:\Python27\lib\site-packages\pycallgraph\output\", line 63, in sanity_check
  File "C:\Python27\lib\site-packages\pycallgraph\output\", line 96, in ensure_binary
    'The command "{}" is required to be in your path.'.format(cmd))
pycallgraph.exceptions.PyCallGraphException: The command "dot" is required to be in your path.

I've installed graphviz:

pip install graphviz

But the problem remained.

Then I've downloaded current stable release of graphviz (graphviz-2.38.msi) from official site:

And installed it.
You can check that graphviz for python is trying to execute dot.exe tool (subprocess.Popen(cmd) in graphviz\, so to solve the problem you can go to the graphviz directory and call pycallgraph:
cd c:\Program Files (x86)\Graphviz2.38\bin
c:\Program Files (x86)\Graphviz2.38\bin>python c:\Python27\Scripts\pycallgraph graphviz -o main.png -- e:\

After that you'll get main.png image inside C:\Program Files (x86)\Graphviz2.38\bin\ directory.

PS. You can add graphviz directory to PATH environment variable.

Saturday, March 21, 2015

Piter Py #2

Сегодня прошла вторая ежегодная конференция Piter Py #2. Было очень много хороших докладов и интересных людей, так что было интересно и весело!
Раздаточный материал
Мне понравилось! Организаторы молодцы, что приглашают ребят из разных городов, и стран.
Удалось лично пообщаться с людьми, с которыми был знаком только по интернету и видел на их доклады с других мероприятий и конференций.
Приятно было послушать хэдлайнеров конференции - это Armin Ronacher и Hynek Schlawack (по-русски Хинек, читается всё-таки через "и").

Из всех докладов больше всего понравился доклад Армина про опенсорс, прямо заслушался. Очень много правильных советов и интересных примеров из опыта. После презентации удалось задать вопрос про лицензии собственных опенсорс наработок.
Также из тех докладов, которые слушал, запомнились доклады Hynek Schlawack, Ивана Ремизова, Кирилла Борисова, Григория Петрова и Александра Щепановского.
Со многими докладчиками удалось поговорить на обеде и в перерывах. Было очень приятно со всеми познакомится и интересно всех послушать.

В перерывах между докладами все общались, ходили смотреть затмение, фотографировались, играли в игры от спонсоров, было весело :) Ещё был прекрасный обед!

Отдельное спасибо хочу сказать Григорию Петрову за помощь и советы в подготовке собственного доклада.

Ссылки на слайды и презентации с PiterPy #2 которые спикеры уже опубликовали:

Алексей Зиновьев - Путь скользкого Python и дао толстой Pandas: дай свои данные, rRrrrrr...

Armin Ronacher - Developing an Open Source Library / 10 Years of Python Libraries

Максим Климишин - Трансдюсеры и Python

Алексей Пирогов - Легковесный Dependency Injection

Николай Телепенин - Генерация RESTfull приложений

Александр Щепановский - Метапрограммирование за гранью приличия

Михаил Кривушин - Pyenv и Pundle: альтернатива virtualenv

Дмитрий Алимов - Разработка фреймворка на Python для автоматизации тестирования STB (Set-Top Boxes)

Николай Телепенин - Похвастайся своим swaggerом

Thursday, February 19, 2015

Chip 'n Dale in Ruby

# Ch-ch-ch-chip 'n Dale
rescue => rangers
# Ch-ch-ch-chip 'n Dale when there's danger

Saturday, January 31, 2015

Edward Lear's poems

Из детства ещё помню, найти бы эти записи, что показывали по ТВ, в таком переводе:

Где же где в каком искать далеке,
Зеленоликих, синеруких плывущих в дуршлаге.
Это из The Jumblies.

Чиппати флип, флиппати чип - Меня зовут Мерихлюндия Пип.
Как только его увидали все звери и тут же запели:
Клиппати чип, чиппати клип - Его зовут Мерихлюндия Пип.
Это из The Scroobious Pip.

И ещё алфавит был. Помню, про цинк:
Ц - это цинк...
Это из Alphabet Poem.

Wednesday, January 14, 2015

Статья про dict в Python 2.7

Сегодня опубликовали мою первую статью на хабре «Реализация словаря в Python 2.7».

Monday, January 12, 2015

Python numbers obfuscation

Python numbers obfuscation can be done the next way:

>>> {} > [] # i.e. 'd' > 'l' ([d]ict > [l]ist)
>>> [] > {} # i.e. 'l' > 'd' ([l]ist > [d]ict)

And it's possible to get values 0 and 1 from False and True, by implicit converting bool to int.

>>> False == 0
>>> True == 1

Other numbers could be received by applying logical operations (like "~" = bitwise NOT, or "<<" bitwise left shift).
For example 2:
2 = -(-2) =  -(~1)

Let's take 16 or more first numbers, and all other numbers could be obtained by sum and/or multiplication of these first numbers.
nums = [
    '([]<{})',  # 0
    '([]>{})',  # 1 
    '-~([]>{})',  # 2   
    '-~-~([]>{})',  # 3
    '-~-~-~([]>{})',  # 4
    '-~-~-~-~([]>{})',  # 5
    '-~-~-~-~-~([]>{})',  # 6
    '-~-~-~-~-~-~([]>{})',  # 7
    '-~-~-~-~-~-~-~([]>{})',  # 8
    '~-~([]>{})*~-~([]>{})',  # 9
    '~-~-~-~([]>{})*~([]>{})',  # 10
    '-~-~(~-~([]>{})*~-~([]>{}))',  # 11
    '(-~-~([]>{})<<-~([]>{}))',  # 12
    '-~(-~-~([]>{})<<-~([]>{}))',  # 13
    '-~-~-~-~-~-~([]>{})*-~([]>{})',  # 14
    '~-~([]>{})*~-~-~-~([]>{})',  # 15
    '(-~([]>{})<<-~-~([]>{}))',  # 16
numbers_count = len(nums) - 1

def get_number(num):
    r = ''
    if num > numbers_count:
        if num // numbers_count > 1:
            if num % numbers_count == 0:
                r += '(%s*%s)' % (get_number(num // numbers_count), numbers_count)
                r += '(%s*%s+%s)' % (get_number(num // numbers_count), numbers_count, num % numbers_count)
            r += '(%s+%s)' % (numbers_count, get_number(num - numbers_count))
        r = '%s' % num
    return r

def convert_nums(s):
    res = s
    for n in xrange(numbers_count, -1, -1):
        res = res.replace(str(n), nums[n])
    res = res.replace('+-', '-')
    return res
num = 1234567
num_str = get_number(num)
num_str = num_str[1:-1] if num_str[0] == '(' and num_str[-1] == ')' else num_str
print num_str
obfuscated = convert_nums(num_str)
print obfuscated
print eval(obfuscated)
So here the number 1234567 was successfully obfuscated and restored using eval function.
1234567 = ((((16+2)*16+13)*16+6)*16+8)*16+7 = (((((-~([]>{})<<-~-~([]>{}))-~([]>{}))*(-~([]>{})<<-~-~([]>{}))-~(-~-~([]>{})<<-~([]>{})))*(-~([]>{})<<-~-~([]>{}))-~-~-~-~-~([]>{}))*(-~([]>{})<<-~-~([]>{}))-~-~-~-~-~-~-~([]>{}))*(-~([]>{})<<-~-~([]>{}))-~-~-~-~-~-~([]>{})

For negative numbers you only need to change the sign in front of the parentheses.

But for huge numbers like 12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789, will be an error:
_push: parser stack overflow
Traceback (most recent call last):
  File "", line 51, in 
    print eval(obfuscated)
So be careful and use nested parentheses carefully :)

The presentation with updated version of obfuscator (added Python 3 support):